Look Less, Think Faster: Joint Token-Compute Adaptation for Multimodal LLMs

AI summary

Overview

Research area: Efficient inference for Multimodal Large Language Models (MLLMs), spanning computer vision, vision-language modeling, and adaptive/systems-aware machine learning.

Technical level: Advanced. The paper assumes familiarity with MLLM architectures (vision encoder plus LLM decoder), transformer inference mechanics such as prefill FLOPs, attention heads and FFN width, token pruning, and techniques like Gumbel-sigmoid relaxation and straight-through estimators.

Scope: This paper proposes SmartVL, a unified adaptive inference framework that jointly controls how many visual tokens are kept and how much LLM computation is executed, conditioned on both the input content and a target FLOPs budget.

What This Paper Is About

Multimodal LLMs spend a fixed amount of computation regardless of how simple or complex the input is, because a standard vision encoder such as CLIP ViT-L/14 produces 576 patch tokens per image and every one of them is processed through a 32-or-more-layer language model. Prior efficiency methods attack these costs one axis at a time — either pruning visual tokens or skipping layers and heads — but they ignore the fact that the right amount of visual detail and the right amount of LLM depth and width depend on each other. This paper introduces SmartVL, which learns to allocate a given compute budget across all three axes (sequence length, depth, width) simultaneously so that accuracy–efficiency tradeoffs improve over decoupled alternatives.

Key Contributions

1. A unified adaptive inference framework that jointly controls sequence length (number of retained vision tokens), model depth (executed transformer layers), and model width (active attention heads and FFN channels) under a single target compute budget. The authors frame this as addressing a fundamental cross-stage coupling that prior work treats as independent dimensions.

2. Two lightweight controllers connected by a shared budget encoding: a vision-side token controller that selects informative visual tokens inside the ViT, and an LLM-side compute controller that decides which layers and attention head groups to activate. The compute controller is explicitly conditioned on the token survival rate produced by the vision controller.

3. A joint training recipe built around a differentiable FLOPs estimator, per-token Gumbel-sigmoid sampling with a straight-through estimator, and an asymmetric budget violation loss that penalizes exceeding the budget quadratically and under-utilization linearly, with a scheduled warmup weight and uniform budget sampling during training.

4. An empirical demonstration across seven MLLM benchmarks that joint token-compute scheduling produces stronger accuracy–efficiency Pareto frontiers and a wider usable adaptation range than token-only pruning (FastV, LLaVA-PruMerge+), compute-only control (AdaLLaVA), or a naively composed token-plus-compute pipeline (AdaLLaVA-PruMerge).

Main Findings

• Joint control beats decoupled control on VQAv2: At 50% FLOPs, SmartVL reaches 74.4% accuracy versus 67.9% for AdaLLaVA and 74.5% for AdaLLaVA-PruMerge. AdaLLaVA drops sharply at its minimum operating point of 50% budget, whereas SmartVL maintains strong accuracy from 20% to 100% of full budget.

• Gains persist at higher budgets: At 70% FLOPs, SmartVL scores 75.7% versus 74.4% for AdaLLaVA, a 1.3-point gap. At full budget, SmartVL matches the base LLaVA-1.5 model at 76.5%.

• Aggregate improvements over AdaLLaVA: The authors report a 6.6% accuracy gain at 50% compute over AdaLLaVA, maintained up to full capacity, and an average improvement of 7.8% at roughly 50% compute budget across seven benchmarks.

• Wider adaptation range: SmartVL spans from 2.5T FLOPs up to the full 8.5T capacity, while LLaVA-PruMerge+ is restricted to a 3T to 4.8T window.

• Large margins on text-heavy and real-world tasks: On TextVQA and VizWiz, AdaLLaVA degrades severely at 50% budget (29.8% and 34.3% respectively), while SmartVL maintains 54.4% and 55.1% at the same FLOP level.

• Mixed but competitive results elsewhere: On GQA at 50%, SmartVL reaches 59.8%, slightly trailing AdaLLaVA-PruMerge (60.1%) but surpassing AdaLLaVA (56.8%). On ScienceQA and POPE all three methods are comparable, with SmartVL showing modest mid-range gains. On MMBench, SmartVL reaches 62.0% at 50%, compared to 63.1% for AdaLLaVA-PruMerge and 63.3% for AdaLLaVA. The paper states that SmartVL is most competitive on VQA-style tasks.

• Depth control matters more than width control: On TextVQA at 50% FLOPs, the layer-level variant SmartVL-L reaches 54.1%, beating AdaLLaVA-PruMerge (51.3%) and avoiding AdaLLaVA's collapse (29.8%). At 70% FLOPs, SmartVL-L scores 57.0% versus 54.3% and 50.8%. At full capacity SmartVL-L reaches 58.1%. The layer-head variant SmartVL-LH trades a marginal drop at full budget, scoring 57.5%, which the authors interpret as evidence that MLLM redundancy lives mainly in sequence and depth, while full-width attention remains important.

• The optimal allocation depends on the task and budget: In an ablation sweeping token retention ratio T in {0.5, 0.6, 0.7, 0.8, 0.9} and compute retention ratio C in {0.5, 0.6, 0.7, 0.8}, POPE consistently favors high visual coverage with minimal LLM depth (T = 0.8, C = 0.5), while VQAv2 and TextVQA require progressively deeper reasoning as budget grows (T = 0.9, C = 0.8 at 7 TFLOPs). At comparable FLOPs around 4 TFLOPs, shifting budget between vision tokens and LLM compute changes predictions on the same input, and the pattern differs across tasks.

• The learned token controller is budget-conditioned: Compared against FastV and LLaVA-PruMerge+ on VQAv2 (with the compute controller disabled), a single trained token controller produces different token subsets for different budgets and preserves accuracy better in the high-FLOPs regime, while outperforming both baselines clearly in the low-FLOPs regime. Fixed-pruning baselines instead form discrete retention-ratio operating points.

• 13B scaling not reported in the available content: The paper text indicates it instantiates SmartVL on the LLaVA-1.5-13B backbone, but the provided content is truncated at that point, so no 13B results are available to summarize here.

Methodology in Plain English

MLLM inference cost during the prefill stage breaks down in three places: how many vision tokens go into the model, how many transformer layers actually run, and how wide each layer runs (how many attention heads and FFN channels are active). The authors treat these as one combined search space rather than three separate problems.

Their framework has two small decision-making modules. The first is a token controller. A vector encoding the target budget — a number saying what fraction of full-model FLOPs is allowed, converted into the vision encoder's hidden space — is appended as an extra token to the vision encoder's input. Because it attends to all the image patches, it ends up carrying both the picture's content and the budget. A linear layer then turns that state into a score for every patch token, and each token is kept or dropped probabilistically. During training, Gumbel noise plus a straight-through estimator lets gradients flow through these binary keep/drop decisions; at inference, the noise is removed and tokens whose score falls at or below 0.5 are physically discarded. The language model's RoPE positional encoding handles the resulting shortened sequence without re-indexing.

The second module is the compute controller. It reuses the same sinusoidal encoding of the budget, but maps it into the LLM's embedding dimension with a separate network, and appends it to the prompt. The first P transformer layers always run at full capacity as a baseline. After those prefix layers, the controller's state is combined with a scalar representing what fraction of visual tokens survived — this is what links the two stages, and a stop-gradient is applied to that scalar so gradients from the LLM's architectural choices do not destabilize the vision controller. From this fused state, the controller independently gates each remaining layer and each group of attention heads. Two granularities are tested: SmartVL-L, which only gates whole layers (depth), and SmartVL-LH, which partitions the attention heads into groups (for LLaMA-7B, 32 heads into 4 groups) and gates them, where a group's mask simultaneously controls its attention heads and the matching FFN channels so that computation shrinks in balance. If every group in a layer is off, the whole layer is skipped, so depth and width control fall out of one formulation.

Training the whole thing requires a differentiable handle on cost, so the authors write a closed-form FLOPs estimate (Eq. 8) using the differentiable token survival sum and the layer activity fractions, and normalize it against the full-model cost. This estimate feeds an asymmetric budget violation loss: overshooting the budget is penalized quadratically, while falling short is only penalized linearly, leaving a tolerance band of [b − μ, b] where no penalty applies. The penalty weight ramps in linearly with training progress, because enforcing the budget too early pushes the network into a degenerate sparse state before it has learned which features matter. At each training step, a budget is sampled uniformly from a range and applied to the whole batch, and the vision encoder is frozen while the LLM, projector, and both controllers are trained end to end. At inference, masks become deterministic, and if they still exceed the target budget, a projection step first discards the lowest-confidence visual tokens and only reduces layers or heads if token-level adjustment is not enough.

Why This Matters

Impact on research. The paper argues that visual token redundancy and LLM reasoning complexity are coupled, and that treating them independently caps how good an accuracy–efficiency tradeoff can be. It provides a concrete training recipe — shared budget encoding, differentiable latency estimation, Gumbel-sigmoid gating, and an asymmetric budget loss — for optimizing over a compound discrete space without collapsing. It also supplies a controlled comparison showing that depth control carries most of the gains while width control can be dropped at little cost, which is a useful negative-ish result for anyone designing similar controllers.

Real-world applications.

• Real-time visual assistants and robotics, where a fixed latency ceiling must be met on every request but images vary widely in complexity.
• Mobile and edge deployment of multimodal models, where available compute fluctuates with system load and a single model must serve multiple operating points.
• Accessibility tools for blind and low-vision users, a use case directly represented by the VizWiz benchmark, where responsiveness matters alongside accuracy.
• Cloud serving of multimodal APIs, where a shared model must satisfy different latency tiers and hardware budgets without maintaining separate checkpoints per configuration.

Industry relevance. Because the framework keeps one set of weights and changes only the target budget at inference — the controller temperatures are fixed across benchmarks and budgets — it maps cleanly onto the common industrial need to serve a single model across a spectrum of latency and cost tiers.

Future Directions

• Reporting the promised LLaVA-1.5-13B results, which are announced in the paper but fall outside the provided content, to establish whether the joint scheduling advantage holds at larger scale.
• Investigating why the layer-head variant loses ground at full budget relative to the layer-only variant, and whether width control can be made selectively beneficial rather than marginal.
• Extending the joint allocation to additional axes such as quantization precision or KV-cache management, which the paper cites as complementary but does not jointly control.
• Making budget projection smarter, since the current inference-time rule prioritizes truncating low-confidence tokens before touching layers or heads — an ordering the authors justify empirically rather than derive.
• Improving on the models where SmartVL only matches or slightly trails baselines (MMBench, GQA at 50%), where AdaLLaVA and AdaLLaVA-PruMerge remain competitive.

Target Audience

This paper is most valuable to researchers and engineers working on efficient multimodal LLM inference — particularly those building adaptive, budget-conditioned inference systems or serving infrastructure where latency targets vary per request. It is also relevant to practitioners doing model compression and conditional computation who want a template for jointly optimizing multiple compute axes with differentiable cost estimates, and to readers tracking how far token pruning and layer/head skipping can be pushed when combined rather than deployed in isolation. A background in transformer inference and multimodal architectures is needed to follow the controller design and training objective.

Authors’ abstract

Multimodal Large Language Models (MLLMs) have recently demonstrated strong performance across vision-language tasks. However, their high inference cost, arising from both the large number of input visual tokens and the heavy computation of the large language model (LLM), remains a key barrier to practical deployment. Recent work attempts to reduce the cost by adaptively optimizing individual dimensions, e.g., pruning redundant visual tokens or skipping LLM layers and heads. Nonetheless, prior approaches typically treat these dimensions independently and overlook a fundamental coupling: the available compute resources must be dynamically allocated across all dimensions based on the input content. To bridge the gap, we propose SmartVL, a unified adaptive inference framework that jointly controls vision token number and model compute capability in response to varying input contents and compute budgets. SmartVL introduces a vision-side token controller that dynamically selects informative visual tokens and an LLM-side compute controller that adaptively adjusts LLM computation. Importantly, these controllers are trained to coordinate with each other so that the overall inference cost satisfies a target budget. To allow this joint scheduling, we connect the controllers using a shared budget encoding and leverage a differentiable latency estimator for end-to-end training. This design enables SmartVL to learn cross-stage allocation strategies that adapt to both input complexity and runtime compute constraints. Experiments across multiple MLLM benchmarks demonstrate that, with joint scheduling, SmartVL consistently outperforms prior adaptive methods and achieves superior accuracy-efficiency Pareto frontiers. Project page: https://www.schaterji.io/publications/2026/jointtokencompute.