Look Only When Needed: Certificate-Guided Visual Routing for Efficient Diffusion VLMs

AI summary

Overview

Research area: Efficient inference for diffusion vision-language models (DVLMs) — specifically, deciding when a denoising step actually needs to look at the image.

Technical level: Intermediate. The paper assumes familiarity with masked/diffusion language model decoding (iterative unmasking of response tokens), vision-language model input layout (visual tokens, prompt tokens, response tokens), and key-value caching. The core idea itself is simple enough to explain without that background.

Scope (one sentence): The paper measures how often the image changes a diffusion VLM's decoding decision, then turns that measurement into a training-free, threshold-free decoding rule — called LOWN — that skips the image whenever a cheap text-only prediction can be certified as unchanged, retaining 99.6% of LLaDA-V's average score on ten benchmarks at 0.353× the decoding FLOPs.

What This Paper Is About

Diffusion VLMs like LLaDA-V re-process their entire input — image plus prompt plus partially decoded response — at every denoising step, and visual tokens make up 89–97% of that input on the benchmarks the authors study. Pruning methods shorten the visual sequence, but they permanently discard information and still touch the remaining image at every step. The authors instead ask a different question: at a given decoding step, does removing the image entirely change which token the model would unmask? Their goal is a decoding procedure that consults the image only on the steps where it matters, and that also uses the same signal to decide how many tokens to unmask per step.

Key Contributions

1. A measurement of visual redundancy in DVLM decoding. By running a FULL forward (image included) and a TEXT-ONLY forward (image removed, position IDs preserved) side by side along LLaDA-V's own trajectory on ten benchmarks, the authors show that the two agree on the unmasked top-1 token at 83.4% of positions, that 85.0% of final tokens were already the top-1 prediction before the step that unmasked them, and that the image changes an unmasked decision on only 22.8% of steps on average — ranging from 7.9% on MMBench to 43.8% on DocVQA.

2. LOWN, a certificate-guided decoding rule. The last image-conditioned predictions and confidences are stored as an "anchor"; a cheap TEXT-ONLY forward certifies anchor tokens whose prediction it agrees with, and the longest confidence-ordered certified prefix decides whether the step skips the FULL forward entirely. The method requires no training, no task-specific threshold, and changes neither model parameters nor attention pattern.

3. Joint control of image access and unmasking width. Because a TEXT-ONLY step accepts the whole certified prefix rather than the baseline's fixed k tokens, LOWN can unmask more than k tokens per step while never exceeding the baseline's step count; on FULL steps it unmasks in "disagreement-first" order so that image-dependent tokens are not deferred.

4. Evidence that the rule composes and transfers. LOWN retains 99.6% of LLaDA-V's score at 0.353× FLOPs, scores highest on eight of ten benchmarks against accelerated baselines given at least its FLOPs budget, composes with RedVTP, ZOO-Prune and Fast-dLLM key-value caching, and transfers unchanged to LaViDa and to VidLaDA-8B on video benchmarks.

Main Findings

• The image rarely decides the token. Averaged over ten benchmarks, 83.4% of unmasked positions receive the same top-1 token from the FULL and TEXT-ONLY forwards; per-benchmark agreement ranges from 71.8% to 92.1%, and the share of final tokens that were already top-1 before unmasking ranges from 74.0% to 90.2%.
• Image dependence is uneven, not periodic. The share of image-dependent steps ranges from 7.9% (MMBench) to 43.8% (DocVQA), with a 22.8% mean; on eight of ten benchmarks disagreement concentrates in early denoising steps, but DeCapBench is given as a counterexample to that pattern. This is the authors' argument against a fixed schedule.
• Headline efficiency. Standalone LOWN keeps 99.6% of LLaDA-V's average score at 0.353× the decoding FLOPs; capping each benchmark at 100% (so gains above baseline cannot offset losses) gives 98.6%. It has the best score on eight of the ten benchmarks among methods given at least its FLOPs budget, and uses one tuning-free rule whereas KLASS uses benchmark-specific thresholds.
• Scheduling beats fixed and random access. Against a random schedule matched to LOWN's average FULL-forward count, LOWN improves task-normalized quality by 4.9–13.0% on MathVista, LLaVA-Bench and DeCapBench. Period 4 uses similar compute on MathVista but scores 4.0 points lower (52.00 vs 56.00).
• Most of the saving is skipping the FULL forward. Visual routing alone cuts FLOPs by 27–68%; adaptive acceptance (whole-prefix acceptance plus disagreement-first order) removes a further 21–43% of the remaining cost, scores higher on seven of ten benchmarks, and loses at most 2.8% normalized quality elsewhere.
• Composition with pruning is favourable. LOWN + RedVTP at 50% retention reaches a normalized average of 98.3 at 0.223× FLOPs and scores highest on all ten benchmarks; LOWN cuts a further 57.6% of RedVTP's FLOPs and 60.6% of ZOO-Prune's (33% retention). On MathVista, RedVTP at 50% retention with LOWN scores 54.9 at 0.195× FLOPs, versus 51.7 at 0.354× for RedVTP at 30% retention — less pruning plus routing beats more pruning.
• Composition with key-value caching helps but costs quality. Adding LOWN to Fast-dLLM reduces generation time by 38.1–50.1% and FLOPs by 44.1–58.9% relative to Fast-dLLM. On all three tested tasks the combined score is lower than Fast-dLLM alone, with a 5.0-point drop on LLaVA-Bench.
• Transfers across backbones. On LaViDa, LOWN retains 99.7% of the baseline average at 0.174× FLOPs; on MMBench and GQA it scores 77.18 vs 76.95 and 56.54 vs 56.69 at 0.088× and 0.091× FLOPs. Adding LOWN to RedVTP at 75% or ZOO-Prune at 33% retention cuts their FLOPs by a further 77.3% and 72.2% at quality indices of 99.5 and 99.8.
• Video results. On VidLaDA-8B under a 128-token decoding protocol, LOWN uses 0.255× the baseline FLOPs on EgoSchema (60.2 vs 60.6) and 0.272× on MLVU (43.3 vs 41.8, an improvement).
• Wall-clock. Across the ten LLaDA-V benchmarks, standalone LOWN reduces generation time by 57.4% on average and by up to 73.2%.
• Certificates are usually fresh and usually right. With a one-step-old anchor, a fresh FULL forward overturns 4.1–11.0% of certified tokens, rising with anchor age to 25.3% on DocVQA at eight steps old; on LOWN's own trajectories the rate is 2.2–4.7%, and 92–97% of accepted tokens use an anchor from the preceding step, none older than five. An oracle schedule that touches the image only where it flips a decision costs 0.33× versus LOWN's measured 0.35× — and is not a strict lower bound, since it still runs one forward per baseline step.
• Qualitative contrast with pruning. In the paper's example, LLaDA-V and LOWN answer correctly using the relevant visual evidence while RedVTP, having pruned it, answers fluently but incorrectly; LOWN's FULL steps concentrate on visually grounded content and the surrounding language is handled text-only.

Methodology in Plain English

The authors start with a diagnostic rather than a method. They take LLaDA-V's normal decoding run and, at every step, also compute what the model would predict with the visual tokens simply deleted — keeping every other token's position ID unchanged so the comparison is apples-to-apples. The image-free pass never influences the decoding state, so the trajectory stays the baseline's by construction; the authors just log where the two passes disagree and which positions the image-bearing pass unmasks. That log yields the 83.4% agreement figure and shows that image dependence varies a lot by benchmark and by step.

The method then reuses that same cheap comparison as an online test. Whenever the model does a full image-bearing forward, LOWN saves its top-1 predictions and confidences for all still-masked positions — the anchor. On the next step it runs only the cheap text-only forward and, scanning masked positions in order of anchor confidence, asks: does this text-only prediction still match the anchor? A match is treated as a certificate that the image was not needed and the older prediction is still valid. The longest unbroken run of matches from the top of that confidence order becomes the set of tokens unmasked at this step. If that run is at least as long as the baseline's fixed per-step unmasking count k, the step ends with no image pass at all and may unmask more than k tokens. If the run is too short, the step pays for a full image-bearing forward, refreshes the anchor entirely, and unmasks k positions — putting the positions where text-only and full predictions disagreed first, so image-dependent content is resolved immediately. A small "repetition guard" stops the certified run before a token that duplicates an already-unmasked neighbour, which prevents repeated words from leaking out of the independently predicted prefix.

Cost accounting follows directly: relative FLOPs are the FULL-forward frequency f plus the relative step count σ times the text-only unit cost ρ, and ρ is 3–10% because the image is most of the sequence. Pruning composes because it shrinks the FULL term while LOWN shrinks f. Evaluation uses ten image benchmarks (MathVista, ScienceQA, ChartQA, MME, MMBench, DocVQA, GQA, LLaVA-Bench, CapArena, DeCapBench) plus EgoSchema and MLVU for video, with baselines RedVTP, ZOO-Prune, KLASS, D³ToM and Fast-dLLM; comparators are assigned at least LOWN's FLOPs budget, chosen by FLOPs rather than by task score, so the comparison is conservative in LOWN's disfavour.

Why This Matters

Impact on research. Efficient-inference work on diffusion VLMs has mostly attacked the cost of a forward (pruning visual tokens, caching intermediate states) or the number of steps (multi-token unmasking schedules). This paper

Authors’ abstract

Diffusion vision-language models (DVLMs) process their entire input at every denoising step. Since visual tokens make up 89-97% of this input on the benchmarks we study, most of the decoding computation is spent repeatedly processing the same image. Visual token pruning methods partially relieve this burden by removing part of the visual tokens, but they lose the removed information and still process the remaining image at every step. To measure how often the image is actually needed, we compare LLaDA-V's predictions with and without the image at every decoding step and find that removing the image changes a token being unmasked at only 22.8% of steps on average and at as few as 7.9%. We therefore propose LOWN, a decoding method that decides at every step whether to process the image and how many tokens to unmask. LOWN predicts the masked tokens without the image and processes the image only when this prediction disagrees with the last image-conditioned prediction. Agreement certifies the earlier prediction for reuse, and the number of certified tokens sets how many are unmasked in that step. We further observe that the need for the image differs across benchmarks, so LOWN's adaptive decision is needed rather than a fixed schedule. Without training or a tuned threshold, LOWN retains 99.6% of LLaDA-V's score on ten benchmarks at 35.3% of the decoding FLOPs on average. LOWN also generalizes across LLaDA-based DVLMs, composes orthogonally with visual token pruning, and runs under key-value caching for further efficiency gains. Code and project page are available at https://hyeon9698.github.io/LOWN/