Do Vision Language Models Need to Process Image Tokens?

AI summary

Overview

• Research area: Multimodal representation analysis for vision language models (VLMs) — computer vision and multimodal LLM interpretability/efficiency.
• Technical level: Intermediate. The core ideas are intuitive (when does looking deeper at an image stop helping?), but the paper uses formal representation metrics such as matrix entropy, intrinsic dimensionality, and trajectory curvature.
• Scope (one sentence): The paper systematically investigates whether image tokens in VLMs need to be processed through the full transformer depth, or whether visual representations stabilize early and can be truncated in a task-dependent way.

What This Paper Is About

Vision language models feed a dense sequence of image tokens through every layer of a large language model decoder, which is computationally expensive, yet it is unclear whether all that depth actually matters for visual information. The authors ask when visual representations stop changing in meaningful ways, whether layers become functionally interchangeable for image tokens, and whether cutting off visual processing at some depth breaks the model's outputs. Their goal is to replace assumptions about "visual redundancy" with a measurable, representation-level account of when image tokens matter and when they do not.

Key Contributions

1. A representation-centric characterization of image-token dynamics: Using matrix entropy (Von Neumann entropy and effective rank), intrinsic dimensionality, and trajectory curvature, the authors show that image tokens rapidly converge to a bounded-complexity regime while textual tokens continue to be restructured across depth — a pattern observed across six models.
2. A layer-substitution protocol testing functional interchangeability: The authors construct hybrid states combining image tokens from an earlier layer with textual tokens from a later layer and show that image-token substitution preserves output semantics (similarity ≈ 1.0 across layer gaps), whereas textual substitution degrades steadily with layer gap.
3. A depth-truncation study establishing task-dependent necessity: By removing all image tokens beyond a cut layer, the authors show that single-token prediction tolerates early truncation, while multi-token generation (VQA open-form answers, captioning) requires sustained visual depth.
4. An adaptation analysis (recoverability, stability, and compute): Distillation-based LoRA fine-tuning of truncated models partially restores base-model behavior, reduces cross-decoding variability, and shifts the compute-performance frontier — with efficiency gains coming mainly from the prefill stage rather than decode.

Main Findings

• Visual representations stabilize early: After a brief adjustment phase, image-token matrix entropy remains confined within a narrow band, effective rank plateaus, intrinsic dimensionality stabilizes, and curvature stays nearly constant. Textual tokens, by contrast, show sustained entropy fluctuations, alternating expansion and compression in intrinsic dimensionality, and larger, more variable curvature — especially in intermediate layers.
• Stabilization implies functional interchangeability for image tokens: In layer-substitution experiments with Qwen2.5-VL-7B-Instruct, image-token similarity heatmaps stay consistently strong and similarity as a function of layer gap is nearly flat (≈ 1.0). Text-token substitution shows clear off-diagonal degradation that increases with depth difference. The pattern holds across model families (LLaVA-1.5-7B, InternVL3-8B) and tasks.
• Interchangeability is not redundancy: The authors are explicit that interchangeable visual tokens are not necessarily dispensable; beyond a certain depth, further processing yields minimal semantic modification.
• Necessity of visual depth is task-dependent: Under image-token removal, single-token accuracy degrades smoothly and recovers monotonically with retained visual depth. Full-string correctness (Index + Answer Exact Match) shows substantially larger degradation. Captioning is consistently the most sensitive, with semantic similarity dropping sharply under shallow truncation and approaching full-model performance only when visual tokens are preserved through later stages. BLEU and ROUGE (ROUGE-1/2/L) show the same monotonic trend.
• Model differences under truncation: Across models, LLaVA exhibits the strongest recovery as visual depth increases, while InternVL shows the weakest recovery, indicating greater sensitivity to early visual truncation.
• Partial recoverability through fine-tuning: With LoRA fine-tuning (LLaVA-1.5-7B and Qwen2.5-VL-7B-Instruct), captioning on Flickr8k recovers substantially across most truncation depths, with deeper truncation enabling near-complete alignment to base-model outputs. ChartQA (without answer choices) improves in exact match but recovers far less. Recoverability increases monotonically with retained depth (dR/dK > 0), and asymptotically captioning approaches full-model alignment while ChartQA remains below it under early truncation.
• Recovery also stabilizes generation: Truncated base models show high variability across decoding strategies (greedy, beam, nucleus, top-k, temperature sampling), especially at shallow depths, while fine-tuned models display a markedly reduced coefficient of variation across all depths.
• Compute gains come from prefill, not decode: Total GFLOPs increase almost linearly with retained visual depth. Fine-tuning shifts the compute-performance frontier so that fine-tuned models achieve higher accuracy per unit of compute than truncated base models at the same budget. The prefill-stage change dominates the decode-stage change; specific GFLOP savings figures are not reported in the paper.
• Reasoning does not compensate for lost visual depth: On the M3CoT dataset, with structured reasoning chains producing <summary>, <caption>, <reasoning>, and <conclusion> outputs, a consistent hierarchy emerges: Caption degrades most, followed by Reasoning, with Summary comparatively stable. Under aggressive truncation, single-token exact match exceeds reasoning-chain exact match (A_single(K) > A_chain(K)), meaning structured decoding increases depth dependence rather than reducing it.
• Intermediate quality and final answers can decouple: Degradation in explanation quality does not always cause final answer failure, indicating partial decoupling between representational fidelity and decision correctness.
• Structural stabilization is not inferential sufficiency: Stabilized representations can support coarse, single-token decisions, but iterative, grounded reasoning still requires sustained visual depth.

Methodology in Plain English

The authors study three VLM families — LLaVA-1.5, Qwen2.5-VL-Instruct, and InternVL — covering parameter scales from 3B to 72B, and use identical images across captioning and VQA-style prompts so that differences come from the task, not the input.

They track how token representations change layer by layer using three measurements: (1) matrix entropy, which describes how concentrated or spread out the representation's energy is across directions, plus effective rank derived from it; (2) intrinsic dimensionality, which estimates the effective degrees of freedom of the representation; and (3) trajectory curvature, the mean angular deviation between successive layer-to-layer updates for each token, capturing how sharply a token's direction changes.

To test function rather than geometry, they build hybrid representations: for layers l_a < l_b, they take image tokens from layer l_a and text tokens from layer l_b, forward the combination through the remaining decoder layers, and compare the output to the unmodified model using semantic similarity (captioning) or Exact Match (VQA).

To test necessity, they select a cut layer and delete all image-token activations beyond it, consistently adjusting position IDs, attention masks, cache positions, and positional embeddings, then sweep the cut layer across depth. They then fine-tune truncated models with LoRA using the full base model's own outputs as targets (a distillation objective) rather than human annotations, evaluated on ChartQA without answer choices and Flickr8k captioning. Finally, on M3CoT, they compare single-token answer prediction against full reasoning-chain generation to see whether explicit reasoning buffers against lost visual depth.

Why This Matters

The paper reframes image-token efficiency from a heuristic pruning problem into a measurable property of multimodal representations. It provides a principled diagnostic — representational stabilization — for identifying depth regions where visual processing yields bounded change, and it shows that necessity, not just redundancy, is what should drive architectural decisions.

Real-world applications:

• Task-aware inference optimization: Deploying short-answer or multiple-choice multimodal systems with less visual depth processing while preserving full depth for captioning and open-form generation.
• Latency- and compute-constrained deployment: Since savings concentrate in the prefill stage, systems that process many images per query (large-scale or batch inference) see the largest benefit.
• Document and chart question answering: Understanding where fine-grained visual-text grounding becomes irreplaceable, since ChartQA-style exact-match tasks were the least recoverable under truncation.
• Multimodal assistants and captioning services: Where descriptive, coarse-semantic generation proved largely redistributable after adaptation.

Industry relevance: the results inform architecture and serving decisions for multimodal models, suggesting that uniform deep visual processing may over-allocate compute, and that fine-tuning a deliberately truncated model can be a practical route to a better compute-performance trade-off.

Future Directions

• Designing architectures around representational stabilization rather than post-hoc truncation, allocating visual depth where tasks actually require it.
• Determining practical selection criteria for how deep to retain visual tokens per task, given that the paper shows necessity scales with output complexity and evaluation strictness.
• Closing the irreversibility gap for precise grounding tasks such as ChartQA, where aggressive early truncation could not be fully recovered through fine-tuning.
• Testing whether the observed structural decoupling between representation geometry, functional necessity, and computational allocation generalizes to additional architectures, modalities, and reasoning formats beyond those studied.

Target Audience

Researchers working on multimodal LLM architectures, interpretability, and inference efficiency; engineers designing or serving vision language systems under compute or latency budgets; and practitioners deciding how much visual processing their applications actually need. Readers with some familiarity with transformer internals and representation analysis will get the most from the methodology sections, though the core conclusions are accessible to anyone working with VLMs.

Authors’ abstract

Vision Language Models (VLMs) have achieved remarkable success by integrating visual encoders with large language models (LLMs). While VLMs process dense image tokens across deep transformer stacks (incurring substantial computational overhead), it remains fundamentally unclear whether sustained image-token processing is necessary for their performance or visual representations meaningfully evolve from early to later layers. In this work, we systematically investigate the functional role of image tokens in VLMs and show that visual representations rapidly converge to a bounded-complexity regime, \ie their entropy stabilizes, intrinsic dimensionality compresses, and trajectory curvature approaches a near-constant profile. In contrast, textual representations continue to undergo substantial restructuring across depth. Once stabilized, visual representations become largely interchangeable between layers, indicating limited additional transformation in deeper stages. Further, depth-wise visual truncation reveals that the necessity of visual processing is task-dependent, where single-token predictions remain comparatively robust to truncated visual depth, but multi-token generation require sustained access to visual representations. Under deterministic decoding, reducing visual depth perturbs intermediate reasoning trajectories more strongly than final outputs, suggesting that image tokens influence the structure of reasoning more than the ultimate conclusions. Collectively, these findings \textbf{question the assumption} that deeper visual processing is uniformly essential in VLMs, challenging the current paradigm of multimodal LLM architectures.