When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMs

AI summary

Overview

Research area: Efficient inference for Vision Large Language Models (VLLMs), specifically training-free visual token pruning.

Technical level: Intermediate. The paper assumes familiarity with transformer decoder layers, attention, and multimodal architectures, but its core argument is conceptual rather than mathematical.

Scope: The paper diagnoses why existing visual token pruning methods stop outperforming random pruning in deep decoder layers, proposes an information-theoretic metric to explain the effect, and shows that adding random pruning to existing methods improves both accuracy and efficiency.

What This Paper Is About

VLLMs convert images into hundreds or thousands of visual tokens before feeding them to a language decoder, which makes inference slow. Many training-free pruning methods exist to remove redundant visual tokens, but the authors observe that in deeper decoder layers these methods perform the same as, or worse than, simply removing tokens at random. The paper's goal is to explain this failure by measuring how much information each visual token actually carries at each layer, and to use that explanation to build a better pruning strategy.

Key Contributions

1. A proposed information metric for visual tokens. The authors estimate the information of a visual token by the change in the model's output probability on the ground-truth answer when that token is removed at a given layer. Removing tokens the metric scores as low-information consistently improves model performance.

2. The "information horizon" concept. The authors show that visual token information becomes uniform across tokens and then drops to near zero at an intermediate decoder layer. Beyond this horizon, visual tokens can be discarded without harming performance.

3. Evidence that the horizon is dynamic. The layer at which information vanishes depends on task visual complexity (OCR-style tasks need deeper tokens than knowledge QA) and on model visual capability (Qwen2.5-VL-7B exploits deeper tokens than LLaVA-1.5-7B).

4. A hybrid pruning recipe. Using existing pruning methods in shallow layers and random pruning in deep layers improves accuracy over the base pruning methods alone, and outperforms withdrawing all visual tokens at a fixed layer.

Main Findings

• Deep-layer pruning matches or loses to random. On LLaVA-1.5-7B, no evaluated pruning method beats random pruning in deeper layers (e.g., the 16th to 20th) across three benchmarks. On Qwen-2.5-VL-7B, random pruning becomes competitive after the 21st layer on MME and after the 24th layer on TextVQA.

• Information collapses with depth. Visual tokens capture varying amounts of information across layers 1 to 7, with high-information tokens clustered on key regions. From the 9th layer variability starts diminishing, and by the 16th layer visual tokens uniformly capture negligible information.

• Diversity-based methods beat importance-based ones in shallow layers. Across three pruning ratios, DivPrune and DART consistently outperform SparseVLMs, SGL and FastV. Random pruning surpasses all baseline methods by the 14th layer.

• Information horizon locations depend on task. For LLaVA-1.5-7B on MME, the mean information of all visual tokens nearly reaches zero at layer 16, and removing tokens there gives performance close to the original model. On TextVQA this occurs at the 24th layer.

• Model capability shifts the horizon. With Qwen-2.5-VL-7B, pruning all visual tokens at the 20th layer gives the highest accuracy for ScienceQA, MME and POPE, while for OCRBench and TextVQA this occurs around the 27th layer. For LLaVA-1.5-7B, the same comparison gives the 15th versus the 24th layer.

• Stronger models use deeper visual tokens. Qwen-2.5-VL-7B reaches 96.4% on MME versus 68.6% for LLaVA-1.5-7B when using visual tokens at the 20th layer. On TextVQA and OCRBench, Qwen-2.5-VL-7B attains approximately 80% accuracy by the 27th layer, whereas LLaVA-1.5-7B remains below 40% beyond the 24th layer.

• Random pruning improves existing methods. On Qwen2.5-VL-7B at 50% token retention, DART + Random raises OCRBench from 75.5% to 77.9%, and average retention across 7 benchmarks rises from 92.7% to 93.9%. On LLaVA-1.5-7B, DivPrune + Random reaches 61.3% on MMBench versus 54.6% for DivPrune alone at 88.9% pruning.

• Random pruning beats withdrawing all tokens. At the same 88.9% pruning ratio, DART + Random achieves 98.2% and 91.5% relative performance at 192 and 64 retained tokens, compared with 83.2% and 59.3% for VTW. On TextVQA, DART + Random reaches 82.7% versus 76.3% for DART + VTW.

• Efficiency gains. DART + Random cuts CUDA latency and FLOPs by 73.0% and 74.4% while maintaining 91.6% of LLaVA-1.5-7B's original performance. At 64 retained tokens it lowers FLOPs from 2.44T to 2.36T while improving accuracy from 50.4% to 53.4%.

• State-of-the-art combination. DivPrune with random pruning maintains 96.9% of Qwen-2.5-VL-7B performance while pruning 50% of visual tokens.

Methodology in Plain English

The authors first build a way to score a visual token's usefulness. At a chosen decoder layer, they delete every visual token except one, run the model forward, and record the probability it assigns to the correct first answer token. They then delete that last remaining visual token too, forcing the model to answer from text alone, and record the probability again. The difference between the two probabilities is the information score for that token at that layer. Low scores mean the token contributes little to the correct answer.

They validate the score by ranking visual tokens with it, removing 75% and 88% of the lowest-scoring ones (leaving 144 and 72 of LLaVA-1.5-7B's 576 tokens) at different layers, and checking MME and TextVQA. Removing low-information tokens consistently beat the unpruned model.

They then measure the total information retained by existing pruning methods — DivPrune, FastV, SparseVLMs, DART and SGL at 90%, 75% and 50% pruning ratios on 200 randomly chosen MME samples — and compare this against random pruning. They also track the mean and variance of token information per layer to locate the horizon, and sweep all six benchmarks while removing every visual token at each layer in turn.

Finally, they test hybrid schemes: a method such as FastV, DART or DivPrune prunes in a shallow layer, then either all remaining tokens are withdrawn (following VTW) or a portion is removed at random in a deep layer. These are run on Qwen2.5-VL-7B (28 decoder layers) and LLaVA-1.5-7B (32 decoder layers), with efficiency measured on a single NVIDIA A6000-40GB GPU using TextVQA.

Why This Matters

Impact on research. The paper reframes deep-layer token redundancy as an information phenomenon rather than an attention artifact, and shows that attention-based importance and similarity-based diversity signals both become uninformative once token information flattens. It also contradicts the fixed-layer assumption behind methods such as VTW by showing the horizon moves with task and model.

Real-world applications:

• Serving VLLMs under latency and memory budgets, where halving visual tokens can roughly halve per-layer attention cost without retraining.
• Document and text-in-image pipelines such as DocVQA, InfoVQA and OCRBench, where the paper shows deeper visual tokens remain informative and need gentler pruning.
• Edge or on-device multimodal assistants, where the memory and latency reductions (storage from 346.2 MB to 90.1 MB, latency from 272.4 ms to 162.2 ms for DART + Random at 64 tokens) matter directly.
• General-purpose vision-language question answering and hallucination detection, where random pruning in deep layers simplifies the pruning pipeline without accuracy loss.

Industry relevance. Random pruning adds no computation and is compatible with FlashAttention, unlike importance-based methods that require access to text-visual attention weights. That makes the approach easy to slot into existing serving stacks. The code is available at https://github.com/YahongWang1/Information-Horizon.

Future Directions

• Detecting the horizon automatically. The paper locates horizons empirically per model and task; an open question is whether the layer can be predicted at inference time from the input rather than fixed in advance.

• Explaining why information vanishes. The mechanism by which visual token information becomes uniform and then zero is described observationally, not mechanistically; understanding it could lead to architectures that delay the horizon.

• Better shallow-layer pruning. The information metric itself is expensive, since it requires per-token forward passes; a cheaper proxy that retains its ranking quality would make it usable directly at scale.

• Extension beyond static images. The study covers image benchmarks (MME, ScienceQA, POPE, TextVQA, OCRBench, OCRVQA and others); whether an analogous horizon exists for video or multi-image inputs is not reported.

Target Audience

Researchers and engineers working on efficient multimodal inference, particularly those building or benchmarking training-free token pruning and visual token compression methods. It is also relevant to practitioners deploying VLLMs under latency, memory or cost constraints, and to readers interested in interpretability of how vision information flows through a language decoder.

Authors’ abstract

Vision Large Language Models (VLLMs) incur high computational costs due to their reliance on hundreds of visual tokens to represent images. While token pruning offers a promising solution for accelerating inference, this paper, however, identifies a key observation: in deeper layers (e.g., beyond the 20th), existing training-free pruning methods perform no better than random pruning. We hypothesize that this degradation is caused by \textbf{``vanishing token information''}, where visual tokens progressively lose their salience with increasing network depth. To validate this hypothesis, we quantify a token's information content by measuring the change in the model output probabilities upon its removal. Using this proposed metric, our analysis of the information of visual tokens across layers reveals three key findings: (1) As layers deepen, the information of visual tokens gradually becomes uniform and eventually vanishes at an intermediate layer, which we term as ``information horizon", beyond which the visual tokens become redundant; (2) The position of this horizon is not static; it extends deeper for visually intensive tasks, such as Optical Character Recognition (OCR), compared to more general tasks like Visual Question Answering (VQA); (3) This horizon is also strongly correlated with model capacity, as stronger VLLMs (e.g., Qwen2.5-VL) employ deeper visual tokens than weaker models (e.g., LLaVA-1.5). Based on our findings, we show that simple random pruning in deep layers efficiently balances performance and efficiency. Moreover, integrating random pruning consistently enhances existing methods. Using DivPrune with random pruning achieves state-of-the-art results, maintaining 96.9\% of Qwen-2.5-VL-7B performance while pruning 50\% of visual tokens. The code is available at https://github.com/YahongWang1/Information-Horizon.