SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs

AI summary

Overview

Research area: Efficient inference for Multimodal Large Language Models (MLLMs) — specifically, visual token pruning in the Computer Vision category (arXiv:2510.24214v1 [cs.CV], 28 Oct 2025).

Technical level: Intermediate. The paper assumes familiarity with transformer attention, vision encoders such as CLIP ViT, and standard multimodal benchmarks, but the central idea (balancing saliency against semantic coverage) is conceptually simple.

One-sentence scope: The paper proposes SCOPE, a training-free visual token pruning strategy that jointly optimizes token saliency (attention scores) and semantic coverage (pairwise token similarity) to cut the number of visual tokens fed into LLaVA-1.5, LLaVA-Next, and Video-LLaVA with minimal accuracy loss.

What This Paper Is About

MLLMs convert images into long sequences of visual tokens, and the cost of self-attention grows quadratically with sequence length, so inference becomes expensive — the paper notes a ViT encoder on a 448×448 image can generate over 1,000 visual tokens, LLaVA produces 576 tokens per image, and LLaVA-Next produces 2,880 tokens (576 × 5) after splitting an image into roughly 4 sub-images plus the original. Existing pruning methods rank tokens by attention (saliency) alone and keep the top-k, which concentrates selection on a few dominant patches and discards context needed for full visual understanding. The goal of SCOPE is to select a small token subset that is both highly salient and semantically representative of the whole image.

Key Contributions

1. Diagnosis of saliency-based pruning. The paper identifies two limitations: saliency-based methods compromise semantic completeness by concentrating on a narrow subset of tokens, and they suffer from highly skewed attention distributions where a few tokens dominate while the rest have nearly uniform (flat) attention values, reducing discriminability. The authors quantify this with a new θ-coverage metric and show that "Saliency Only" selection has low coverage across θ values and even performs worse than random selection on the MME benchmark.

2. The SCOPE selection framework. A set-coverage score defined from pairwise token cosine similarities, a token-coverage gain measuring the marginal coverage added by including a candidate token, and a SCOPE score that multiplies the coverage gain by the token's attention score raised to a scaling factor α.

3. An iterative, training-free pruning algorithm. Algorithm 1 greedily selects the token with the highest SCOPE score, then updates coverage scores, repeating until the target budget K is reached. The module is inserted after the vision encoder and requires no additional model training.

4. Extensive validation across models and modalities. Experiments integrate SCOPE into LLaVA-1.5, LLaVA-Next, and Video-LLaVA, covering seven image benchmarks and four video benchmarks, with comparisons against FastV, SparseVLM, VisionZip, and PDrop.

Main Findings

• Large token reduction with small accuracy loss on LLaVA-1.5 7B. SCOPE achieves a 9× reduction in visual tokens while retaining 96.0% of the original performance. (The introduction and Section 3.3 also describe this as a "factor of 8" reduction while preserving over 96%; both figures appear in the paper.)
• Consistent wins at every budget. At 192 tokens (a 66.7% reduction from 576), SCOPE reaches 99.5% average relative accuracy, surpassing FastV (+6.0%), SparseVLM (+3.0%), and VisionZip (+1.5%). At 128 tokens (77.8% reduction) it reaches 98.1%. At 64 tokens (88.9% reduction) it holds 96.0%, versus VisionZip at 93.5% and SparseVLM at 85.1%. PDrop at 64 tokens is reported as 98.7% average relative accuracy with a much lower raw GQA score of 41.9.
• Occasional gains over the full-token upper bound. With 192 tokens on LLaVA-1.5 7B, relative accuracies of 100.2% on POPE and 104.5% on MMVet are reported, which the authors attribute to removing interference from redundant visual tokens.
• Strong results on LLaVA-Next 7B (2,880 default tokens). At 640 tokens (77.8% reduction) SCOPE reaches 98.9% average relative accuracy; at 320 tokens (88.9% reduction) 97.1%; at 160 tokens (94.4% reduction) 95.1%, compared with SparseVLM at 86.9% and VisionZip at 92.5%.
• Near-lossless video results on Video-LLaVA. Reducing video tokens from 2,048 to 136, SCOPE reports TGIF 47.1, MSVD 69.2, MSRVTT 55.9, and ActivityNet 44.9, against the Video-LLaVA baseline of 47.1, 69.8, 56.7, and 43.1 respectively. The authors state SCOPE achieves the best performance among compared methods (FastV, SparseVLM, VisionZip).
• Stability under extreme compression. In Figure 4, as token counts are reduced to as few as 8, SCOPE's margin over VisionZip grows, while VisionZip drops sharply in low-token regimes.
• Ablation confirms complementarity. On GQA, MMB, MME, POPE, and TextVQA, the full method (58.3, 61.7, 1698, 83.9, 56.6) beats coverage-only (58.1, 60.8, 1687, 82.1, 56.3), saliency-only (55.0, 60.8, 1665, 76.8, 55.4), and random selection (55.5, 54.0, 1556, 75.2, 48.4). The token budget used for this ablation is not specified in the available content.
• Efficiency versus PDrop on LLaVA-NeXT 7B. Reducing tokens from 2,880 to 160 (over 18× compression), SCOPE keeps POPE at 81.3 versus 86.4 for the full model and 53.2 for PDrop, with latency of 188.8 s versus 601.9 s for the full model (3.2× speedup); PDrop is slightly faster at 184.0 s (3.3×).
• Lower θ-coverage for saliency-only selection. Figure 2, measured on MME with 64 tokens selected from 576 in LLaVA 1.5 7B, shows saliency-only selection has consistently low θ-coverage and performs below random selection, while SCOPE achieves significantly higher coverage.
• Visualization behavior. Saliency-only concentrates on salient objects (such as a cat and a banana); coverage-only spreads tokens across the image, preserving global context but possibly missing object detail; SCOPE keeps high token density on salient patches while strategically retaining a sparse set of background tokens.

Methodology in Plain English

The method treats token selection as a coverage-maximization problem rather than a ranking problem.

1. Measure similarity between tokens. Every pair of visual tokens is compared with cosine similarity, producing a similarity matrix over the full token set V.
2. Define coverage. For a token u, its coverage by a selected set S is the maximum cosine similarity between u and any token in S. The total set-coverage f(S) is the sum of these maximum similarities over all tokens in V. Intuitively, a token is "covered" if something similar to it has been selected.
3. Compute the gain of adding a token. The token-coverage gain Δ(v; S) is how much total coverage increases if candidate token v is added — the sum over all tokens of how much their best-match similarity improves.
4. Blend in saliency. The gain alone ignores how information-rich a token is, so the authors multiply it by the token's attention score raised to a scaling factor α: Δ(v; S) · A_v^α. This is the SCOPE score.
5. Select greedily and iteratively. The token with the highest SCOPE score is added to S, coverage scores are updated, and the loop repeats until K tokens are retained.

Implementation details: the pruning module sits after the vision encoder; saliency is the attention weight of visual tokens with respect to the CLS token at the second-to-last layer (layer −2) of the vision encoder; α defaults to 1.0; the codebase is built on the lmms-evals package; experiments run on 4×A100 GPUs with inference batch size 1. No training is required.

Benchmarks used: GQA, MMBench, POPE, ScienceQA, TextVQA, SEEDBench, and MMVet for images, plus TGIF, MSVD, MSRVTT, and ActivityNet for video. Baselines: FastV, SparseVLM, VisionZip, and PDrop. The paper states that the GQA evaluation uses the "testdev_balanced_instructions" subset with 12,578 samples, MMBench contains approximately 3,000 multiple-choice questions, and MME is evaluated on the dev split with 4,377 samples across 14 subtasks.

Why This Matters

Impact on research. The paper reframes visual token pruning from "which tokens are most attended?" to "which tokens best represent the whole image?", supplying a quantitative definition (θ-coverage) that other pruning methods can be measured against. The finding that saliency-only selection can underperform random selection in coverage terms is a direct challenge to a widely used design pattern.

Real-world applications:

• Deploying multimodal assistants on edge devices, robots, and other hardware where long visual token sequences are prohibitively expensive, as the introduction motivates.
• High-resolution image understanding, where LLaVA-Next-style tiling produces 2,880 tokens per image and pruning with over 18× compression is shown to be feasible.
• Video question answering and video-language assistants, where token counts multiply across frames; SCOPE reduces Video-LLaVA's 2,048 video tokens to 136 while nearly matching baseline scores.
• Serving MLLMs at scale, where per-request latency (601.9 s to 188.8 s in the reported LLaVA-NeXT 7B experiment) translates directly into throughput and cost.

Industry relevance. Because SCOPE is training-free and inserts after the vision encoder, it can be applied to existing deployed checkpoints without retraining or fine-tuning, and it is model-agnostic across LLaVA-1.5, LLaVA-Next, and Video-LLaVA. The latency and POPE numbers on LLaVA-NeXT 7B indicate that saliency-only pruning such as PDrop degrades accuracy sharply under aggressive budgets, which matters for production systems that must trade accuracy against cost.

Future Directions

• Scaling to larger models. Appendix C.1 is stated to report results on LLaVA-1.5 13B and LLaVA-Next 13B; those numbers are not included in the available content, so whether the gains hold at larger scale is not verifiable here.
• Sensitivity to the scaling factor α. The appendix is said to include a hyperparameter analysis, but only the default α = 1.0 is described in the main text. How α trades saliency against coverage, and whether it should adapt per task or per layer, remains open.
• Where in the pipeline to prune. SCOPE is inserted after the vision encoder; whether pruning inside the encoder or at later LLM layers would change the saliency-coverage trade-off is not explored.
• Robust selection under flat attention. The paper documents skewed attention distributions but does not specify how SCOPE behaves when nearly all attention values are flat, nor how sensitive the coverage gain is to the cosine-similarity geometry of tokens from different encoders.
• Limitations and broader impact. The appendix states that broader impact and limitations are discussed, but that discussion is not present in the provided content.

Target Audience

Researchers and engineers working on inference efficiency for multimodal LLMs, particularly those implementing token pruning, compression, or selection in vision-language pipelines. It is also useful for practitioners deploying LLaVA-family models on latency- or memory-constrained hardware, and for students studying attention-based token selection who want a concrete example of replacing pure top-k saliency ranking with a diversity- or coverage-aware objective. Readers need basic familiarity with transformer attention and multimodal benchmarks to follow the results tables.

Authors’ abstract

Multimodal Large Language Models (MLLMs) typically process a large number of visual tokens, leading to considerable computational overhead, even though many of these tokens are redundant. Existing visual token pruning methods primarily focus on selecting the most salient tokens based on attention scores, resulting in the semantic incompleteness of the selected tokens. In this paper, we propose a novel visual token pruning strategy, called \textbf{S}aliency-\textbf{C}overage \textbf{O}riented token \textbf{P}runing for \textbf{E}fficient MLLMs (SCOPE), to jointly model both the saliency and coverage of the selected visual tokens to better preserve semantic completeness. Specifically, we introduce a set-coverage for a given set of selected tokens, computed based on the token relationships. We then define a token-coverage gain for each unselected token, quantifying how much additional coverage would be obtained by including it. By integrating the saliency score into the token-coverage gain, we propose our SCOPE score and iteratively select the token with the highest SCOPE score. We conduct extensive experiments on multiple vision-language understanding benchmarks using the LLaVA-1.5 and LLaVA-Next models. Experimental results demonstrate that our method consistently outperforms prior approaches. Our code is available at \href{https://github.com/kinredon/SCOPE}{https://github.com/kinredon/SCOPE}.