PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference

AI summary

Overview

Research area: Efficient multimodal inference — specifically visual token compression for Vision-Language Models (VLMs) in computer vision.

Technical level: Advanced. The paper assumes familiarity with Vision Transformer (ViT) encoding, LLM prefill, cross-attention and self-attention, and token-budget terminology, though it defines its own mechanisms clearly.

Scope: A training-free, plug-and-play inference framework that compresses visual information both before and after the vision encoder to accelerate high-resolution VLM inference.

What This Paper Is About

High-resolution images and videos produce enormous visual token sequences in VLMs, making inference slow. Existing pruning methods cut tokens only after the vision encoder, which leaves the cost of encoding high-resolution pixels untouched and, under tight budgets, destroys fine-grained details such as text strokes and chart elements. PACE aims to attack both bottlenecks at once by condensing the input before encoding and extracting the most useful tokens after encoding.

Key Contributions

1. Identification of the dual bottleneck. The authors show that conventional visual token pruning ignores the vision-encoder overhead entirely and degrades sharply on detail-sensitive tasks because it cannot jointly preserve holistic layout and fine-grained detail.
2. A unified Condense-and-Extract framework. PACE combines a pre-encoder Adaptive Pixel Compressor (APC) with a post-encoder Dynamic Dual-Attention Extractor (DDAE), the latter fusing ViT self-attention with LLM cross-attention via confidence-weighted attention fusion.
3. Superior performance-efficiency trade-off. On Qwen2.5-VL-7B, PACE retains over 93% of original performance while discarding 90% of visual tokens, with a 3.1 times TTFT acceleration (reported as 3.13 times in the efficiency profiling).
4. Demonstrated orthogonality. APC is shown to compose with existing post-encoder pruning methods (VisionZip and MMTok), improving them on detail-sensitive benchmarks.

Main Findings

• Performance at a 10% token budget: On Qwen2.5-VL-7B under the fixed-resolution setting, PACE retains 93.8% of the full model's normalized average performance, versus 87.7% for PACE without APC, 81.4% for SparseVLM, 81.1% for VisionZip, 80.4% for MMTok, 79.9% for FastV, 72.6% for DART, and 72.5% for DivPrune. The paper reports average gains of 12.7 and 13.9 percentage points over VisionZip and FastV respectively.
• Stronger retention at tighter budgets: At 5% retention PACE averages 84.3%, compared with 76.9% for PACE without APC, 70.5% for VisionZip, 70.3% for SparseVLM, 69.6% for FastV, 67.3% for MMTok, 62.2% for DART, and 62.0% for DivPrune. At 20% retention PACE averages 98.6%.
• Detail-sensitive tasks are where baselines fail: At 10% retention, PACE reaches DocVQA 69.55 versus 59.56 for FastV and 49.55 for VisionZip. The paper notes DocVQA still trails Vanilla (94.74) because exact answers can depend on microscopic characters that become ambiguous after condensation.
• Robust scaling across continuous budgets: Sweeping from 100% down to 5% on POPE, MMStar and OCRBench, PACE at 10% achieves 84.99, 59.08 and 70.90, beating the strongest baseline by 1.44, 4.18 and 17.90 absolute points. At 5% it exceeds the best baseline by 2.00, 7.39 and 14.70 points.
• Latency reduction (10% budget, single RTX 4090): Vision encoder latency drops from 148.84 ms to 49.47 ms (3.01 times), LLM prefill from 217.05 ms to 32.69 ms (6.64 times), and TTFT from 365.89 ms to 116.79 ms (3.13 times). APC adds an average 34.62 ms preview-and-resizing overhead. Average TTFT speedup rises from 1.10 times at 80% retention to 1.67 times, 2.64 times and 3.13 times at 50%, 20% and 10%.
• APC is orthogonal to existing pruners: Adding APC at a 10% budget improves VisionZip by more than 15 points on ChartQA (53.08 to 68.96) and OCRBench (49.40 to 67.50), plus roughly 10 points on DocVQA (49.55 to 60.45). MMTok gains 15.80 points on ChartQA and 15.50 on OCRBench at the same budget; gains persist at 5%.
• Adaptive beats static resolution: Under the dynamic-resolution setting, fixed resolution at 50% gives a marginal 0.36-point gain over adaptive resolution on ChartQA at 10% but loses 2.74 points on RealWorldQA, which the authors attribute to rigid global scaling overfitting specific domains.
• Dual attention matters: Relying only on LLM attention degrades ChartQA by over 10 points; relying only on ViT attention weakens task alignment. Dynamic fusion is the most balanced option tested against single-modality and fixed-weight (0.5/0.5) fusion baselines.
• Extraction depth trades quality for latency: At depth 2, prefill is 35.26 ms with a 2.11 times speedup and ChartQA 73.52; at depth 24, ChartQA rises to 78.96 but prefill grows to 68.12 ms and speedup drops to 1.09 times, measured against a 74.52 ms PACE-without-DDAE baseline.
• Cross-backbone generalization: On InternVL3.5-4B, across 25%, 20% and 10% retention, PACE attains normalized averages of 85.2%, 80.9% and 69.5%, exceeding the strongest corresponding baseline by 5.1, 4.7 and 4.0 points.
• Preview signal choice: A controlled ablation (Appendix B.2) confirms the semantic feature preview is more reliable than RGB, entropy, edge-density and Laplacian statistics.
• End-to-end generation is not accelerated: PACE does not speed up autoregressive decoding, which is why the paper reports time to first token rather than end-to-end generation time. TPOT measurements are mentioned as being collected but no TPOT values are reported in the available content.

Methodology in Plain English

PACE has two stages that bracket the vision encoder.

Condense (before encoding). The Adaptive Pixel Compressor first runs one shallow ViT block to obtain a cheap "preview" of the image's visual features. From this preview it computes two scores. The global information density score measures how non-redundant the image is, by averaging pairwise cosine similarity between normalized tokens — a high similarity average (large uniform areas) means high redundancy and low density. The local detail contrast score measures how far individual tokens deviate from the image-wide average token; the authors average the distances of the top 10% most deviant tokens (so a single noisy outlier cannot dominate), and scale the result. The two scores are blended with a weighting hyperparameter into a single target retention ratio, which determines how much the raw image is resized (bicubic interpolation, both width and height scaled by the square root of the ratio) before encoding. If a system memory budget caps the allowed scaling below that ratio, the image is resized down to the hard budget instead.

Extract (after encoding). The Dynamic Dual-Attention Extractor gathers two attention signals: semantic relevance from an early LLM layer's cross-modal attention, and visual density from the final ViT block's self-attention. Each is min-max normalized to [0,1]. Because a sharper (higher standard deviation) distribution indicates more confident focus on a concise set of regions, the standard deviations of the two normalized maps are passed through a softmax with a temperature parameter to produce fusion weights. The weighted combination yields a final saliency score, and the top-K tokens are kept for the LLM context.

Evaluation. All experiments use the open-source lmms-eval framework with greedy decoding on Qwen2.5-VL-3B and 7B, plus InternVL3.5-4B for cross-backbone checks, compared against FastV, SparseVLM, DivPrune, DART, VisionZip and MMTok across nine benchmarks. Latency is profiled on a single RTX 4090.

Why This Matters

Impact on research. The paper reframes visual token compression as a two-sided problem rather than a purely post-encoder one, and shows that a pre-encoder module can be dropped into existing pruning pipelines as an orthogonal add-on. That gives the efficiency community a concrete demonstration that encoder-side cost is a first-class target, not a fixed constant.

Real-world applications:

• Document and form processing, where DocVQA-style scanned-document reading is latency-bound on high-resolution pages.
• Chart and figure understanding, where grid lines, axis labels and numerical structure are exactly the cues that aggressive post-encoder pruning discards.
• OCR-heavy mobile or edge assistants, where scene text and dense text regions must survive compression.
• High-resolution visual question answering and video understanding pipelines, where the paper notes extensive video also produces massive visual token sequences.

Industry relevance. The method is training-free and requires no architectural modification, so it can be applied to already-deployed models without retraining. The reported 3.01 times encoder and 6.64 times prefill reductions at 10% retention translate directly into lower serving latency and higher throughput for GPU-bound multimodal serving stacks. The paper also cautions that APC's encoder-side gains depend on the backbone: on fixed-grid VLMs, DDAE still reduces prefill cost but APC provides no encoder-side benefit.

Future Directions

• Query-conditioned or uncertainty-triggered recovery. The authors explicitly propose recovering high-resolution crops when condensation has made evidence ambiguous, since APC is query-agnostic and one-shot and DDAE cannot reconstruct detail that resizing already destroyed.
• Extending encoder-side gains to fixed-grid VLMs. APC only helps the encoder when cutting pixels or tiles also cuts encoder token count, leaving multi-tile or fixed-grid designs unaddressed.
• Reducing preview overhead across backbones. The authors note preview overhead varies with architecture and resolution, so the Qwen2.5-VL speedups may not transfer unchanged elsewhere.
• Accelerating decoding, not just prefill. Since PACE does not speed up autoregressive decoding, end-to-end generation latency remains an open target.
• Adaptive hyperparameter selection. The paper's sensitivity study shows the global-local weight trades off layout/relation-sensitive MMStar categories against coarse perception and math reasoning, and recommends a lower weight for high-stakes detail-sensitive use.

Target Audience

Researchers and engineers working on efficient multimodal inference, visual token compression, or VLM serving infrastructure. It is also relevant to practitioners deploying high-resolution document, chart or OCR workloads under latency constraints, and to readers interested in training-free methods that compose with existing pruning pipelines. Readers without background in ViT/LLM internals will need to consult the referenced pruning literature to place the results in context.

Authors’ abstract

Vision-Language Models (VLMs) demonstrate exceptional visual reasoning capabilities, yet their inference costs escalate rapidly with the proliferation of visual tokens. Existing visual token pruning methods exhibit two fundamental limitations. First, most approaches operate exclusively post-vision encoder, leaving the substantial latency of the visual encoding phase unoptimized. Second, under strict token budgets, these methods often fail to jointly preserve holistic visual contexts and fine-grained details, leading to performance degradation. To address these bottlenecks, we propose PACE (Pixel-Adaptive Condense and Extract), a training-free inference framework that accelerates both the vision encoder and the Large Language Model (LLM) via a unified Condense-and-Extract paradigm. During the Condense stage, an Adaptive Pixel Compressor (APC) evaluates visual information density prior to encoding, adaptively downsampling redundant inputs, curtailing encoder computation while preserving global context and essential visual cues. In the Extract stage, a Dynamic Dual-Attention Extractor (DDAE) selectively retains visual tokens via a fusion of internal visual signals from the encoder and semantic signals from the LLM, safeguarding task-critical details. By integrating PACE into Qwen2.5-VL-7B, the model retains 93.8% of its original performance while utilizing only 10% of the visual tokens, yielding a 3.1x speedup in time to first token (TTFT). Our code is available at https://github.com/jjL357/PACE.