Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Prior

AI summary

Overview

Research area: Computer vision / multimodal AI — specifically efficiency optimization for Video Large Language Models (VLLMs) via dynamic visual token compression. Technical level: Advanced (assumes familiarity with transformer attention, token pruning, and video LLM architectures). Scope: The paper introduces DyToK, a training-free method that uses VLLM attention layers as a built-in "keyframe prior" to decide how many visual tokens to keep per video frame, and evaluates it as a plug-in for existing compression methods on three long-video benchmarks.

What This Paper Is About

Video LLMs are slow because long videos produce enormous visual token sequences, and self-attention cost grows quadratically with sequence length. Existing fixes either prune tokens using a single fixed ratio for every frame (ignoring that some frames matter far more than others), or they select keyframes before feature encoding using auxiliary models, which adds cost rather than removing it. DyToK asks a sharper question — not "how many tokens to drop" but "which frames deserve more tokens" — and answers it by reading the model's own attention distribution over frames, allocating larger token budgets to query-relevant frames and smaller budgets to redundant ones.

Key Contributions

1. Empirical discovery of a keyframe prior in VLLMs. The authors show that attention from the final text token to visual tokens peaks on query-relevant frames even when the model produces a wrong answer, indicating an inherent task-dependent keyframe bias in the attention mechanism itself.
2. DyToK, a training-free dynamic compression paradigm. The method converts cross-modal attention into per-frame importance weights and uses them to allocate a global token budget non-uniformly across frames, rather than applying one fixed retention ratio to all frames.
3. Demonstrated plug-and-play compatibility. DyToK integrates with encoder feature-based methods (VisionZip, DyCoke-encoder) and LLM attention-based methods (FastV, DyCoke-LLM) without retraining, and works across multiple VLLMs including LLaVA-OneVision and Qwen2.5-VL.
4. State-of-the-art efficiency-accuracy tradeoffs. The paper reports 4.3× faster inference while preserving accuracy, with gains that grow larger as compression becomes more aggressive.

Main Findings

• Attention identifies keyframes even when answers are wrong. Figure 1 shows LLaVA-OneVision's answers to video QA tasks with averaged attention from the final text token to visual tokens across all layers; the top-8 frames by attention score are arranged in time order with ground-truth keyframes highlighted in red, and the attention still pinpoints relevant frames regardless of answer correctness.
• Deeper layers give better keyframe priors. With a 20% retention ratio and 32 frames, layer 20 achieved the best result at an average score of 53.1 (98.2% of the unpruned baseline) and layer 23 reached 52.0 (96.1%), versus layer 0 at 48.9 (90.5%), layer 4 at 50.4 (93.1%), layer 8 at 50.0 (92.3%), layer 12 at 50.9 (94.1%), and layer 16 at 51.6 (95.4%). The paper notes that using the deepest layer to guide the first layer nearly doubles inference costs, motivating a separate assistant model.
• A 14× smaller assistant model works nearly as well. The Tiny setting uses LLaVA-OneVision-0.5B while the Base setting uses LLaVA-OneVision-7B. Differences across retention levels were −0.6 at 25%, +0.3 at 50%, −1.5 at 75%, −0.9 at 80%, and −1.5 at 90%.
• Large gains on encoder feature-based compression. At 50% compression (3136 tokens), DyToK improves VisionZip† by 4.2% average accuracy; at 90% compression (448 tokens) the improvement reaches 18.9%, and using the 7B prior reaches +20.4%. DyCoke gains were +1.2 at 25%, +1.5 at 50%, +1.3 at 75%, +0.0 at 80%, and +1.3 at 90%.
• Consistent gains on LLM attention-based compression. At 85% compression, DyToK improves FastV by 6.1% accuracy across benchmarks; gains were +0.5 at 25%, +1.8 at 50%, and +2.2 at 75%. At a 90% pruning rate, the token budget allocation reduces FastV's retained visual tokens to zero, so those results are omitted.
• Headline improvements in the introduction. Under 20% token retention, DyToK surpasses uncompressed baselines by 2.6% accuracy on LongVideoBench, and at extreme compression ratios (10% retention) it achieves a 24.0% performance gain over competitors.
• Practical implementation choice. To avoid manual tuning, the authors uniformly average attention scores from the last one-third of layers for all models.

Methodology in Plain English

DyToK has two stages, and neither requires training.

Stage 1 — estimating which frames matter. The system looks at the attention weights between the last token of the user's question and all the visual tokens. Frames whose visual tokens receive high attention are judged important; those receiving low attention are judged redundant. Because using the full-size model's deep layers would require running the whole model anyway, the authors run a small "assistant" model from the same family (LLaVA-OneVision-0.5B instead of 7B) and average attention only from its deepest third of layers. The result is a normalized importance weight for each frame.

Stage 2 — spending the token budget. Given a global token budget, each frame's initial allocation is its importance weight multiplied by the total budget, rounded down. Leftover tokens are distributed to the frames whose fractional remainders were closest to earning another token, and a per-frame cap prevents any single frame from hoarding tokens — excess is redistributed to frames that still have capacity. Finally, any existing compression function (encoder-based or attention-based) is applied per frame with that frame's individual budget.

The effect is that a critical frame might keep most of its 196 tokens while an irrelevant frame keeps very few, instead of both being cut to the same fixed fraction.

Why This Matters

Impact on research: The paper challenges the binary frame-selection paradigm and the uniform-ratio token-pruning paradigm simultaneously. It reframes compression as a budget allocation problem rather than a selection problem, and it shows that the keyframe signal already exists inside the model's attention — no extra detector, no fine-tuning. Because it is modular, it can be layered onto existing pruning methods as an add-on, which makes it directly comparable with a broad class of prior work.

Real-world applications:

• Long-video question answering for surveillance, where only a handful of minutes in an hour of footage correspond to a query.
• Video archive search and media asset management, where users need to query hours of stored content conversationally.
• Assistive and clinical video review, where a specialist asks targeted questions about a long recorded session.
• On-device or edge video assistants, where memory and compute budgets make fixed per-frame compression wasteful.

Industry relevance: The 4.3× inference speedup and compatibility with both encoder-based and attention-based pipelines make DyToK attractive for deployment, since it does not require retraining existing VLLMs or replacing existing compression stacks. The 14× smaller assistant model is also relevant to cost-sensitive serving, where running a second large model for routing would negate the savings.

Future Directions

• Removing the assistant model entirely. The authors explicitly state in the limitation section that this work has not yet proposed a better method to avoid introducing an additional model, and that future efforts will focus on addressing this.
• Extending beyond the last-third layer heuristic. The paper adopts a uniform averaging of the last one-third of layers for generalizability; a learned or adaptive layer-selection criterion remains open.
• Scaling to more VLLM families and longer inputs. The main text presents key results only for LLaVA-OneVision; Qwen2.5-VL, extended video lengths, and broader model sizes are reported in appendices (A.2, A.3, A.4) rather than the main text.
• Behavior at extreme compression. At 90% pruning, FastV's retained visual tokens drop to zero and DyToK has nothing to reallocate, so that case is omitted — indicating a boundary condition where budget reallocation alone cannot help.

Target Audience

Researchers and engineers working on efficient multimodal inference, video understanding, or LLM serving infrastructure. It is most useful to readers who already understand transformer attention and token pruning and want a concrete, training-free drop-in technique. Readers new to video LLMs will find the motivation section accessible, but the methodology and experimental tables assume intermediate-to-advanced background.

Authors’ abstract

Recent advances in Video Large Language Models (VLLMs) have achieved remarkable video understanding capabilities, yet face critical efficiency bottlenecks due to quadratic computational growth with lengthy visual token sequences of long videos. While existing keyframe sampling methods can improve temporal modeling efficiency, additional computational cost is introduced before feature encoding, and the binary frame selection paradigm is found suboptimal. Therefore, in this work, we propose Dynamic Token compression via LLM-guided Keyframe prior (DyToK), a training-free paradigm that enables dynamic token compression by harnessing VLLMs' inherent attention mechanisms. Our analysis reveals that VLLM attention layers naturally encoding query-conditioned keyframe priors, by which DyToK dynamically adjusts per-frame token retention ratios, prioritizing semantically rich frames while suppressing redundancies. Extensive experiments demonstrate that DyToK achieves state-of-the-art efficiency-accuracy tradeoffs. DyToK shows plug-and-play compatibility with existing compression methods, such as VisionZip and FastV, attaining 4.3x faster inference while preserving accuracy across multiple VLLMs, such as LLaVA-OneVision and Qwen2.5-VL. Code is available at https://github.com/yu-lin-li/DyToK .