Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Models

AI summary

Overview

Research area: Efficient inference for large vision-language models (LVLMs), specifically compressing the key-value (KV) cache used during autoregressive decoding.

Technical level: Advanced. The paper builds on attention-score analysis and introduces a per-attention-head mathematical criterion, though the underlying intuition (keep what matters, keep what is different) is easy to grasp.

Scope: The paper analyzes semantic redundancy in multi-modal KV caches and proposes MixKV, a plug-and-play scoring function that mixes importance with diversity to improve existing KV compression methods.

What This Paper Is About

Long visual inputs (high-resolution images, long videos) produce huge numbers of KV pairs in an LVLM's language model, creating a memory bottleneck. Existing KV compression methods keep only the "most important" pairs, but the authors show that importance alone keeps many near-duplicate, semantically similar pairs and misses part of the original information distribution. MixKV adds a diversity term, weighted adaptively per attention head, so each head keeps a set of KV pairs that better covers the full semantic range of the uncompressed cache.

Key Contributions

1. Semantic redundancy analysis. The authors measure KV cache similarity in LVLMs versus text-only LLMs and report that VL model keys peak around 0.6–0.8 average similarity versus 0.2–0.4 for the text model (a 2–3× increase), and that this redundancy varies strongly across attention heads (some heads exceed 0.9 average similarity, others stay below 0.3).
2. Evidence that importance-only selection under-covers. A t-SNE visualization (Figure 3) shows SnapKV focusing on a small portion of the KV distribution and missing the coverage of the full uncompressed cache.
3. MixKV, a head-wise adaptive mixing mechanism. A redundancy estimate computed per layer and head — the off-diagonal average similarity of normalized keys — sets a weight that blends importance and diversity scores in a single comprehensive score.
4. Comprehensive validation. MixKV is integrated with SnapKV, PyramidKV, AdaKV, and SparseMM across multiple LVLMs, image-understanding benchmarks, a GUI grounding benchmark, and long-context text benchmarks, with consistent gains and comparable inference efficiency.

Main Findings

• Redundancy is higher in multi-modal than in text-only caches. Keys in Qwen2-VL peak around 0.6–0.8 average similarity while Qwen2 keys peak around 0.2–0.4, and Qwen2-VL shows denser high-similarity regions in the same layer and head.
• Redundancy varies by head. For Qwen2-VL-7B, some heads exceed 0.9 average cosine similarity while others stay below 0.3, consistently across vision-language tasks — motivating per-head rather than uniform weighting.
• Importance-only compression loses semantic coverage. The t-SNE figure shows SnapKV (blue stars) covering only a small part of the full KV distribution (light gray circles), which MixKV widens.
• Extreme compression gains. At budget = 64, MixKV improves baseline methods by an average of 5.1% across five multi-modal understanding benchmarks (DocVQA, OCRBench, TextVQA, ChartQA, TextCaps).
• Large gains on GUI grounding. On ScreenSpot-v2 with Qwen2.5-VL-7B-Instruct at budget 128, MixKV improves SnapKV's average precision from 75.3% to 83.3% (+7.9%), and improves AdaKV by 4.4% at 128 and 9.0% at 64.
• Consistent per-benchmark improvements. On LLaVA-NeXT-Mistral-7B at budget 64, SnapKV gains +1.5 (DocVQA), +4.2 (OCRBench), +3.0 (TextVQA), +0.9 (ChartQA), and TextCaps rises from 0.444 to 0.514 with MixKV.
• Smaller but positive gains on strong baselines. SparseMM, which already allocates head-wise budgets, improves modestly (for example +0.2 and +1.6 on LLaVA-NeXT-Mistral-7B OCRBench at budgets 256 and 64), with a few negative deltas such as −1.1 on OCRBench at budget 256.
• Task-dependent behavior on text. On LongBench with Mistral-7B-Instruct-v0.2 and Llama-3.1-8B-Instruct, MixKV improves average scores, with clearer gains in Information Aggregation (summarization) and occasional declines in Information Localization, which the authors attribute to lower head-wise redundancy in LLMs making diversity a distraction.
• Importance design choice matters. Ablations (Table 4) indicate that combining extrinsic attention importance with value norm (VNorm) beats combining it with key norm (KNorm), and that online per-sample head weights are compared against offline, OCRBench-derived, sample-shared head weights. The table's numeric results are cut off in the provided text.

Methodology in Plain English

The authors start from the standard picture of an LVLM: a vision encoder and projector turn an image or video into visual tokens, which the language model processes alongside text tokens, storing keys and values for every token in the pre-filling phase.

1. Score importance. They combine two views of importance: an extrinsic score, the average attention a key receives from an observation window at the end of the prompt (default length 32), and an intrinsic score, the norm of the value vector. The value norm is normalized to [0, 1] and rescaled so its magnitudes match the attention scores before adding.
2. Score diversity. For each head, they normalize every key vector, average all keys into one global key, and use the negative cosine similarity between each key and that average. Keys far from the average are considered diverse. This costs linear time in sequence length.
3. Measure head redundancy. Using the normalized keys, the total pairwise similarity sum equals T² times the squared norm of the average key, so the off-diagonal average similarity can be computed without building the full T×T similarity matrix.
4. Mix adaptively. The comprehensive score is (1 − r) × importance + r × scaled diversity, where r is the head's redundancy estimate. Highly redundant heads lean toward diversity; low-redundancy heads lean toward importance.
5. Compress. For each layer and head, the top B pairs by comprehensive score are kept, where B is the KV cache budget. MixKV changes only the evaluation function, not the compression operator, so it plugs into existing pipelines.

Why This Matters

KV cache compression is what makes long-context multi-modal inference affordable, and most current methods treat "importance" as the only signal. This paper shows that a second, cheap signal — semantic diversity — is systematically missing, and that LVLMs have a distinctive redundancy structure that text-only LLMs do not. That reframes the compression objective from "keep the most attended tokens" to "approximate the original information distribution."

Real-world applications:

• On-device GUI agents. The authors explicitly tie the +7.9% average ScreenSpot-v2 gain for SnapKV at budget 128 to edge-side deployment, where memory is tight.
• High-resolution document and OCR pipelines. DocVQA, OCRBench, TextVQA, ChartQA, and TextCaps are the exact workloads where aggressive compression usually hurts most.
• Multi-turn mobile or web assistants. These need the KV cache compressed to keep latency and GPU memory low during long interactions.
• Long-context text serving. The LongBench results with Mistral-7B-Instruct-v0.2 and Llama-3.1-8B-Instruct show the same scoring function transfers to text-only LLMs.

Industry relevance: any team serving LVLMs under a fixed GPU memory budget can drop MixKV into an existing compression method without retraining and without offline calibration, since head redundancy is estimated on the fly per sample. That is a low-cost path to either higher throughput or better accuracy at the same memory.

Future Directions

• Video evaluation. The paper discusses VideoLLMs such as LLaVA-Video and Video-XL-2, but the reported experiments cover image understanding, GUI grounding, and text; video benchmarks are not evaluated in the provided content.
• When diversity hurts. LongBench Information Localization results suggest diversity can dilute attention when head redundancy is low. Characterizing and predicting those cases, or adapting the mixing per task, is an open question.
• Combining with orthogonal efficiency techniques. The related-work section positions KV cache quantization, token compression, sparse and linear attention, and state-space models as complementary; how MixKV stacks with them is not reported.
• Scaling evidence. The paper states that Tables 6 and 7 verify robustness on larger and MoE-based models (InternVL3-38B and Qwen3-VL-30B-A3B-Instruct); those tables are not included in the provided content, so their specific numbers are not available here.
• Cheaper or richer redundancy estimates. The current weight is a single scalar per head derived from average key similarity; whether finer-grained or layer-level estimates help is not addressed.

Target Audience

Researchers and engineers working on efficient LLM/LVLM inference, KV cache management, and long-context multi-modal serving. It is also useful for practitioners deploying vision-language models on memory-constrained or edge hardware, and for readers who already know methods such as SnapKV, PyramidKV, AdaKV, or SparseMM and want to understand what a diversity-aware extension adds. Readers without a background in attention mechanisms will find the math in Section 3.3 demanding, though the motivation and experimental tables are readable on their own.

Authors’ abstract

Recent large vision-language models (LVLMs) demonstrate remarkable capabilities in processing extended multi-modal sequences, yet the resulting key-value (KV) cache expansion creates a critical memory bottleneck that fundamentally limits deployment scalability. While existing KV cache compression methods focus on retaining high-importance KV pairs to minimize storage, they often overlook the modality-specific semantic redundancy patterns that emerge distinctively in multi-modal KV caches. In this work, we first analyze how, beyond simple importance, the KV cache in LVLMs exhibits varying levels of redundancy across attention heads. We show that relying solely on importance can only cover a subset of the full KV cache information distribution, leading to potential loss of semantic coverage. To address this, we propose MixKV, a novel method that mixes importance with diversity for optimized KV cache compression in LVLMs. MixKV adapts to head-wise semantic redundancy, selectively balancing diversity and importance when compressing KV pairs. Extensive experiments demonstrate that MixKV consistently enhances existing methods across multiple LVLMs. Under extreme compression (budget=64), MixKV improves baseline methods by an average of 5.1% across five multi-modal understanding benchmarks and achieves remarkable gains of 8.0% and 9.0% for SnapKV and AdaKV on GUI grounding tasks, all while maintaining comparable inference efficiency. Furthermore, MixKV extends seamlessly to LLMs with comparable performance gains. Our code is available at https://github.com/xuyang-liu16/MixKV.