C3: Speech-Guided KV Retrieval for Long-Context SpeechLLM Inference

AI summary

Overview

Research area: Efficient inference for speech large language models (SpeechLLMs) — specifically KV-cache retrieval for long-context audio. Technical level: Advanced (assumes familiarity with transformer attention, KV caching, and grouped-query attention). Scope: The paper diagnoses why text-oriented KV retrieval fails on audio, proposes a speech-specific retrieval method called C³, and evaluates it on four SpeechLLMs across recognition, translation, and summarization.

What This Paper Is About

SpeechLLMs encode audio at a much higher temporal rate than text, so the key-value (KV) cache grows quickly with input duration and becomes the main memory and decoding bottleneck for long recordings. KV retrieval keeps the full cache but lets each decoding step attend to only a small, query-dependent subset. Existing retrieval methods were designed for text and, the authors show, can misalign the decoder from the audio and cause errors to compound autoregressively into "generation drift." The goal is a retrieval scheme that matches the actual structure of audio attention and stays accurate under very tight retrieval budgets.

Key Contributions

1. Diagnosis of retrieval-induced generation drift. The authors identify a failure mode where retrieval errors compound autoregressively, and derive a bound showing the attention-output perturbation scales with the omitted attention mass (1 − ρ_t) times the diameter of the value vectors D_t.
2. Three empirical properties of audio attention that motivate speech-specific design: unequal concentration across query heads sharing a KV head, non-contiguous audio support, and temporal continuity of high-attention regions across decoding steps.
3. The C³ method (Cluster–Compete–Compete), which combines hierarchical key-space clustering during prefill, shared retrieval competition among query heads within a KV group (max-based scoring), and adaptive cluster resolution via paired split–merge updates under a fixed retrieval budget.
4. A broad evaluation across four SpeechLLMs (Qwen3-Omni-30B-A3B, Qwen2.5-Omni-3B, Gemma-4-E4B, Voxtral-Mini-3B) on long-form ASR, multilingual ASR, speech translation, and summarization, plus a needle-in-a-haystack passcode test and latency/memory measurements.

Main Findings

• Strongest overall accuracy under matched budgets: C³ is the best retrieval method in 31 of the 32 settings shown in Table 1, with the largest gains under tight budgets.
• Dramatic tight-budget gains on Qwen3-Omni: On LS-Long at a 128-KV budget, C³ reduces WER from 180.94% with ClusterKV to 7.07%. At 512 KV it reaches 1.98% versus 2.02% for full attention.
• Large translation gains: On Qwen3-Omni speech translation at a 256-KV budget, chrF rises from 25.91 (ClusterKV) to 55.79; at 1024 KV, 71.96 versus 70.15 for ClusterKV and 74.75 for full attention.
• Consistent across models: On Qwen2.5-Omni-3B LS-Long at 128 KV, C³ gives 40.48 WER versus 133.13 for ClusterKV and 51.49 for UNIQUE. On Gemma-4-E4B at 128 KV, 14.63 versus 95.72 for ClusterKV; on Voxtral-Mini-3B at 128 KV, 41.86 versus 212.96.
• Not uniformly best in every cell: In a few settings the ablated variants are ahead — for example, on Qwen2.5-Omni-3B LS-Long at 128 KV, the "+C2+C3" configuration from the C1 (ClusterKV) base gives 40.48 but the "+C2" variant gives 64.73, while on the Quest/mean base the "+C2+C3" result is 63.63 and "+bias (UNIQUE)" is 51.49. On Gemma-4-E4B summarization at 256 KV, C³ gives 19.83 versus 20.19 for ClusterKV.
• Needle-in-a-haystack: On 120 recordings of roughly 13–16 minutes containing three spoken seven-digit passcodes, C³ reaches 95% exact-match accuracy with only 32 retrieved KVs and lossless recovery with 64 KVs on Qwen3-Omni, while ClusterKV and temporal baselines need several hundred KVs to approach the same accuracy.
• Latency crossover: With a 512-KV budget on Qwen3-Omni-30B-A3B, retrieval adds overhead at short contexts but both C1+C2 and full C³ become faster than full attention at approximately 8K audio tokens, measured up to 32K tokens.
• Memory scaling: Offloading the full audio KV bank and inactive hierarchy nodes to host memory reduces accelerator-resident audio state from O(N) to O(A + B), where N is the number of audio KVs, A the active-frontier size, and B the retrieval budget.
• Evidence for the three attention properties: At one displayed layer, the concentrated head group (about one third of query heads) has mean normalized attention entropy of approximately 0.40 versus 0.79 for the diffuse group, and places 85% of attention mass on audio versus about 12%. The fraction of top-64 attention mass outside the dominant band averages 34% for recognition, 29% for translation, and 78% for summarization. Mean Jaccard overlap of top-64 sets at decoding-step offsets of 1, 2, 4, 8 is 0.58/0.46/0.34/0.24 for recognition, 0.56/0.44/0.33/0.22 for translation, and 0.45/0.34/0.24/0.16 for summarization; the peak position of audio-relevant heads moves a median of about four audio frames (0.3 s) per decoding step.
• A concrete drift example: In an ASR case, the retrieved region advances with the transcript but jumps to a later audio segment at decoding step 276, after which the transcript loses alignment and emits an end-of-sequence token prematurely.

Methodology in Plain English

The method works per KV head and per layer, in stages.

First, during prefill, the audio keys for each KV head are recursively split with k-means into a binary hierarchy, where each node is summarized by its centroid. A fixed number of nodes from an intermediate level form an "active frontier" that partitions all audio tokens — every token belongs to exactly one active node — but nodes can sit at different depths, so some regions are described finely and others coarsely.

Second, at each decoding step, each active node is scored by the best cosine similarity between its centroid and any of the query heads that share that KV head. Taking the maximum (rather than averaging) means one strongly matching, concentrated head can promote a cluster without diluting it. All query heads in that group then share a single ranking and one common budget, instead of retrieving separately.

Third, nodes are consumed in descending score order until the per-KV-head budget B is filled, and exact attention is computed over that bounded set of at most B audio KVs.

Fourth, the frontier adapts. Each node keeps an exponential moving average of its relevance score; at each step a high-priority node is split into its two children and a low-priority sibling pair is merged into its parent, keeping the frontier size fixed while shifting fine resolution toward regions that are currently relevant. Newly activated nodes inherit the relevance state of the nodes they replace.

Cost-wise, full attention scores N audio KVs, whereas C³ scores only A centroids and then runs exact attention over at most B retrieved KVs, with A ≈ N/g for initial granularity g. Implementation uses one-token pages in a paged KV cache, FlashInfer's batch decode kernel, and custom Triton kernels for gathering and for fused split–merge updates; no retraining or architecture change is needed, and retrieval is bypassed when the audio has no more than B tokens. Default settings are average granularity g = 16, EMA strength λ = 0.5, and two paired split–merge updates per decoding step; k-means uses eight iterations per split with random seed 0.

Evaluation covers LS-Long (270 recordings), a multilingual benchmark built from MLS by concatenating same-chapter segments into roughly four-minute recordings in Dutch, German, French, and Spanish (658 examples), speech translation on 100 Marco-LongSpeech examples from each of six languages (Spanish, German, French, Dutch, Italian, Indonesian), and summarization on the 545 examples in the first Marco-LongSpeech release partition. Metrics are WER for ASR, chrF for translation, and ROUGE-L for summarization; budgets of 128, 256, 512, and 1024 audio KVs are matched against full attention, Quest, UNIQUE, and ClusterKV.

Why This Matters

Impact on research. The paper reframes long-context SpeechLLM efficiency as a problem of audio attention structure rather than generic sparsity. The drift analysis gives a quantitative reason why tight-budget retrieval fails — omitted attention mass compounds across decoding steps — and the three identified properties (head concentration asymmetry, non-contiguous support, temporal continuity) are reusable design targets for future speech-specific cache methods. The result that key-space clustering beats fixed temporal chunking is a direct challenge to the page-based assumptions inherited from text LLM work.

Real-world applications:

• Transcribing hour-scale recordings such as meetings, lectures, and call-center logs without keeping the entire audio cache resident on the accelerator.
• Multilingual speech translation of long audio, where the reported chrF improvement at a 256-KV budget (25.91 to 55.79 on Qwen3-Omni) is the difference between unusable and useful output.
• Summarization of long recordings where evidence is scattered — the needle-in-a-haystack result shows isolated facts such as passcodes or figures can be recovered with as few as 32–64 retrieved KVs.
• Deployment on memory-constrained or cost-sensitive hardware, where reducing accelerator-resident audio state from O(N) to O(A + B) lowers the barrier to serving long-form speech.

Industry relevance. The method is training-free and patches only the selection of audio KVs, leaving non-audio prompt and generated-token KVs on the model's original path — a drop-in optimization for serving stacks. The latency crossover at roughly 8K audio tokens and the offloading strategy matter directly for throughput and GPU-memory budgeting in production. Two of the authors' affiliations are Amazon AGI, and the accelerator setups (NVIDIA H200, Ada 6000) reflect deployment-relevant conditions.

Future Directions

• Generalization beyond the four models: the authors note that accuracy/efficiency trade-offs may vary with other model architectures, audio encoders, and generation settings, and that hybrid architectures with fewer full-attention layers expose less KV-cache state to optimize.
• Truly long contexts: at fixed cluster granularity the number of active representatives grows with audio length, so extremely long audio may require coarser granularity or an additional hierarchical search stage; system results beyond 32K audio tokens are analytical projections rather than measured generation quality.
• Hardware and kernel portability: the context length at which retrieval beats full attention is measured on H200 and Ada 6000 and may shift on other accelerators or with more specialized kernels.
• Retrieval quality under tighter and adaptive budgets: since the largest gains appear at tight budgets, and a few table cells favor ablated variants, further work on budget allocation across layers and heads, and on closing the remaining gap to full attention at large budgets, is a natural next step.

Target Audience

Researchers and engineers working on efficient LLM inference, KV-cache management, and speech-language models, particularly those deploying long-form audio systems under memory or latency constraints. It is also valuable for readers interested in how attention structure differs between modalities, since the diagnostic analysis (head concentration, non-contiguity, temporal continuity, drift) stands on its own apart from the proposed method. Readers without background in attention mechanics and KV caching will find the method sections dense, though the diagnostic findings are accessible.

Authors’ abstract

Speech language models encode audio at a much higher rate than text, causing the KV cache to dominate the memory, bandwidth, and decoding cost of long-form generation. KV retrieval can reduce per-step access by attending to only a query-dependent subset of the cache, but existing methods are designed primarily for text and do not account for the structure of audio attention. We first show that retrieval errors may compound autoregressively into generation drift. We then analyze full audio attention during long-form generation and identify three properties relevant to retrieval: unequal attention concentration across query heads, non-contiguous audio support, and temporal continuity across decoding steps. Guided by these observations, we propose $C^3$(Cluster-Compete-Compete), which organizes audio KVs in a hierarchical key-space index, lets query heads sharing a KV head compete for a common retrieval budget, and adaptively reallocates retrieval resolution through paired split-merge updates. Across long-form speech recognition, translation, and summarization on four SpeechLLMs, $C^3$ achieves the strongest overall performance among retrieval methods under comparable final-KV access budgets, with particularly large gains in the tight-budget regime, while improving long-context decoding efficiency and accelerator-memory scaling.