KV Cache Transform Coding for Compact Storage in LLM Inference

AI summary

Overview

• Research area: Efficient LLM inference and serving — specifically key-value (KV) cache compression and memory management.
• Technical level: Advanced (the core idea is graspable by an intermediate reader, but the method relies on PCA/SVD, dynamic-programming bit allocation, and quantization theory).
• Scope: The paper introduces kvtc, a transform-coding pipeline that compresses KV caches for on-GPU and off-GPU storage, and evaluates it on models from 1.5B to 70B parameters across math, knowledge, long-context, and coding benchmarks.

What This Paper Is About

Serving LLMs at scale means every token of context occupies KV cache memory — for example, 128 MiB per 1K tokens for Llama 3.1 8B and 320 MiB for Llama 3.3 70B Instruct in 16-bit. In multi-turn chat and iterative code editing, a cache is reused across turns when the prompt prefix matches, but stale caches consume scarce GPU memory, force offloading, or force expensive recomputation. The paper's goal is a lightweight compressor that shrinks KV caches for storage and transfer without changing the model's weights, so that caches can be retained longer and moved more cheaply while accuracy is preserved.

Key Contributions

1. A transform-coding scheme for KV caches (kvtc) that combines PCA-based feature decorrelation, adaptive quantization with a dynamic-programming bit allocator, and entropy coding (DEFLATE via nvCOMP), while leaving model parameters unchanged.
2. A one-time calibration procedure instead of per-prompt decomposition: the PCA basis and bit allocation are computed once per model and compression ratio from a calibration set, then reused for all requests — unlike SVD-based baselines that recompute a decomposition per prompt.
3. Direct empirical evidence of cross-head redundancy: keys (and, less strongly, values) from different attention heads occupy a shared subspace up to an orthogonal transformation, which motivates both cross-head and cross-layer concatenation before decomposition.
4. Systematic evaluation against inference-time baselines (GEAR, KIVI, FP8, H2O, TOVA, xKV, and the trained eviction method DMS) showing higher compression at comparable or better accuracy, plus latency and calibration-cost measurements.

Main Findings

• Compression levels: kvtc achieves up to 20× compression while maintaining reasoning and long-context accuracy, and 40× or higher for specific use cases. A kvtc 16× setting corresponds to approximately 20× after DEFLATE.
• Near-vanilla accuracy at 16×: kvtc 16× consistently stays within less than 1 score point (accuracy or F1) of the vanilla models across the general-purpose benchmarks, based on reported compression ratios of roughly 17–21× after DEFLATE.
• High-ratio behavior: kvtc maintains high accuracy on the tested models even at 32× and 64×, with the 64× setting showing degradation on the harder retrieval tasks (for example, Llama 3.1 8B LITM drops from 99.4 for vanilla to 90.2 at the highest tested ratio).
• Baseline comparison: quantization methods GEAR and KIVI show degradation on GSM8K and the Lost-in-the-Middle task at 5× CR; eviction methods H2O and TOVA perform poorly as generic KV cache compressors; xKV performs well across most tasks except Qasper. In some cases kvtc at very high compression ratios surpasses the vanilla model's score.
• Reasoning models: on DeepSeek-R1-distilled Qwen 2.5, kvtc 8× costs 0.3 percentage points on LiveCodeBench for the 1.5B model and 0.2 percentage points for the 7B model. On Qwen 2.5 R1 7B, vanilla AIME24 is 50.9 ± 4.9 and kvtc 8× is 52.5 ± 3.6.
• Small caches benefit too: the Qwen 2.5 R1 1.5B cache is 29 KiB/token versus 131 KiB/token for Llama 3.1 8B, and a 9× compression

Authors’ abstract

Serving large language models (LLMs) at scale necessitates efficient key-value (KV) cache management. KV caches can be reused across conversation turns via shared-prefix prompts that are common in iterative code editing and chat. However, stale caches consume scarce GPU memory, require offloading, or force recomputation. We present KVTC, a lightweight transform coder that compresses KV caches for compact on-GPU and off-GPU storage. Drawing on classical media compression, KVTC combines PCA-based feature decorrelation, adaptive quantization, and entropy coding. It requires only a brief initial calibration and leaves model parameters unchanged. By exploiting redundancies in KV caches, KVTC achieves up to 20$\times$ compression while maintaining reasoning and long-context accuracy, and 40$\times$ or higher for specific use cases. We test KVTC with Llama 3, Mistral NeMo, and R1-Qwen 2.5 models across benchmarks including AIME25, GSM8K, LiveCodeBench, LongBench, MATH-500, MMLU, Qasper and RULER. It consistently outperforms inference-time baselines such as token eviction, quantization, and SVD-based methods, while achieving higher compression ratios. These results support KVTC as a practical building block for memory-efficient LLM serving with reusable KV caches.