LowRAM: Low-Rank Keys and Amortised Values for Composable KV Cache Compression at Test-Time

AI summary

Overview

Research area: Efficient inference for large language models — specifically post-training compression of the key-value (KV) cache, the memory structure that dominates activation memory during long-context autoregressive decoding.

Technical level: Intermediate. The paper assumes familiarity with transformer attention, KV caching, grouped-query attention, low-rank (SVD) factorisation, and gradient-based adaptation, but its central idea — that values are largely predictable from keys — is intuitive and explained from first principles.

Scope: The paper introduces LowRAM, an asymmetric KV cache compression pipeline that compresses values by learning a per-head predictor from keys (storing residuals only for poorly reconstructed tokens) while compressing keys with cross-layer low-rank decomposition, and evaluates it on three model families across RULER and LongBench.

What This Paper Is About

Serving transformers at long context lengths is bottlenecked by the KV cache, which grows linearly with context length and batch size and comes to dominate the model's memory footprint. Existing post-training compression methods attack redundancy across tokens (eviction), across numerical precision (quantisation), or across feature dimensions (low-rank decomposition), but leave one axis largely unexplored: the redundancy between keys and values. The paper's goal is to exploit that redundancy — treating keys and values as structurally distinct objects that should be compressed differently — to achieve aggressive, composable compression on frozen pretrained models without retraining.

Key Contributions

1. Value cache amortisation. The paper identifies within-head key-to-value predictability as an underexplored source of KV cache redundancy, and replaces most stored values with a lightweight per-head predictor fitted at test time, retaining residual corrections only for the tokens it reconstructs worst — giving 4× near-lossless value compression.

2. LowRAM, an asymmetric pipeline. The authors pair amortised values with cross-layer low-rank key compression (xKV, Chang et al., 2026), justified by a source-target analysis showing that values are better reconstructed from keys than the reverse, and that key errors are more harmful than value errors.

3. Reduced overhead via meta-learning and selective reconstruction. A meta-learned initialisation (First-Order MAML) cuts per-context adaptation to 2 gradient updates and brings time-to-first-token overhead to 1.3× the uncompressed baseline, down from 26.3× (150 updates) and 55.5× (300 updates); selective reconstruction of only query-relevant chunks yields up to 5× higher peak decode throughput at 4× compression.

4. Composability. LowRAM stacks with quantisation and eviction, reaching 26.9× compression while retaining 94.5% of baseline performance on LongBench contexts longer than 10K tokens (Mistral-7B).

Main Findings

• The best prediction direction is current-layer keys to current-layer values. In a source-target analysis on Mistral-7B at a fixed 3× compression ratio — comparing previous-layer keys, previous-layer values, their concatenation, current-layer values, and current-layer keys as sources — reconstructing current-layer values from current-layer keys (K_L → V_L) recovered the highest downstream LongBench performance.

• Reconstruction error is a weak proxy for downstream quality. Source-target pairs with similar reconstruction errors exhibit significantly different recovered performance, and the differing scales of keys and values make absolute error hard to compare. The authors therefore base design decisions on recovered downstream performance rather than error.

• Near-lossless at moderate compression across three model families. At 3× compression, Mistral-7B and Llama-3.1-8B retain 97.9–99.4% of baseline. At 4×, recovery is 96.0–99.6% except for Llama-3.1-8B on RULER, which falls to 93.9% (attributed to the reduced residual budget). Qwen3.5-4B matches baseline on RULER (100.0/100.0/100.1 at 2×/3×/4×) and recovers 94.4–97.8% on LongBench.

• Amortisation beats low-rank compression of values in isolation. With keys kept exact on Llama-3.1-8B, value amortisation retains 98.2% of RULER and 99.3% of LongBench at 4× compression; xKV on the same value cache reaches 99.3% on RULER but only 95.8% on LongBench, staying behind at every LongBench ratio.

• The key-to-value map is strongly head-specific. Sharing one predictor across a layer's heads costs almost 20 points (Table 7), which the authors read as evidence the mapping is per-head.

• Residual placement matters. At a fixed budget, error-ranked residual selection recovers almost all accuracy, whereas random selection loses most of it.

• Quantisation composes additively; eviction trades off. Adding quantisation to a 4× LowRAM cache on Mistral/LongBench raises compression to 6.4× and recovery from 97.9% to 99.4%; for Llama/RULER at 4× it lifts recovery from 93.9% to 99.4%. Eviction shortens the sequence without shrinking the fixed network cost: raising the LowRAM ratio from 2× to 4× with eviction fixed at 2× reduces RULER recovery by 7.4 points (Mistral) and 17.4 points (Llama), versus 1.2 and 3.7 on LongBench.

• Extreme compression is achievable on long contexts. On LongBench sequences longer than 10K tokens, the maximum compression retaining at least 90% of Mistral-7B performance is 10.0× for eviction (91.6%), 7.5× for quantisation (92.3%), 9.0× for LowRAM (90.7%), and 26.9× for LowRAM-EQ (94.5%).

• Overhead mechanisms cost a small amount of accuracy. At 4× compression on Llama-3.1-8B, selective reconstruction matches or exceeds full reconstruction (96.3 vs 96.9 at 16K, 96.8 vs 100.0 at 32K, 92.4 vs 95.8 at 64K on RULER; equal at 95.3 on LongBench >10K without meta-init). Meta-initialisation costs 1.4–4.4 points on RULER and 2.4 points on LongBench (>10K) relative to 300-update adaptation, in exchange for the TTFT reduction.

• LowRAM suits long contexts. The predictor's parameters are a fixed cost amortised only after a few thousand tokens; at 4× compression this fixed cost dominates below roughly 2–3K tokens, which explains why LowRAM degrades more on 4K RULER and combines less cleanly with eviction.

Methodology in Plain English

The core intuition. Keys and values in an attention head are produced from the same hidden representation, so a value vector should be largely recoverable from its key. If that is true, you do not need to store values at all — you need to store a small function that regenerates them from keys you are already keeping.

Step 1 — Pick the right direction. Before committing, the authors ran a controlled comparison: for each layer, train the same predictor to reconstruct either keys or values, feeding it various sources (previous-layer keys, previous-layer values, their concatenation, current-layer values, current-layer keys), all under an identical 3× memory budget with the residual budget held fixed. They judged configurations on recovered LongBench score, not just reconstruction error.

Step 2 — Amortise the values. For each layer and each KV head, they fit a two-layer MLP (hidden width equal to the key dimension, ReLU, MSE loss, Adam at 0.001) mapping keys to values. Two tricks help it start well: the linear residual branch is initialised from the Moore-Penrose pseudoinverse of the key projection times the value projection (W_K^+ W_V, a "Slim Attention" prior), which is the best linear key-to-value map the model weights admit, and the whole network is warm-started from a one-off offline pass (32 batches of 4096-token FineWeb-Edu sequences, 25 gradient steps, under five minutes on one GPU, run once per model). After fitting, they score every token by squared reconstruction error, keep exact residuals for the top-p fraction, and discard the rest. Memory is then the MLP parameters plus a sparse residual store plus indices.

Step 3 — Compress the keys separately. Because keys must be retained anyway (both for attention and as inputs to the predictor), they need their own scheme. The authors use xKV, which shares a single low-rank SVD basis across groups of adjacent layers. Before either compression step they invert RoPE, since pre-RoPE representations compress better, and reapply the rotation at attention time. During decoding, newly generated key-values are stored exactly.

Step 4 — Cut the runtime overhead. Fitting an MLP per context is expensive if the context is queried once. Two mechanisms address this. Meta-learned initialisation uses First-Order MAML to learn starting parameters that adapt in 2 gradient updates instead of 150 or 300. Selective reconstruction avoids rebuilding the whole cache each decoding step: the prefix is partitioned into chunks represented by mean-key landmarks, outlier chunks and a local window are kept exactly, and only the top-k chunks selected by the current query are reconstructed, with a persistent buffer reusing selections across steps. The exact key-cache SVD is also replaced with a randomised Cholesky-QR decomposition for a 4× speedup with no downstream loss.

Step 5 — Stack with existing methods. Eviction is applied before fitting (a small CNN predictor scores tokens by predicted future attention, retaining attention sinks — first 8 tokens — and a local window of the last 64 tokens unconditionally), so the MLP only handles surviving tokens. Quantisation uses TurboQuant without the 1-bit QJL error-correction step, applied only to the context-length-scaling A factor of the key factorisation, with asymmetric precision: 4-bit for A and 2-bit for value residuals.

Evaluation. RULER (4K, 16K, 32K, 64K), LongBench, and LongBench filtered to sequences of at least 10K tokens, across Mistral-7B-Instruct-v0.3, Llama-3.1-8B-Instruct, and Qwen3.5-4B (a hybrid architecture where LowRAM is applied only to the full-attention layers, with compression ratios reported relative to that subset's cache). All results are reported as recovered performance relative to the uncompressed baseline, with compression ratios computed via byte-level accounting as the harmonic mean of the key and value ratios.

Why This Matters

Research impact. The paper opens a compression axis that the eviction / quantisation / low-rank literature has largely left alone: redundancy between keys and values rather than within either. Its source-target analysis also produces a methodological point with wider reach — reconstruction error and downstream accuracy can diverge sharply, so compression work that optimises MSE may be optimising the wrong target. The finding that key errors and value errors have asymmetric downstream consequences connects to prior observations on asymmetric KV quantisation.

Real-world applications:

• Long-document question answering, where a large context is ingested once and queried repeatedly — the setting where the paper argues full per-context fitting is most attractive, since the fitting cost is paid once.
• Shared-prefix serving, where many user requests share a system prompt or retrieved document set, letting one compressed cache amortise across requests.
• Long-context retrieval and RAG pipelines over contexts in the 10K–262K token range, where the paper's peak-throughput advantage is largest.
• Multi-query or batch-heavy inference on memory-constrained accelerators, where KV cache size caps batch size — LowRAM-SR supports a batch size of 28 at all three evaluated context lengths, and at 262K tokens reaches roughly 5× the peak throughput of the exact-cache baseline.

Industry relevance. The method is post-training and operates on frozen open-weight models with grouped-query attention, which is how most deployed open models are structured — so it does not require pretraining, architectural change, or access to model weights beyond the projection matrices. The composability with quantisation and eviction means it can be layered onto existing inference-stack optimisations rather than replacing them.

Future Directions

• Long generations. The authors have not evaluated long generations, which would require periodically compressing the suffix of newly generated tokens (for example every N tokens) rather than relying on a single prefill-time fit.

• What persists across contexts. It is unclear how much of the learned parametrisation is shared. Gains from warm-starting and the Slim Attention initialisation suggest part of the key-to-value relationship is model-level and shared; gains from test-time adaptation indicate a remaining context-specific component. Splitting these two would allow more efficient allocation of memory and compute.

• Stability of hard-to-predict tokens. Whether the tokens with the largest reconstruction error are stable across contexts is an open question the authors flag, with implications for how residual budgets should be spent.

• Better pairing with eviction. Because eviction shortens the sequence while leaving the predictor's fixed parameter cost unchanged, the two interact badly at high compression and short lengths; a scheme that scales or shares the predictor under aggressive eviction is a natural extension.

Target Audience

Researchers and engineers working on LLM inference efficiency, KV cache compression, and long-context serving — particularly those who need post-training methods applicable to frozen open-weight models. It is also useful for practitioners building inference stacks that already use quantisation or eviction and want an additional orthogonal axis, and for readers interested in test-time adaptation and learned associative memory as a substitute for stored state. Readers without background in attention mechanics or low-rank factorisation will need to consult the referenced prior work (xKV, Slim Attention, TurboQuant) to follow the implementation details.

Authors’ abstract

Autoregressive decoding relies on the KV cache, whose memory footprint grows linearly with context length and caps batch size and throughput. Existing post-training methods exploit redundancy across tokens, layers, and feature dimensions, or lower numerical precision. We identify a complementary axis: within each attention head, values are largely predictable from their corresponding keys. We exploit this with value cache amortisation, which fits a lightweight per-head…