Real Long-Term Memory for AI: A 50-Million-Token Window That Is Faster and Cheaper Than Recompute

AI summary

Overview

Research area: Natural Language Processing / large language model serving systems — specifically long-context inference, key-value (KV) cache persistence, and memory for served models.

Technical level: Advanced. The paper assumes familiarity with transformer attention, KV caching, prefill cost, time-to-first-token (TTFT), and GPU serving stacks such as vLLM.

Scope: The paper measures a disk-backed KV memory layer, released as the package galahad-kv, on a 50,000,000-token corpus served from one NVIDIA H100, and compares restoring stored KV state against recomputing it.

What This Paper Is About

Transformer language models can only attend to tokens inside their context window, and serving systems rebuild the KV attention state of a prompt every time it is sent, so the model pays full prefill cost again for text it has already processed. The paper describes a memory layer that writes the KV state of each ~16,000-token block to encrypted local NVMe storage and later restores any block byte-exact, with zero recompute and flat GPU memory. The goal is to let a served model reuse material from far outside its native 16k window while making inference faster and cheaper than recomputing the same tokens.

Key Contributions

1. A demonstrated 50-million-token memory window on a real served model, with byte-exact restore at every depth probed and constant VRAM. A block is processed once and can afterwards be reused without recompute.
2. A measurement of the cost of reuse against the cost of recompute: restoring a block is 2.8× to 4.3× faster and takes 8.8× to 12.3× less GPU energy than recomputing the same tokens, with restore time staying flat as depth increases.
3. A cheat-resistant benchmark protocol with a countermeasure for each of five known ways a long-context benchmark can be gamed, plus a harness that lets a third party rebuild the experiment on a single GPU.
4. A characterization of KV footprint under sliding-window attention: 36.4 KiB/token for the 12B model (versus ~160 KiB for full attention) and 134 KiB/token for the 31B FP8 model.

Main Findings

• Zero-recompute restore at every depth: All probed blocks were restored from the encrypted store without recompute — 100 of 100, at depths from 0 to 50M tokens — for both Gemma 4 12B and Gemma 4 31B. Probes covered 100 depths between 0 and 49,488,000 tokens.
• Text recall differs by model, not by memory: On identical stored blocks, the 12B model answered 82/100 correctly and the 31B model 98/100, with zero hallucinations on either model. Both refused in the negative-control cases (0/20 fabricated).
• Restore beats recompute on time and energy: For Gemma 4 12B, restore median was 0.266 s / 59 J versus recompute median 0.760 s / 521 J (2.8× speedup, 8.8× energy reduction). For Gemma 4 31B, restore median was 0.347 s / 80 J versus recompute median 1.475 s / 976 J (4.25× speedup, 12.3× energy reduction).
• Depth independence: A block from 50M tokens ago is recalled as quickly as the first block. Low-third to high-third TTFT was 0.193 → 0.274 s for the 12B and 0.343 → 0.375 s for the 31B. Restore TTFT p90 was 0.284 s (12B) and 0.579 s (31B).
• Larger models gain more: Because 31B blocks are larger and more expensive to recompute, the advantage of restoring them is greater (4.25× / 12.3× against 2.8× / 8.8× for the 12B), so reuse scales with model size.
• Flat VRAM during deposit: Peak VRAM was 70,864 MB for the 12B and 75,111 MB for the 31B, both flat over the whole 50M-token stream, corresponding to an O(1) moving window.
• Deposit cost: 20,839 tok/s for the 12B and 10,700 tok/s for the 31B, producing encrypted stores of 1.86 TB and 6.25 TB respectively, each with 3,125 MRLNCRY1 blocks.
• Store audit: All 3,125 blocks carry the MRLNCRY1 header, bytes on disk match the KV footprint math, and the number of blocks equals the number deposited, meaning no block was pruned or evicted.
• Misses are in-block errors: In every miss the model selected a wrong line that was genuinely in the block — either a same-format decoy or another in-block number — rather than fabricating a code.

Methodology in Plain English

The system is a plug-in (GalahadConnector) loaded by vLLM that performs two operations. On deposit, it writes the per-block KV of a request to an encrypted on-disk store in the MRLNCRY1 format using AES-256-GCM with one key per organisation, then frees it, so only one block is resident in VRAM at a time. On restore, which happens on a later request, it grafts the KV of a stored block back so the forward pass continues as if the model had just computed that block, without recomputing its tokens.

The evaluation is deliberately hardened. Before ingestion, a runtime mutator replaces names, dates, amounts and e-mails in the real source with random nonces so pretraining cannot supply an answer. All 50M tokens are deposited blind, with no question in context, and questions are sent only after the memory is fully written. One readable, high-entropy needle (a named record with a 4-digit code) is planted per block at a known depth, alongside same-format decoy records with other codes, so a keyword match returns a decoy and only actually reading the right record yields the correct code. Telemetry is audited by requiring every block to carry the MRLNCRY1 magic and match the KV footprint math.

Two metrics are recorded separately per probe: (A) KV restore, which checks that context tokens were served from the store with no recompute (a property of the engine), and (B) text recall, which checks whether the model emitted the exact planted code (a property of the model). Energy is taken from the GPU's NVML hardware counter, idle-subtracted, and the answer key is held out of the engine. As a negative control, each block is also asked for a record it does not contain, where a correct system must refuse rather than fabricate. The corpus was 50,000,000 tokens = 3,125 blocks × 16,000 tokens, streamed from WildChat (40%), permissively-licensed source code (25%), arXiv papers (20%) and educational web text (15%). Temperature was 0 throughout.

Why This Matters

The paper separates two properties that long-context work often conflates: the engine's exactness (restore is 100/100 at every depth on both models, with flat VRAM) and the model's reading quality (82/100 versus 98/100 on identical stored blocks). It also reverses the usual cost story for long context — durable memory makes inference both cheaper (8.8× to 12.3× less energy) and faster (2.8× to 4.3×), while making outcomes smarter in the specific sense that the model answers from facts it processed millions of tokens earlier. The weights are not changed; what changes is the set of facts the model can act on.

Real-world applications suggested by the measured setup:

• Long-lived assistant memory: the store is durable and encrypted at rest with one key per organisation, so it can hold an organisation's or person's own material across restarts.
• Large-scale document and codebase recall: 50M tokens of mixed web chat, source code, arXiv papers and educational text were stored and queried at depths up to 49,488,000 tokens.
• Cost reduction in production serving: avoiding re-prefill of already-seen text cuts GPU energy per reused block by 8.8× to 12.3× on the measured models.
• Model escalation: a cheaper model can serve while a stronger one is escalated to, with both free of recompute and both reading the same durable memory.

Industry relevance lies in the deployment constraints the paper states plainly: the method runs on any single ≥24 GB NVIDIA GPU, requires local NVMe (a network volume voids the latency and energy figures), and trades disk for reuse at 1.86 TB (12B) or 6.25 TB (31B) for 50M tokens.

Future Directions

• Combining this window with learned retrieval: selecting a passage by meaning (the Blaise component) is explicitly out of scope here, and combining it with this window at 50M scale is left for future work.
• Widening evidence on exactness: the paper does not re-establish byte-exact logit equality of grafted KV, deferring that to the companion Taliesin work; the engine property measured directly here is 100/100 zero-recompute restore with zero fabrication.
• Stress-testing depth beyond 50M: the paper demonstrates a 50M-token window, roughly 8× larger than its earlier 6M-token report, raising the question of how restore time and store auditability behave at still greater scale.
• Separating attention from memory: the paper states it does not widen single-pass attention and that one block is resident at a time, leaving open how processed, held-in-VRAM and restored states should be composed in a serving system.

An open methodological question is also noted: a per-probe restore takes ~0.3 s, which approaches the resolution of the NVML counter, so absolute per-probe joules carry a wider band than the restore-vs-recompute ratios.

Target Audience

This paper is most useful to inference and serving engineers working on KV cache management, offloading and long-context deployment; to researchers studying context extension, retrieval-augmented generation and agent memory; and to infrastructure teams weighing GPU energy and time-to-first-token costs for models with fixed context windows. Readers wanting to reproduce it need access to a single ≥24 GB NVIDIA GPU, local NVMe storage, and the public software listed in the paper (galahad-kv==1.31.4, vllm==0.31.0, transformers, datasets, nvidia-ml-py).

Authors’ abstract

A large language model can only use the text that fits in its context window, and it recomputes its internal key-value (KV) state for a prompt every time the prompt is sent. We test a memory layer, the public package galahad-kv, that saves the KV state of each block of about 16,000 tokens to encrypted local NVMe disk and loads it back later, byte-exact, without recomputing it. We ran it on 50,000,000 tokens of real public text, served through vLLM on one NVIDIA H100, with Gemma 4 12B and Gemma 4 31B. Every block we probed was loaded back from the encrypted store with no recompute (100 of 100, at depths from 0 to 50M tokens) on both models. Loading a block was 2.8x to 4.3x faster than recomputing it and used 8.8x to 12.3x less GPU energy, and GPU memory stayed flat over the whole 50M-token stream. Asked about facts planted millions of tokens earlier, the 12B model gave the right answer 82 times out of 100 and the 31B model 98 times out of 100. Neither model made up an answer. The limits are as follows. This is reuse of stored state, not a wider attention window: one block is loaded at a time, and how well a question is answered depends on the model. Writing the memory is a one-time cost, and the store takes terabytes of local NVMe disk. We describe the test protocol, which is built to resist common ways of gaming long-context benchmarks, and give a single-GPU reproduction that uses public software and a free licence for the package.