Working Around the Compute Ceiling: Byte-Exact Memory in Galahad Makes LLM Reading a One-Time Cost LLM Reading a One-Time Cost

AI summary

Overview

• Research area: Natural Language Processing / LLM inference systems — specifically key–value (KV) cache reuse and retrieval for LLM serving engines.
• Technical level: Advanced. The paper assumes familiarity with transformer prefill, attention key–value state, paged/prefix caching, retrieval-augmented generation and GPU serving runtimes such as vLLM, SGLang and llama.cpp.
• Scope: A system description and empirical evaluation of Galahad, a memory layer that makes repeated reading of the same text a one-time cost by storing exact KV state and exact document text.

What This Paper Is About

Serving engines are stateless across requests, so a model asked a second question about a document recomputes that document's attention state from the first token. The authors measured that on seven real-world datasets, 98.7% of prompt tokens were text the model had already read. Galahad's goal is to make that reading a one-time cost: its Taliesin component saves KV state for a block of text and loads it when the same bytes appear again, and its Blaise component passes the model only the document section a question needs.

Key Contributions

1. A memory layer delivered as one shared library that plugs into three production runtimes — vLLM (as a KV connector), SGLang (as a HiCache storage backend) and llama.cpp (through slot save and restore) — with no modification to model weights or runtime source code, and which falls back to normal computation on any load failure.
2. Evidence that restored state is exact: bit-identical logits after save and restore, and byte-equal prefill across four model architectures (Llama 3.1 70B, Gemma 4 E4B, Qwen 3.6 35B-A3B, Ministral 3 8B).
3. Separate measurements of accuracy, latency and GPU energy for Taliesin and Blaise on a controlled 100-fact recall test and on 349 questions drawn from seven real-world datasets, including an external RAGFlow baseline and coverage of 30 of 30 tested models.
4. An account of the design boundaries of the approach — block-level reuse granularity, fail-closed behaviour, and the limitation that memory begins from the text it is given — and why these follow from how transformers and serving engines work.

Main Findings

• Most prompt work is repeated reading. On seven real-world datasets, 98.69% of prompt tokens were loaded from Taliesin rather than recomputed (llama.cpp 99.49%, vLLM 99.37%, SGLang 97.21%). In the recall test on llama.cpp, each question sent 53,219 prompt tokens on average, of which 99.5% were loaded from Taliesin.
• KV persistence alone recovers accuracy the truncated baseline cannot reach. Without Galahad, the baseline used a 12,000-token context window and saw only the last 12% of the 96,726-token corpus, answering 10 of 100. Taliesin alone answered 98 of 100 on llama.cpp, 99 of 100 on vLLM and 100 of 100 on SGLang, against 10 in every no-Galahad row.
• Taliesin was faster and cheaper despite reading far more. On llama.cpp the model processed 5.5 times as many prompt tokens as the truncated baseline (9,700 per question) and still finished 3.1 times faster (3.01 s against 9.25 s) with 79% less GPU energy (572 J against 2,754 J).
• Blaise adds a second, larger saving. With Blaise the model read about 668 tokens per question instead of the 96,726-token corpus and answered 100 of 100 on all three runtimes at 0.59–0.64 s and 200–213 J per question. On vLLM that was 13.7 times less time and 92% less GPU energy than the 8.11 s and 2,402 J baseline, while answering 100 questions instead of 10.
• The two memories compound. On llama.cpp, Taliesin alone was 3.1 times faster than the baseline, and giving the model 668 tokens instead of 96,726 took that to 14.5 times.
• A tuned retrieval baseline answered fewer questions. RAGFlow v0.27.2, with lightly tuned retrieval settings on vLLM on an L40S, answered 77 of 100. The paper attributes this to selection: passing less text helps only when the selected text contains the answer.
• Restored state is bit-identical. All 262,144 of 262,144 output logits matched after process restart, rehydration from disk and hot-load (27 checks, Gemma 4 12B Q8_0, llama.cpp). Deterministic prefill gave 20 of 20 byte-equal SHA-256 hashes of logits on two fresh prefills of the same 1,024-token prompt.
• State survives hardware and process movement. KV state saved on an A6000 and loaded on an RTX 4090 — and the reverse — produced 64 of 64 identical greedy tokens; stored bytes round-tripped 3 of 3 on each of 10 GPU types; 276 of 276 tokens were bit-identical after restart; encryption round-trip moved 9.77 GiB with 0 failures alongside 4,167 NIST CAVP vectors with 0 failures.
• Reuse works on unfamiliar data. Across 349 questions from seven sources and three runtimes, Taliesin alone answered 329–333 questions (94–95%). Adding Blaise answered 319–321 (91–92%) and was 3–10 times faster per question than Taliesin alone.
• Fail-closed behaviour is demonstrated. With one in three loads deliberately failed on an H100, throughput stayed above the no-Galahad baseline: 2.716 against 2.643 turns per second. A forced kill (SIGKILL) left 0 damaged records.
• Throughput and latency improve. On an H100 with Qwen3-30B-A3B (16 sessions of 14,000 tokens, concurrency 8), throughput rose from 2.643 to 3.424 turns per second (+29.6%). Time to first token fell on all seven GPU types tested; examples are 1,138.8 ms to 391.2 ms (2.91 times) on an RTX PRO 4500 and 251.2 ms to 191.0 ms (1.32 times) on an H100 SXM.
• Long windows run at flat GPU memory. On an A40 with Gemma 4 and llama.cpp, a corpus of 200 blocks of 30,000 tokens (5.97 million tokens) answered 5 of 5 probes, with access time between 0.551 and 0.593 s and GPU memory between 24,433 and 24,696 MB. A ladder from 1M to 50M tokens answered 24 of 25 probes — the miss was a parsing error in the test harness — at peak GPU memory of 33,813–34,080 MB.
• The corpus pays for itself quickly. Storing the 96,726-token corpus took one prefill: 96 s and 27.8 kJ on llama.cpp, 108 s and 28.4 kJ on vLLM. On llama.cpp each later question saved 2,182 J and 6.2 s, so energy was recovered after 13 questions and time after 16. Indexing the corpus in Blaise took 0.1 s and computed no tokens on the GPU.
• Row-level reuse is not possible. Rotary position encodings make each KV row depend on its position: 0 of 344,064 rows matched in a direct test. Galahad therefore reuses whole blocks, where positions match exactly.
• Coverage held across models and runtimes. On 4 times A10G with vLLM 0.30.0 and the release library (ABI 1.31), 30 of 30 models answered, saved, loaded and hit saved state. A tensor-parallel run (Qwen3 14B, TP=2) succeeded on 5 of 5 fresh starts. On DeepSeek-V4-Flash (284B) with vLLM 0.29 on 2 times H200, Taliesin and Blaise together answered 98 of 100 recall questions at 0.133 s and 143 J per question.
• Messy extraction is a hard limit. In 9 of 50 messy-PDF questions the answer was lost during PDF text extraction before any model saw it. On the 41 answerable questions, Taliesin only scored 37 and Taliesin + Blaise scored 30, which the paper uses to illustrate that no memory layer can restore text an upstream parser dropped.
• Safety mechanisms were tested adversarially. For each of four defences (confirmation hash, licence signature, tenant separation, byte comparison on load), the corresponding attack was run three times — defence on, defence off, defence restored — and all four met the criterion. A separate pass with ten analysis tools, including address sanitizers, found one out-of-bounds read, fixed before release.

Methodology in Plain English

The authors split the problem into two independently testable memories and measured them separately so the contribution of each is visible.

Taliesin works after the runtime has computed the KV state for a block of prompt text: it writes that state to storage under a fingerprint of the exact input bytes, the model and the tenant, and loads it on a later request containing the same block. The design rule is "serve without memory, never with wrong memory" — a missing, damaged or failing record triggers recomputation, so memory can never change an answer. Blaise keeps each document as byte-exact text divided into sections, selects one section for a question on the CPU, and passes only that section to the model.

For evaluation, the authors built a controlled recall test: 100 facts with invented names and random numbers inserted into 13 Wikipedia articles (434 KB, 96,726 tokens in 11 blocks), with each fact queried in paraphrase at depths from 1% to 99% of the corpus, using Gemma 4 31B (4-bit) on an RTX A6000. They report three configurations — Taliesin alone, Taliesin with Blaise, and no Galahad — and, unless noted, emptied the runtime's own cache before every question so any reuse had to come from Galahad. They then tested on 349 questions from seven datasets they had not tuned on. All benchmarks ran on rented cloud GPUs; no configuration was told where an answer was located; every retry and loaded block counted toward time, energy and token totals; time is the median per question and energy comes from the GPU's own counter (NVML), with idle power measured separately and included. Decoding was greedy at temperature 0.

Why This Matters

The paper reframes inference cost: today a request costs as much as the total text in its prompt, whereas with persistent memory it costs as much as the text the model has not read before. It also changes which models are practical, since a smaller model with exact memory of a large corpus can answer questions that would otherwise need a much longer context window.

The work is positioned explicitly against the argument of Varin Sikka and Vishal Sikka that a transformer's O(N² · d) computation bound limits which tasks it can carry out or verify. The authors do not try to raise that ceiling. They argue that repeated work, search and verification are moved away from the model: state is stored and loaded rather than recomputed, section selection runs on the CPU, and correctness is decided by byte comparison and cryptographic hashes rather than by the model judging its own output.

Real-world applications named in the paper:

• Support assistants that answer hundreds of questions against the same manuals.
• Coding agents that re-send the same repository files on every step.
• Contract review tools that ask dozens of questions of one agreement.
• Agent and long-document workloads more generally, which the paper groups with support, code and legal tasks as the workloads that return to the same documents.

Industry relevance: Galahad ships as one shared library (libgalahad.so) for Linux x86-64 with two runtime dependencies (OpenSSL's libcrypto and zstd) and requires no changes to model weights or runtime source code, so it targets existing production serving stacks rather than requiring a new engine. The paper reports a customer-style clean install in which, on SGLang 0.5.20, 1,792 of 1,824 prompt tokens came from Galahad after SGLang's own cache was emptied, and a GKE cluster with an L4 GPU where 63.9% of lookups hit saved state across a pod restart.

Future Directions

• Benchmarking Blaise's second mode, in which the model itself reads Blaise's index of the corpus and Taliesin keeps that reading so it is paid once; this paper reports only the first mode.
• Extending section selection to poorly extracted PDFs, since the paper shows that in 9 of 50 messy-PDF questions the answer was already lost by the extractor.
• Reporting results from more models and from independent operators, beyond the author-run experiments on rented cloud GPUs.
• Open questions raised by the design boundaries: the paper does not describe the storage format, the internal structure of Blaise, or the security design in implementation detail, stating that these are covered by pending patent applications.

Target Audience

Machine-learning systems engineers and inference-serving specialists working on KV caching, prefix reuse and retrieval pipelines; researchers studying long-context inference and the cost of prefill; and technical decision-makers evaluating whether persistent memory can reduce serving cost for document-heavy workloads. Readers need prior familiarity with transformer attention, prefill versus decode, and serving runtimes such as vLLM, SGLang and llama.cpp to follow the measurements.

Authors’ abstract

A transformer language model performs a bounded amount of computation per token, and recent work by Vishal Sikka, former CEO of Infosys, argues that this bound limits which tasks a model can carry out or verify (arXiv:2507.07505). We ask how much of the budget beneath that ceiling is spent on work the model has already done. Serving is stateless across requests: a model that answers a second question about a document recomputes the document's attention state from the first token. On seven real-world datasets, 98.7% of prompt tokens were text the model had already read. We present Galahad, a memory layer for vLLM, SGLang and llama.cpp that makes this reading a one-time cost. Taliesin saves the model's key-value (KV) state for a block of text and loads it on the next request that contains the same bytes, instead of recomputing it. Blaise keeps the documents themselves and passes the model only the section a question needs. On a recall test with 100 facts hidden in a 97,000-token corpus (Gemma 4 31B), Taliesin alone let the model attend to the whole corpus and answered 98 of 100 on llama.cpp at 3.0 s and 572 J per question, against 10 of 100, 9.3 s and 2,754 J for the same model without Galahad, which could hold only the last 12,000 tokens. With Blaise added, the model read about 668 tokens per question and answered 100 of 100 on all three runtimes at 0.59-0.64 s and 200-213 J; a tuned RAGFlow pipeline answered 77. Storing the corpus is a one-time cost of about 100 s and 28 kJ, whose energy is recovered after 13 questions. Restored state is bit-identical: all 262,144 output logits matched after restart, rehydration and hot-load. Galahad worked with all 30 models we tested under vLLM, and it fails closed: any load that does not pass its checks is recomputed. Together these results move LLM serving from stateless to stateful inference.