Persistent Memory in Multi-Agent LLM Inference: What It Costs, What It Buys, and When You Can Tell

AI summary

Overview

Research area: Systems and evaluation methodology for multi-agent LLM inference, specifically KV-cache memory management, agent decomposition, and persistent memory tiers.

Technical level: Intermediate. The core ideas are stated plainly, but the paper leans on measurement statistics (percentile and cluster bootstrap, confidence intervals, within-configuration standard deviation) and on serving-stack details (KV geometry, attention layers, edge accelerators). A reader comfortable with LLM serving and basic statistics will get the most out of it.

Scope (one sentence): The paper measures, on a single three-tier agent architecture, whether decomposing long-context inference across agents bounds peak KV-cache memory (it does) and whether an added persistent reasoning-trace tier improves accuracy (it does not, and the paper argues the evaluation regime makes that null structural rather than incidental).

What This Paper Is About

Long-context inference is limited by the KV cache, which grows with context and is provisioned against its peak rather than its average, often exceeding the VRAM of an edge accelerator. Many systems respond by splitting a task across cooperating agents, and many of those then add a persistent tier that stores and recalls reasoning traces, usually justified by an ablation showing an accuracy gain when the tier is enabled. This paper measures both claims on one architecture: it asks what the decomposition buys in peak cache, what the persistent tier costs, and whether the standard ablation is capable of detecting what it claims to detect.

Key Contributions

1. A measured, ablated peak-KV result for agent decomposition. The authors report that decomposition alone bounds the largest single call's KV footprint, independent of any persistent-recall tier, and that a retrieval-matched RAG baseline does not: RAG changes which evidence enters a call, while decomposition changes how many calls it is spread across. They state that to their knowledge no prior work reports an agent architecture's peak KV as a measured, ablated systems quantity.

2. A null result for the persistent tier, with a stated bound. Across eight controlled dataset pairs at n=100 per arm, enabling persistent trace recall cost +0.368 MiB of peak cache (95% CI [+0.167, +0.590]) and produced an accuracy change of +0.015 (95% CI [−0.011, +0.046]). The authors do not claim the effect is zero; they claim an effect larger than ≈0.046 is excluded and the point estimate is not separable from noise.

3. A structural explanation of why the null arises. The paper argues that single-question benchmarks supply each item with its own evidence and score it independently, and that correctness requires resetting stored traces between conditions, so recall has nothing informative to retrieve. Measured with the retriever's own predicate (cosine 0.6, sentence-transformers/all-MiniLM-L6-v2), recall was reachable for 24% of questions as sent (20% once shared instruction boilerplate is stripped), making the aggregate figure an intention-to-treat estimate.

4. Four named conditions an agent-memory ablation must satisfy, plus detection procedures. C1 (the component executed), C2 (runs were independent), C3 (only the named axis varied), and C4 (the effect exceeds the measurement floor), each of which was satisfied in appearance and violated in fact, along with detection methods that require no knowledge of the specific defect.

Main Findings

• Decomposition bounds peak KV. Peak KV per query was 14.3 MiB [13.0, 15.4] for CoA-PKV against 35.5 MiB [34.0, 37.0] for the closed-book single-pass baseline (Flat) and 35.3 MiB [30.9, 39.8] for retrieval-matched RAG — a reduction of 59.9% and 59.7% respectively. The CoA-PKV interval clears both baselines with no overlap, while Flat and RAG overlap each other.

• The bounded working set is bought with more calls. Cumulative KV traffic is ≈1.6 times higher for CoA-PKV, so the claim is lower peak at higher total traffic, which matters only when KV-cache memory binds.

• Latency also dropped, but the figure is qualified. Per-query latency was 67 s (Flat), 52 s (RAG), and 29 s (CoA-PKV): −56.7% versus Flat and −44.2% versus RAG. The authors note these timings come from one contiguous service window, before the ≈3.5 times serving-stack drift of a later campaign, which is why they lead with peak KV.

• RAG filtering does not shrink the largest call. The retrieval-matched RAG pipeline still concatenates its passages into one generation call, producing a peak statistically indistinguishable from closed-book single-pass.

• Persistent recall costs memory. Enabling recall added +0.368 MiB [ +0.167, +0.590] of peak cache, a metric whose interval excludes zero. Recall prepends retrieved traces to the reasoner's prompt, growing prompts by ≈2% against per-query peaks of 13.6–21.9 MiB.

• Persistent recall buys no detectable accuracy. The accuracy delta was +0.015 [−0.011, +0.046]. The paper reports this is convention-dependent: scoring failed questions as 0 gives +0.015, while dropping questions either arm failed gives +0.003.

• The subgroup that can actually test efficacy is underpowered. Restricted to questions where recall could fire, the effect was −0.012 [−0.073, +0.046] (n=138) — a wider interval than the aggregate — while questions where it provably could not fire acted as a built-in control and gave +0.009 [−0.007, +0.026].

• Reachability varies sharply by dataset. As-sent reachability was 0% (HotpotQA), 27% (GPQA), 86% (MMLU-Pro), 2% (LB-HotpotQA), 10% (LB-2WikiMultihopQA), 16% (LB-MuSiQue), and 2% (LB-All). Stripping boilerplate halved LB-2WikiMultihopQA (10% to 1%) and LB-MuSiQue (16% to 7%), meaning roughly half their recall was firing on template text. Over the six cells, 136 of 575 answered questions were reachable as sent.

• MMLU-Pro is the sharpest null. It had the highest as-sent reachability at 86% (unchanged after stripping boilerplate, because items genuinely share subject vocabulary) and an accuracy delta of −0.012 — recall fired most and still did not help.

• C1: the component never ran. The persistent tier wrote traces that nothing ever read, so both arms of every prior memory ablation ran with no memory. Three further components were similarly inert: a client-side ablation flag for a server-side agent, a reranker whose 404s became "keep existing order," and a cost accumulator off the benchmark path that zeroed 38 published cells.

• C2: runs were not independent. With traces keyed by question embedding, a fixed-seed re-run retrieves its own previous answers; the authors measured 100/100 of a run's questions already carrying a stored trace, from 48,915 traces over 2,856 questions. The signature was detectable retrospectively: within groups sharing code version, n and seed, scores rose monotonically with run order in 6 of 6 groups on the most entity-dense dataset (p ≈ 0.0006). Correcting it moved one dataset 0.2949 → 0.2282, downward.

• C3: more than one axis varied. The original ablation switch disabled memory and moved execution from a service to in-process and changed the evidence source and the dependency set. An uncorrected MMLU-Pro pair read +0.370 — a 37-point apparent memory win produced by an option-letter extraction step fixed in one arm and not the other; with both arms on the same parser the cell reads +0.010.

• C3 also left a time confound it could not resolve. The memory-enabled arm ran first in all eight pairs, while the serving stack slowed ≈3.5 times. Within a pair the arms ran at the same speed, but memory-off failed more: 20 orchestrator give-ups to 1, with client timeouts level at 3. Across the eight pooled pairs' 794 paired questions, memory-on and memory-off failed 7 and 23 times: 3 versus 3 client timeouts (900 s), 1 versus 20 orchestrator give-ups (HTTP 500), and 3 versus 0 server disconnects.

• C4: the floor was higher than the effects. Pooled within-configuration standard deviation was 0.046 and 0.052 on the two multi-hop datasets, giving 95% bands of ±0.09–0.10; every architecture-versus-baseline delta the authors had been prepared to quote was smaller.

• The floor is not intrinsic. Re-measured under the trace-reset protocol with four replicates each, within-configuration standard deviation was 0.003 (MuSiQue) and 0.007 (2WikiMQA), and on a subsequent repair of the serving stack the four replicates of each were identical (σ=0) — 95% bands of ±0.006–0.014 rather than ±0.09.

• Corrections moved cells in both directions. Two corrections raised scores (output schema + aligned prompt +0.145; scoped retrieval +0.082/+0.032/+0.045), one lowered them (trace-reset protocol −0.067), and two intended improvements were regressions that were reverted (disabling reasoner CoT −0.209; lexical reranker −4 of 40 gold-in-top-5), which the authors offer as evidence of measurement rather than tuning.

• Sample size alone reversed a comparison. A comparison reading 0.704 > 0.614 at n=50 read 0.622 < 0.646 at n=100.

• One excluded cell would have produced a spurious significant benefit. LB-All, reported but not pooled because it overlaps three pooled subsets, had an arm asymmetry of 1 versus 17 failed questions driven by an orchestrator outage in the second arm. Its accuracy delta reads +0.071 — and had it been pooled, the across-cell accuracy interval would have shifted to [+0.001, +0.065] and excluded zero.

• Token estimation is a poor proxy for peak. The ratio of a query's pre-run token estimate to its measured peak was 6.3–12.2 times for CoA-PKV, versus 1.1–1.7 times for Flat and RAG, showing most of the ten-or-so decomposed calls generate far below their declared budget.

• A critic failure visible only in the logs. In the reported "good scenario" (question about My Neighbor Totoro's production studio, gold 1985), the answer critic scored the Reasoner's first, correct pass as "score=0.00 threshold=0.70 — needs improvement: Answer is factually incorrect," and was overridden downstream. The paper flags this as genuine system behaviour worth its own investigation.

• One metric can hide three failure mechanisms. On the reported "weak scenario" (gold IRA), all three reasoning traces converged on the correct entity before failing differently downstream: CoA-PKV's Synthesiser output an unrelated name ("Seán Kelly"), while CoA-NoKV and RAG produced "Provisional Irish Republican Army," a semantically right answer scored at zero against the gold string "IRA."

Methodology in Plain English

The authors build one system and vary one axis on it. The system, CoA-PKV, has three memory tiers: L1 is the ephemeral per-call KV cache; L2 is a Redis-backed key-value store passing typed Communication Units between agents; L3 is a Qdrant vector database serving both semantic evidence filtering and, when enabled, persistent trace recall and write. Six nodes run over these tiers — Decomposer, Retriever, Reasoner, Critic, Synthesiser, and a separate answer critic. Inference runs on two on-premises edge accelerators: an NVIDIA Jetson Orin Nano Super (8 GB) running llama.cpp with Qwen3.5-2B, and an NVIDIA DGX Spark running vLLM with Qwen3.5-35B-A3B-FP8 (35B total parameters, ≈3B active). The served model's hybrid attention keeps a KV cache on only 10 of 40 layers, giving the 2 × 10 × 2 × 256 × 1 B (fp8) = 10,240 B/token constant used to convert every KV figure.

To test the decomposition claim, they compare peak KV — the largest single call's (prompt + generated) footprint within a query, not the sum over calls — against a closed-book single-pass baseline (Flat) and a retrieval-matched RAG pipeline sharing the same reasoner, critic and evidence pool, over HotpotQA and LB-HotpotQA at n=100.

To test the memory claim, they ablate exactly two operations, search_outputs and store_qkv, producing CoA-PKV and CoA-NoKV, which are otherwise identical. Pairs are matched on dataset, n, seed, evidence scoping and endpoint; traces are cleared between conditions (evidence is not); and isolation is verified live by a probe requiring the recall path to be present when enabled and absent when disabled. Per-cell intervals are percentile bootstrap over per-question paired deltas (B=10⁴); the across-cell interval is a two-stage cluster bootstrap that resamples cells then questions, because questions within a cell share a difficulty profile.

The most distinctive methodological move is the audit itself. The authors reconstruct how they arrived at their own result and report four "conditions" that were satisfied in appearance and violated in fact, each of which independently produced an apparent memory benefit. For each condition they give a mechanical detection procedure — a counter the run summary refuses to omit (C1), a pre-run query of the store with the evaluation questions (C2), a diff of the arms' serialised requests plus counterbalanced ordering (C3), and band estimation from replicates rather than a within-run bootstrap (C4). They also test their own explanation of the null by re-implementing the retriever's predicate and splitting questions into those where recall could provably fire and those where it provably could not.

Why This Matters

Impact on research. The paper's central methodological claim is the contrapositive of a common practice: a non-zero memory

Authors’ abstract

Decomposing long-context inference across cooperating agents bounds the active KV cache per call rather than total evidence, which matters when KV-cache memory binds. Many such systems add a persistent tier storing and recalling reasoning traces, usually validated by an ablation reporting an accuracy gain. We measure both on one three-tier agent architecture. Decomposition delivers: peak KV working set of 14.3 MiB per query against 35.5 and 35.3 MiB for single-pass and retrieval-augmented baselines. The persistent tier does not: across eight controlled dataset pairs at n=100 per arm it costs +0.368 MiB [+0.167, +0.590] of peak cache and produces no detectable accuracy change (+0.015, 95% CI [-0.011, +0.046]). We argue the null is structural: single-question benchmarks supply each item with its own evidence and score it independently, and correctness requires resetting stored traces between conditions, so recall has nothing informative to retrieve. Reaching it took four measurement corrections -- three inflating the apparent benefit, the fourth making an effect that size look resolvable -- none visible in the results table. We give the conditions an agent-memory ablation must satisfy and detection procedures that need no knowledge of the specific defect.