Memory Tax: Dissecting the Net Value of Agent Memory

AI summary

Overview

Research area: Long-term memory for LLM agents, and retrieval-augmented generation (RAG) as a baseline for evaluating it. Technical level: Intermediate — the ideas are conceptually simple, but the paper assumes familiarity with agent memory architectures, retrieval pipelines, and LLM-as-judge benchmarking. Scope (one sentence): A large-scale empirical study comparing eleven dedicated agent-memory systems against naive chunk-based RAG on three conversational-memory benchmarks and three model backbones, which introduces "memory benefit" and "memory tax" as a way to account for the net value of building memory at all.

What This Paper Is About

Agent memory systems have grown elaborate — flat stores, hierarchical tiers, graphs, and trained policies — but they are usually compared against each other rather than against simply retrieving chunks from the raw conversation history. The authors ask a basic question that this literature leaves open: what is the net value of building memory, over and above a naive RAG baseline that keeps the raw history intact and retrieves from it directly? They answer by measuring both the queries memory fixes that RAG misses (memory benefit) and the queries memory breaks that RAG already solved (memory tax), then trace where the tax comes from.

Key Contributions

1. A holistic reassessment of memory's net value. The paper evaluates eleven open-source memory systems across four design families (flat store, hierarchical/lifecycle, graph-based, trained policy) on LongMemEval-s, PersonaMem-v2, and MINT-Eval, with Qwen3.5-4B, Qwen3.6-27B, and GPT-5.6 Luna (low) as backbones, using naive RAG as the reference. Many systems fall below RAG despite higher cost.
2. A decomposition of net value into memory benefit and memory tax. Using per-query comparison against RAG (1-round), the authors separate the queries memory gains from those it loses, and frame both as consequences of the deviations from raw history that any non-trivial memory process introduces.
3. A systematic dissection of the tax from the system side. The tax is analyzed through faithfulness (how much information survives construction, measured by replaying QA with gold memory units and by tracing sampled atomic facts), accessibility (how easily preserved evidence is retrieved, measured by iterative retrieval replay and query–evidence similarity drift), and robustness (how much performance shifts when the construction or QA backbone changes).
4. Generalization and design implications. The benefit–tax lens is extended to experiential memory (ALFWorld, WebShop, Mind2Web, BigCodeBench, AppWorld, τ²-Bench) and world-model memory (Context-as-Memory, MemoBench), and the paper derives three lightweight remedies: recoverable deviations, recursive/self-referential memory, and modeling future demand.

Main Findings

• Naive RAG is a strong baseline, and many memory systems lose to it. On LongMemEval-s with Qwen3.6-27B, RAG (2-round) scores 81.4 while MAGMA scores 68.2, MemoryOS 73.4, SimpleMem 73.6, and OpenViking 58.8. The largest single deficit in the main table is OpenViking's 31.8 against RAG's 72.2 on LongMemEval-s with Qwen3.5-4B (−40.4). Deltas in the main table are computed against RAG (2-round).

• Memory is not uniformly worse — it wins in specific places. MemOS beats the reference on PersonaMem-v2 with both Qwen backbones (+1.0 with Qwen3.5-4B, +1.5 with Qwen3.6-27B), Memory-R2 gains +2.0 there with Qwen3.5-4B, and on MINT-Eval with GPT-5.6 Luna, Mem-T gains +3.9 and PlugMem +3.3. These wins sit alongside larger losses in the same columns, which is exactly the benefit-versus-tax structure the paper formalizes.

• Faithfulness: evidence is lost or degraded during memory construction. Tracing sampled atomic facts, the fraction with no loss ranges from 39.4% (OpenViking) and 65.6% (SimpleMem) up to 98.4% (MemoryOS) and 93.6% (MAGMA). OpenViking loses 38.0% of facts entirely (no memory unit) and 13.0% suffer major loss; SimpleMem partially loses 27.6%; Memory-R2 partially loses 14.8% and loses 6.0% entirely.

• Accessibility: preserved evidence can still be hard to reach. Replaying retrieval iteratively shows memory systems recover fewer queries than RAG in the first round, so some evidence is buried deeper after construction. Case inspection identifies three recurring patterns — evidence fragmented across units, diluted with unrelated content, or rephrased — and a query–gold-unit similarity comparison shows the query alignment of the same evidence shifting relative to raw turns.

• Construction loss is not the whole story. Replaying QA with oracle gold memory units reduces the measured tax (red boxes in the paper's Figure 3) but does not eliminate it. Some systems (MAGMA, LightMem) reach gold-memory scores close to gold-raw-turn scores yet sit far above their original scores, pointing to access failure rather than information loss.

• Robustness: memory amplifies backbone sensitivity. Holding the QA backbone fixed, changing the construction model causes noticeable variation. Holding construction fixed, changing the QA backbone causes larger performance swings than naive RAG does for many systems, meaning memory adds fragility beyond raw-history retrieval.

• Benchmark headroom is an evaluation-side driver of the tax. Across a broader set of long-term memory benchmarks, an extremely simplified IDF-keyword baseline exceeds 50% absolute score on some benchmarks and RAG-Hybrid reaches about 70%. When retrieval already solves most queries, memory has little room to add benefit and even modest losses become visible.

• The benefit–tax view transfers to other memory forms. In experiential memory, distilling long traces into workflows or strategies beats naive RAG on web tasks, while on tool-use several memory methods fall behind it. In world-model memory, directly retrieving and reinjecting past observations (I3DM, naive frames) gives the best long-horizon consistency — SSIM .560 / 1−LPIPS .704 on C-as-M-eval and SSIM .430 / 1−LPIPS .476 on MemoBench — ahead of raw KV (WorldKV .372/.459 and .379/.420), learned KV (ReWorld .378/.510 and .387/.422), visual latents (MagicWorld .388/.341 and .433/.412; FramePack .217/.312 and .279/.303), and memory tokens (HyDRA .320/.319 and .355/.326).

• Simple remedies reduce the tax. Retrieving from the source history alongside memory retrieval and jointly reranking before QA, and writing past queries plus retrieved evidence back into the memory store, both show consistent improvements and reduced memory tax across several systems.

Methodology in Plain English

The authors pick the most common memory setting — a personal agent with a long chat history — and hold everything else constant so that memory systems can be compared fairly against raw-history retrieval.

They use three benchmarks. LongMemEval-s has 500 questions over histories of roughly 50 sessions (~115k tokens each). PersonaMem-v2 poses four-way multiple-choice questions about a simulated user's implicit persona and preferences; the authors evaluate a 20% stratified subset (40 of 200 corpora, 934 questions). MINT-Eval tests memory under multi-target interference with an ordered stream of steps; they use a 10% stratified sample (60 of 595 corpora, 1,557 questions). For GPT-5.6 Luna they further use a 100-corpus subset of LongMemEval-s.

Their reference is a carefully specified naive RAG: the corpus is chunked along conversation turns without crossing session boundaries, short turns are merged into chunks of about 1024 characters with the last turn carried over as overlap, turns over 8192 characters are split into 8192-character windows with 1024-character overlap, and each chunk gets a session date-time header. BM25 and dense retrieval with Qwen3-Embedding-0.6B each return top-k chunks; the deduplicated union is reranked by Qwen3-Reranker-0.6B and the top-k chunks go to the backbone. A two-round variant adds a follow-up query if the first round is judged insufficient.

The fairness problem they solve is that memory units vary from a single turn to a whole session summary, so the same top-k can mean very different amounts of retrieved text. They define Retrieval Scope (RS) — the tokens in memory units exposed to the LLM from retrieval through answering, divided by the total tokens in the system's retrievable store — and tune each system's top-k, usually upward from its default, so its RS is no lower than RAG's one-round RS and roughly comparable across systems. Systems with only a handful of memory units keep a fixed top-k.

To separate benefit from tax, they compare memory and RAG query by query and plot the overlaps. For faithfulness, they identify the gold raw turns containing the evidence and the corresponding gold memory units, then replay QA with each alone: gold memory units remove retrieval failures, so whatever tax remains comes from construction. At the micro level they sample atomic facts and annotate loss as none, partial, major, or no memory unit. For accessibility, they replay retrieval for multiple rounds to see how many rounds it takes to recover all gold memory units, and compare (query, gold raw turn) similarity with (query, gold memory unit) similarity. For robustness, they build memory with each of three backbones and evaluate each memory with different QA backbones.

Answers are scored by an LLM-as-judge backed by Qwen3.6-27B rather than by F1 or BLEU, with agreement against human labels of 95.1% on LongMemEval-s, 100.0% on PersonaMem-v2 (multiple choice, scored by option letter), and 97.3% on MINT-Eval. Cost is tracked via total input/output tokens across ingestion and QA, end-to-end wall-clock time normalized to single-concurrency equivalent, LLM call counts, and memory store size.

Why This Matters

Memory research has been optimizing within a family of memory designs while rarely checking whether the family beats a well-tuned, cheap baseline. This paper makes the RAG baseline explicit and turns it into a measuring instrument: net value is defined relative to it, and the losses get a name and a diagnosis. That reframes progress as "does this beat retrieval, and where does it lose to it" rather than "does this beat the previous memory system." It also gives benchmark designers a concrete diagnostic — if an IDF-keyword baseline already clears 50% on a benchmark, that benchmark cannot clearly show memory's value, so the reported gains are compressed and the losses dominate.

Real-world applications:

• Personal assistants and companion chatbots that carry months of conversation history, where the question is whether to invest in extraction, summarization, and consolidation or just retrieve earlier turns.
• Customer-support and enterprise agents that must answer from long account histories, where losing a single piece of evidence during summarization is a correctness failure, not a stylistic one.
• Coding and tool-use agents that accumulate experience across tasks, where the paper finds memory tax is largest and several memory methods trail naive trajectory retrieval.
• Video and interactive world models that need long-horizon scene consistency, where simply reinjecting past observations outperformed KV-cache, latent, and memory-token representations on the reported metrics.

Industry relevance: teams building agent platforms now ship memory as a product feature with real ingestion and storage costs. The paper's cost metrics (tokens, wall-clock, LLM calls, store size) and its Retrieval Scope protocol give practitioners a way to report whether a memory layer actually pays for itself, and its two lightweight remedies — a recovery path back to raw history, and feeding past queries and retrieved evidence back into the store — are drop-in changes rather than new architectures.

Future Directions

• Extend the lens beyond conversational and visual memory. The authors explicitly leave omni-modal, latent, and parametric memory to future work, expecting benefit and tax to appear along those axes too.
• Move past static, offline, English benchmarks. The limitations section notes that long-lived deployments with interleaved queries and multilingual interactions are underexplored, and that stronger frontier models than GPT-5.6 Luna (low) were not tested due to cost.
• Model future demand instead of processing everything ahead of time. The paper contrasts ahead-of-time memory, which must speculate about future queries and grows expensive, with just-in-time memory, which is targeted but adds user-facing latency, and proposes predicting future information needs from past interests and access patterns — effectively treating memory as an implicit recommendation system.
• Close the loop between access and construction. The recursive, self-referential proposal — where past queries, evidences, and retrieval paths become source material for memory — suggests inter-query test-time scaling as an open research direction.

Target Audience

Researchers and engineers working on LLM agents, long-context systems, or retrieval pipelines who need to decide whether to build a memory layer and how to evaluate one honestly. It is especially useful for benchmark designers and for practitioners who ship memory as a product feature, since the Retrieval Scope protocol and the RAG-anchored reporting convention are directly adoptable. Readers new to the area will find the benefit/tax framing intuitive, but will need some background in retrieval pipelines to follow the faithfulness and accessibility analyses.

Authors’ abstract

Memory is increasingly used in long-horizon agents, yet the net value of building memory over direct retrieval from raw history remains largely unclear. Through a large-scale evaluation across multiple benchmarks and model backbones, we find many dedicated memory systems underperform naive RAG in many cases, indicating a negative net value. Taking RAG as a reference, we dissect the net value of memory into the gains and losses over direct retrieval, termed memory benefit and memory tax. These gains and losses arise because memory process changes how the raw information is organized and accessed. We view such changes as the double-edged deviations from the raw history. They are originally designed and introduced to benefit the task, for example by reorganizing facts or distilling preferences, but some may also remove or obscure evidence, giving rise to memory tax. Since memory benefit reflects the expected gains of its design, we then focus more on the overlooked memory tax, study how it arises from the system and evaluation side, and generalize the concept to more application scenarios of memory. We hope the notion of memory tax can provide a new lens and shift more attention from standalone performance to the net value of memory, informing more reliable memory designs and evaluations. We will open-source the code and results to the community. Project Page: https://hexuan21.github.io/MemoryTax/