Is Complex Agentic Memory Always Needed? Naive RAG Is Stronger Than We Thought

AI summary

Overview

Research area: LLM long-term memory and retrieval-augmented generation (RAG) for long conversations.

Technical level: Intermediate — assumes familiarity with retrieval pipelines (embeddings, rerankers) and LLM evaluation benchmarks, but the core argument is accessible.

Scope: A controlled empirical comparison of a deliberately naive raw-history RAG baseline against twelve agentic memory systems across four long-conversation benchmarks, plus diagnostic analyses of why the simple baseline holds up.

What This Paper Is About

Many recent "agentic memory" systems for long-term conversation add complexity — LLM-based fact extraction, mutation of stored memories, graph construction, hierarchical consolidation, and learned retrieval. The paper asks whether that complexity is actually necessary. The authors build Raw-RAG, a minimal baseline that indexes the raw conversation history (with timestamps and image captions), retrieves with dense similarity plus a cross-encoder reranker, and hands the retrieved windows to a reader LLM — and then test it head-to-head against the complex systems.

Key Contributions

1. Challenges the assumed need for explicit memory mutation. The authors audit Mem0's write behavior and show UPDATE and DELETE decisions are rare (4.39% of 33,902 write decisions on LoCoMo; 2.46% of 428,858 on LongMemEval), and that disabling mutation improves LoCoMo accuracy from 67.36% to 69.42%.
2. Establishes Raw-RAG as a strong reference baseline. Across LoCoMo, LongMemEval, MEME, and MemoryAgentBench, Raw-RAG ranks first among compared memory systems in six of seven benchmark–model settings, beating the strongest compared agentic system by 0.13–5.53 points.
3. Identifies evidence access as the dominant bottleneck. Measured coverage (does retrieval contain sufficient evidence?) varies far more across systems than utilization (accuracy given sufficient evidence), and Raw-RAG leads on coverage.
4. Shows that preserving raw information during ingestion matters. Even an oracle with complete access to the transformed memory stores of seven systems does not close the gap to a raw-evidence control.

Main Findings

• Naive RAG wins most comparisons. Raw-RAG leads LoCoMo by 0.65–0.95 points, LongMemEval by 0.13 points, GPT-4o-mini MEME by 2.24 points, and MemoryAgentBench by 4.48–5.53 points. The one loss is DeepSeek-V4-Flash MEME, where MemMachine leads by 1.52 points (44.71 vs. 43.19).
• Full context is still the ceiling in one hard setting. On DeepSeek-V4-Flash MEME, Full-context leads Raw-RAG by 19.98 points — retrieval, not memory architecture, is doing the work, but it is not free.
• Mutation is rare and often ineffective. Beyond the low UPDATE/DELETE rates, "effective mutations" (persisted text-changing updates or deletions) account for only 1.90% of LoCoMo decisions and 1.34% of LongMemEval decisions; many proposed updates are duplicates or malformed.
• Capable readers resolve state changes from raw evidence. In a MEME phase-visible oracle setting (all evidence available, no retrieval error), DeepSeek-V4-Flash reaches 82.32% on cascade questions and 91.54% on invalidation; overall scores are 50.50 (GPT-4o-mini), 85.48 (DeepSeek-V4-Flash), and 70.25 (Qwen3.5-35B-A3B).
• Coverage, not utilization, separates systems. On MemoryAgentBench, coverage ranges 19.07–45.29% while utilization ranges only 80.00–89.71%. On MEME, coverage varies by 40.66 points versus 21.04 for utilization. Raw-RAG has the highest coverage on both (45.29% and 76.35%).
• Reranking matters; exact chunk size does not. Removing the reranker costs 3.92 points (95% CI [−5.80, −2.06]) and returning only one item costs 11.73 points, while chunk sizes from 128 to 1,024 tokens do not significantly change the total. Final-k values of 5, 10, 30, and 60 stay within 1.3 points of the default k=15; top-120 loses 14.57 points.
• Adding memory management to Raw-RAG hurts. Atomic-fact extraction drops MemoryAgentBench total by 19.62 points (test-time learning falls from 47.68% to 15.37%), summary augmentation by 5.81, graph-expanded retrieval by 5.78, and display-turn deduplication by 2.39 — all against a 48.32% baseline.
• Raw history preserves reasoning paths, not just answers. On MEME tracking and cascade after-state questions, Raw-RAG has the highest conditional path coverage P(path | result) at 62.34%, versus 56.70% for MemMachine and 9.62% for MemU — meaning systems can retrieve the answer value yet omit the temporal or dependency chain needed to justify it.
• Ingestion-time loss is hard to undo. A complete-store oracle improves all seven tested systems, but Raw-RAG and MemMachine (which retain raw conversational content) reach 57.54% and 57.46% versus 61.60% for the raw-evidence control. Without the packing ceiling, all seven stores remain 7.58–29.38 points below raw evidence.
• Raw-RAG is far cheaper. On a 10-episode, 110-question MEME slice, total Add-to-ready plus Search time is 305.91 seconds, versus 1,561.97 (EMem), 2,769.56 (MemOS), 3,515.74 (MemMachine), 4,142.95 (Mem0), and 9,301.48 (EverMemOS) — 5.1× faster than the next system and 30.4× faster than EverMemOS. On the 1,710-message BEAM-1M stream, cumulative Add time runs from 0.33 hours (Raw-RAG) to 26.95 hours (Memora), with MemOS the next-fastest completed run at 3.25 hours.

Methodology in Plain English

The authors build the simplest defensible retrieval pipeline. Each conversation is serialized with message timestamps and image captions, cut into 256-token windows with 32-token overlap, and embedded into a single flat index. At query time they pull the 128 nearest windows by cosine similarity, rescore each question–window pair with a cross-encoder, and pass the top 15 windows to the reader LLM, which must sort out temporal ordering and state changes itself. No extraction, no summarization, no graph, no mutation.

They then evaluate this against twelve memory systems (EverMemOS, MemOS, EMem, SimpleMem, MemU, MemMachine, Mem0, Memora, Nemori, MemoryOS, A-MEM, RecMem) plus Full-context, on LoCoMo (10 conversations, 1,540 non-adversarial questions), LongMemEval (500 examples), MEME (100 episodes, 1,188 phase-specific questions), and MemoryAgentBench (2,071 questions from 14 sources), using matched evaluation settings where possible.

Two preliminary diagnostics motivate the design: an audit of Mem0's write decisions on all LoCoMo conversations and all 500 LongMemEval examples, and a MEME oracle experiment where readers get all evidence with no retrieval error. A review of 14 prior works identifies coarse retrieval units, restricted retrieval depth, or missing reranking in several evaluated baselines. Finally, they decompose performance into coverage (was sufficient evidence retrieved?) and utilization (was it answered correctly given sufficient evidence?), and run a complete-store oracle to test whether retrieval access or ingestion-time information loss is the binding constraint. Ablations use 20,000 paired, source-stratified bootstrap resamples with Holm correction.

Why This Matters

Impact on research: The paper argues that a well-specified naive RAG baseline should be mandatory in memory-system papers, and that reported gains over weak baselines (coarse chunks, shallow retrieval, no reranking) may reflect baseline weakness rather than architectural benefit. It also reframes the problem: the bottleneck is evidence access and relational completeness, not memory management.

Real-world applications:

• Personal assistants and customer-support agents over long histories, where ingestion cost and latency dominate deployment economics — 0.33 hours versus 26.95 hours of ingestion on a one-million-token history is a product-level difference.
• Enterprise knowledge and ticket-history search, where preserving raw source text also preserves auditability and lets downstream readers reconstruct why a fact is current.
• Cost-constrained or on-premise deployments that cannot afford 21.6–109.3M LLM tokens and 2.9–8.8 hours to build a memory store, compared with roughly 12 minutes for raw-history indexing.
• Regulated domains where deletion and provenance requirements make LLM-rewritten memory stores harder to defend than indexed source records.

Industry relevance: The results suggest that much of the value attributed to elaborate memory architectures is recoverable with standard dense retrieval plus reranking, which is cheap, well-understood, and already in most production stacks.

Future Directions

• Test beyond offline conversation. The authors note their evidence covers only public offline benchmarks and does not establish behavior for interactive, multimodal, or broader operational settings.
• Probe reader capability as a variable. State reconstruction from raw evidence depends on the reader; whether explicit state maintenance helps weaker readers or reduces repeated reasoning across queries is left open.
• Map when structure does pay off. The paper calls for identifying when structured or compressed memories beat raw retrieval, weighing accuracy against latency and compute.
• Address storage, privacy, and deletion. Keeping raw history raises storage and privacy concerns; future work should incorporate explicit privacy and deletion constraints.

Target Audience

Researchers and engineers building LLM memory or long-context retrieval systems, benchmark designers who need to know which baselines are strong enough to be meaningful, and practitioners choosing between a lightweight RAG index and a full agentic memory pipeline for production. Readers evaluating memory-system claims will benefit most from the coverage/utilization decomposition and the complete-store oracle methodology.

Authors’ abstract

LLM memory is widely used in long-term conversation understanding. Agentic memory systems often use mutation, structural organization, learned retrieval strategies, or other complex operations. Is this complexity always necessary? We study Raw-RAG, a naive RAG baseline using dense retrieval and reranking over raw history to supply evidence to a reader LLM. Across LoCoMo, LongMemEval, MEME, and MemoryAgentBench, Raw-RAG ranks first among the compared memory systems in six of seven benchmark-model settings, outperforming the strongest compared agentic memory system by 0.13-5.53 points. We investigate why raw-history retrieval remains competitive. Evidence analysis shows that its higher coverage of necessary evidence can offset lower utilization. Useful evidence includes both answer values and the relations and dependency paths needed to derive them. Even complete access to transformed memory stores leaves an accuracy gap relative to raw evidence, highlighting the importance of preserving raw information. These findings challenge the necessity of complex agentic memory for the evaluated conversation tasks and establish naive RAG as an essential baseline.