Goal-Directed Search Outperforms Goal-Agnostic Memory Compression in Long-Context Memory Tasks

AI summary

Overview

Research area: Long-context memory for large language model agents (retrieval, agentic tool use, and reinforcement learning for reasoning).

Technical level: Intermediate. The paper assumes familiarity with retrieval-augmented generation, reinforcement learning with verifiable rewards, and LLM agent tool-calling, but its core argument is conceptual rather than mathematically heavy.

Scope: The paper introduces and evaluates SUMER (Search in Uncompressed Memory via Experience Replay), a reinforcement-learning agent that searches raw, uncompressed conversational memory instead of relying on hand-designed memory compression, and shows it reaches state-of-the-art results on the LoCoMo long-context conversational QA benchmark.

What This Paper Is About

Most LLM "memory" systems compress conversations into summaries, notes, or structured stores using fixed create/read/update/delete rules before anyone knows what question will later be asked. Because compression is lossy and goal-agnostic, it can throw away details that a future query needs. The authors ask whether it is better to skip compression entirely and instead train an agent to search the original, uncompressed conversation on demand, guided by the actual question. Their answer, tested on the LoCoMo benchmark, is yes: a trained search policy over raw memory outperforms both compression-based memory frameworks and feeding the whole conversation as context.

Key Contributions

1. SUMER, an end-to-end reinforcement learning agent trained with verifiable reward (RLVR) that learns to call two tools — search_memory (semantic and keyword modes) and submit_answer — over a memory bank built from individual, unmodified conversation messages.
2. A state-of-the-art result on LoCoMo, where SUMER with Qwen2.5-7B-Instruct reached overall scores of 48.65 token-level F1, 43.44 BLEU-1, and 66.79 LLM-judge correctness, a 43% gain over the prior best as reported by the authors.
3. A controlled comparison against goal-agnostic compression, covering RAG, full-context prompting, Langmem, A-MEM, Mem0, and MemMachine, plus a pre-RL version of the same agent (SUMER-Base) to isolate the effect of the learned search policy.
4. Ablation experiments that remove temporal context around retrieved memories, keyword search, or semantic search, quantifying how each tool contributes to accuracy and to the number of search turns required.
5. An open-source release of the SUMER code and all implemented baselines at https://github.com/zycyc/SUMER, along with a call for new benchmarks that are more dynamic and autonomously scalable.

Main Findings

• Search over raw memory wins overall. SUMER-GRPO achieved the best overall scores on nine held-out LoCoMo conversations: F1 48.65, B1 43.44, and judge accuracy 66.79, outperforming every compression-based baseline and the full-context baseline.
• Large margin over the strongest compression baseline. Against MemMachine, SUMER-GRPO improved overall F1 from 41.09 to 48.65 (+7.56), B1 from 33.77 to 43.44 (+9.67), and judge accuracy from 33.70 to 66.79 (+33.09, roughly double).
• Training the search policy matters, not just the architecture. SUMER-Base (before RL) scored 28.07 F1 / 23.95 B1 / 48.55 J overall. GRPO training added +18.24 in judge accuracy (a 37.57% relative improvement as stated in the main text; the appendix table reports 37.56%), along with gains in F1 and B1.
• The agent avoids overfitting to the training conversation. Mean reward climbed from around 0 to around 0.8 over 400 training steps, while validation judge accuracy rose from 48.55 to 66.79.
• Single-hop questions: SUMER-GRPO reached 61.82 F1 / 56.55 B1 / 79.53 J, improving judge accuracy by more than 15 points over the best non-RL variant.
• Multi-hop questions are hardest. SUMER-GRPO reached 44.83 J, the best judge accuracy of any method, but had a modest trade-off in F1 (28.45) and B1 (21.85) relative to MemMachine (32.86 F1 / 23.18 B1).
• Open-domain questions: smaller absolute gains, with SUMER-GRPO at 19.98 F1 / 17.45 B1 / 39.53 J, matching or slightly exceeding the best prior judge accuracy while scoring higher on F1 and B1.
• Temporal reasoning: SUMER-GRPO reached 42.23 F1 / 37.66 B1 / 62.72 J, with large margins over all other baselines.
• Beating full context. The full-context baseline scored 21.37 F1 / 15.01 B1 / 46.69 J overall, below SUMER-GRPO on all three metrics and below even SUMER-Base on judge accuracy (48.55).
• Ablation: semantic search is the most important single tool. Removing it dropped final judge accuracy to 61.38 and raised the average number of search turns to 26.34, the worst efficiency of the ablated variants.
• Ablation: keyword search is complementary. Removing it caused the mildest degradation (65.01 J, 12.94 turns), suggesting semantic retrieval does most of the work while keyword search adds precision in some cases.
• Ablation: local temporal context buys efficiency. Removing the surrounding messages around a retrieved memory left accuracy close to full SUMER (64.64 J) but nearly tripled the turns needed (29.94 versus 10.22 for full SUMER).
• RL helps even with crippled tools. Every ablated variant improved substantially under GRPO: No Context from 38.32 to 64.64 J (+68.67% relative), No Semantic from 40.47 to 61.38 J (+51.65%), and No Keyword from 49.52 to 65.01 J (+31.29%).

Methodology in Plain English

The researchers took the LoCoMo dataset, which contains ten long multi-session conversations, and stored every individual message as-is in a simple memory bank, keeping metadata such as speaker, session, and timestamp. Each message was embedded into a 1024-dimensional vector using Qwen3-Embedding-0.6B.

An LLM agent (Qwen-2.5-7B-Instruct) is given a question and a description of the memory database, then interacts over multiple turns. It has two tools. search_memory supports semantic search (finding the k most similar memories to a natural-language query by cosine similarity) and keyword search (returning all memories containing specified keywords in content or metadata), and both modes can be filtered by speaker or session. Retrieved memories are expanded into "memory groups" by concatenating the two messages immediately before and after the match. submit_answer ends the episode and produces the final response.

Training uses Group Relative Policy Optimization (GRPO) with verifiable rewards. For each question, eight trajectories are sampled, their terminal rewards are normalized into advantages, and the policy is updated. A crucial detail is multi-turn masking: text the agent generates contributes to the loss, while prompts and tool responses are visible as context but masked out, so the model learns to search and reason rather than to predict tool output. The agent can make up to five parallel tool calls per turn and up to 20 turns per trajectory.

The reward is the product of an LLM-judge correctness signal (binary CORRECT/WRONG from gpt-oss-120b, which accepts paraphrases that are factually right) and a token-level F1 score between the prediction and the gold answer, which discourages verbose answers. Failing to submit an answer yields a reward of -1, teaching the agent to stop searching once it has enough evidence.

One of the ten conversations (conv-48, containing 191 questions) was used for training; the other nine were held out for validation. Training ran on 8 NVIDIA H100 GPUs (80GB each) using the VERL framework, with a learning rate of 1×10⁻⁶, temperature 1.0 for training exploration and greedy decoding for validation, and evaluation every 50 steps. Notably, the researchers used Qwen3-Embedding-0.6B instead of text-embedding-3-small and gpt-oss-120b instead of gpt-4o-mini, which they say sped up iteration but prevents direct numerical comparison with prior reported results.

Why This Matters

The paper's central claim is a design-philosophy argument, not just a benchmark win: for long-context memory tasks, deciding what to keep before you know what will be asked (goal-agnostic compression) discards information that a question-conditioned search policy can recover. Because the agent searches raw data only when needed, the burden shifts from human-engineered memory schemas to learned behavior, echoing the "bitter lesson" idea that search and learning tend to beat hand-crafted representations as tasks scale. It also suggests that current long-context memory benchmarks may under-measure world modeling and schema learning, because they resemble extended pattern matching over relatively short conversational horizons.

Potential real-world applications:

• Personal AI assistants that must recall facts from months or years of chat history without pre-summarizing everything.
• Customer support and CRM agents that need to retrieve specific details from long interaction logs.
• Enterprise or research agents that answer questions over large collections of raw documents and meeting transcripts.
• Agentic frameworks that need a memory layer whose retrieval behavior adapts to the task rather than following fixed add/update/delete heuristics.

Industry relevance: The comparison set reads like a roster of commercial and open memory infrastructure (Mem0, MemMachine, Langmem, A-MEM), which makes the result directly relevant to teams building agent memory products. The trade-offs are real, though: searching raw uncompressed data costs retrieval and inference at query time, whereas compression costs an up-front processing pass and discards information permanently. The paper notes that its LoCoMo conversations do not even exceed the base model's context window, so its conclusions are strongest for the "moderately long" regime rather than for histories vastly longer than any context window.

Future Directions

• More expressive search policies and tool use. The authors emphasize that SUMER's search procedure is deliberately minimal, and suggest richer tooling plus tighter integration between retrieval, world modeling, and planning could improve results further.
• Benchmarks where compression should actually help. The paper calls for evaluation settings with histories far beyond the context window and conditions where distilling stable facts or schemas is plausibly beneficial, and where naive search alone may underperform.
• Lifelong-agent capabilities. The authors argue a genuinely strong lifelong agent needs to continually update an internal world state, reliably reuse past experience to avoid repeating mistakes, and extract cross-experience patterns for genuine generalization — none of which LoCoMo meaningfully probes.
• Apples-to-apples replication. Because the study used Qwen models and gpt-oss-120b rather than the GPT-4o-mini and text-embedding-3-small configurations of prior work, absolute numbers are not directly comparable; retraining under matched configurations is an open task.

Target Audience

This paper is most useful to researchers and engineers working on LLM agents, retrieval-augmented generation, and long-term memory systems, particularly those designing or evaluating memory infrastructure. It also speaks to reinforcement learning practitioners applying RLVR and GRPO to multi-turn tool-use tasks, and to benchmark designers interested in what current long-context evaluations fail to capture. Readers looking for a new search algorithm will not find one here; the contribution is the empirical and conceptual case that goal-directed search over uncompressed memory beats goal-agnostic compression, at least on the LoCoMo conversational QA benchmark.

Authors’ abstract

How to enable human-like long-term memory in large language models (LLMs) has been a central question for unlocking more general capabilities such as few-shot generalization. Existing memory frameworks and benchmarks focus on finding the optimal memory compression algorithm for higher performance in tasks that require recollection and sometimes further reasoning. However, such efforts have ended up building more human bias into the compression algorithm, through the search for the best prompts and memory architectures that suit specific benchmarks, rather than finding a general solution that would work on other data distributions. On the other hand, goal-directed search on uncompressed information could potentially exhibit superior performance because compression is lossy, and a predefined compression algorithm will not fit all raw data distributions. Here we present SUMER (Search in Uncompressed Memory via Experience Replay), an end-to-end reinforcement learning agent with verifiable reward (RLVR) that learns to use search tools to gather information and answer a target question. On the LoCoMo dataset for long-context conversation understanding, SUMER with Qwen2.5-7B-Instruct learned to use search tools and outperformed all other biased memory compression approaches and also the full-context baseline, reaching SOTA performance (43% gain over the prior best). We demonstrate that a simple search method applied to raw data outperforms goal-agnostic and biased compression algorithms in current long-context memory tasks, arguing for new paradigms and benchmarks that are more dynamic and autonomously scalable. Code for SUMER and all implemented baselines is publicly available at https://github.com/zycyc/SUMER.