MemSteer: Turning User Histories into Supervision for Agent Memory Optimization

AI summary

Overview

Research area: AI agents and long-term memory systems — specifically, automatic optimization of memory designs (storage, retrieval, and context construction) for personal-assistant agents.

Technical level: Advanced. The paper assumes familiarity with retrieval-augmented generation, LLM-based evaluation, and iterative prompt/program optimization.

Scope: MemSteer is a pipeline that converts unlabeled user histories into question–answer pairs with source-grounded evidence, plus MemSteer-Opt, an optimizer that uses this generated supervision to revise memory designs without touching model weights; it is evaluated on ATM-Bench and RHELM.

What This Paper Is About

Agents that help a user over months or years must store, retrieve, and assemble that user's records (conversations, emails, documents, images, videos) into useful context, and the best memory design differs from user to user. Adapting a design requires feedback, which normally means costly human-written QA pairs plus supporting evidence for every new history. The paper asks whether unlabeled user histories can themselves be turned into reliable QA–evidence supervision, and whether that supervision is good enough to drive memory-design optimization comparable to using official human-annotated QA.

Key Contributions

1. Memory-design optimization from unlabeled histories. MemSteer builds QA–evidence supervision from raw user records so that memory designs can be evaluated and improved without updating any model parameters and without manual annotation of the target history.
2. Evidence-grounded answer planning and verification. Each item is produced from an explicit answer plan that states how facts across related records jointly support a reference answer; the plan guides question generation and is then checked by source-grounded auditing and blind answer recovery.
3. MemSteer-Opt, an evidence-guided optimizer. A diagnosis step attributes each wrong answer to a storage, retrieval, or answering failure, samples cases stratified by failure type, and proposes design revisions that are accepted only if validation accuracy improves.
4. Empirical validation across optimizers and users. Generated supervision is competitive with the Official-QA reference on ATM-Bench and RHELM, also improves two existing optimizers (M⋆ and MemEvolve) within their own design spaces, and supports cross-benchmark transfer and continued adaptation to new users.

Main Findings

• Generated supervision improves designs substantially. Relative to Base-BM25, MemSteer-Opt raises ATM-Bench Question Score by 13.4 points (54.9 → 68.3) and Hard-set QS by 20.9 points (8.2 → 29.1); on RHELM it improves accuracy by 4.1 points without external data (22.3 → 26.4) and 6.6 points with it (33.3 → 39.9).
• It is competitive with human annotation. On ATM-Bench, MemSteer scores marginally above Official-QA on all four metrics, with QS margins of 1.1 points on the main set (68.3 vs 67.3) and 1.3 points on Hard (29.1 vs 27.9). On RHELM it is 0.1 and 0.8 accuracy points below Official-QA without and with external data (26.4 vs 26.5; 39.9 vs 40.7). Notably, Official-QA trains on 600 of the 1,013 ATM items and 788 of the 1,305 RHELM items that remain in its evaluation sets, whereas MemSteer uses no evaluation items for training.
• Other optimizers benefit too. From their own start systems, M⋆ gains 6.0 main-set QS points with MemSteer versus 4.4 with Official-QA; MemEvolve gains 18.6 with either source. Gains are not uniform: on Hard, MemEvolve reaches 9.5 QS (+2.3) and its R@10 drops by 0.1 points.
• Competitive with existing memory systems. On ATM-Bench, MemSteer-Opt reaches the highest QS among non-oracle systems using DeepSeek-V4 (68.3, versus 65.3 for Memexa and 71.8 for the gold-evidence Oracle). On RHELM with external data, 39.9 exceeds the strongest listed RAG, long-context, or memory-system baseline of 33.9 (GPT-4.1-mini long-context); without external data, 26.4 remains competitive with the strongest listed baseline, RAG at k=50 (28.9).
• Designs transfer across benchmarks. A design frozen after ATM-Bench optimization beats Base-BM25 on RHELM by 4.1 and 5.3 accuracy points without and with external data (26.4 and 38.6), and further optimization with either RHELM Official-QA or RHELM-generated QA adds more (MemSteer: 30.6 and 42.8).
• Cross-user transfer is mixed; personal continuation helps. Applying a frozen six-user RHELM design (34.3 pooled source-user accuracy vs 31.5 for Base-BM25) to two held-out personas slightly helps David R. Ellis (37.3 → 38.6) but hurts Martin Keller (41.2 → 37.7). Continuing optimization with supervision generated from each target user's own history improves both, to 44.3 and 39.2 respectively.
• Verification and planning both matter. MemSteer achieves 68.3 QS and 90.9 average QA-quality score while keeping 47.9% of candidates, versus Naive-QA at 54.2 QS and Ragas-QA at 63.0 QS; adding the checks lifts Naive-QA to 64.8 and Ragas-QA to 65.1. Two-stage plan-then-question generation scores 85.9 QA quality versus 73.1 for joint plan–question and 67.8 for direct generation, at a higher cost of 105.1k versus 68.2k and 64.0k tokens per question.
• Diagnosis beats raw evidence. Adding gold evidence to trace-based feedback gives only limited improvement, explicit failure attribution yields larger gains (particularly in evidence retrieval), and diagnosis combined with stratified sampling achieves the highest downstream QS.
• A modest training budget suffices. Going from 100 to 200 training questions produces the largest gains; beyond 200, retrieval recall largely plateaus while QS continues to improve on ATM-Bench and ATM-Bench-Hard.

Methodology in Plain English

Building supervision (MemSteer). Each user's history is organized as a graph of records, with edges between records that share a conversation, sit in adjacent document sections, reference each other, or overlap lexically. Starting from a seed record, the system walks the graph to collect a connected, budget-limited bundle of records, with citations pointing back to exact source spans. An LLM planner then writes an answer plan: what the question should ask, what the reference answer is, which facts support it and where they come from, what each evidence record contributes, and what operations combine the facts. Plans that fail automatic structural and provenance checks are discarded before any question is written; a separate call then turns the accepted plan into a standalone question.

Checking it. Two safeguards follow. A reviewer LLM audits the question against the plan and the bundle, judging support for every fact and every evidence record's stated contribution. Then a "blind" reader that sees only the question, the answer type, and the source records must (a) declare the question answerable, (b) reproduce the reference answer exactly under type-specific normalization, and (c) cite every selected evidence record. Failing candidates are dropped. The survivors are split into disjoint training and validation sets.

Using it (MemSteer-Opt). A memory design is a triple of storage, retrieval, and context-construction choices. For each question the system stores, retrieves, assembles context, and answers; correctness is binary. Designs are compared by validation accuracy, with the answering model, its prompt, and the context budget held fixed. For each training miss, the evidence annotations reveal whether the needed records were present in the final context (an answering failure), unrecoverable from memory (storage failure), or recoverable but not retrieved (retrieval failure). Up to ten cases per failure category are fed, along with the current design and revision history, to an LLM proposer, which suggests a revised design and a rationale. The candidate is adopted only if validation accuracy strictly improves; otherwise the current design is kept. Each setting runs 20 reflective evolution iterations. The same generated QA can instead be handed to M⋆ or MemEvolve, which keep their own design spaces and procedures.

Why This Matters

Impact on research. The paper reframes memory-design optimization as a supervision problem rather than a modeling problem: it shows that self-generated, evidence-cited QA can substitute for human annotation in the optimization loop, and that evidence annotations have a second use beyond scoring — localizing where in the pipeline a design failed. It also introduces a reusable quality-control recipe (answer plans plus blind answer recovery) for synthetic QA.

Real-world applications:

• Personal assistant agents that must reason across years of a user's email, photos, and videos, where no labeled QA exists for that individual.
• Enterprise assistants over shared mailboxes and document repositories, where privacy rules make sending data out for annotation impractical.
• Customer-support and account-management agents that must adapt to per-client histories and where one retrieval recipe will not fit all accounts.
• Long-horizon consumer agents where a per-user redesign must be produced cheaply, since the paper shows continued optimization from a target user's own history improves a transferred design.

Industry relevance. The approach is a drop-in feedback layer for existing memory stacks: it requires no weight updates, works with third-party optimizers, and delivers measurable gains from a simple BM25 baseline — relevant to teams shipping memory features who cannot afford per-customer annotation. The cost profile (token cost per generated question, 20 iteration rounds) and the paper's explicit ethics caution about generated QA combining sensitive facts also matter for production deployment.

Future Directions

• Lower supervision cost. Two-stage plan-then-question generation gives the best quality but costs 105.1k tokens per question versus 64.0k for direct generation; closing that gap without losing quality is an open engineering question.
• Better cross-user transfer and adaptation. Direct transfer of a shared design hurt one of two held-out personas (Martin Keller, 41.2 → 37.7), so how much to share versus personalize, and when personal continuation is required, remains unresolved.
• Robustness of LLM-based verification. The pipeline depends on remote LLM APIs, and the authors note that outputs may vary with model nondeterminism and provider updates; how stable the accepted annotation set is across judge models (they use GPT-5-mini, GPT-4.1, GPT-4.1-mini, and DeepSeek-V4-Flash in different roles) is not established.
• Generalization beyond two benchmarks. Evaluation covers ATM-Bench and RHELM; the authors deliberately avoid LoCoMo and LongMemEval as near-saturated, leaving open whether the diagnosis taxonomy transfers to other memory workloads and modalities.

Target Audience

Researchers and engineers working on agent memory, retrieval-augmented generation, and self-improving agent architectures; practitioners building personal or enterprise memory systems who need feedback signals without annotation budgets; and readers interested in synthetic-data quality control and automated design search over LLM pipelines. Readers without background in retrieval systems and LLM-as-judge evaluation will find the experimental sections dense.

Authors’ abstract

An effective agent memory system should adapt to a user's history and task requirements. Human-annotated supervision can guide this adaptation by providing feedback for evaluating and improving memory designs. Yet suitable labels are rarely available for new user histories, and manually constructing QA pairs and supporting evidence is costly. We introduce MemSteer, which turns unlabeled user histories into question-answer (QA) supervision with supporting evidence for memory-design optimization. MemSteer constructs an evidence-grounded answer plan that specifies how facts across related records jointly support a reference answer. The plan guides question generation, while source-grounded auditing and blind answer recovery check the resulting annotations. To study the utility of this supervision, we develop MemSteer-Opt, which combines QA-based design evaluation with evidence-guided failure diagnosis to revise memory store, retrieval, and context construction without updating model parameters. Experiments on ATM-Bench and RHELM show that this supervision is competitive with official-QA-guided optimization. Relative to Base-BM25, MemSteer-Opt improves ATM-Bench Question Score by 13.4 points and RHELM accuracy by 4.1 and 6.6 points without and with external data, respectively. The supervision also benefits two existing optimizers on ATM-Bench, while new-user experiments demonstrate continued design optimization using supervision generated from the target histories.