AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversations

AI summary

Overview

Research area: Natural Language Processing — specifically memory management for LLM-based conversational assistants and benchmarking methodology for long-horizon dialogue.

Technical level: Advanced. The paper assumes familiarity with retrieval-augmented generation, agentic memory architectures, LLM-as-judge evaluation, and reinforcement-learning-style notions of on-policy versus off-policy data.

One-sentence scope: The paper introduces AMemGym, an interactive, fully automated environment that evaluates and optimizes conversational memory by grounding free-form LLM-simulated user interactions in a structured, pre-planned evolution of user states.

What This Paper Is About

Assistants built on large language models need memory to serve users across long conversations, but existing memory benchmarks evaluate them on static, pre-generated dialogue data that the assistant did not produce itself — a practice the authors call off-policy evaluation. The paper argues this is unrealistic, biased, and expensive to scale, and proposes instead an interactive environment where an LLM-simulated user role-plays over a planned trajectory of evolving user states, so that the assistant's own conversational choices shape the evaluation. The goal is a benchmark that is simultaneously reliable, diagnostic, and usable as a feedback signal for improving memory strategies.

Key Contributions

1. AMemGym, an on-policy evaluation framework for conversational memory. It grounds free-form interactions between an assistant and a simulated user in a structured state evolution, producing a scalable and diagnostically rich environment for assessing memory in personalization settings.

2. Empirical demonstration of off-policy bias plus the first extensive on-policy evaluation of popular memory systems. The authors show reuse bias in off-policy evaluation and measure native LLMs, RAG, agentic-write variants, and established memory agent frameworks.

3. Diagnostic metric decomposition. Failures are attributed to three operational stages — write, read, and utilization — with a normalized memory score that isolates memory ability from raw task performance.

4. A proof of concept for agent self-evolution. An agent uses environmental feedback inside AMemGym to autonomously refine its memory management policy, expressed as a natural-language prompt.

Main Findings

• Off-policy evaluation introduces reuse bias that can mislead optimization. Rankings of memory configurations shift between evaluation modes. For example, AWE-(2,4,30) scores .291 on-policy but .253 off-policy (Δ .038), while RAG-(2,4,30) scores .227 on-policy and .241 off-policy (Δ .014). Trends for the ns and topk parameters differ between the two settings.
• The off-policy bias is less pronounced for native LLMs than for memory agents. The exception noted is gemini-2.5-flash-lite, which scores .269 on-policy but .204 off-policy (Δ .065). Dialogue understanding, the authors suggest, can proxy for long-horizon interaction in LLM comparison except in such cases.
• LLMs handle precise information well in short contexts but degrade sharply over long interactions. All evaluated LLMs achieve an upper bound of S_UB > 0.8, yet as history grows most models fall below 50% of their upper bounds, and some perform no better than random guessing in later periods.
• Carefully designed agentic memory helps. AWE variants achieve the highest memory scores, outperforming native LLMs and standard RAG, indicating that selective, agentic information curation beats storing all raw history. AWI may lose crucial information through aggressive filtering.
• Failure modes trade off against each other. Tailored retrieval or compression improves utilization at the cost of read efficiency. Lower update frequency and larger short-term memory increase read failures. Top-k has minimal impact on read and utilization but a non-monotonic effect on write, trading recall of critical information against signal-to-noise ratio.
• Diagnostic failure rates by strategy (mean over all periods): LLM — write .301, read .087, utilization .244; RAG — .377, .172, .067; AWE — .338, .159, .074; AWI — .286, .245, .122.
• Self-evolution works, primarily by improving writing. Memory scores rise from .172 (No Evolution) to .197 (Question Only) and .197 (Complete Feedback). The write failure rate drops from .293 to .291 and then .263 with Complete Feedback, while read failures remain stable (.242, .235, .237) and utilization failures are .118, .110, and .136 respectively.
• Data quality meta-evaluation supports reliability. State exposure was judged successful on 200 queries with a 99.1% average quality score and Gwet's AC1 of 96.8%; conversational state integrity reached 99.2% consistency with a Gwet's AC1 of 98.2% across 748 annotated items from 40 conversations; on 100 questions, inter-annotator agreement was 0.92 and agreement between LLM answers and each human was 0.96 and 0.94.
• Costs are low. Offline structured data generation uses roughly 0.14M input tokens and 15.2K output tokens, costing about $0.40 per instance with gpt-4.1. The user simulator adds roughly 74.5K input tokens and 2.7K output tokens, about $0.17 with gpt-4.1 or $0.02 with deepseek-v3, versus roughly $13.0 to evaluate gpt-4.1 itself.

Methodology in Plain English

The authors work backwards from the answer. Instead of collecting conversations and hoping to find testable facts in them, they start by sampling a user profile from a pool of 100K personas in the Nemotron-Personas dataset, then sample evaluation questions for that profile. An LLM extracts which information types each question needs, and these are merged into a canonical global state schema of M variables with discrete value sets.

Next, they simulate how the user's states evolve over N_p periods. Each period has a state vector assigning a value to every variable, each transition is motivated by a narrative life event, and the resulting trajectory is the ground truth. From that trajectory they generate natural-language utterances that implicitly expose small subsets of the current state, and they generate personalized answers for each question and each state variant, accepting an answer only if an LLM classifier can recover the intended variant from the question-answer pair.

During evaluation, a user LLM role-plays with the assistant, seeded by the fixed state-bearing utterances so that key states appear reliably in the dialogue. After each period the assistant answers all evaluation questions. Overall accuracy is reported directly; a normalized memory score rescales accuracy between a random lower bound and an upper bound obtained by simply handing the assistant the ground-truth states, bypassing retrieval. Failures are then attributed to write, read, or utilization by querying individual state values and comparing against the nearest write position.

Two configurations are used: base (10 periods, 2 required states per question, 4 interaction turns per exposure) requiring a 128K+ context window, and extra (20, 3, 10) requiring 512K+ context window. Both use 20 randomly sampled user profiles with 10 questions each, for 200 questions tested at N_p+1 positions. Four memory implementations are compared — native LLM, standard RAG, Agentic Write (External), and Agentic Write (In-Context) — all using gpt-4.1-mini for generation and text-embedding-3-small for embeddings, with the default AWE configuration AWE-(2,4,30). Established frameworks Mem0-G, Nemori, and A-Mem are also included. The benchmark itself covers 20 unique user profiles, ages 18–85 across 6 age groups, 9 education categories, and 16 occupations; the base version has 11 periods per user, 4.29 sessions per period, and 47.15 turns per user, while the extra version has 21 periods, 3.89 sessions per period, and 81.60 turns per user.

Why This Matters

Impact on research. The paper challenges the default evaluation protocol for conversational memory — static, manually curated or semi-automated off-policy data — and shows empirically that this protocol can reorder system rankings. The write/read/utilization decomposition gives researchers a way to say not just that a memory system failed, but where.

Real-world applications

• Personal assistants that must track shifting user preferences, habits, and plans across months of conversation.
• Recommendation systems that adapt suggestions to a user's latest implicit state rather than a fixed profile.
• Customer-support agents that must retain incident history and prior resolutions without swamping their context window.
• Coaching, tutoring, or health-oriented assistants where the advice depends on a user's evolving circumstances.

Industry relevance. The benchmark is fully automated and cheap: about $0.40 per instance for data construction and $0.17 for the simulated user with gpt-4.1. The authors argue the on-policy overhead is negligible relative to the inference cost of the models under test. The self-evolution result suggests a practical path for product teams to tune memory update policies from evaluation feedback rather than manual prompt engineering, and the work was conducted with authors affiliated with Meituan and The Hong Kong University of Science and Technology.

Future Directions

1. Broadening coverage. The benchmark uses 20 user profiles; scaling persona diversity and domain coverage is an open direction, and the authors note custom sampling strategies could target specific real-world distributions.
2. Extending self-evolution. The reported self-evolution result is explicitly a proof of concept, targeting only the in-context agent's memory buffer prompt. Whether it scales to external-memory agents, retrieval policies, and longer evolution cycles is unresolved.
3. Understanding when off-policy proxies hold. The paper finds off-policy bias is small for native LLMs but large for memory agents and for at least one LLM (gemini-2.5-flash-lite), leaving the boundary conditions to be characterized.
4. Exploring the memory parameter space. The authors report trade-offs among update frequency, short-term memory size, and retrieved memory count; the full extra-configuration results and other configurable parameters are deferred to appendix material not included in the provided content.

Target Audience

Researchers and engineers building long-horizon conversational memory systems, benchmark designers interested in on-policy and simulation-based evaluation, and practitioners who need to choose or tune a memory architecture — such as RAG, agentic write, or a memory agent framework — for a deployed assistant.

Authors’ abstract

Long-horizon interactions between users and LLM-based assistants necessitate effective memory management, yet current approaches face challenges in training and evaluation of memory. Existing memory benchmarks rely on static, off-policy data as context, limiting evaluation reliability and scalability. To address these gaps, we introduce AMemGym, an interactive environment enabling on-policy evaluation and optimization for memory-driven personalization. AMemGym employs structured data sampling to predefine user profiles, state-dependent questions, and state evolution trajectories, enabling cost-effective generation of high-quality, evaluation-aligned interactions. LLM-simulated users expose latent states through role-play while maintaining structured state consistency. Comprehensive metrics based on structured data guide both assessment and optimization of assistants. Extensive experiments reveal performance gaps in existing memory systems (e.g., RAG, long-context LLMs, and agentic memory) and corresponding reasons. AMemGym not only enables effective selection among competing approaches but also can potentially drive the self-evolution of memory management strategies. By bridging structured state evolution with free-form interactions, our framework provides a scalable, diagnostically rich environment for advancing memory capabilities in conversational agents.