Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agents

AI summary

Overview

• Research area: Large language model (LLM) agents, agent memory management, and reinforcement learning (RL) for agentic decision-making.
• Technical level: Advanced. The paper assumes familiarity with LLM agent architectures, retrieval-augmented generation, context-window constraints, and policy-gradient reinforcement learning (specifically GRPO).
• Scope: The paper proposes AgeMem, a framework that folds both long-term memory (LTM) and short-term memory (STM) management into a single learnable LLM policy exposed through six tool-based memory operations, trained with a three-stage progressive RL curriculum and a step-wise GRPO variant.

What This Paper Is About

LLM agents operating over long horizons are bottlenecked by finite context windows, so how they store, retrieve, compress, and discard information determines how well they perform. Existing systems almost always handle long-term memory and short-term memory as separate, loosely coupled modules driven by handcrafted schedules or auxiliary "memory manager" models, which limits adaptability and blocks end-to-end optimization. AgeMem's goal is to make memory management an intrinsic, learnable part of the agent's own policy rather than an external heuristic pipeline.

Key Contributions

1. A unified agentic memory framework (AgeMem): LTM and STM are managed jointly inside the agent's decision-making process, letting the LLM autonomously decide when, what, and how to store, retrieve, update, summarize, filter, or discard information.
2. A six-tool memory interface: LTM is manipulated through Add, Update, and Delete; STM through Retrieve, Summary, and Filter. These operations are added directly to the agent's action space instead of being controlled by external heuristics.
3. A three-stage progressive RL strategy with step-wise GRPO: The model first learns LTM storage, then STM context control under distractors, then coordination of both under full task settings. Step-wise GRPO broadcasts the terminal trajectory reward back to every intermediate step to address sparse and discontinuous rewards induced by memory operations.
4. Comprehensive evaluation: Experiments on five long-horizon benchmarks with two LLM backbones (Qwen2.5-7B-Instruct and Qwen3-4B-Instruct) against four representative memory-agent baselines plus a no-RL ablation.

Main Findings

• Highest average task performance on both backbones: AgeMem reaches 41.96% average on Qwen2.5-7B-Instruct and 54.31% on Qwen3-4B-Instruct, which the paper reports as relative gains of 49.59% and 23.52% over the no-memory baseline, respectively.
• Gains over the strongest baselines: Compared with the best baselines (Mem0 and A-Mem), AgeMem improves by 4.82 and 8.57 percentage points on average.
• RL training is what drives the gain: RL contributes 8.53 and 8.72 percentage points over AgeMem-noRL on the two backbones. On Qwen2.5-7B-Instruct, AgeMem-noRL averages 33.43% while full AgeMem averages 41.96%.
• Higher-quality stored memory: On HotpotQA, AgeMem achieves the highest Memory Quality (MQ) scores of 0.533 and 0.605 on the two backbones, judged by how relevant stored memories are to HotpotQA ground-truth facts.
• More efficient context usage: AgeMem uses 2,117 tokens on average versus 2,186 for AgeMem-RAG on Qwen2.5-7B-Instruct (a 3.1% reduction), and 2,191 tokens versus 2,310 on Qwen3-4B-Instruct (a 5.1% reduction).
• RL changes memory behavior, not just performance: On HotpotQA, Add Memory calls rise from 0.92 to 1.64 on Qwen2.5-7B-Instruct, Update Memory appears after training (0.13 versus nearly zero), and Filter Context rises from 0.02 to 0.31. Retrieve Memory decreases (2.31 to 1.95 on Qwen2.5; 4.62 to 4.35 on Qwen3), which the paper interprets as a shift from reactive repeated retrieval to selective, query-driven retrieval. Total tool calls rise from 4.33 to 4.92 (Qwen2.5) and from 7.50 to 8.67 (Qwen3).
• Ablations confirm every component matters (Qwen2.5-7B-Instruct): Adding LTM alone (+LT) yields gains of +10.6%, +14.2%, and +7.4% over the baseline on three representative datasets; adding RL (+LT/RL) improves further, especially on HotpotQA (+6.3%); the full system (+LT/ST/RL) improves by +13.9%, +21.7%, and +16.1%. STM tools give the largest additional boost on SciWorld (+3.1%) and HotpotQA (+2.4%).
• Multi-component rewards beat task-only rewards: On HotpotQA, the All-Returns reward strategy achieves an LLM-as-a-Judge score of 0.544 versus 0.509 for Answer-Only, and a memory quality of 0.533 versus 0.479, using 2,117 versus 2,078 tokens and 4.92 versus 3.93 tool calls. The paper reports noticeably faster convergence and higher final performance for All-Returns.
• Low sensitivity to the FILTER threshold: Performance is stable for θ_f in [0.4, 0.8], with Judge scores of 0.524, 0.551, 0.544, 0.530, and 0.531 and MQ of 0.511, 0.550, 0.533, 0.526, and 0.510 for θ_f = 0.4, 0.5, 0.6, 0.7, and 0.8 respectively; average token counts stay near 2,089 to 2,149 throughout.

Methodology in Plain English

AgeMem treats memory as part of the agent's action space rather than a background process. At each step the agent sees a state made of three things: the current conversation context (STM), the persistent memory store (LTM), and the task specification. Alongside ordinary language generation, the agent can pick one of six tools — three that edit persistent memory (Add, Update, Delete) and three that act on the active context (Retrieve brings the top-k semantically similar memories into context, Summary compresses a span of history, and Filter removes messages whose semantic similarity to a criterion exceeds a threshold θ_f).

Training runs over trajectories split into three consecutive stages, with LTM persisting across all of them but context reset between Stage 1 and Stage 2 to stop the agent from shortcutting the final task with leftover context:

• Stage 1 (LTM construction): the agent has a casual conversation containing useful information and may store salient facts.
• Stage 2 (STM control under distractors): the agent is fed synthetic distractor messages unrelated to the target query (generated by a procedure called DistractorGen) and must filter or summarize them away.
• Stage 3 (integrated reasoning): the agent receives the real query and must retrieve from LTM, manage context, and answer.

The trajectories are generated with K independent rollouts per task. Because rewards arrive only at the end of a trajectory, the authors use a step-wise variant of GRPO: compute a group-normalized advantage from the terminal reward, then broadcast that same advantage to every earlier step in the trajectory, including Stage 1 storage decisions and Stage 2 filtering decisions. This gives long-range credit assignment across heterogeneous memory actions.

The composite reward combines a task-completion term (an LLM-judge score in [0,1] against the expected answer), a context-management term (compression efficiency, preventive summarization/filtering, and preservation of query-relevant content), a memory-management term (storage quality, meaningful updates/deletes, and semantic relevance of retrieved memories), and penalty terms for context overflow or exceeding the interaction limit.

Notably, AgeMem is fine-tuned with RL only on the HotpotQA training set — because HotpotQA provides both questions and supporting facts, which automatically supplies Stage 1 context — and is then evaluated directly on all five datasets.

Why This Matters

• Research impact: The paper reframes memory from an external engineering component into a single learnable control problem under delayed supervision, and shows that jointly optimizing heterogeneous memory actions against one terminal reward outperforms optimizing them in isolation. The step-wise GRPO formulation also offers a template for other agentic settings where rewards are sparse and decisions are discontinuous.
• Real-world applications:
  • Long-running personal assistants that accumulate user preferences and task history across many sessions but must operate inside a bounded context window.
  • Customer-service or support agents that need to preserve durable account facts while suppressing irrelevant chat noise in an active conversation.
  • Embodied and game agents — the paper's benchmark suite covers embodied action (ALFWorld), game-based reasoning (SciWorld, BabyAI), planning (PDDL), and knowledge-intensive QA (HotpotQA).
  • Knowledge-intensive research or analytics agents that must retrieve supporting facts selectively instead of repeatedly re-querying an external store.
• Industry relevance: The framework removes the need for an auxiliary expert LLM to control memory, which the authors explicitly cite as a practical deployment constraint (higher inference cost and training complexity). Reducing average prompt tokens while improving task performance is directly relevant to serving costs for long-horizon agentic workloads.

Future Directions

• Finer-grained memory control: The current implementation uses a fixed set of memory management tools; the limitations section identifies extending this to more fine-grained control as future work.
• Evaluation beyond controlled settings: The authors state that their five benchmarks, while showing zero-shot cross-domain transfer, remain relatively controlled compared to open-ended deployment, and call for evaluation in persistent, long-term dialogue or real-user interaction scenarios.
• Broader curriculum sources: Training currently relies on HotpotQA as the source of three-stage trajectories; extending the curriculum to other data sources with richer interaction structures is proposed to broaden applicability.
• Open questions raised by the design: How the DistractorGen procedure and the three-stage curriculum scale to domains without QA-style supporting facts, and how the reward weights that balance task, context, and memory quality should be set for different deployment regimes, are not resolved by this paper (specific training hyperparameters such as the number of rollouts K, batch size B, or the reward weights are not reported in the provided content).

Target Audience

Researchers and engineers working on LLM agents, agentic memory systems, and RL fine-tuning for multi-step reasoning. It is most valuable to readers who already understand retrieval-augmented generation, context-window management, and policy-gradient RL, and who want a concrete design for teaching an agent to manage its own memory end-to-end rather than bolting on an external memory module. Readers looking for dataset statistics, full hyperparameter settings, and case studies will need the paper's appendices, which are referenced but not included in the provided content.

Authors’ abstract

Large language model (LLM) agents face fundamental limitations in long-horizon reasoning due to finite context windows, making effective memory management critical. Existing methods typically handle long-term memory (LTM) and short-term memory (STM) as separate components, relying on heuristics or auxiliary controllers, which limits adaptability and end-to-end optimization. In this paper, we propose Agentic Memory (AgeMem), a unified framework that integrates LTM and STM management directly into the agent's policy. AgeMem exposes memory operations as tool-based actions, enabling the LLM agent to autonomously decide what and when to store, retrieve, update, summarize, or discard information. To train such unified behaviors, we propose a three-stage progressive reinforcement learning strategy and design a step-wise GRPO to address sparse and discontinuous rewards induced by memory operations. Experiments on five long-horizon benchmarks demonstrate that AgeMem consistently outperforms strong memory-augmented baselines across multiple LLM backbones, achieving improved task performance, higher-quality long-term memory, and more efficient context usage.