Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolution

AI summary

Overview

• Research area: Agent memory systems for large language model (LLM) agents — specifically procedural memory, the "how-to" knowledge an agent accumulates from past task executions.
• Technical level: Intermediate. The paper assumes familiarity with LLM agents, retrieval-augmented generation, embeddings, and tool-use benchmarks, but the mechanisms are described conceptually enough for a motivated non-specialist.
• Scope: The paper proposes ReMe ("Remember Me, Refine Me"), a framework that manages an agent's procedural memory across its full lifecycle — acquisition, reuse, and refinement — and evaluates it on BFCL-V3 and AppWorld with the Qwen3 model family.

What This Paper Is About

Most existing agent memory systems follow a "passive accumulation" paradigm: they store past trajectories or workflow summaries and treat memory as a static, append-only archive. This causes three problems — coarse-grained experiences that bury the core logic, retrieved experiences applied without adaptation to the new task, and memory pools that degrade into "a mixture of valid insights and toxic noise" because nothing is ever removed. ReMe's goal is to make memory an actively evolving substrate that distills fine-grained experience, adapts it to each new task, and prunes what stops working.

Key Contributions

1. The ReMe framework: a full procedural-memory lifecycle combining multi-faceted experience distillation, context-adaptive reuse, and utility-based refinement, closing the loop that the authors say resolves the "passive accumulation" dilemma.
2. reme.library dataset: a released fine-grained procedural memory dataset built from diverse agentic tasks, containing structured success patterns and failure lessons, offered as a community resource.
3. Empirical state-of-the-art results on BFCL-V3 and AppWorld across Qwen3-8B, Qwen3-14B, and Qwen3-32B, plus six additional LLM backbones (GPT-4.1-2025-04-14, o4-mini-2025-04-16, Qwen3-Max-Preview, Kimi-K2-Thinking, DeepSeek-V3.2, GLM-4.7).
4. Demonstration of a memory-scaling effect, where a smaller model equipped with ReMe surpasses a larger memoryless model, which the authors frame as a computationally efficient path to lifelong agent learning.

Main Findings

• Headline gains over no memory: Qwen3-8B with ReMe improves over the No Memory baseline by 8.83% in Avg@4 and 7.29% in Pass@4 on average across the two benchmarks.
• Dynamic beats fixed: ReMe (dynamic), which updates the experience pool during execution, outperforms ReMe (fixed), which does not, across all model sizes and both benchmarks.
• Memory substitutes for scale: Qwen3-8B + ReMe (dynamic) reaches 55.03% Pass@4 versus 54.65% for vanilla Qwen3-14B. Qwen3-14B + ReMe (dynamic) beats Qwen3-32B without memory on Avg@4 (44.66% vs. 40.89%) and Pass@4 (63.71% vs. 61.52%).
• Best full results (ReMe dynamic): Qwen3-32B reaches 56.17% Avg@4 / 76.44% Pass@4 on BFCL-V3 and 42.02% Avg@4 / 63.49% Pass@4 on AppWorld, with the highest overall averages of 49.10% Avg@4 / 69.97% Pass@4.
• Baseline instability: LangMem performs well on BFCL-V3 but drops significantly on AppWorld, especially for smaller models, whereas ReMe (dynamic) is consistent across both.
• Fine-grained extraction wins: Keypoint-level experiences beat trajectory-level extraction. For Qwen3-8B, keypoint-level yields +4.17 Avg@4 / +6.22 Pass@4 over no memory versus +2.67 / +0.45 for trajectory-level.
• Quality over quantity in pool updates: Selective addition (only successful trajectories become experiences) beats full addition by 3.50% Avg@4 and 2.66% Pass@4 on BFCL-V3 with Qwen3-8B.
• Each refinement component adds value (Qwen3-8B, BFCL-V3): full addition 40.83%/62.00%; selective addition 44.33%/64.66%; plus failure-aware reflection 45.00%/64.66%; plus utility-based deletion 45.17%/68.00%.
• Reranking and rewriting both help (Qwen3-8B, BFCL-V3): No Memory 24.41%/28.50%; +ReMe 27.17%/34.66%; +rerank 28.91%/36.67%; +rewrite 28.67%/37.33%; both 29.00%/40.67%. (Note this ablation uses a different setup: thinking mode disabled, ReMe(fixed).)
• Usage-scenario indexing is the best retrieval key: for Qwen3-8B it gives 44.50% Avg@4 / 65.77% Pass@4, versus 44.00%/63.33% for raw task query, 42.50%/63.77% for generalized query, and 44.22%/65.33% for query keywords. The same ordering advantage holds for Qwen3-14B and Qwen3-32B.
• A stronger summarizer helps the agent: with LLM_execute fixed at Qwen3-8B, scaling LLM_summ from 8B to 14B to 32B raises Avg@4 from 44.50% to 46.33% (Δ=1.83↑) to 47.83% (Δ=3.33↑), and Pass@4 from 65.77% to 66.00% (Δ=0.23↑) to 68.00% (Δ=2.23↑).
• Retrieval saturates: performance rises with the number of retrieved experiences, K, from 0 to 10, then saturates; beyond that, more retrieval can hurt because of noisy experiences — K=5 is used in the main experiments.
• Low overhead: on AppWorld, average inference latency per task is 21.42 seconds without memory and 23.96 seconds with ReMe (+2.54).
• Fewer and different failures (Qwen3-8B on BFCL-V3): total failure cases drop from 62 (No Memory) to 47 (ReMe); ReMe fixes 17 baseline-specific errors while introducing only 2 new ones. Reasoning Error falls from 22 to 14, with a moderate reduction in Action Omission.
• Consistent gains on other backbones (BFCL-V3 Avg@4): GPT-4.1 48.25% → 53.33% (fixed) → 54.67% (dynamic); o4-mini 54.67% → 57.78% → 60.00%; Qwen3-Max 59.00% → 61.83% → 64.00%; Kimi-K2-Thinking 57.17% → 62.17% → 66.00%; DeepSeek-V3.2 53.06% → 56.00% → 57.33%; GLM-4.7 68.00% → 70.00% → 73.83%.
• Results on the full BFCL V3 Multi-Turn category (750 tasks): Qwen3-8B goes from 37.78% overall to 43.02% (+5.24%), with Base 59.55% → 65.77%, Miss Param 31.00% → 38.50%, Miss Func 19.00% → 21.50%, and Long Context 47.00% → 52.00%.

Methodology in Plain English

ReMe treats an experience not as a saved transcript but as a structured record: E = ⟨ω, e, κ, c, τ⟩, where ω says when to use it, e is the core content, κ is a set of categorization keywords, c is a confidence score between 0 and 1, and τ lists the tools involved.

Acquisition. The executing model runs each training task multiple times (N=8 samples, temperature 0.9) to collect a spread of good and bad trajectories. A summarizer model then reads them three ways: it recognizes success patterns, analyzes failures for pitfalls, and compares the best and worst runs to isolate the decision that made the difference. An LLM-as-a-Judge step validates whether each extracted experience is actionable, and near-duplicates are removed by comparing embedding cosine similarity against a threshold. Surviving experiences are indexed by the embedding of their usage scenario ω and stored in a vector database.

Reuse. For a new task, the system embeds the task and retrieves the top-K (K=5) most similar experiences by cosine similarity. An optional reranker model re-evaluates their relevance to the current task's specific constraints, and a rewriting module fuses the retrieved experiences into a single task-specific guidance text rather than pasting them in raw.

Refinement. After each task, only successful trajectories are distilled and added (the authors found that a single failed trajectory at run time gives too little context for reliable failure analysis). Failures instead trigger a failure-aware reflection: the summarizer extracts lessons, the agent retries on them, and if the retry succeeds the lessons enter memory — otherwise they are discarded. Self-reflection is capped at 3 attempts. Separately, every experience tracks how many times it was retrieved (f) and how often its recall contributed to success (u); once f ≥ α = 5, an experience whose u/f ≤ β = 0.5 is deleted.

Experimental setup. For BFCL-V3, 50 tasks from the base multi-turn category seed the initial pool and 150 tasks form the evaluation set. For AppWorld, 90 training tasks seed the pool and 168 test-normal tasks are evaluated. Metrics are Avg@4 (average task success rate across four independent trials) and Pass@4 (probability at least one of four trials succeeds), each averaged over three independent runs with standard deviation reported. LLM_summ = LLM_execute for the self-evolution experiments, text-embedding-v4 with its default 1024 dimensions is used for indexing, and agent iterations are capped at 30.

Why This Matters

Research impact. The paper reframes agent memory as a lifecycle problem rather than a storage problem, and provides evidence — the memory-scaling effect — that a well-maintained memory can substitute for raw parameter count. It also contributes an open dataset (reme.library) and code (github.com/agentscope-ai/ReMe), and its ablations give concrete design guidance: use fine-grained keypoints, index by usage scenario, add only from successes at runtime, and delete low-utility entries.

Real-world applications:

• Long-running customer-service or personal-assistant agents that must improve from repeated interactions without retraining the underlying model.
• Enterprise workflow automation (e.g., tool-chaining over email, payments, and calendar APIs) where latency budgets matter — ReMe adds only 2.54 seconds per AppWorld task.
• Cost-constrained deployments that need near-frontier performance on smaller, cheaper models; ReMe lets Qwen3-8B match or exceed memoryless Qwen3-14B.
• Non-stationary environments where task distributions shift over time, since the utility-based deletion mechanism is designed to prevent the memory pool from filling with outdated entries.

Industry relevance. The approach requires no gradient updates and no retraining, only prompt-level context augmentation, which makes it straightforward to bolt onto existing agent stacks. The reported comparisons against LangChain's LangMem and A-Mem target the tools practitioners already use.

Future Directions

• Flexible, context-aware retrieval. The authors note ReMe currently retrieves once at the beginning of each task; dynamically incorporating experiences mid-task is the most direct extension they raise.
• Stronger experience validation. Validation currently rests mainly on an LLM-as-a-Judge approach, which the authors say may overlook nuanced aspects of experience quality and relevance.
• Advanced summarization for small models. Because summarizer capability strongly influenced results (Avg@4 rose from 44.50% to 47.83% as LLM_summ scaled from Qwen3-8B to Qwen3-32B), the authors suggest designing summarization strategies that let small models capture the same benefit.
• Open question on failure learning. The paper shows single failed trajectories at run time are too sparse for reliable failure analysis, but retains failure-aware reflection as a successful-only admission path. How to make run-time failure learning as reliable as multi-trajectory offline analysis remains unresolved.

Target Audience

Researchers and engineers working on LLM agents, tool use, and memory-augmented reasoning will get the most from this paper, particularly those building long-running agents that need to improve online without retraining. It is also useful for practitioners choosing between memory frameworks, since it benchmarks directly against A-Mem and LangMem, and for anyone studying compute-efficient alternatives to scaling model parameters. Readers should be comfortable with embedding-based retrieval and agent benchmark terminology.

Authors’ abstract

Procedural memory enables large language model (LLM) agents to internalize "how-to" knowledge, theoretically reducing redundant trial-and-error. However, existing frameworks predominantly suffer from a "passive accumulation" paradigm, treating memory as a static append-only archive. To bridge the gap between static storage and dynamic reasoning, we propose $\textbf{ReMe}$ ($\textit{Remember Me, Refine Me}$), a comprehensive framework for experience-driven agent evolution. ReMe innovates across the memory lifecycle via three mechanisms: 1) $\textit{multi-faceted distillation}$, which extracts fine-grained experiences by recognizing success patterns, analyzing failure triggers and generating comparative insights; 2) $\textit{context-adaptive reuse}$, which tailors historical insights to new contexts via scenario-aware indexing; and 3) $\textit{utility-based refinement}$, which autonomously adds valid memories and prunes outdated ones to maintain a compact, high-quality experience pool. Extensive experiments on BFCL-V3 and AppWorld demonstrate that ReMe establishes a new state-of-the-art in agent memory system. Crucially, we observe a significant memory-scaling effect: Qwen3-8B equipped with ReMe outperforms larger, memoryless Qwen3-14B, suggesting that self-evolving memory provides a computation-efficient pathway for lifelong learning. We release our code and the $\texttt{reme.library}$ dataset to facilitate further research.