Evolving Memory, Learning to Meta-Control: Test-Time Adaptation for Experience-augmented LLM Agents

AI summary

Overview

Research area: Machine learning — specifically test-time adaptation and memory/experience management for self-evolving LLM agents, combining retrieval-augmented prompting with contextual-bandit meta-control.

Technical level: Advanced. The paper assumes familiarity with LLM agent pipelines, retrieval over dense embeddings, and linear contextual bandits (Thompson Sampling, LinUCB).

Scope: The paper proposes OMCE (Online Meta-Control of Experience), a framework that learns when to use accumulated experience, which form of it to inject, and when to consolidate new guidance, rather than following a fixed retrieve-inject-reflect pipeline.

What This Paper Is About

Self-evolving LLM agents accumulate experience from streams of tasks, but most existing frameworks apply that experience through a fixed routine: always retrieve, always inject, always reflect — regardless of whether the retrieved memory actually helps the current query. The paper argues that scheduling experience is itself a learnable dimension of self-evolution, and formalizes it as three coupled decisions: whether to invoke experience at all, which memory format to use (concrete cases versus distilled guidance), and when to distill new guidance from accumulated episodes. OMCE addresses this with a lightweight contextual-bandit controller that routes each query among four "cognitive actions" and updates both the memory bank and the routing policy from task feedback during test-time adaptation.

Key Contributions

1. A reframing of experience use as meta-control. The authors convert the always-on memory pipeline into a control problem over the full lifecycle of experience, defined by three requirements: adaptive invocation, selective form, and learnable consolidation.

2. The OMCE framework. A lightweight meta-controller implemented as a contextual bandit selects among four actions — DIRECT (no experience), EPISODE (inject retrieved episodic cases), GUIDE (inject retrieved guidance), and DISTILL (distill a new guidance entry from retrieved episodes, then inject it) — using a nine-dimensional meta-state vector summarizing retrieval signals, with no additional LLM call for action selection.

3. Coupled memory evolution and meta-evolution. After each task, episodic records (trajectory, reward in [0,1], case-specific insights) are written, injected guidance entries are refined using the realized trajectory and reward, and the bandit's action-wise sufficient statistics are updated with a forgetting factor so older evidence is downweighted as the memory distribution shifts.

4. An empirical study across four benchmarks and four backbones (MultiChallenge, BBEH, BigCodeBench, SWE-bench Verified; GPT-5.2, DeepSeek-V3.2, DeepSeek-V4-pro, Kimi-K2.5), including routing ablations, shared-memory controls, a frozen-controller control, and a token-cost analysis.

Main Findings

• Learned routing raises first-attempt success. OMCE attains the highest first-attempt success (OSR) on eleven of sixteen benchmark–backbone configurations, and improves first-attempt success by up to 20.5 percentage points over the strongest compared baseline.

• Clear wins on multi-turn conversation. On MultiChallenge, OMCE leads ReasoningBank, ACE, and MemP on all four backbones, by 13.5 to 20.5 points over the best of the three; its weakest backbone result there (67.4%) still exceeds the strongest baseline on any backbone by 10.7 points.

• Consistent but smaller wins on software engineering. On SWE-bench Verified, OMCE leads the same three baselines on all four backbones by 1.8 to 10.0 points.

• Mixed results elsewhere. On BBEH, OMCE matches or leads the strongest baseline on three backbones (53.3% with GPT-5.2 versus 52.4% for ACE) and trails by 0.1 point on Kimi-K2.5. On BigCodeBench, OMCE leads in OSR with GPT-5.2 but trails MemP on the other three backbones.

• Bypassing memory is sometimes the right call. Because DIRECT is available, the better OMCE variant on BigCodeBench matches or exceeds the Base LLM's OSR on three backbones and trails it by 0.2 point on the fourth, whereas ACE falls 9.9 to 11.1 points below the Base LLM on three.

• No fixed experience strategy is consistently best. On GPT-5.2, Fixed-EPISODE exceeds Fixed-GUIDE&DISTILL by 7.2 OSR points on BBEH (50.3 vs. 43.1), and the ordering reverses on MultiChallenge, where Fixed-GUIDE&DISTILL leads by 7.7 points (63.7 vs. 56.0). The BBEH ordering of the two fixed policies even flips between GPT-5.2 and DeepSeek-V3.2.

• Learning beats hand-crafted rules. The learned controllers exceed the tuned heuristic OMCE-HEU by 0.8 to 7.7 points on all four benchmarks with GPT-5.2 (widest margin on BBEH, 53.3 vs. 45.6). Across the full grid, the better learned variant outperforms OMCE-HEU on 14 of 16 benchmark–backbone pairs; TS and UCB each outperform it on 11 of 16.

• Gains come from routing, not just memory content. Holding memory fixed at OMCE's snapshot, OMCE exceeds Always-Both by 3.6 to 16.0 OSR points and Random by 4.8 to 11.8, with higher CSR on all four benchmarks. Indiscriminate retrieval can be worse than no memory: Always-Both reaches 48.8 OSR on SWE-bench Verified (below the Base LLM's 52.8) and 47.5 on BigCodeBench (against 48.1). With memory kept evolving, Always-Both-and-Update trails OMCE on all four benchmarks by 2.2 to 9.8 OSR points.

• The controller must keep adapting. Freezing controller parameters at mid-stream while memory keeps evolving costs first-attempt success on all four benchmarks — by 5.5 points on MultiChallenge, 3.5 on BBEH, 1.8 on BigCodeBench, and 0.6 on SWE-bench Verified. CSR falls on three benchmarks by 0.2 to 3.1 points, but rises on BigCodeBench from 59.2 to 65.4.

• Favorable token economics. With GPT-5.2, OMCE reduces per-task token use by a factor of 5.7–62.5 relative to ACE. MemP remains the most token-efficient method because it performs no test-time scaling; OMCE sits between MemP and ACE on all four benchmarks and backbones.

Methodology in Plain English

The authors build an agent with two memory stores and a small decision-maker on top.

Two memory stores. Episodic memory keeps concrete past cases: the full trajectory, the reward received, and a handful of case-specific insights extracted by an LLM. Guidance memory keeps distilled principles that summarize recurring patterns across cases and are meant to transfer more broadly. Both use the same two-part format — a short description of the scenario where the entry applies (used as the retrieval cue) and the detailed principle (the content injected into the prompt).

A small decision-maker, not another LLM call. For each incoming query, the system retrieves the top 3 entries from each memory source by cosine similarity of dense embeddings. Rather than passing raw memory text to a controller, it compresses the retrieval results into a nine-dimensional vector: eight normalized features plus a bias term. The features cover Relevance (mean retrieval scores from each source), Quality (quality scores and relevance-weighted scores, where episodic quality maps success to 1.0, partial/unknown to 0.5, failure to 0.0), and Diversity (the balance of successful versus failure-side episodic evidence, and the normalized entropy of outcome types). The two diversity features are masked so only the DISTILL action can see them.

Four actions. The controller picks DIRECT (no injected experience), EPISODE (inject retrieved cases), GUIDE (inject retrieved guidance), or DISTILL (first generate and store a new guidance entry from retrieved episodes, then inject it). Only the chosen experience is added to the task prompt.

Learning from feedback. Each action has its own linear reward model; the controller picks the action with the highest estimated reward plus an exploration term, using either Thompson Sampling (OMCE-TS) or LinUCB (OMCE-UCB). Because only the chosen action is observed, the bandit updates only that action's statistics, while a forgetting factor discounts all action models so stale evidence fades as the memory distribution changes. Action selection and updates involve only small matrices and vectors.

Memory upkeep. After execution, an LLM extracts case-specific insights and the full outcome is written to episodic memory regardless of success or failure. If GUIDE or DISTILL was used, the injected guidance entries are refined with the realized trajectory and reward and written back. An optional extension allows up to P attempts per task, injecting experience from failed attempts into the next try.

Evaluation. All methods run in an online adaptation setting where tasks arrive sequentially and the agent may update memory or policy after each instance. The primary metric is one-shot success rate (first-attempt success), with cumulative success rate reported within a shared budget of P = 4 attempts. Baselines span four regimes: no memory (Base LLM), within-instance retry (Reflexion), cross-task adaptation (MemP), and cross-task adaptation with multi-attempt inference (ACE, ReasoningBank).

Why This Matters

Research impact. The paper shifts attention from how much experience an agent accumulates to how well it schedules that experience. This is a distinct axis from retrieval-quality work such as MemRL or benchmark work such as Evo-Memory, and it connects memory-augmented agents to the meta-control literature on tool use, deliberation budgets, and test-time compute. The finding that indiscriminate retrieval can underperform no retrieval at all is a useful corrective for the field's default "always retrieve" assumption.

Real-world applications:

• Coding assistants and repository-level engineering agents, where retrieved past patches help on some tasks but can mislead on others (SWE-bench Verified, BigCodeBench).
• Multi-turn conversational agents that must notice when a user revises an earlier constraint — the paper's case study shows a fixed pipeline insisting on "June" while the learned controller routes to contradiction-reconciling guidance and picks up "July."
• Cost-sensitive deployments at scale, where a bandit over a nine-dimensional vector avoids an extra LLM call per routing decision and cuts per-task tokens by 5.7–62.5 relative to ACE.
• Long-running assistants with persistent memory, where the choice of storing raw cases versus distilled principles directly affects how fast the memory store grows.

Industry relevance. Token cost per task is a first-order constraint for production agents, and the paper's efficiency analysis positions OMCE near retrieval baselines rather than near the expensive full-playbook prompting of ACE. The controller is lightweight enough (linear models over eight normalized features) to run alongside an existing retrieval stack.

Future Directions

• Handling sparse or delayed feedback. The authors note that contextual-bandit updates become noisy when rewards are sparse or delayed, and that abrupt distribution shifts can make both memory and meta-policy stale. Robust credit assignment under these conditions is unresolved.

• Counterfactual signals for unchosen actions. Because the bandit observes only the selected action, the controller lacks information about the roads not taken, which can slow recovery from early suboptimal choices. The paper suggests using the LLM's reflective capabilities to simulate counterfactual rewards for unchosen actions.

• Memory maintenance as memory grows. Memory banks expand indefinitely, so retrieval may suffer from redundancy or semantic collisions. Explicit aging, eviction, auditing, and pruning are proposed as complements to the current meta-control framework.

• Generalizing the controlled analyses. The memory-use and controller-freezing comparisons are run only on GPT-5.2, while the routing comparison spans all four backbones; the authors call for testing the same interventions on the remaining backbones.

Target Audience

Researchers and advanced practitioners working on LLM agents, agent memory systems, and test-time adaptation will get the most from this paper, particularly those already familiar with retrieval-augmented prompting pipelines and contextual bandits. Engineers building production agents with persistent memory and hard token budgets will find the efficiency analysis and the DIRECT escape hatch practically relevant. Readers new to bandit methods will need to consult the appendices for the Thompson Sampling and LinUCB variants, the full meta-state definitions, and the finite-time analysis.

Authors’ abstract

Self-evolving agents accumulate experience from task streams, but existing frameworks typically rely on fixed pipelines to retrieve and utilize this experience, neglecting when and how to deploy it. We introduce Online Meta-Control of Experience (OMCE), a framework that updates experience memory and learns a meta-control policy from task feedback during test-time adaptation. OMCE employs a lightweight contextual-bandit meta-controller that uses compact retrieval signals to select among four cognitive actions: DIRECT, EPISODE, GUIDE, and DISTILL. This routing mechanism operationalizes three coupled decisions: whether to invoke experience, which memory format to utilize, and when to consolidate new guidance. OMCE evolves online, leveraging task feedback to update the memory bank and refine the routing policy via contextual bandit updates. Experiments on four online adaptation benchmarks across four backbones show that OMCE improves first-attempt success by up to 20.5 percentage points over the strongest compared baseline, while reducing per-task token use by a factor of 5.7 to 62.5 relative to ACE with GPT-5.2. Controlled ablations further support the benefits of learned routing and continued routing policy adaptation.