More Memory, Worse Agents: Error Reproduction and Anti-Persistence in LLM Agents

AI summary

Overview

Research area: Artificial intelligence / LLM agents — specifically memory-based self-improvement, where an agent improves across tasks without changing model weights by storing and reusing its own experience.

Technical level: Intermediate. The paper combines an engineering design (a memory module) with a small formal appendix, and it assumes familiarity with agent loops, retrieval, and benchmark evaluation.

Scope: A single paper-length study arguing and testing that inferred abstractions (rules, summaries, skills) should be generated on demand and discarded rather than stored, across three benchmarks and three backbones.

What This Paper Is About

LLM agents can improve over time by writing their own past executions into an external memory and reusing them later. A key design choice is whether memory should hold only raw trajectories and outcomes, or also the higher-level rules, summaries, and skills inferred from them. The authors argue that storing inferred abstractions creates a feedback loop they call error reproduction: a faulty rule shapes an execution, that execution is stored, and later abstraction from the stored execution regenerates the same faulty rule — so the error survives even after the original rule is deleted. The paper proposes a memory design, Anti-Persistence, that keeps trajectories but makes abstraction temporary, and tests whether this yields better held-out success at lower cost.

Key Contributions

1. A memory design that separates experience accumulation from abstraction persistence. The agent retains raw trajectories, task outcomes, and an outcome log of mode choices, but keeps the persistent abstraction store empty: synthesized guidance is used for the current task only and then discarded.

2. An adaptive mode selector for on-demand abstraction. A lightweight contextual-bandit controller chooses per task among four modes — none (use retrieved records directly, no synthesis call), imitation (extract reusable action patterns and their conditions from successful trajectories), strategic (organize higher-level subgoals and plans), and corrective (derive avoidance guidance from failed trajectories, optionally contrasted with a successful one). Selection uses similarity- and recency-weighted past rewards plus a UCB-style exploration bonus.

3. A formal account of error reproduction in abstraction memory. The appendix defines operationally biased ("flagged") abstractions, historical provenance of execution records, a conditional reproduction coefficient α = Pr(flagged output | flagged-parent exposure), and a conditional admission probability η, then shows in Proposition 1 that positive q (exposure), α, and η imply a positive probability of generating and admitting a flagged abstraction. Remark 1 shows that deleting a parent abstraction does not remove its stored descendants or the trajectories it shaped.

4. Empirical evaluation across three benchmarks and three backbones. Held-out success, LLM-call cost, per-iteration dynamics, and ablations on recency decay, mode selection, and cost weight, plus a controlled persist-versus-discard comparison.

Main Findings

• Anti-persistence led in seven of nine benchmark–backbone settings at K = 10 iterations (Table 1). Averaged equally across settings it reached 59.9%, versus 53.5% for trajectory memory, 49.0% for abstraction memory, and 49.3% for joint memory. It beat trajectory and abstraction memory in all nine settings and joint memory in eight.

• It also beat the strongest agent baseline. Reflexion, the strongest baseline on average, reached 52.9%.

• Stored abstractions did not help consistently. Abstraction memory and joint memory both underperformed plain trajectory memory in seven of the nine settings.

• Lowest average LLM-call cost. Anti-persistence had the lowest mean per-task LLM-call cost among the four memory configurations while also having the highest mean success; the paper attributes this to none-mode reuse of trajectories without a synthesis call and to shorter executions in some settings. The actual call counts appear only in a figure, not as numbers in the text.

• Discarding guidance beat persisting it in all four tested settings (Table 2): Llama-4-Scout/ALFWorld 54.7 ± 2.9 vs 26.7 ± 6.1 (Δ +28.0); Llama-4-Scout/WebShop 46.3 ± 1.8 vs 34.0 ± 4.0 (Δ +12.3); Qwen3-Coder-Next/ALFWorld 96.0 ± 1.4 vs 92.7 ± 1.2 (Δ +3.3); Qwen3-Coder-Next/WebShop 54.0 ± 2.0 vs 47.3 ± 3.1 (Δ +6.7).

• Anti-persistence reached the highest final mean success in all six settings tracked across iterations, though it did not lead at every iteration; joint memory on Llama declined in later rounds, particularly on ScienceWorld.

• It beat AutoRefine in all six settings. AutoRefine, a maintenance-based method, improved on abstraction memory in one of six settings, matched it in one, and was worse in four.

• Adaptive mode selection mattered. On Qwen3-Coder-Next, the bandit scored 96.0% on ALFWorld and 54.0% on WebShop, above fixed strategic, imitation, and corrective policies and random selection; the best fixed policy differed by benchmark (strategic on ALFWorld, imitation on WebShop).

• Recency decay had a sweet spot. On Qwen3-Coder-Next, the default γ = 0.998 gave 96.0% (ALFWorld) and 54.0% (WebShop); γ = 0.980 gave 90.0% and 48.7%; γ = 0.990 gave 87.3% and 44.7%; no decay (γ = 1) gave 88.7% and 48.7%. Success did not vary monotonically with γ.

• A modest synthesis-cost penalty helped. The default λ = 0.05 gave the best mean success on both ALFWorld (96.0%) and WebShop (54.0%). Removing the penalty (λ = 0) increased synthesis-mode usage but lowered success; stronger penalties (λ ∈ {0.20, 0.50}) reduced both.

• The controller used multiple modes. Across all nine backbone × benchmark combinations, none had a nonzero mean selection frequency; stronger backbones favored imitation on WebShop, while Llama-4-Scout favored strategic, and ALFWorld and ScienceWorld were more balanced.

• The evidence for error reproduction is associational. The paper reports associations between earlier and subsequently generated abstractions consistent with the mechanism, but explicitly does not identify causal effects of individual abstractions.

Methodology in Plain English

The agent's model weights never change. Instead, each task's execution — task description, full trajectory, and outcome — is written into a trajectory memory, and each mode choice with its outcome is written into a separate outcome log.

For a new task, the agent does two retrievals. First it pulls up to B̂ = 10 similar entries from the outcome log to decide how to use experience. Each entry gets a weight from task similarity and a recency discount, and each attempt's reward is its outcome minus a penalty (λ) for the extra synthesis calls it consumed, clipped to [−1, 1]. Mode scores are weighted averages of those rewards plus an exploration bonus favoring rarely tried modes; the highest-scoring mode wins.

Second, it retrieves up to B = 5 trajectories under the chosen mode (plus one extra successful trajectory as a contrast in corrective mode). In none mode those records go straight into the prompt. In the three synthesis modes, a fixed procedure turns them into temporary guidance, which is used for this task only and then thrown away. Only the resulting trajectory and outcome persist — so guidance can still influence later tasks indirectly, through the executions it shaped, but never as a stored object.

Experiments run K = 10 iterations per method. Memory updates come only from training episodes; after each iteration the full held-out set is evaluated with memory, abstraction store, outcome log, retrieval metadata, and pruning decisions frozen, and evaluation logs never feed back into memory. Results are over 3 seeds ({42, 43, 44}) with standard deviations as error bars. Comparisons include four memory configurations (trajectory, abstraction, joint, anti-persistence) and four agent baselines (ReAct, Reflexion, ExpeL, AutoRefine), with controlled variants that differ only in whether synthesized guidance is discarded or stored.

Why This Matters

Impact on research. The paper reframes a common assumption in agent-memory work — that distilling experience into rules and skills is straightforwardly beneficial — as a risk-bearing design choice. The distinction between error propagation (an existing error affects behavior) and error reproduction (an error gets re-encoded into a new abstraction via the trajectories it shaped) is a useful analytical lens, and the persist-versus-discard ablation is a clean way to isolate the effect of retention policy from the effect of synthesis quality.

Real-world applications:

• Long-running customer-support or coding agents that accumulate logs across thousands of sessions and currently cache "lessons learned" that may silently reinforce a bad workaround.
• Robotic and simulation-based task agents (household instruction following, web navigation, science-experiment simulation — the three benchmark domains here) where a stale heuristic can cause premature termination.
• Enterprise copilots with shared team memory, where one confidently wrong stored rule could spread to every agent that retrieves it.
• Continual-learning pipelines that avoid fine-tuning for cost or compliance reasons and instead rely on external memory, where the persistence policy is the primary safety lever.

Industry relevance. The cost result matters commercially: temporary guidance did not raise per-task LLM calls and was the cheapest of the four configurations, so the safer design was also the cheaper one in these tests. The mode selector also gives practitioners a tunable knob (λ) for trading synthesis spend against success, with a default (λ = 0.05) that was best in the tested settings.

Future Directions

• Determine when verification justifies selective retention. The conclusion explicitly raises studying when a synthesized abstraction can be validated well enough to be stored, rather than always discarded.

• Establish causality rather than association. The provenance analysis links earlier and later abstractions only associatively; controlled experiments that inject known-bad rules and trace their descendants would test error reproduction directly.

• Distinguish premature termination from efficiency. Shorter rollouts under joint memory coincided with lower success, which is "consistent with" premature termination but not shown to be its cause — a diagnostic study is open.

• Test stronger maintenance policies. The AutoRefine comparison changed retrieval and synthesis as well as maintenance, so the paper notes it does not isolate maintenance effects or rule out more effective pruning, scoring, and merging schemes; the theory also explicitly does not claim all maintenance policies fail.

• Check generality beyond the tested grid. Results are for three benchmarks, three backbones, K = 10 iterations, and 3 seeds; the authors note that the preferred design remains setting-dependent and that per-iteration trajectories varied by backbone and benchmark.

Target Audience

Researchers and engineers building LLM agents with external memory, especially those designing self-improvement loops without weight updates. It suits practitioners choosing what to persist in production memory stores, and theory-minded readers interested in feedback dynamics in iterative self-generated data. Readers wanting a fully specified reproduction of the persistence ablation should note that some details — including the benchmark partition sizes and the exact per-task call counts — appear only in tables and figures not included in this excerpt.

Authors’ abstract

LLM agents can improve across tasks without updating model parameters by storing experience from their own executions in external memory and reusing it to guide later decisions. A central design question is whether to retain only observed trajectories and outcomes or also the rules, summaries, and skills inferred from them. Retaining these abstractions allows agents to reuse strategies without repeatedly deriving them from raw trajectories. However, errors in stored abstractions can propagate to subsequent decisions through retrieval and reuse. They can also shape execution trajectories, which then become the basis for new abstractions that reproduce the original errors. We term this feedback mechanism error reproduction: errors are regenerated through a cycle of abstraction, execution, and re-abstraction, extending their persistence beyond direct reuse of the original faulty knowledge. We propose Anti-Persistence, a memory design that accumulates trajectories and outcomes while reconstructing task-specific guidance on demand. An adaptive selector uses prior outcomes to decide whether and how to synthesize guidance, which is discarded after execution. This prevents its direct reuse as a stored object, although indirect feedback through retained trajectories remains possible. Across different benchmarks and backbones, Anti-Persistence achieves higher overall held-out success and the lowest average LLM-call cost among the evaluated memory designs. These results support separating experience accumulation from abstraction persistence in memorybased self-improvement.