Recurrent Harness for Agentic Continual Learning

AI summary

Overview

• Research area: Machine learning / language model agents — specifically continual learning for LLM agents operating over long interaction streams, and the "agent harness" (memory, reflection, and prompt scaffolding) that surrounds a fixed backbone model.
• Technical level: Intermediate. The core idea is conceptual and narrative rather than mathematical, but it assumes familiarity with retrieval, in-context learning, memory/reflection agents, and benchmark evaluation protocols.
• Scope: The paper proposes ReHarness, a recurrent harness that adds a backward revision step to the usual forward read/write loop, and evaluates it on three continual learning benchmarks across three LLM backbones.

What This Paper Is About

Language model agents increasingly run over long streams of tasks where the data distribution shifts over time, so they must adapt online and remember useful experience. Existing agent harnesses evolve only forward: they write memories or reflections after each instance and retrieve them later, but never revisit whether an earlier reflection actually helped. The paper's goal is to close that loop by adding a backward revision step, so that downstream outcomes are used to correct previously written reflections — all without replaying past instances, using validation sets, or updating model parameters.

Key Contributions

1. ReHarness, a recurrent harness framework. It combines forward read and forward write (as in prior stateful harnesses) with a backward revision step that revises, after each instance, the specific reflections that were retrieved for that instance.

2. Strictly online backward feedback in text. The revision agent π_R revises a reflection R_i into R′_i using the reflection's source trajectory τ_i and the current instance's trajectory τ_t as feedback, with the constraint that any new content must remain grounded in τ_i to prevent information leakage from τ_t.

3. Empirical validation across three benchmarks and three backbones. ReHarness is compared against Mem0, RBank, DoT, ACE, GEPA-Online, Top-K Raw, and Full Raw on CL-Bench, MemoryBench, and MemoryArena using Claude Opus 4.8, GPT-5.5, and Gemini 3.7 Flash.

4. Analysis of grounding, revision capability, and continual-learning dynamics. The paper measures source-grounding/leakage/unsupported claim rates, runs a counterfactual replay to test whether revisions are meaningful, and decomposes gains into stability and plasticity.

Main Findings

• Best average normalized gain. ReHarness achieves an overall mean normalized gain of 0.240, the highest in Table 1, and ranks first or tied in eight of the nine benchmark–backbone combinations. For comparison, Full Raw reaches 0.164, Top-K Raw 0.168, Mem0 0.131, GEPA-Online 0.092, DoT 0.071, ACE 0.043, and RBank 0.034 on average.

• Statistically significant against all baselines. Using two-sided paired Wilcoxon signed-rank tests with Bonferroni correction, ReHarness significantly outperforms all baselines (p_adj < 0.05), including Top-K Raw (p_adj = 0.039) and Full Raw (p_adj = 0.043).

• Raw-context baselines are strong, but inconsistent. Top-K Raw and Full Raw outperform most structured memory and reflection methods on average, especially on CL-Bench and MemoryArena. RBank, DoT, ACE, and GEPA-Online sometimes yield negative gains, and GEPA-Online degrades substantially on MemoryBench with Gemini.

• Revisions stay source-grounded. Across all backbones, over 95% of factual claims in revised reflections remain grounded in their source trajectory, with leakage rates of 0.8%–2.1% and unsupported rates of 0.5%–2.6%. Grounding decreases by at most 0.3% per revision (Claude 0.953 grounded, GPT 0.987, Gemini 0.963).

• Counterfactual replay confirms revisions are meaningful. On CL-Bench with Claude, applying the revised harness to the same instance (counterfactual replay, which violates the streaming protocol) yields an average of 0.335, versus 0.217 for ReHarness and 0.144 for a no-revision baseline. Gains over no-revision appear across all tasks, especially database exploration (0.361 vs 0.279), sales prediction (0.408 vs 0.258), and blind spectrum monitoring (0.162 vs 0.033).

• Better stability and plasticity. On CL-Bench, ReHarness achieves the highest average stability gain (0.015) and the highest plasticity gain (0.241). Mem0, RBank, DoT, ACE, and GEPA-Online all show negative average stability gains, meaning their forward-only states can become detrimental after distribution shifts.

• Token and cost efficiency. Over one rollout across all CL-Bench tasks with Claude, ReHarness uses roughly 37.19M input and 4.38M output tokens (~41.57M total). Top-K Raw uses ~52.20M input and ~0.46M output (~52.66M total), and Full Raw uses ~141.68M input and ~0.44M output (~142.12M total) — 1.3× and 3.4× ReHarness's total tokens respectively. With prompt caching and Claude API pricing, estimated costs are about $85 for Top-K Raw, $254 for ReHarness, and $314 for Full Raw, while normalized gain on that setting is 0.165, 0.217, and 0.191 respectively.

• Small revision sets suffice. Normalized gain rises sharply from k = 1 to k = 5, stays relatively stable for k ∈ {5, 10, 20}, and declines at larger k. The paper fixes k = 5 in main experiments. Removing revised reflections as demonstrations for the memory agent drops CL-Bench gain with Claude from 0.217 to 0.191.

• Task-level variation. ReHarness leads or stays close on most CL-Bench tasks but shows only marginal gains on exploitable poker. On MemoryBench, most methods do well on HelloBench tasks, but only ReHarness achieves consistent gains across the WritingBench tasks. On MemoryArena, Mem0 is notably strong on travel planning and GEPA-Online performs well on progressive web search.

• Known weaknesses. The paper reports that ReHarness underperforms on exploitable poker, where hand-level rewards are weakly coupled to strategy quality because of hidden cards and chance. It also still stores raw trajectories for future revision, incurring storage overhead.

Methodology in Plain English

The setup is a stream of instances. For each instance, a decision agent receives a query and answers it using a few retrieved reflections from the harness state (retrieval uses the top five entries ranked by Qwen3-Embedding-8B). This is the forward read. After the environment gives a reward, the system does two things:

1. Backward revision. A revision agent looks at each reflection that was retrieved for this instance, along with the original trajectory that reflection came from and the outcome just observed. It rewrites the reflection so that, in hindsight, it would have been more useful. Crucially, the new reflection must stay grounded in the original trajectory — the current trajectory is used only as feedback, not as a source of new facts. This prevents information leakage.

2. Forward write. A memory agent reflects on the just-completed trajectory to write a new reflection, using the revised reflections as demonstrations of what a good revision looks like.

The resulting state carries forward to the next instance. Nothing is trained, no parameters change, no past instance is replayed, and no validation set is used — everything happens in text under a strictly online streaming protocol.

To evaluate, tasks from each benchmark are standardized into streams, each interaction unit is an instance, and the team reports normalized gain: (r_method − r_stateless) / (r_max − r_stateless), averaged over each stream, with three independently sampled stream rollouts sharing the same initialization and seed per rollout.

Why This Matters

Impact on research. The paper argues that forward-only evolution of agent harness state is structurally incomplete: an overgeneralized reflection written early can persist and keep misleading the agent. Putting backward revision into the textual harness state gives an adaptation signal that works without gradients, without replay, and without validation data — a combination that frontier API-based models otherwise make hard. It also reframes continual learning for agents as an editing problem over accumulated text rather than a parameter-update problem.

Real-world applications (drawn from the benchmarks the paper evaluates on):

• Software engineering: codebase adaptation, where an agent's rules about a repository must be revised as the code evolves.
• Data analysis and databases: exploratory querying of unfamiliar SQLite databases, including handling schema shifts such as renamed or newly added columns.
• Wireless/spectrum monitoring: inferring persistent transmitters from sequential scans, where hallucinating transmitters is as costly as missing them.
• Long-horizon planning and assistance: travel planning, web shopping, and progressive web search, plus long-form writing tasks where accumulated style or content guidance can go stale.

Industry relevance. The paper reports that ReHarness uses fewer total tokens than the strongest full-history raw-context baseline despite generating more output tokens, and is substantially cheaper in dollar terms ($254 vs $314 on one CL-Bench rollout) while achieving higher gain. For teams deploying long-running agents on paid APIs, that combination of better adaptation and lower context cost is directly operational. The revision mechanism also requires no fine-tuning, so it can be layered onto existing API-based agent products.

Future Directions

1. Noisy-reward credit assignment. ReHarness struggles on stochastic settings like exploitable poker, where downstream rewards are weakly coupled to strategy quality. The paper explicitly calls for addressing this in future work — for example by making revision more robust to noisy feedback signals.

2. Reducing storage overhead. ReHarness must retain raw source trajectories so reflections can later be revised against them. The paper flags this as an unresolved cost and an intended future direction.

3. Choosing how much to revise. The k-sweep shows gains taper and then decline at large k, with the authors attributing this to unnecessary revision and reflection drift. Automatically selecting which and how many reflections to revise — rather than a fixed k = 5 — is an open question.

4. Extending recurrent-harness adaptation more broadly. Since the framework is agnostic to the specific memory or reflection scheme, testing backward revision on top of other harness designs, or combining it with other forms of textual feedback, remains unexplored here.

Target Audience

Researchers and engineers working on LLM agents, agent memory, and test-time or continual adaptation, particularly those building long-running agents on top of frozen API-based models. It is also useful for practitioners who care about the cost of long context, since it reports concrete token and dollar comparisons. Readers looking for a neural-network or gradient-based continual learning method will not find one here — the entire contribution lives in text-space harness management.

Authors’ abstract

Language model agents increasingly operate over long interaction streams, requiring online adaptation to distribution shifts through continual learning. However, existing agent harnesses typically either evolve their states only through forward information flow or rely on additional replay or validation data for harness optimization. To address this, we propose ReHarness, a recurrent harness framework for agentic continual learning that complements forward read and write with backward revision under a strictly online streaming protocol. We ask what information could have been captured differently in the harness state to better inform the current instance in the stream. Specifically, after each instance, we leverage the resulting trajectory as feedback to revise the reflections retrieved for that instance, thereby propagating feedback backward through the textual harness state. Across three continual learning benchmarks and three language model backbones, ReHarness consistently outperforms strong raw-context baselines, memory- and reflection-based methods, and prompt optimization approaches. Further analysis shows that ReHarness produces source-grounded and meaningful revisions, corrects overgeneralized or outdated experience under distribution shifts, and improves both stability and plasticity during continual learning.