RRSI: Regularized Recursive Self-Improvement of Agent Harnesses

AI summary

Overview

• Research area: LLM agent systems and automated agent-harness engineering (machine learning / cs.LG), with connections to recursive self-improvement and generalization.
• Technical level: Intermediate to Advanced. The paper assumes familiarity with LLM agents, agent "harnesses" (prompts, control flow, tooling, memory, context management), evaluation benchmarks, and regularization intuitions from classical machine learning.
• Scope: The paper diagnoses why iterative, feedback-driven editing of an agent harness overfits a finite evolve set, and proposes a set of regularization constraints on the proposal and selection steps that make the evolved harness transfer to unseen benchmarks at lower token cost.

What This Paper Is About

An LLM agent's capability depends heavily on its harness — the prompts, control flow, tools, memory and context management wrapped around a frozen backbone model — and recent methods automate harness improvement by repeatedly proposing and keeping edits that raise a benchmark score. That loop is a practical form of recursive self-improvement at the agent-system level, but because it reuses a finite evolve set adaptively across rounds, improvements can reflect benchmark-specific fitting, evaluation noise, or accumulated complexity rather than reusable mechanisms. The paper's goal is to keep the harness edit space fully open while regularizing how evolve-set feedback is converted into permanent harness changes, so that gains survive on out-of-distribution benchmarks.

Key Contributions

1. Identifies overfitting as a central failure mode of harness-based recursive self-improvement. The authors characterize three coupled behaviors that widen the evolve-to-transfer gap: benchmark-specific fitting, noise chasing, and complexity accumulation.
2. Proposes RRSI, a framework that regularizes both sides of the evolution loop — constraining how proposal capacity is spent and which empirically measured improvements are allowed to become permanent harness state — without restricting which harness components may be edited.
3. Maps the regularizers onto classical analogues: an L0-style cardinality constraint on edits per update, Lasso/L1-style structural pruning of unproductive components, and Ridge/L2-style shrinkage on aggregate resource footprint (measured as policy-token cost).
4. Demonstrates transfer and efficiency across eight benchmarks in three domains, showing gains on the evolve split, on in-distribution held-out data, and on five out-of-distribution benchmarks, with fewer policy tokens than unregularized evolution.

Main Findings

• Gains on the split that is evolved against are modest, but held-out gains are real. Evolve-set gains were 6.0 points on Terminal-Bench 2.1, 4.9 on EngDesign, and 1.1 on Harvey LAB. Held-out improvements include 1.8 points on SWE-bench Verified, 2.3 on the in-distribution held-out split of Harvey LAB, and 4.3 Medal points on Frontier-Eng (a 24.3% relative improvement).
• Out-of-distribution gains range from 3.5 to 4.7 points across the three agentic workspace out-of-distribution benchmarks (7.2% to 13.1% relative). No held-out split regressed.
• Baselines overfit and invert on transfer. In Table 1, prior methods worked well on the Harvey LAB evolve split (e.g., Meta-Harness at 93.0 versus RRSI at 90.5), but out of distribution Meta-Harness added only 0.9 points to the out-of-distribution average, HarnessX landed on the base, and AHE and TTHE finished below the unevolved harness H0, TTHE by 1.7 points. RRSI's out-of-distribution average was 43.6 against 39.7 for H0.
• Ablation shows the regularizers trade evolve-set score for transfer. Removing both groups raised the evolve-set score to 92.8 (the highest of any arm) while leaving the out-of-distribution average at 40.3, within a point of the unevolved harness, at 3.80 million tokens per trial versus 2.42 for RRSI. Removing acceptance constraints alone raised evolve score from 90.5 to 91.5 while out-of-distribution average fell from 43.6 to 41.0 and token cost rose by half.
• Results are not tied to one policy family. Under Gemini 3.5 Flash, Terminal-Bench 2.1 rose from 64.6 to 78.7 and SWE-bench Verified from 76.8 to 79.0. Under Claude Opus 4.8, Terminal-Bench 2.1 rose from 74.2 to 80.2 and SWE-bench Verified from 82.0 to 83.8.
• The evolved harness helps a backbone that never participated in the search. A harness evolved with Gemini 3.5 Flash was run unchanged with Gemini 3.1 Flash Lite, raising Terminal-Bench 2.1 from 11.2 to 14.6 — a 30.4% relative gain on a base score less than a fifth of the search policy's.
• RRSI produces the lightest evolved harness. All four baselines spend more policy tokens per trial for a lower out-of-distribution average; AHE is the extreme case at 3.82 million tokens per trial, 58% more than RRSI, for 4.4 points less out of distribution. RRSI runs 26.3 steps per trial against 27.3 to 34.6 for prior methods. No evolved harness was as cheap as H0 at 1.56 million tokens and 21.2 steps.
• Deterministic grading rules out judge-pleasing as the explanation. EngDesign and Frontier-Eng tasks are graded by frozen simulators or testbenches rather than judge models, and the gains survive there unchanged.

Methodology in Plain English

The authors start from the standard harness-evolution loop: run the current harness on a fixed evolve set, summarize the trajectories into feedback, have an LLM propose candidate harnesses, score the candidates on the same evolve set, and promote the best candidate to become the next incumbent. They leave this loop and the edit space intact — prompts, control flow, configuration, context management, tools, skills, memory, and subagents can all be modified, added, or removed — and instead regularize how the search moves through that space.

On the proposal side, three constraints apply. First, an annealed edit budget limits how many independently attributable edits a single candidate may bundle, following a cosine schedule that starts large (allowing coordinated early exploration) and shrinks toward a small minimum in later rounds. Second, the system records the full history of every evaluated candidate — the component changed, the hypothesis tested, the source diff, the score and cost changes, and whether it was accepted — so rejected mechanisms remain negative evidence and successful ones retain explicit credit. Third, when progress over the previous w rounds stays inside the empirical noise band, a small portion of the proposal budget is reserved for harness components not yet exercised, to prevent the search from collapsing onto one edit family.

On the selection side, a candidate must clear several non-compensatory criteria. A critic reads each candidate diff before evaluation and rejects edits that encode task names, entity names, task-specific values, answers, or inert machinery — screening before scoring matters because a leaking candidate never receives the inflated score that would make it attractive later. A noise-adjusted floor requires a candidate's score to be at least the best score seen so far minus an empirical noise band δ, estimated by repeatedly evaluating the unchanged base harness beforehand. Candidates whose gain exceeds δ must also satisfy a cost-growth condition: the relative increase in policy-token cost must be no greater than β0 + β1 times the score gain, with β0 and β1 selected on the evolve set and held fixed for all transfer evaluations. Finally, a pruning rule tracks whether recently exercised components produced strictly positive measured gains over a fixed window, reporting persistently unproductive components to the proposer as deletion targets.

Evaluation spans three domains. Coding evolves on Terminal-Bench 2.1 (89 containerized terminal tasks verified by their own unit tests) and transfers to SWE-bench Verified. Agentic workspace evolves on Harvey LAB (160 tasks split into a 120-task evolve set and a 40-task held-out set, across 25 practice areas) and transfers to JobBench, GDPval (185 tasks) and APEX-Agents (480 tasks). Engineering design evolves on 61 EngDesign tasks graded by their own frozen simulators and transfers to Frontier-Eng (47 tasks in the v1 set, 26 domains, with 38 tasks contributing credit). The policy is frozen throughout at Claude Opus 4.8, which also serves as proposer, analyst, and leakage critic; base harnesses are Terminus-2 for coding, a ReAct loop over an MCP tool gateway and a dynamic toolbelt, and ReSum-style context management for Harvey LAB and EngDesign. Four baselines are compared: Meta-Harness, AHE, TTHE, and HarnessX, all starting from the same base harness with the same frozen policy, evolve set, and candidate budget.

Why This Matters

The paper reframes a widely used agent-improvement recipe as an adaptive overfitting problem and shows that the fix is not to restrict what can be edited but to restrict how evidence is turned into permanent change. For research, it supplies a concrete diagnostic — evolve-set gains that do not survive transfer, including baselines that end below their starting harness — and a set of regularization mechanisms that can be bolted onto existing harness-evolution method families rather than replacing them. The mapping to L0, L1, and L2 intuitions gives the area a shared vocabulary that previous empirical work lacked.

Real-world applications:

• Automated coding agents: a harness evolved on one terminal-task suite retaining a 1.8-point gain on repository-level bug fixing it was never scored on (and 2.2 points under a different policy family) directly reduces the cost of re-engineering scaffolds for each new coding benchmark.
• Professional knowledge work: legal and office-task agents, where RRSI produced gains of 3.5 to 4.7 points across JobBench, GDPval, and APEX-Agents, and where a harness must move between verifiers that are judge-based rubrics rather than unit tests.
• Engineering design and optimization: deterministic simulator-graded tasks (EngDesign, Frontier-Eng), where the same regularization yields a 4.3 Medal-point improvement without exploiting judge behavior.
• Cost-constrained deployment: because the regularizers include a cost-growth constraint and a pruning rule, the resulting harness uses roughly 30% fewer policy tokens than unregularized evolution while scoring higher out of distribution.

Industry relevance is direct: harness engineering is currently manual labor tied to a specific backbone model, so its cost is paid again with every model release. A method that automates harness evolution while producing mechanisms that survive a change of benchmark — and, per the cross-model experiment, a change of backbone — reduces that recurring cost and makes agent infrastructure more portable across model families and capability tiers.

Future Directions

• Weight-level self-improvement. The paper explicitly limits its scope to harness-level RSI with frozen backbone models; whether the same regularizers transfer to settings where model weights are updated during evolution is left open.
• Dependence on the evolve set and hyperparameters. RRSI still relies on a finite evolve set and several regularization hyperparameters (the budget schedule, noise band, β0 and β1, pruning window), so effectiveness may depend on feedback-signal quality and search budget — a natural target for adaptive or learned settings of these values.
• Broader architectural validation. The authors note that wider validation is needed across substantially different agent architectures, tool ecosystems, and longer-running self-improvement processes than the three domains studied here.
• Relationship to concurrent generalization work. The paper positions its dynamics-level regularization as orthogonal to methods that change what is edited, such as explicitly optimizing for generalization or replacing greedy selection with a diversity-preserving archive — combining the two directions is an obvious unexplored axis.

Target Audience

Researchers and engineers working on LLM agent systems, automated prompt or scaffold optimization, and self-improving agents will get the most from this paper, since the core argument concerns search dynamics over an editable harness rather than any single component. Practitioners building production agents on frozen backbones also benefit from the cost and cross-model transfer results. Readers without background in agent harnesses, benchmark evaluation protocols, or the regularization analogy (L0/L1/L2) will find the method understandable but the motivation and the significance of the baseline inversions harder to appreciate.

Authors’ abstract

An LLM agent's capability is largely magnified by its harness, namely the prompts, control flow, tooling, memory, and context management surrounding the frozen backbone model. Recent methods increasingly automate this process by iteratively proposing and selecting component-wise edits of an agent harness, practically establishing a form of recursive self-improvement (RSI) at the agent-system level. However, such recursive evolution may overfit by memorizing the training tasks, showing large in-distribution gains that shrink or even vanish on out-of-distribution benchmarks. We introduce Regularized Recursive Self-Improvement of Agent Harnesses (RRSI), which incorporates the principles of regularizations into harness self-improvement by constraining the evolution candidate proposal and selection. The proposer operates with a temporally annealed budget, limiting how many edits a candidate can bundle, and it encourages unexplored trajectories based on evolution history. The selector is equipped with a critic and a pruner: the critic screens benchmark-specific proposals, while the pruner, removes changes that are too small, too expensive, or no longer useful. Together these constraints favor reusable agent mechanisms over benchmark-specific ones or even noises. Across eight benchmarks spanning coding, agentic workspace and engineering design tasks, RRSI gains up to 14.1 points on the split it evolves against and up to 4.7 points on the five out-of-distribution benchmarks, while producing a harness that runs on 30% fewer policy tokens than the unregularized evolution. Code is available at https://github.com/google-research/rrsi and project page is https://regularized-rsi.com/.