ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement

AI summary

Overview

• Research area: Natural Language Processing, specifically LLM-based CLI agents and Recursive Self-Improvement (RSI) of agent harnesses for long-horizon coding and terminal tasks.
• Technical level: Advanced. The paper assumes familiarity with agent harness architecture, trajectory-based reinforcement signals, benchmark protocols, and module-level software decomposition.
• Scope: The paper proposes ModularRSI, a benchmark-disjoint, contrastive, and modular framework for evolving agent harnesses, and evaluates whether the resulting improvements generalize to unseen tasks, domains, and foundation models.

What This Paper Is About

Agent harnesses govern how a CLI agent executes tasks: how it loops, calls tools, manages context, reads environment feedback, and decides when a task is done. Prior work lets harnesses self-improve from execution experience, but it is hard to tell whether the improvements are genuinely reusable or merely overfitted to the benchmark used for evolution. ModularRSI addresses this by contrasting successful and failed trajectories of the same task, aggregating evidence across tasks, and evolving five separate harness modules independently before integrating them, all using 2,000 evolution tasks that are disjoint from the downstream evaluation benchmarks.

Key Contributions

1. A contrastive, modular credit-assignment framework. ModularRSI combines same-task trajectory contrast, cross-task evidence aggregation, and module-restricted evolution to convert coarse task-level success/failure rewards into localized harness modification signals, reducing task-specific and cross-mechanism interference.
2. A benchmark-disjoint evolution protocol. The authors independently curate 2,000 executable evolution tasks from external sources (GitHub, Hugging Face, Kaggle, Linux kernel documentation), guided only by high-level domain labels derived from benchmark task categories, with instance-level similarity filtering and fine-grained domain analysis to minimize overlap with downstream benchmarks.
3. Extensive evaluation on TerminalBench 2.0 and SWE-Bench Verified. The evolved harness improves performance on unseen in-domain and out-of-domain tasks and transfers across different foundation models, with controlled studies showing that independently evolving and merging modules outperforms joint or non-modular evolution.
4. An open release. All code and datasets are available at https://github.com/IQuestLab/ModularRSI.

Main Findings

• In-domain gains over the un-evolved harness. With DeepSeek-V4-Flash-Preview as the backbone, evolution on TB-related data raises TerminalBench 2.0 accuracy from 47.57 to 52.43, and evolution on SWE-related data raises SWE-Bench-Verified accuracy from 73.40 to 76.45.
• Gains transfer across domains. The harness evolved on TB-related data reaches 75.80 accuracy on SWE-Bench-Verified (from 73.40), and the harness evolved on SWE-related data reaches 49.40 accuracy on TerminalBench 2.0 (from 47.57).
• Reliability improves, not just average accuracy. On TerminalBench 2.0, Pass^3 rises from 30.34 to 35.96 under TB-related evolution, indicating fewer stochastic failures across repeated trials.
• Improvements transfer across foundation models. Freezing the harness evolved with DeepSeek-V4-Flash-Preview and applying it to other models on TerminalBench 2.0 raises GLM-5.2 accuracy from 59.55 to 61.80 and MiniMax-2.5 accuracy from 41.57 to 44.94; on DeepSeek-V4-Flash the accuracy goes from 47.57 to 52.43.
• Modularity matters. On TerminalBench 2.0, ModularRSI reaches 52.43 accuracy, while non-modular evolution falls to 46.44 and joint all-module evolution falls to 44.19, both below the 47.57 baseline.
• Modules contribute different benefits. Every single-module variant beats the baseline on accuracy. Agent Loop gives the largest single-module accuracy gain (50.56), while Observation Management substantially reduces the average number of execution steps (22.50 versus the 34.70 baseline). Cross-module integration then lifts accuracy to 52.43 and Pass^3 to 35.96.
• Existing RSI methods gain little under the benchmark-disjoint protocol. Under a unified protocol using DeepSeek-V4-Flash-0731 with web search disabled, Meta-Harness reaches 62.92 accuracy and AHE 62.54 against a Terminus-2 baseline of 61.79, whereas ModularRSI reaches 67.42, with Pass^3 of 56.18 and Pass@3 of 78.65.
• Contrastive pairs become scarcer as evolution proceeds. The proportion of contrastive trajectory pairs among trajectory groups decreases across epochs on TerminalBench 2.0, which the authors interpret as the harness absorbing the reusable behavioral improvements these contrasts reveal.
• Evolution data difficulty distribution matters. A Medium-centered difficulty distribution yields 76.45 accuracy on SWE-Bench Verified, versus 74.25 for a Hard & Easy distribution, a gap of 2.20 percentage points.

Methodology in Plain English

The starting point is the Terminus-2 harness from Harbor, whose behavioral mechanisms are reorganized into five modules: Agent Loop, Tool Use, Observation Management, Context Management, and Task Completion Detection. Infrastructure concerns such as sandbox initialization, parallel execution, and LLM communication are deliberately excluded from evolution.

For each evolution task, the agent is rolled out K times and each trajectory receives a binary reward from the task evaluator. Tasks are then sorted into three groups: Positive (all rollouts succeed), Contrastive (some succeed, some fail), and Negative (all fail). A Trajectory Memory stores historical trajectories and rewards so that earlier epochs can supply contrast when current rollouts cannot. A Code-Modify Agent — the agent under evolution itself — compares paired successful and failed trajectories of the same task to find function-level factors behind different outcomes. For all-failing tasks it first looks for a previously successful trajectory of the same task; if none exists, it performs single-sided diagnosis for evident deficiencies such as repetitive loops, incorrect tool usage, ineffective recovery, or premature termination. For all-succeeding tasks it looks for efficiency opportunities such as redundant actions, repetitive exploration, or unnecessary tool calls. Findings are consolidated into structured JSON.

Modification targets are chosen by consolidating semantically similar diagnoses, then voting by the number of distinct supporting tasks, which favors changes backed across multiple tasks over instance-specific failures. An Evolution History per function records prior code changes and introduced functionality to reduce redundant or oscillating edits. The five modules are evolved independently without sharing intermediate updates, so they can proceed in parallel.

Every proposed change must pass three validation gates. Program Check runs static checks including AST validation, import checks, protocol compliance, discovery-contract verification, and static self-attribute audits. Diff Review asks the Code-Modify Agent whether the diff encodes task-specific solutions, heuristics, or conditions unlikely to generalize. Execution Validation samples two tasks from the current batch and runs them with the updated harness. Failures at any gate trigger a rollback using recorded diffs. After all modules are evolved, a Cross-Module Integration epoch removes duplicated mechanisms, clarifies module responsibilities, and adjusts coordination logic. A Function Merge step removes redundant functions, and a Task-Aware Function Composer activates only task-relevant functions per task. The resulting function library is then frozen for evaluation.

The evolution dataset of 2,000 executable tasks was built by extracting high-level domain labels from the TerminalBench and SWE-Bench families, retrieving public resources from external sources using those labels as search guidance, and constructing tasks in Harbor format. Quality control combined LLM-based filtering on environment completeness, practicality and non-triviality, and evaluator validity; executable validation requiring the reference solution.sh to pass all tests while a no-op submission cannot receive a positive reward; manual review; and LLM-based semantic similarity screening against downstream benchmarks. All evolution used 3 epochs, with 120 instances selected from each of the TB-related and SWE-related subsets, DeepSeek-V4-Flash-Preview and DeepSeek-V4-Flash-0731 as primary evolution models, a TPM limit of 2 million tokens, and a batch size of 10.

Why This Matters

The paper's central move is separating evolution data from evaluation data, then showing that harness improvements still transfer. That is a meaningful methodological shift for self-improvement research, where gains observed on the same benchmark used for evolution are hard to attribute.

• Research impact: It reframes harness RSI as a credit-assignment problem and provides a standardized, benchmark-disjoint evolution resource (2,000 instances) for comparing methods on equal footing. It also shows that modular restriction is not merely an engineering convenience — joint and non-modular evolution both scored below the un-evolved baseline on TerminalBench 2.0.
• Real-world applications:
  • Long-horizon software engineering agents operating on real repositories, where reliability across repeated attempts matters as much as peak accuracy.
  • Terminal automation, where reducing the average number of interaction steps directly lowers compute and latency costs — Observation Management cut StepNum from 34.70 to 22.50.
  • Cross-model deployment, where a harness evolved once with one foundation model can be reused with others rather than re-tuned per model.
  • Post-deployment improvement of agent products, since harness-level gains do not require retraining model weights.
• Industry relevance: Because the evolved function library is frozen and transferable across GLM-5.2, MiniMax-2.5, and DeepSeek-V4-Flash, harness improvements behave like a deployable artifact rather than a training-run side effect. For agent platform vendors, that suggests a cheaper path to better agents than model retraining. The finding that benchmark-derived evolution data yields limited gains under a disjoint protocol is also a direct caution for teams currently tuning harnesses on their own evaluation sets.

Future Directions

• Isolate contrastive analysis. The authors explicitly state they do not run a dedicated ablation isolating the contribution of contrastive trajectory analysis; the paper supports it with trajectory statistics and case studies instead.
• Scale beyond the subset. The main evolution experiments use only a subset of the 2,000 curated instances for computational reasons, so the behavior of the framework across the full evolution set remains untested.
• Broaden method comparison. The paper compares against AHE and Meta-Harness adapted to the Harbor framework and the same 120 evolution instances; comparisons with a wider set of the frameworks listed in Table 1 remain open.
• Understand what governs transfer. The Medium-centered versus Hard & Easy result shows difficulty distribution changes downstream generalization by 2.20 percentage points on SWE-Bench Verified, leaving open what other properties of evolution data determine whether improvements generalize.

Target Audience

Researchers and engineers working on LLM agent systems, agent harness design, and self-improving agents will benefit most. It is also relevant to practitioners building coding or terminal agents who want to improve reliability and efficiency without retraining foundation models, and to evaluation researchers interested in protocols that separate evolution data from benchmark data. Readers without background in agent architectures, trajectory-based learning, or benchmark design will find the method sections demanding.

Authors’ abstract

Recent work extends recursive self-improvement (RSI) to agent harnesses for long-horizon coding and terminal tasks, enabling agents to improve execution mechanisms from experience. However, generalizable harness RSI remains challenging. First, evolving harnesses on evaluation benchmarks or their subsets makes it difficult to distinguish reusable improvements from benchmark-specific adaptation. Second, single-trajectory updates can conflate systematic harness deficiencies with instance-specific reasoning and solution details, producing modifications that transfer poorly to unseen tasks. Third, localizing recurring behavioral deficiencies within monolithic harnesses is difficult, while whole-harness optimization can entangle unrelated mechanisms and complicate attribution and validation. We propose ModularRSI, a benchmark-disjoint, contrastive, and modular framework for generalizable harness evolution. ModularRSI contrasts successful and failed trajectories for the same task and aggregates evidence across tasks to identify recurring behavioral deficiencies. It decomposes the evolvable harness into five functional modules: Agent Loop, Tool Use, Observation Management, Context Management, and Task Completion Detection. Each module evolves independently within a restricted modification scope, followed by an integration stage that combines the evolved modules into a unified harness and resolves potential conflicts. To support benchmark-disjoint evolution, we curate 2,000 executable evolution tasks from external sources that are disjoint from downstream evaluation benchmarks. Experiments on TB2.0 and SWE-Bench Verified show consistent improvements on unseen in-domain and cross-domain tasks, with the evolved harness also transferring across different foundation models.