Harness Self-Evolution for Long-Horizon Terminal Agents

AI summary

Overview

Research area: LLM agent engineering — specifically the harness (the runtime layer that executes commands, builds observations, manages context, and decides when a task is complete) for long-horizon terminal agents.

Technical level: Intermediate. Readers need familiarity with LLM agents, tool/command execution loops, and standard pass-rate metrics, but the paper is not mathematically heavy.

Scope: A framework called RiseHarness that iteratively rewrites an agent's harness code (with model weights frozen) so that a fixed model solves long-horizon terminal tasks more consistently across repeated runs, not just occasionally.

What This Paper Is About

Long-horizon terminal agents can solve a task in one run and fail the same task in the next, because a single bad command or missed observation changes the environment and context for everything that follows. The authors argue this gap between "can solve" and "solves reliably" is largely a property of the harness, not the model, and that the harness can therefore be improved directly. RiseHarness is a loop that mines tasks with mixed success and failure ("flipping tasks"), diagnoses why the harness let a failure happen, borrows implementation ideas from other agent frameworks, and keeps only changes that measurably improve repeated-run performance — all without touching model weights.

Key Contributions

1. An evidence-driven harness evolution framework. RiseHarness iteratively retains harness modifications that improve measured performance, using successful runs of the same task as references for analyzing failures, with model weights held fixed throughout.
2. Behavioral diagnosis from paired runs plus internal execution. The framework combines agent trajectories with a runtime monitor's harness traces and a "Harness Handbook" that maps behavioral responsibilities to implementing source code, so that a proposed behavioral need is checked against what the harness actually did before any change is made.
3. Harness modification informed by external implementations. Proposals are generated by searching other harness frameworks, source code, and technical documentation for relevant designs, then adapting them to the current execution flow, interaction protocol, available runtime signals, and observed model behavior.
4. Reliability and transfer evaluation. Beyond average success, the paper measures repeated-run reliability (Pass^r, Flip), runs ablations isolating harness evidence and external research, compares harnesses evolved separately per model, and tests transfer to a benchmark never used for evolution.

Main Findings

• Gains over the seed harness for GPT-6-sol: RiseHarness raises Terminal-Bench Hard (TBH) Pass@1 from 50.25 to 55.50 (+5.25) and Pass^4 from 39.00 to 46.00 (+7), while Flip drops from 20.00 to 14.00 (-6). On Terminal-Bench 2.1 (TB2.1) Pass@1 goes from 78.65 to 81.74 (+3.09) and Pass^4 from 68.54 to 69.66 (+1.12).
• Larger gains for DeepSeek-V4-Flash: On TBH, Pass@1 rises from 56.75 to 64.00 (+7.25) and Pass^4 from 37.00 to 51.00 (+14), with Flip falling from 34.00 to 22.00 (-12). Pass@4 rises only from 71.00 to 73.00 (+2), which the authors interpret as the gains coming mainly from reliability rather than broader task coverage.
• Competitive against prior harness methods: On GPT-6-sol TBH, RiseHarness reaches 55.50 Pass@1 versus 50.75 for ModularRSI and 54.75 for AHE Harness; on DeepSeek-V4-Flash TBH it reaches 64.00 versus 58.00 and 59.75 respectively. Results are not uniformly best on TB2.1: for GPT-6-sol, ModularRSI reports higher Pass@4 (91.01) and AHE reports higher Pass^4 on DeepSeek TB2.1 (62.92 vs 67.42 for RiseHarness — the latter being higher; note ModularRSI's 53.93 is lower).
• Harness traces and the Handbook contribute: Removing harness traces and the Harness Handbook (keeping agent traces and source code) costs the DeepSeek-V4-Flash variant 7% Pass@1 and 8% Pass^4 on TBH relative to the full method, and the full method also leads on TB2.1.
• External research contributes separately: Removing external design search (keeping dual traces and the Handbook) costs 5.50% Pass@1 and 6% Pass^4 on TBH, with lower Pass@4 and higher Flip; the full method also leads on TB2.1.
• Adaptation is model-specific: DeepSeek-V4-Flash's retained changes strengthen the evidence required before declaring completion; GPT-6-sol's changes target session recovery, observation construction, completion context, workspace diffs, and prior-work evidence. Swapping the evolved harnesses between the two models reduces performance on both benchmarks.
• Gains transfer to unseen tasks: Harnesses evolved only on TBH were frozen and evaluated on TB2.1 with the same model; both models retained gains in average success and all-success task fraction. The paper reports that directly applying the TBH-selected final DeepSeek version to TB2.1 improves Pass@1 by 3.09% and Pass^4 by 6.74% over Terminus-2.
• Version rankings differ across benchmarks: For DeepSeek, the three-mechanism version performs best on TB2.1, while the four-mechanism final version performs best on TBH.

Methodology in Plain English

The setup fixes the model, the task set, the task protocol, the verifier, and the acceptance rules. Only harness prompts and executable interaction logic are editable, and the model that writes the proposals may differ from the model being evaluated.

Step 1 — Record two kinds of trace. The team runs the seed harness (Terminus-2) repeatedly (at least two complete repetitions per task in the reference corpus; four in evaluation) and records both the usual agent trajectory and a harness trace produced by a runtime monitor that logs arguments, return values, exceptions, caller relationships, and source locations at selected function boundaries. Run identifiers link the two, so an analyst can follow a model request through internal processing to the feedback the model actually saw.

Step 2 — Find flipping tasks. The analysis focuses on tasks with both successes and failures under the identical configuration. A success proves the system can do the task and serves as a concrete reference for what went wrong in the failure. The flipping set is defined by 0 < (number of successful repetitions) < (total repetitions).

Step 3 — Diagnose behaviorally. Four operations — Compare, Wish, Revise, Group — compare paired runs, state what the harness should do under an observable condition, then check the harness trace to see whether the needed information was even available and whether the existing mechanism already did it. Findings are then pooled across tasks into groups that could be fixed by one shared change, reviewed for whether the common wish really applies to all members, and ranked by distinct-task coverage, distinct failing runs, and finding count.

Step 4 — Research elsewhere, then adapt. For each selected group, the system turns the behavioral wish into search questions, looks at how other harnesses handle the same problem (their execution flow, maintained state, feedback to the model), and designs a change fitted to the current harness's structure and the observed model behavior. It also checks traces of tasks that succeeded every time, to avoid disrupting behavior that already works.

Step 5 — Implement, screen, evaluate, keep or discard. Candidates are built together with an updated Handbook and trace configuration, smoke-checked, and (when enabled) behaviorally tested against per-mechanism contracts with triggers, observable effects, hooks, and state. Survivors go through targeted screening and then a full repeated evaluation on the development set. A candidate is adopted only if records are valid, the required run set is complete, and the aggregate pass rate beats the declared baseline; otherwise the current version is kept, and the system may revisit the proposal or repair the implementation. Accepted versions, Handbooks, and traces become the evidence base for the next round.

Measurement. Four metrics are reported at r = 4 repetitions: Pass@1 (average pass rate over tasks and repetitions), Pass^r (fraction of tasks passing in all r runs), Pass@r (fraction passing at least once), and Flip = Pass@r − Pass^r. The paper cautions that a lower Flip alone can mean tasks became all-failures rather than more reliable, so Flip must be read together with Pass^r.

Why This Matters

Impact on research. The paper reframes agent reliability as a harness-engineering problem that can be attacked without training runs, and it supplies measurement discipline — separating average success from all-run consistency, and treating Flip as ambiguous unless paired with Pass^r. It also provides a concrete negative result for one-size-fits-all harnesses: identical mechanisms do not transfer between models, so harness adaptation should be model-specific. The dual-trace plus Handbook linkage is a reusable diagnostic instrument for any system where the runtime layer, not the model, decides what the model sees.

Real-world applications:

• DevOps and infrastructure automation, where an agent that edits configs and runs shell commands must succeed repeatedly and safely, not just once in a demo.
• Software maintenance and CI repair, where agents operate in terminals over long multi-step workflows and an early bad command corrupts the workspace for every later step.
• Data pipeline and ETL operations, where completion must be verified rather than assumed — the exact behavior DeepSeek-V4-Flash's evolved harness enforces.
• Customer-facing autonomous support tooling, where consistent completion and honest "not done yet" signals matter more than peak capability.

Industry relevance. The method requires no model retraining, no gradient access, and no weight changes, so it can be applied to proprietary models available only through an API. It also converts failure analysis into a repeatable, auditable loop that produces versioned artifacts (source, Handbook, traces, evaluation records), which fits how platform teams already ship and roll back changes.

Future Directions

• Extending across more models, task distributions, and agent frameworks. The authors state this explicitly, since the current evidence covers GPT-6-sol and DeepSeek-V4-Flash (with GPT-5.6-Terra in the appendix) and a single seed harness.
• Reducing the evaluation cost of harness evolution. Every candidate requires repeated full runs on the development set; the paper names this as an open problem rather than a solved one.
• Predicting which harness a model needs, instead of evolving one per model. Swapped harnesses underperform, which raises the question of whether model behavior can be profiled up front to choose or initialize harness mechanisms.
• Understanding when benchmark-specific version rankings generalize. DeepSeek's best version differed between TBH and TB2.1, so how version selection should be done when the target distribution is unknown remains unsettled.

Target Audience

Researchers and engineers building or maintaining LLM agent runtimes — particularly those working on terminal/shell agents, agent scaffolds, and reliability evaluation. It is most useful to practitioners who own harness code and can change prompts and execution logic but cannot fine-tune the underlying model, and to evaluation researchers interested in metrics that separate capability from consistency. Readers looking for training-based improvements or new model architectures will not find them here.

Authors’ abstract

Long-horizon terminal agents may solve a task in one run yet fail across repeated runs. Stochastic errors can propagate through actions and context, creating a gap between capability and reliable execution. We present RiseHarness, a framework for iteratively improving agent harnesses with model weights fixed. It focuses on flipping tasks with both successful and failed runs under the same configuration, using successful runs as references for failure analysis. RiseHarness combines agent trajectories with internal harness traces from a runtime monitor to examine how model actions interact with command execution, feedback, context management, and completion decisions. A Harness Handbook links this evidence to harness behaviors and source code, helping identify recurring problems in mechanisms shared across tasks. RiseHarness then searches other harness frameworks for relevant implementations and adapts them as design references to the current model and execution protocol. Candidate changes undergo behavioral checks and repeated task evaluation; versions with measured gains seed the next iteration. This loop integrates suitable mechanisms from different frameworks into a single harness and uses new execution evidence to guide further changes. We evaluate task success and reliability across models, ablate harness execution evidence and external research, and test transfer to unseen benchmarks with the same model. RiseHarness treats a specific model as the unit of harness adaptation, studying how changes to shared interaction mechanisms can make existing capabilities more reliable across tasks.