HarnessPro: Recursive Harness Self-Improvement with Pareto Optimization for Performance-Cost Trade-offs

AI summary

Overview

Research area: Machine learning — specifically autonomous AI agents, LLM agent harness engineering, and recursive self-improvement (RSI) of agent scaffolding.

Technical level: Advanced. The paper assumes familiarity with LLM agent architectures, benchmark evaluation, Pareto optimization, and multi-agent LLM pipelines.

Scope: The paper introduces HarnessPro, a label-free multi-agent method that recursively rewrites an agent's software harness to improve task pass rates while lowering execution cost, returning three harnesses (lite, medium, heavy) spanning different performance–cost trade-offs.

What This Paper Is About

An "agent harness" is the software layer around a language model — its prompts, tools, skills, memory, hooks, and verification rules — that determines what the model sees and can do. Recent recursive self-improvement methods make harnesses better at solving tasks, but they typically do so by spending more tokens on exploration and verification, so gains arrive with higher execution cost and latency. HarnessPro asks whether recursive harness self-improvement can deliver better performance at lower cost, and answers by optimizing both metrics jointly and returning a small family of harnesses rather than a single one.

Key Contributions

1. A multi-agent, cost-aware harness evolution framework. HarnessPro analyzes execution trajectories, plans and implements harness edits under three different performance–cost priorities, and iterates, so that each epoch's feedback drives the next round of updates.

2. Test-based candidate evaluation instead of LLM judging. Rather than asking an LLM to compare trajectories directly (as RHO does), a dedicated test-synthesis agent writes fixed tests from task descriptions and the initial harness's trajectory, with no access to ground-truth solutions or official tests. The paper reports failure recall rising from 41.7% to 83.3% compared with direct LLM judgments.

3. Pareto-based selection producing three deployment tiers. Candidates below the initial pass-rate floor or Pareto-dominated are discarded, and normalized weighted squared-distance selection retains cost-focused (lite), balanced (medium), and performance-focused (heavy) harnesses for the next epoch.

4. Empirical results across three benchmarks and two backbone stacks. Evaluated on Terminal-Bench 2, SWE-Bench Pro, and ProgramBench with GPT-5.5 + Codex and GLM-5.2 + Claude Code, the heavy mode gains an average 2.0 percentage points in pass rate and an average 26.1% per-task cost reduction relative to RHO (GPT-5.5 + Codex).

Main Findings

• Better performance at lower cost than RHO (GPT-5.5 + Codex). On Terminal-Bench 2, HarnessPro-Heavy reaches 80.8% pass rate versus RHO's 76.3% (a 4.5 percentage point gain) while cutting mean cost per task by 33.9% ($0.72 vs. $1.09). On ProgramBench, 69.6% versus RHO's 68.8% at 17.8% lower mean cost ($2.95 vs. $3.59).

• The medium tier trades a little accuracy for large savings. On SWE-Bench Pro, HarnessPro-Medium reaches 79.0% versus RHO's 80.0% while reducing mean cost per task by 53.0%, from $2.68 to $1.26.

• Lite is cheaper than doing nothing. The lite mode outperforms the unmodified Vanilla harness on all three benchmarks while spending 17.7–57.4% less.

• Gains hold on a second backbone stack. With GLM-5.2 + Claude Code, HarnessPro-Heavy beats RHO on all three benchmarks at lower cost: 81.4% vs. 78.0% on Terminal-Bench 2 ($0.62 vs. $1.00), 77.3% vs. 75.0% on SWE-Bench Pro ($1.01 vs. $2.17), and 64.4% vs. 63.3% on ProgramBench ($14.95 vs. $17.00).

• Recursion beats a cost-aware single round. Against RHO+Iteration, HarnessPro-Medium matches its Terminal-Bench 2 performance at 31.7% lower cost, and HarnessPro-Heavy raises ProgramBench pass rate by 1.3 percentage points while lowering mean cost by 5.8%.

• Latency improves as a side effect. Although latency is treated as an auxiliary metric rather than an objective, cost reductions usually come with faster completion — for example, SWE-Bench Pro mean latency falls from RHO's 520 seconds to 412 seconds.

• Evolved harnesses work through dynamic budgeting, not brute compression. They pick a reasoning-effort level and tool-call budget from each task description, and runtime hooks enforce it. On Terminal-Bench 2 with the medium mode's standard profile, the agent is reminded to issue parallel tool calls and avoid redundant checks after 14 tool calls, pushed toward completing and verifying after 26, and blocked after 44. The harnesses also compress tool output (RTK-style) and streamline developer instructions.

• Single strategies transplanted into RHO do not reproduce the gains. Adding dynamic routing to RHO on SWE-Bench Pro gives 76.0% at $2.13, and adding budget hooks gives 75.0% at $1.75 — both cheaper but below RHO's own 80.0% pass rate. Fixing low reasoning effort drops pass rate to 59.0%.

• Both components matter in ablation. Removing the planner drops SWE-Bench Pro medium from 79.0% to 73.0% while cost rises from $1.26 to $1.42. Removing the test synthesizer is more damaging, with pass rates falling 3.4–10.0 percentage points across both benchmarks and all three modes.

• The Pareto frontier advances across epochs. On Terminal-Bench 2, heavy rises from 74.6% at epoch 1 to a mean of 80.8% at epoch 3 while cost falls from $0.91 to $0.72; lite and medium hold pass rate from epoch 2 to 3 while costs drop from $0.61 to $0.43 and $0.68 to $0.56. On SWE-Bench Pro, medium improves at the expense of cost in epoch 2 (70.0% → 77.0%, $1.57 → $1.66) before improving both in epoch 3 (79.0% at $1.26).

Methodology in Plain English

The setup starts from an existing harness and a benchmark split into training, validation, and test tasks. From each benchmark, a 10-task training set is chosen with a Determinantal Point Process to balance difficulty and diversity; 59, 100, and 100 tasks are held out for testing on Terminal-Bench 2, SWE-Bench Pro, and ProgramBench respectively, and the remainder forms the validation set. The test set is only touched at the very end.

For every training task, the current harnesses each run three independent rollouts (G = 3). A trajectory analyzer looks at all three rollouts together, rather than one, and reports what went wrong, how much it cost in performance and money, and where the evidence sits in the trajectory files (diagnoses are summarized if they exceed 64K tokens).

An evolution planner then writes three plans per candidate, each under a different instruction: one prioritizing cost reduction, one balancing the two, and one prioritizing performance. The planner is shown the candidate's own code, its diagnoses and metrics, and its code differences from the other candidates, so it can borrow what works. It is told to propose generalizable changes, and never sees the validation or test sets.

A harness editor implements each plan in a separate workspace and commits the result to Git, so edits are traceable and reversible. New candidates are pooled with their parents.

A test synthesizer writes test cases once, from each task description and the initial harness's trajectory, and those tests stay fixed throughout evolution so all candidates are compared on the same footing. Every candidate is run once per validation task and scored on pass rate, cost (computed from uncached input, cached input, and output tokens priced separately), and latency.

Candidates below the initial harness's pass rate, or Pareto-dominated by another candidate, are thrown out. From what remains, each of the three tiers independently picks the harness minimizing a weighted squared distance in normalized performance and cost space, with different weights per tier. Those three harnesses become the candidate set for the next epoch. Evolution runs three epochs on Terminal-Bench 2 and SWE-Bench Pro and four on ProgramBench, with up to three harnesses retained after each epoch.

Why This Matters

Research impact. Most harness self-improvement work treats execution cost as an afterthought, so reported gains can be purchased with tokens. HarnessPro reframes harness evolution as a multi-objective problem over the whole harness — instructions, skills, tools, hooks, and verification — and shows that the Pareto frontier can be advanced in the direction of both axes rather than traded along a fixed frontier. It also supplies evidence that LLM-as-judge selection is a weak link in label-free pipelines, and that synthesized tests are a substantially better substitute in this setting.

Real-world applications:

• Coding agents in enterprise repositories, where per-task API spend and latency dominate the economics of running agents over many issues.
• CI and automated program repair, where an agent must be budgeted per pull request or per failing test.
• Terminal and DevOps automation, where long-running agents can accumulate large tool-output contexts and where cutting overhead matters as much as accuracy.
• Tiered agent products, where the same system must serve a cheap default tier and a premium high-accuracy tier without maintaining separate hand-built harnesses.

Industry relevance. The output of the method is precisely what a product team needs: three deployable harness variants mapped to budget levels, plus a mechanism (task-conditioned reasoning effort and tool budgets, enforced by runtime hooks) that any agent stack could adopt. The finding that naive cost cuts like fixed low reasoning effort collapse performance to 59.0% is a direct warning against blunt budget slashing. The method is label-free, so it can be run against proprietary tasks without ground-truth answers.

Future Directions

• Scaling the training coreset. Evolution here runs on a 10-task training set selected by DPP; whether larger or differently selected coresets change the Pareto frontier, or reduce the epoch-to-epoch cost bumps seen on SWE-Bench Pro, is untested in this paper.

• Better synthetic tests. The test synthesizer is the single most damaging component to remove, yet the generated tests are created once from the initial harness's trajectory and then frozen; iteratively improving test quality, or validating it against held-out behavior, is an open direction.

• Extending beyond three tiers and two objectives. The selection scheme is defined over pass rate and cost with three weightings; adding latency, or exposing a continuous budget knob rather than three discrete modes, would make the output more directly usable in deployment.

• Testing transfer and generalization. The paper reports no results on how a harness evolved for one benchmark or model transfers to another, and the content provided does not report an explicit limitations section or a study of failure modes when the label-free tests are systematically wrong.

Target Audience

Researchers and engineers working on LLM agents, agent scaffolding, and inference-time budget allocation will get the most from this paper, particularly those who already know RHO or similar trajectory-driven harness optimization methods. It is also relevant to practitioners deploying coding or terminal agents at scale who need to justify per-task API spend, and to anyone building label-free evaluation pipelines, since the test-synthesis result (41.7% → 83.3% failure recall) stands on its own. Readers without background in agent harnesses or Pareto optimization should expect a steep entry point.

Authors’ abstract

Harness development plays a crucial role in building capable AI agents. Recent approaches improve harnesses through recursive self-improvement (RSI), but their performance gains often come with higher execution costs. We introduce HarnessPro, a label-free method for evolving harnesses to improve task performance while reducing execution cost. In this setting, candidate harnesses are evaluated using LLM-generated tests without access to ground-truth solutions or official tests. Specialized agents diagnose execution trajectories, plan updates with different performance-cost priorities, and implement them as candidate harnesses. Rather than returning a single harness, HarnessPro uses normalized weighted squared-distance selection to provide cost-focused (lite), balanced (medium), and performance-focused (heavy) harnesses for different deployment budgets. Selected harnesses enter the next iteration, where execution feedback guides further updates. Our experiments cover Terminal-Bench 2, SWE-Bench Pro, and ProgramBench, using GPT-5.5 with Codex and GLM-5.2 with Claude Code. Across these benchmarks using GPT-5.5 with Codex, the heavy mode of HarnessPro achieves an average pass-rate gain of 2.0 percentage points and an average per-task cost reduction of 26.1% relative to RHO. This advantage also holds when using GLM-5.2 with Claude Code. Code is available at https://github.com/pzs19/HarnessPro