Self-Evolving Coding Agents via Harness-Policy Optimization

AI summary

Overview

Research area: Large language model (LLM) coding agents — specifically the joint optimization of an agent's policy (the underlying model) and its harness (the external scaffold controlling context, tools, feedback, and termination), using reinforcement learning.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning from verifiable rewards, GRPO-style policy optimization, and software-engineering agent benchmarks such as SWE-Bench.

Scope: The paper proposes Self-HPO, a recursive self-improving loop that alternates harness search with policy RL training on three SWE-Bench benchmarks, using Qwen3.5-35B-A3B with mini-SWE-agent as the base agent.

What This Paper Is About

Coding agents are built from two parts: the policy model and the harness that surrounds it. Prior work improves only one at a time — harness search keeps the model frozen, and policy training keeps the harness frozen — which the authors argue is suboptimal because a harness tuned for one checkpoint may not target the failure modes of an updated policy, and policy learning can only draw on trajectories the fixed harness happens to elicit.

The goal is a coding agent that improves both components itself, in alternation, without needing a separate external model to propose harness changes.

Key Contributions

1. Self-HPO framework. A recursive self-improving loop for coding agents that alternates a harness optimization step (H-step) with a policy optimization step (P-step), where the evolving policy serves as its own harness proposer.
2. Evolving harness search set. Rather than searching on a fixed task set, each H-step targets tasks that were never resolved in any rollout during the preceding P-step, so harness search follows the policy's shifting failure modes.
3. Empirical validation across three benchmarks. Self-HPO beats both policy-only (RLVR) and harness-only (Self-Harness) baselines, raising Qwen3.5-35B-A3B's average resolve rate from 57.8% to 64.8%, with gains transferring to out-of-distribution benchmarks.
4. Component and capability analysis. Cross-evaluation shows each optimized component retains gains when paired with its base counterpart and that the joint gain exceeds the sum of individual gains; Self-HPO also raises pass@1 reliability and the pass@k capability frontier.

Main Findings

• Joint optimization beats one-sided optimization. On SWE-Bench Verified, Self-HPO reaches 69.5% versus 67.5% for RLVR (+2.0) and 65.2% for Self-Harness (+4.3). Its average over three benchmarks rises from 57.8% (Base) to 64.8%, exceeding RLVR's 61.9% by 2.9 points.

• Gains transfer out of distribution. Training uses only Python tasks, yet Self-HPO reaches 59.7% on SWE-Bench Multilingual (+5.5 over Base) and 65.2% on SWE-Bench Pro (+6.9 over Base), beating RLVR by 3.4 points on both. On SWE-Multilingual, Self-Harness improves over Base by only 0.1 point.

• Harness evolution is a better transfer channel than SFT distillation. With Opus 4.8 as external supervisor, Meta-Harness beats SFT by 4.1 points on SWE-Verified and 8.9 points on average; SFT actually falls below Base on SWE-Multilingual and SWE-Pro, while Meta-Harness improves on both. Adding RL policy updates (HPO) pushes further to 71.6% / 61.4% / 66.0% (66.3 average), beating Meta-Harness by 2.5 points on SWE-Verified and 2.8 points on average.

• Repeated alternation keeps paying off. In the optimization traces, Self-Harness drops from 55.0% to 54.3% on SWE-Multilingual after a second harness-only update, whereas Self-HPO's second H-step and second P-step each add gains. Candidate-level traces show the shared first H-step reaching 64.3%, the first policy update reaching 67.8%, the second H-step 68.6%, and the second P-step 69.5%.

• Ablations confirm each design choice. Starting from HPO's 71.6%: replacing RL with SFT in the P-step drops to 67.2% (−4.4); RFT reaches 70.5% (−1.1); a fixed search set gives 70.1% (−1.5); using the target policy as proposer instead of Opus 4.8 gives 69.5% (−2.1). An external proposer helps but is not required.

• Components are complementary. Applying the final harness H₂ to the base policy yields 64.5% (from 61.0%), and deploying the final policy P₂ under the seed harness yields 64.4%. Combining both reaches 69.5% — an 8.5-point gain that exceeds the summed individual gains (3.5 + 3.4 = 6.9) by 1.6 points.

• Reliability and capability frontier both improve. With eight attempts per task, pass@8 rises from 80.0% to 84.8% (+4.8), and Self-HPO stays above Base for every k from 1 to 8. Pass@1 gains are 17.2 points on Medium tasks, 12.9 on Easy, and 6.1 on Hard, while it drops 2.2 points on the already-Resolved group.

• Training signal is less sparse. In the final P-step, zero-success rollout groups fall to 6.5% over the last ten updates, versus remaining mostly between 11% and 17% over RLVR's last 50 updates.

• Harness changes are concrete runtime controls. The evolved harness keeps the seed agent's query/execute loop but adds a budget warning prompting a best-effort submission, a one-time empty-patch guard asking for git add -N, a delayed first submission that prompts checking API names and signatures against the repo and tests, and a fallback that submits the sandbox's Git diff when the patch fails the apply check.

Methodology in Plain English

The setup pairs a policy model with a harness, and measures success as the expected resolve rate — the fraction of tasks whose generated patch passes an executable verifier.

The loop runs in rounds. In the H-step, the current policy is frozen and used twice: as the task solver, and as the harness proposer. It runs on a search set, looks at its own failed trajectories, and writes revisions to the harness prompt or executable runtime logic. Six candidate harnesses (K = 6) are tried per H-step, each revising the current best one and drawing on the accumulated experience of all previously evaluated harnesses and their trajectories; the best-scoring candidate on the search set is kept.

In the P-step, that harness is frozen and the policy is updated with reinforcement learning on 2,484 training tasks drawn from SWE-rebench, SWE-rebench V2, and Scale-SWE. Rewards are binary outcomes from the verifier, optimized with GRPO plus Balanced Adaptive Rollout, which draws extra rollouts to build reward-balanced groups and reduce uninformative zero-variance groups. Each P-step runs 50 RL updates.

The key twist is the search set. It starts as 300 randomly sampled tasks. Afterwards, it is rebuilt from tasks the policy never solved in any rollout during the preceding P-step. Two rounds are run in total, based on a stopping criterion where the hard-task set stabilizes between rounds.

The base agent is Qwen3.5-35B-A3B with mini-SWE-agent as the seed harness. Evaluation covers SWE-Bench Verified (500 tasks, in-distribution), SWE-Bench Multilingual (300 tasks), and SWE-Bench Pro's public V2 set (642 tasks), averaged over three runs (avg@3). An external-model variant, HPO, swaps in Opus 4.8 as the proposer while keeping the same loop.

Why This Matters

Research impact. The paper reframes agent improvement as a two-variable, co-evolving problem rather than a one-sided one, and shows that harness and policy gains are super-additive (8.5 points jointly versus 6.9 summed). It also challenges the common assumption that distilling strong-model trajectories into weights is the best way to transfer capability — here, learning from Opus 4.8 through harness optimization beat SFT, and SFT degraded out-of-domain performance below the base model.

Real-world applications:

• Automated software maintenance agents that resolve GitHub issues in production repositories across many languages.
• Coding assistants that must work within a fixed latency or step budget and benefit from runtime guards like best-effort submission before timeout.
• Tool-using agents outside software (data pipelines, shell automation) where the same harness/policy loop could target execution and termination failures.
• Self-improving internal agent platforms where no stronger external proposer model is available.

Industry relevance. The self-proposer variant removes the dependency on a proprietary frontier model as harness designer, which matters for cost, data governance, and deployment in closed environments. The paper notes that a 35B model (3B active) with Self-HPO outperforms several much larger entries on the SWE-Bench Verified leaderboard as listed in Table 4 — though the table truncates before Self-HPO's own row is shown in the provided text, so its exact leaderboard placement is not reported here.

Future Directions

• Longer optimization horizons. Only two H-step/P-step rounds are run, governed by a stopping criterion based on hard-task set stabilization; whether gains continue to compound over many more rounds is not reported.
• Alternative proposer and P-step designs. The ablations show SFT (−4.4) and RFT (−1.1) trail RL, and self-proposal costs 2.1 points versus Opus 4.8 — whether a trained proposer, or harness edits placed directly in the RL action space, could close that gap remains open.
• Search-set and budget policy. Fixed search sets cost 1.5 points, but how best to select, size, and refresh hard-task sets — and how to allocate the K = 6 candidate budget — is unexplored.
• Beyond coding. All benchmarks are repository-level software tasks; whether the co-evolution pattern transfers to web navigation or computer-use agents is untested.

Target Audience

Researchers and engineers working on LLM agents, agentic reinforcement learning, and automated software engineering — particularly those building or tuning agent scaffolds who want to know whether to invest in harness search, policy training, or both. It is also relevant to practitioners deploying self-improving coding agents under compute and data constraints, since the paper quantifies the trade-off between self-proposal and an external frontier proposer. Readers without RL or SWE-Bench background will find the core idea accessible but the experimental section dense.

Authors’ abstract

Large language model (LLM) agents have shown promising potential in coding tasks, and their performance is jointly determined by the policy model and the harness that mediates their interaction with the environment. Indeed, harness and policy are mutually reinforcing and evolving only one component is suboptimal---a harness tuned for one checkpoint may not target the failure modes of an updated policy, while policy learning remains limited to the trajectories elicited by a fixed harness. In light of this, we propose Self-HPO, a recursive self-improving framework for coding agents that alternates harness and policy optimization, using the evolving policy as its own harness proposer. Specifically, in each round, the current policy first analyzes its own failures and proposes new harness candidates (H-step). The policy is then updated through reinforcement learning (RL) under the selected harness (P-step), and hard tasks identified during training form the next harness search set, keeping harness optimization targeted to the policy's evolving failure modes. Across three coding benchmarks, Self-HPO consistently outperforms harness-only and policy-only baselines, improving Qwen3.5-35B-A3B's average resolve rate from 57.8% to 64.8%, with gains generalizing to out-of-distribution settings. Further analysis shows that both optimized components retain gains even when paired with their base counterparts, while their joint gain exceeds the sum of their individual gains. Self-HPO also improves reliability (Pass@1) and expands the capability frontier (Pass@k) over the base agent. Additionally, we show that with a stronger external model (e.g., Opus 4.8) as the harness proposer, the same framework yields a further performance boost.