Training the Whole Agent, Not Just the Model

AI summary

Overview

Research area: Reinforcement learning for coding agents; specifically the co-design of the language-model policy and the "harness" (tools, runtime hooks, and skill guidance) that surrounds it. Category: cs.SE.

Technical level: Advanced. The paper assumes familiarity with GRPO-style policy-gradient training, agent scaffolds (bash/tool-use environments), and SWE-bench-style software repair benchmarks.

Scope (one sentence): The paper proposes HarnessAdapt-RL, a training procedure that adapts an agent's harness — task-specific skills plus evolved tools and hooks — alongside GRPO policy updates on a 9B base model, and studies which harness components must be kept at deployment.

What This Paper Is About

Coding agents are really a pair: a model that picks actions and a harness that supplies guidance, tools, and runtime feedback. Prior training methods improve the model while holding the harness fixed, even though training trajectories keep exposing flawed repair strategies (wrong diagnoses, symptom-only fixes) and interaction problems (failed edit commands, excessive output, too little verification). The paper asks how to design skills for policy learning, how to evolve tools and hooks as the policy changes, and which harness pieces must survive into deployment.

Key Contributions

1. Skill design for policy learning. The authors compare three ways of generating skills — near-answer hints, mechanism-level skills, and a general skill bank — and find that mechanism-level skills (which explain when and why a repair strategy applies, with paths, symbols, and literal code stripped out) give the best performance once the skills are removed at evaluation.

2. Tool and hook evolution during reinforcement learning. A separate teacher model reads current-policy trajectories, identifies recurring interaction problems, and proposes new tools, hooks, or both. A proposal is accepted only if it improves the observed pass rate or the observed trainable rate without lowering either.

3. Training-aware harness selection. The selection rule explicitly accounts for whether rollouts are usable for GRPO updates ("trainable rate"), not just whether tasks pass, because all-success and all-failure groups carry no reward contrast.

4. Training and deploying the model–harness pair. The paper shows that skill gains persist after skills are removed, while the evolved tools and hooks remain necessary: removing them at deployment costs up to 7.7 percentage points of pass@1 with policy weights unchanged.

Main Findings

• Large gains over GRPO on a 9B model. HarnessAdapt-RL improves pass@1 over GRPO by 9.8, 12.6, and 6.4 percentage points on SWE-bench Verified, Multilingual, and Pro. Average pass@1 is 44.9% versus 35.3% for GRPO (an average gain of +9.6 points in the paper's table).

• It also beats a human-designed harness. Against OpenHands-GRPO (same base model, human-designed OpenHands harness, average 39.0%), HarnessAdapt-RL is higher by 6.2, 4.0, and 7.4 percentage points on the three benchmarks.

• Mechanism-level skills win after removal. With the same base harness, average scores without skills are 38.3% (mechanism-level), 34.1% (skill bank), and 32.7% (near-answer hints), against GRPO's 35.3%. Only mechanism-level skills beat GRPO on all three benchmarks.

• Higher training success can hurt. Near-answer hints raise training success but score 2.6 points below GRPO on average once removed. With mechanism-level skills, 46.5% of sampled groups contain both successes and failures, versus 39.0% with near-answer hints, which produce more all-success groups that give no reward contrast.

• Concrete answers shorten trajectories too much. Near-answer hints reduce average tokens per rollout by 40.8% relative to GRPO, versus 14.5% for mechanism-level skills.

• Evolving during training beats searching before training. HarnessAdapt-RL is 2.4, 4.3, and 2.7 points above Pre-Evolved (same search steps and tasks, but the policy stays at the base model during search).

• Re-evolving in the second epoch helps. HarnessAdapt-RL is 2.2, 4.6, and 2.0 points above H1+Mechanism, which shares the same first epoch and H1 but keeps H1 fixed.

• Evolution targets shift over training. Early accepted changes (context-limited read/search tools, direct editing tools, patch generation and diff display, hooks against repeated actions) raise the pass rate by 13.7% and the trainable rate by 7.4% relative to the current harness. Later changes add automatic related-test execution to patch preparation (pass rate +6.3%, trainable rate +7.4%) and a restore_file tool for undoing failed edits.

• Rejected proposals can return. One hook raised the pass rate by 2.7% but lowered the trainable rate by 7.0% and was rejected; a later proposal combining review with view_diff improved both rates and was accepted.

• The evolved harness is needed at deployment. Holding HarnessAdapt-RL's weights fixed and swapping H2 for H0 drops pass@1 by 4.8, 7.7, and 4.7 points on Verified, Multilingual, and Pro. Skill-Mechanism, trained only under H0, still gains from H2 (52.2 → 54.2 on Verified), but HarnessAdapt-RL's gain from H2 is larger (+4.8 vs +2.0 on Verified).

• The evolved harness can be reused. Frozen-H2+Mechanism, which trains a fresh policy with H2 held fixed, reaches 56.8%, 38.3%, and 38.9% — within 1.0 point of HarnessAdapt-RL on every benchmark, and above Pre-Evolved by 1.6, 5.3, and 1.9 points. This depends on H2 having first been discovered through evolution during training.

Methodology in Plain English

Start with a pretrained 9B model (Qwen3.5-9B) and a minimal base harness (mini-swe-agent) that mostly just runs bash. Train it with GRPO on a fixed pool of software-repair tasks drawn from R2E-Gym-Lite. While training runs, two frozen teachers watch.

The first teacher (GLM-5.3-Flash) looks at two attempts on the same task — ideally one that worked and one that failed — and writes a short "mechanism-level" skill: it explains the repair strategy and when it applies, but deliberately omits file paths, function names, and literal code, so the model still has to locate and write the fix itself.

The second teacher (GPT-5.6 Sol through Codex) looks across many tasks for problems that keep recurring — for example, the policy stops running verification tests late in training even though it has testing tools. It then writes candidate tools (packaged reusable operations) and hooks (runtime messages that nudge the policy). Each candidate harness is compared against the current one using the same policy, tasks, skills, and rollout budget; it is kept only if it improves the observed pass rate or the observed trainable rate without lowering the other. Each epoch runs up to five such evolution rounds, then trains the policy on fresh rollouts, so the next round of proposals is informed by the updated policy's behaviour. At deployment, skills and teachers are dropped, and the trained policy ships with the final tools and hooks.

Why This Matters

The paper reframes agent training: the deliverable is not just a set of weights but a model–harness pair, and the harness must co-evolve with the policy rather than being frozen or hand-designed. It also gives a concrete, measurable argument that training-time scaffolding can be a crutch — near-answer hints looked better during training and produced worse models — which bears directly on how guidance and hints are used in agent RL.

Real-world applications:

• Automated bug-fixing and maintenance agents in production codebases, where tool and feedback design matters as much as model quality.
• Agent platform and scaffold engineering: using trajectory-driven evolution instead of hand-tuning tool interfaces and runtime reminders.
• Deployment planning for trained agents: knowing which components are safe to strip at inference and which must ship with the model.
• Cost-aware training pipelines, since the paper reports that selection uses both task success and the fraction of rollouts actually usable for policy updates.

Industry relevance: The gains are measured against both a minimal harness and a human-designed OpenHands harness on the same 9B model, which speaks to teams choosing between investing in better scaffolds versus bigger models. The paper also reports additional costs of preparation, harness comparison, and teacher calls (Appendix 9.7), and releases weights, the evolved harness, training code, and data.

Future Directions

• Other agent domains. All tasks are software repair from SWE-bench-style benchmarks; the authors state they have not tested whether the procedure produces useful support for web navigation or data analysis.

• Larger and different models. Only one 9B base model is used. It is unknown whether larger models, which may already handle many operations the tools package, would benefit from harness evolution to the same extent, or whether the preferred guidance form would stay the same across model families.

• Training-seed and search variability. Each configuration is trained once; the three repetitions are evaluations of a fixed checkpoint, so the reported confidence intervals do not quantify variability across training seeds or independent harness-search runs.

• Statistical rigor in candidate selection. Acceptance uses observed pass and trainable rates from a finite rollout comparison and is explicitly not a significance test, so rollout noise and repeated candidate search can affect which changes are retained.

Target Audience

Reinforcement-learning and agent researchers working on coding agents; ML engineers building or maintaining agent scaffolds and tool interfaces; and practitioners deciding what to ship at inference time. Readers without background in policy-gradient training or SWE-bench evaluation will find the results intuitive but the method sections demanding.

Authors’ abstract

Training coding agents typically updates the policy while keeping the harness fixed. During training, trajectories reveal flawed repair strategies and interaction problems that vary across tasks and change as the policy learns. We introduce HarnessAdapt-RL, which trains the policy and adapts its harness during reinforcement learning. For flawed repair strategies, a frozen teacher generates task-specific mechanism-level skills. For interaction problems recurring across tasks, a separate teacher model evolves tools and hooks, keeping each change only if it improves the observed pass rate or the observed trainable rate without lowering either. The policy trains with the adapted harness, and its new trajectories guide further adaptation. With a 9B base model, HarnessAdapt-RL improves pass@1 over GRPO by 9.8, 12.6, and 6.4 percentage points on SWE-bench Verified, Multilingual, and Pro, and outperforms GRPO with the human-designed OpenHands harness. Gains from mechanism-level skills persist after their removal at evaluation. Re-evolving tools and hooks as the policy changes outperforms freezing early versions. Removing them at deployment, in contrast, costs up to 7.7 percentage points. Training therefore produces a model-harness pair: the trained policy and the evolved tools and hooks, with skills removed at deployment.