Harness-S: Learning from Sibling Harnesses for Generalizable Agent Training

AI summary

Overview

Research area: Language-model agents — specifically the co-design of agent harnesses (prompts, tool interfaces, context/memory mechanisms, execution rules) with reinforcement-learning policy training.

Technical level: Advanced. The paper assumes familiarity with GRPO/PPO-style policy-gradient training, LLM agent benchmarks, and the notion of an "execution scaffold" separate from model weights.

Scope: The paper proposes Harness-S, a training pipeline that evolves multiple specialized "sibling" harnesses from a common base, uses them only to rescue tasks the base harness fails on, and forces the resulting gains back into a single policy that is then evaluated under the original base harness.

What This Paper Is About

An agent's behavior depends on both its model weights and its harness. The paper observes that when you evolve harness variants from a common base, each variant tends to fix some failures while breaking previously solved tasks, different variants cover each other's errors, and a variant's usefulness shifts as the policy itself learns. Rather than searching for one universally superior harness — or co-evolving harness and policy together, which is expensive and may not transfer — the authors ask whether specialized harnesses can be used selectively to rescue failures of the base system, and whether the capabilities they elicit can be absorbed into a single policy that remains strong under the original base harness.

Key Contributions

1. Diagnosis of three properties of harness evolution. The authors identify and empirically document Locality (edits repair subsets of failures while regressing others), Complementarity (sibling harnesses recover one another's errors, with an oracle union covering more tasks than repeated sampling from the base harness at the same rollout count), and Nonstationarity (a harness's utility relative to the base can reverse across policy checkpoints). Figure 1 illustrates all three on Qwen3.5-4B on AppWorld.

2. Directional Harness Evolution. Starting from a base harness H₀, the pipeline diagnoses failures in execution trajectories, groups related root causes into repair directions, has an editor produce child harnesses targeting limited sets of causes, and selects the next parent set by maximizing joint validation coverage (each validation task counts once, favoring parents whose solved sets are disjoint). All executable, protocol-compliant children are retained in a fixed Sibling Harness Pool used throughout policy training.

3. Complementary Harness Rescue Training. For each task, the policy first draws K rollouts under H₀. Only when all of them fail does the Adaptive Harness Router select the Top-k auxiliary harnesses, and the same policy then makes one fresh rollout per selected harness from the task's initial state; this rescue group replaces the failed group for GRPO updates. This targets learning signal at exactly the tasks where the base group provides no within-task reward contrast.

4. Adaptive Harness Router with policy-aware profile re-anchoring. Harness Behavior Profiles encode "When: condition / Behavior: difference" entries derived by an LLM from paired validation trajectories, and multiple independent LLM voters rank harnesses for a task. Because harness utility is nonstationary, the profiles are periodically refreshed from current-policy validation trajectories. The same router is reused at inference as Rank-1 routing over the full pool including H₀.

Main Findings

• Gains survive removal of the auxiliary harnesses. Under the original H₀, Harness-S reaches AppWorld Pass@3 67.56 ± 1.87 / Avg@3 56.87 ± 1.47 at 4B and 72.52 ± 1.65 / 62.34 ± 1.28 at 9B, versus 61.83 / 51.91 and 65.65 / 54.83 for GRPO with H₀ rescue, and 60.31 / 48.98 and 64.89 / 53.31 for plain GRPO.

• Consistent advantage across all three benchmarks. At 9B under H₀, Harness-S scores SWE-Gym 40.00 ± 2.04 Pass@3 / 32.50 ± 1.51 Avg@3 and DBBench 76.33 ± 2.13 / 73.44 ± 1.67, versus 29.38 / 26.88 and 70.67 / 67.78 for GRPO with H₀ rescue. The 4B results show the same ordering (SWE-Gym 38.75 / 28.75; DBBench 73.33 / 71.00).

• Rank-1 routing adds gains at matched rollout count. Same trained checkpoints scored 68.32 ± 1.74 / 61.45 ± 1.35 (4B) and 75.19 ± 1.56 / 67.05 ± 1.20 (9B) on AppWorld, with improvements in every benchmark–capacity setting. The routed 4B policy exceeds 9B GRPO with H₀ rescue on every reported metric, and the routed 9B policy exceeds all three frontier references on DBBench (GPT-6-Sol 71.33 / 68.89; DeepSeek-v4-Flash 75.00 / 70.44; GLM-5.3-Flash 76.67 / 72.11).

• Beats frozen-harness evolution and joint adaptation. With the policy frozen, Harness-S (HE only) at 9B reaches AppWorld 61.45 / 50.13 under H_rank1 versus Meta-Harness's 53.44 / 42.88 under H_best. Full Harness-S exceeds the policy–harness co-evolution baselines WHALE (68.70 / 58.02 at 9B on AppWorld) and HarnessForge (64.89 / 54.83) despite those methods being scored with a task-independent H_best.

• Better learning signals, not just more sampling. On AppWorld with Qwen3.5-4B, the Effective Group rate (retained groups containing both successes and failures) reaches 79.38% for Harness-S versus 70.94% for GRPO with H₀ rescue and 64.69% for standard GRPO. Effective training-token share reaches 80.88% versus 72.68% for H₀ rescue. Final validation success from the same starting policy: 63.33% versus 54.00% and 48.00%; both fixed harness mixtures finish below plain GRPO.

• Fixed mixtures of harnesses do not capture the benefit. GRPO with mixed harnesses (DSH H₀ + Claude Code) scores 57.63 / 46.69 at 4B on AppWorld versus 67.56 / 56.87 for Harness-S — harness diversity helps mainly when directed at tasks the base group fails.

• Routing strategy matters, and re-anchoring helps. With a fixed rollout budget on AppWorld 4B, Top-8 Adaptive rescue success by ten-step stage is 52.38 / 66.67 / 45.71 / 50.00, versus 35.71 / 24.24 / 26.67 / 38.89 for Top-8 Fixed, and final Avg@3 improves by 4.71 percentage points. Top-8 Adaptive also beats allocating two rollouts to each of four harnesses (LLM Router 4×2), and all rescue strategies beat GRPO (Pass@3 60.31, Avg@3 48.98).

• Complementarity is large but routing must act without outcomes. Under the same final AppWorld policy, individual siblings repair 11–15 base-harness failures, while the nine analyzed siblings repair 29 in union — coverage obtained only by executing each sibling once.

• Behavior changes under H₀ are specific, not just "more tool use." On AppWorld, schema lookup before first API use rises from 22.9% (base) / 38.2% (GRPO) to 67.6%, repeated execute code falls from 30.1% (GRPO) to 9.8%, and explicit pagination loops rise to 38.3%. On SWE-Gym, switching to assertions after blocked tests occurs in 48.5% of Harness-S cases versus 13.3% for GRPO.

• Qualitative transfer to held-out instances. Two AppWorld cases are traced: a set-intersection constraint learned from a successful H2 rescue (used at update 10) reappears on a held-out test instance where GRPO sends an overinclusive union; an account-recovery procedure learned from an H5 rescue on Venmo (used at update 3) transfers to an Amazon instance where GRPO leaves the password unchanged.

• The policy becomes less dependent on routing. The fraction of AppWorld tasks placing H₀ at Rank-1 rises from 1.53% to 22.52% after training, and the Rank-1 margin over H₀ shrinks from 14.21 to 0.76 points in Pass@3 and from 13.49 to 4.58 points in Avg@3.

• Gains persist under unseen harnesses but do not erase harness sensitivity. With no further updates, Harness-S (4B) achieves the highest Pass@3 across benchmarks under DSH H₀, Claude Code, and OpenClaw, and beats mixed-harness GRPO on the external harnesses — yet still scores below DSH H₀ on both external harnesses.

Methodology in Plain English

Think of the harness as the "interface kit" an agent uses: what system prompt it sees, how tools are described, how context is managed, how execution is governed. The authors start with one base kit (H₀) and grow a family of variants from it.

Step one — grow the family. They run the current policy on fresh training tasks, look at the failed trajectories, and cluster the failures into related root causes. An editor is then asked to modify a parent harness to address one such cluster. Candidate children are validated; anything executable and protocol-compliant is kept in a pool. Which parents seed the next generation is decided by a coverage rule: pick the small set of children that together solve the most distinct validation tasks. This preserves specialization instead of collapsing everything into a single "best" harness, and the pool is frozen before policy training begins.

Step two — train with selective rescue. During reinforcement learning, the policy first tries each task K times (K = 8) under the base harness. If it succeeds at least once, that group is used as normal. If every attempt fails, the group carries no useful learning signal — there is nothing to reinforce against. So the router picks a handful of auxiliary harnesses judged likely to help with this particular task, and the same policy makes one fresh attempt under each, starting from the task's initial state. Those rescue attempts, which often contain a mix of successes and failures, replace the dead group. The policy is then updated with GRPO on the mixed group, so it learns from its own actions under alternative execution support.

Step three — route intelligently and keep the routing fresh. The router works from "Harness Behavior Profiles": natural-language entries describing the task conditions under which a given variant helps or hurts relative to H₀, extracted by an LLM from paired validation trajectories. Several independent LLM voters rank candidate harnesses for the incoming task and the ranks are combined into a Top-k selection. Because a harness that helped early in training may hurt later, the profiles are periodically rebuilt from validation trajectories collected with the current policy (profile re-anchoring), and test outcomes are never used.

Step four — test the transfer claim. Everything is evaluated under the original H₀, so any gain must come from the policy having internalized behavior rather than from the auxiliary harness still being present. Separately, the router's top-ranked harness is applied per task at inference (Rank-1 routing) at the same executor rollout count, to measure how much additional benefit remains available if specialized support is allowed.

Setup. Models are Qwen3.5-4B and Qwen3.5-9B; benchmarks are AppWorld (320 train / 150 validation / 262 test), DBBench (1,600 / 200 / 300), and SWE-Gym (240 / 100 / 160). Metrics are Pass@3 and Avg@3. Main-table numbers average three independent randomized runs; all experiments use eight NVIDIA H800 GPUs. Baselines share the same optimizer and differ only in which trajectories are retained.

Why This Matters

The paper reframes harness engineering: instead of asking "which harness is best?", it asks "which harness is best for this task and this policy checkpoint, and can the policy learn from that without keeping the harness?" That shifts harnesses from deployment artifacts into a source of training data and exploration diversity — and it suggests that the expensive co-evolution loop of jointly redesigning harness and policy can be replaced by one offline evolution pass plus selective use.

Real-world applications:

• Software-engineering agents that must diagnose and fix repository issues; the SWE-Gym results and the observed shift toward assertion-based verification and post-blocked-test recovery map directly onto CI/debugging workflows.
• Enterprise application and API automation (AppWorld-style tasks): booking, messaging, account management — where the paper shows schema lookup, pagination, and read-back-after-write behaviors transferring to new instances.
• Database and analytics agents (DBBench), where the routed 9B policy exceeds the frontier-model references reported in the paper.
• Harness/platform teams building agent runtimes: the results argue for shipping a small pool of specialized execution modes plus a router, rather than one monolithic scaffold.

Industry relevance: the "policy remains effective under the original harness" property matters commercially, because harnesses are often fixed by a product integration or a vendor's runtime. If capability gains can be baked into weights rather than into a bespoke scaffold, teams can improve agents without rewriting deployment infrastructure — and can reuse a single evolved pool across successive model checkpoints, since the pool stays fixed during training.

Future Directions

• Lasting transfer and pool reuse across policies. The authors explicitly flag studying whether transferred behaviors persist over longer training and how the same fixed pool can be efficiently reused when the underlying policy changes.
• Cross-harness state exchange. They plan to investigate how independently developed harnesses can hand off task state across execution interfaces — a prerequisite for rescue attempts that continue rather than restart from the initial state.
• Harness Scaling. Measuring how additional repair diversity affects policy gains versus the cost of router calibration is left as planned work; the paper currently reports no scaling curve.
• Residual routing value. Rank-1 routing still retains a 4.58-point Avg@3 margin over H₀ after training, so what exactly the router contributes once the policy has internalized sibling behavior is left open; similarly, the policy still underperforms DSH H₀ on Claude Code and OpenClaw, so transfer is real but incomplete.

Target Audience

Researchers and engineers working on LLM agent training: reinforcement-learning practitioners who build agentic RL pipelines, harness/scaffold designers at agent-platform companies, and teams deciding between prompt/scaffold engineering and weight-level optimization. It is also useful for readers interested in curriculum and exploration design in sparse-reward settings, since the core mechanism is a principled way to manufacture within-task reward contrast. Readers without background in policy-gradient RL will find the empirical behavior analysis (Table 3) and the transfer case studies (Table 4) accessible, but the training machinery assumes intermediate-to-advanced familiarity.

Authors’ abstract

Harness evolution from a common base exhibits Locality, Complementarity, and Nonstationarity: variants repair different failures, and their utility changes as the policy learns. Rather than seeking a universally superior harness, we introduce Harness-S to selectively rescue failures of a base execution system through sibling harnesses and transfer the resulting capabilities into a single policy that remains effective under the original harness. Directional Harness Evolution develops specialized sibling harnesses along distinct repair directions, forming a reusable Sibling Harness Pool that remains fixed during policy training. Complementary Harness Rescue Training invokes this pool only when all initial attempts under the base harness fail. The same policy generates fresh rollouts under selected auxiliary harnesses, and these trajectories replace the failed group for reinforcement learning. An Adaptive Harness Router selects the Top-k variants using Harness Behavior Profiles and voting across multiple LLMs. Policy-aware profile re-anchoring refreshes this evidence as the policy learns. Across AppWorld, SWE-Gym, and DBBench, the trained policy substantially outperforms policy-only training when returned to the original base harness. Behavioral analyses and held-out cases support the transfer of harness-elicited behaviors into the policy itself. Reusing the router for Rank-1 inference provides additional gains at the same executor rollout count.