Harness Scaling for Generalizable Agentic Reinforcement Learning

AI summary

Overview

Research area: Reinforcement learning for large language model (LLM) agents — specifically, how the execution "harness" around a model (system prompt, tool interface, observation format, context management, workflow) affects what an agent learns and how well it generalizes.

Technical level: Advanced. The paper assumes familiarity with policy-gradient RL (PPO, GRPO), agent scaffolds, and factorial experimental design (orthogonal arrays, the Rao bound). The core intuition is accessible, but the method and ablation tables require background.

Scope: The paper proposes "Harness Scaling" — training an agent policy across a structured, factorized space of harness configurations rather than a single fixed harness — and evaluates cross-harness generalization across three benchmarks, two model scales, and two RL algorithms.

What This Paper Is About

Modern LLM agents operate inside a "harness" — the layer that supplies instructions, exposes tools, formats observations, manages context, and dictates workflow. Existing agentic RL typically optimizes a policy through one fixed harness, which can overfit the model to that specific execution setup. The authors ask whether training across many harness configurations (while keeping the task and environment the same) produces agents that generalize to harnesses they have never seen, including external black-box harnesses like Claude Code.

Key Contributions

1. A new scaling dimension. The paper introduces Harness Scaling as distinct from task scaling (what problems agents solve) and environment scaling (where they operate): it scales how the policy executes and interacts with the same task and environment.

2. A structured framework for harness diversity. Harness variation is factorized into five components (system prompt, tool interface, observation, context management, workflow), each with three realizations, giving a 3^5 = 243-configuration space. A strength-2 orthogonal design selects a compact 18-configuration training set with balanced component-level and pairwise coverage.

3. Locality-aware cross-harness training. Each task is paired with K configurations: a uniformly sampled anchor plus its K−1 nearest neighbors under Hamming distance, combining broad global coverage with controlled local variation per task. The procedure is applied unchanged to both PPO and GRPO.

4. Empirical evidence of cross-harness generalization. Across three benchmarks, two Qwen3 model scales, and two RL algorithms, Harness Scaling beats single-harness RL on held-out white-box configurations and transfers better to the unseen Claude Code black-box harness than training directly through an OpenClaw black-box harness.

Main Findings

• Better combinatorial generalization in-space. On 18 held-out harness configurations, averaged over three benchmarks, Harness Scaling gains +9.80 and +3.57 points over single-harness RL with GRPO and PPO on Qwen3-4B-Instruct, and +5.21 and +4.90 points on Qwen3-30B-A3B-Instruct.

• Consistent across model scale and algorithm. Overall averages for Qwen3-4B-Instruct rise from 29.81 to 39.04 (GRPO) and from 33.91 to 36.93 (PPO); for Qwen3-30B-A3B-Instruct from 37.29 to 42.78 (GRPO) and from 36.31 to 41.89 (PPO).

• Transfers to unseen external black-box harnesses. On Claude Code, Harness Scaling reaches average Pass@1 of 37.22 (Qwen3-4B-Instruct) and 41.20 (Qwen3-30B-A3B-Instruct), beating black-box SFT (35.08 and 38.08) and black-box RL (33.38 and 36.28) that were trained directly through OpenClaw.

• More harnesses is not automatically better. A random 18-harness set scores 47.90/42.78 on ClawGym-Bench and 51.28/48.74 on PinchBench (in-space/Claude Code) versus 52.78/45.37 and 56.24/51.47 for the structured set. The exhaustive 243-harness setting — which even includes the white-box evaluation configurations — still performs worse than the structured 18-harness set under the same training budget.

• Three-harness first-order coverage already helps, but structure adds more. Three harnesses covering all component realizations yield +11.43/+4.84 points on ClawGym-Bench and +3.44/+7.86 on PinchBench; the structured 18-harness set adds a further +4.02/+1.57 and +7.20/+2.66 respectively.

• Per-task harness diversity matters, but saturates. K = 1 (different tasks get different harnesses) already beats single-harness RL; K = 2 improves further and consistently beats a matched "Repeat-2" control that reuses the same harness twice; K = 4 gives no further improvement, with K = 2 best overall.

• Locality-aware assignment beats random assignment. Random assignment loses 2.32/2.20 points on ClawGym-Bench and 3.04/1.24 on PinchBench (in-space/black-box).

• Gains are not paid for on the base harness. For Qwen3-30B-A3B-Instruct with GRPO on ClawGym-Bench, base-harness performance changes by only −0.19 points, while in-space rises +10.48 and Claude Code +3.60.

• Behavioral shifts concentrate in execution skills. Analyzing 200 trajectories per evaluated harness, the largest improvements over single-harness RL appear in state tracking, workflow compliance, and error recovery; instruction following and interface handling also improve, with a relatively smaller gain in observation reading.

• Rewards become less harness-sensitive. Comparing paired per-task rewards under two held-out harnesses differing in four components (Harness A = S0T2O1C1W2, Harness B = S2T1O1C2W0), Harness Scaling shifts points toward the upper-right and aligns rewards more closely across configurations.

Methodology in Plain English

Start with one agent setup and break the "harness" into five independent knobs: how the system prompt is phrased, how tools are named and exposed, what extra metadata wraps tool outputs, how much conversation history is kept, and whether the agent plans first or works free-form. Give each knob three settings. That yields 243 possible harnesses for the exact same task and environment.

Training on all 243 is wasteful and, per the ablations, not even better. Instead the authors borrow a technique from experimental design: pick a small set of combinations where every setting of every knob appears equally often, and every pair of settings across two knobs also appears equally often. With five knobs and three settings, the arithmetic forces a minimum of 18 configurations, and those 18 become the training set.

During training, each task is randomly assigned one of those 18 harnesses as an anchor, plus one nearby harness (differing in as few knobs as possible), so the same problem is seen twice under slightly different execution conditions. Random anchors spread tasks across the whole training space; nearest-neighbor pairing keeps the within-task variation controlled. Rollouts from these task–harness pairs feed a standard clipped policy-gradient objective — no change to PPO or GRPO themselves, only to which task–harness pairs generate the data.

Evaluation then happens on 18 different harnesses that never appear in training, and on Claude Code, an external harness outside the constructed space entirely. Harness variants in Appendix A were generated by a model-driven transformation procedure using Opus-4.7, with each transformation touching only its target component.

Why This Matters

Impact on research. The paper reframes agentic RL scaling: beyond collecting more tasks or more environments, one can scale the execution configuration for a fixed task. It also shows that organization matters as much as quantity — an exhaustive 243-harness run underperformed a structured 18-harness run — which is a caution for anyone assuming "more diversity is better."

Real-world applications:

• Portable agents: models that keep working when a vendor changes tool names, prompt phrasing, or output formatting between releases.
• Cross-platform deployment: training once and deploying into third-party agent runtimes (the paper tests Claude Code and OpenClaw) without per-platform retraining.
• Robust enterprise workflows: agents handling professional tasks over heterogeneous workspace files, where context-compaction and workflow policies differ by product.
• Tool-heavy assistants: systems where distractor tools, opaque tool identifiers, and noisy observation metadata are common in production but rare in training.

Industry relevance. Companies building agent products typically train against their own harness and then face degradation when users or partners plug the model into a different scaffold. Structured harness diversity offers a training-time recipe for reducing that brittleness, and the result that structured white-box training beats direct black-box training is directly relevant to teams without access to a target harness's internals.

Future Directions

• Extending beyond five components and three realizations. The paper's design space is 3^5 = 243; whether the orthogonal-design approach scales to larger or differently structured spaces (and how the Rao bound constrains it) is open.

• Better organization schemes than strength-2 balance. Since the exhaustive 243-configuration setting underperformed the structured 18, the question of which higher-order structure (rather than just pairwise balance) best predicts transfer is unresolved.

• Understanding why K saturates. K = 2 beat K = 1 and K = 4, and the mechanism behind this ceiling — budget effects versus redundant signal — is not settled by the reported ablations.

• Broader out-of-space validation. Only Claude Code and OpenClaw are used as external black-box harnesses; whether the transfer result holds across a wider range of production scaffolds remains untested in the reported content.

Target Audience

Researchers and engineers working on agentic reinforcement learning, LLM agent infrastructure, and post-training for tool-using models. It is most useful to readers who already understand policy-gradient RL and agent scaffolds and want a concrete, budget-aware recipe for improving robustness to execution-environment changes — and to practitioners deciding whether to train against one production harness or a designed family of them.

Authors’ abstract

Agent harnesses have become a central component of modern agentic systems, shaping how models use tools, manage context, and coordinate long-horizon interaction. Realizing the benefits of these harnesses depends on how effectively the underlying model can make use of them, motivating reinforcement learning directly through the harness. However, existing approaches typically train through a single fixed harness, which can lead to specialization and limited transfer across harness configurations. To address this, we introduce \textbf{Harness Scaling}, a new scaling dimension for agentic RL that extends training from a single harness to diverse execution configurations. Concretely, we construct a factorized harness space, select a compact training subset with balanced factorial coverage, and organize cross-harness training to combine global coverage with controlled variation for the same task. Across multiple benchmarks, model scales, and RL algorithms, Harness Scaling consistently improves generalization to unseen white-box configurations. Moreover, models trained through structured white-box variation transfer effectively to unseen external black-box harnesses, outperforming models trained through a single complex black-box harness. These results establish harness diversity as a meaningful scaling dimension for agentic RL and highlight the importance of how this diversity is organized for cross-harness generalization.