Your Policy Is Secretly a Harness: The Decision Geometry of Agent Interfaces

AI summary

Overview

Research area: Machine learning — specifically post-training and deployment of language-model agents, reinforcement learning theory (KL-regularized control), and agent interface ("harness") design and evaluation.

Technical level: Advanced. The paper develops formal control-theoretic results (soft Bellman equations, Gibbs-form optimal policies, a regularized performance-gap identity, Bregman divergences) and validates them with exact, small-scale post-training experiments plus LLM-based estimates on coding agents.

Scope (one sentence): The paper shows formally and empirically that a trained agent policy encodes the relative action preferences of the harness it was trained under, derives an exact decomposition of deployed return into a harness "ceiling" minus a policy "mismatch," and uses it to explain why frozen-model comparisons of harnesses mislead and when adaptation recovers the loss.

What This Paper Is About

Language-model agents are post-trained and deployed through a "harness" — the prompts, tools, and control logic that present a task to the model and execute its outputs. Practitioners routinely judge harness revisions by running a fixed, already-trained model under each candidate, which assumes what the model learned under one harness transfers to another. The authors ask what a policy actually learns from its harness, what survives when the harness changes, and how to measure the true cost of a harness change rather than conflating "what the harness offers" with "how well this particular checkpoint uses it."

Key Contributions

1. What a policy learns under a harness. The paper proves that the log odds of an optimal policy recover exactly the relative action scores the harness induces (Proposition 3.1, "behavioral identification"). On MiniWoB++, real post-training moves Qwen3.5-2B's action preferences toward its training harness's optimum and makes those preferences specific to that harness.

2. Why transfer fails. The score error of a policy moved to a new harness is split into three named terms — target shift, training residual, and readout drift — showing that a change in presentation alone can break a well-trained model even when every optimal decision stays the same.

3. What transfer costs and how to measure it. Deployed return equals the harness's ceiling minus the policy's mismatch exactly. From this the authors derive a mismatch threshold below which adapting to a revision that hurt the frozen model becomes a net gain (Corollary 5.2), and show ceiling and mismatch measured before adaptation predict which rejected swaps pay off (Spearman ρ = 0.95).

4. Measuring mismatch in deployment. A sampled estimator of mismatch is validated against known optima, and a teacher-model stand-in for the optimum orders five coding-agent harnesses by Terminal-Bench Lite accuracy exactly (Spearman ρ = 1.0) when ceiling minus mismatch is used, while either term alone misorders them.

Main Findings

• A policy is "secretly a harness." Under KL-regularized control the optimal policy is a softmax over scores of the form F*(s,a) = Q*(s,a) + τ log π_ref(a|s); a policy trained to optimality therefore carries the harness's relative action scores even though it contains none of the harness's code, tools, or execution machinery.

• Presentation-only changes destroy performance. Reversing the order of the observation fields leaves every optimal decision unchanged yet drops a trained model's success from 85.0% to 19.2%, changing its most probable action at 52.0% of states and moving mismatch M from 0.015 to 0.484. Milder variants: menu rendered as a list (9.2% choice change, 86.0% success), shifted codes (27.7%, 57.1%), reversed codes (30.0%, 68.2%), permuted codes (25.3%, 67.6%).

• Post-training sharpens preferences toward the training harness. Under every one of six harnesses (H0–H5) on three MiniWoB++ task families, geometry error falls 69% to 90% below its initialization level, as much on held-out tasks as on training tasks, and under every deployment harness the checkpoint trained under that harness has the lowest error.

• Frozen comparisons reject harnesses that adaptation later vindicates. In the batched-clicks revision (H0 → H1), the new harness raises the ceiling slightly but raises the frozen model's mismatch about ten times as much, so return falls. After equal-budget adaptation under H1, the adapted model beats both the source checkpoint and the equal-budget H0 continuation, with paired intervals excluding zero.

• Nine rejected swaps, seven recoveries. Across harness swaps, nine swaps are both rejected by a frozen comparison and have positive threshold T. After adaptation, the seven that raise the ceiling all beat the source pair; the two that lower it gain nothing. Three of the seven switch to H5, which only adds instructions — so a revision that changes no action can still raise the ceiling through the reference it induces.

• The frozen loss does not predict the eventual gain. H0 → H5 and H4 → H3 lose the same small amount when frozen, yet H0 → H5 ties for the largest gain and H4 → H3 gains nothing. The pre-registered prediction T − r₂ gets the sign of every gain right and orders the eight recorded gains with Spearman ρ = 0.95, versus ρ = 0.17 for the frozen return change.

• Sampled mismatch works. On large configuration changes (click reliability, click cost, prompt presentation), subtracting the sampled mismatch cuts prediction error about a hundredfold versus using the ceiling alone. On the small differences that decide whether adaptation pays off, the mismatch-based prediction is about ten times more accurate than a frequency rollout.

• Coding agents: only the difference ranks correctly. With a frozen Qwen3.5-9B across five harnesses on Terminal-Bench Lite, accuracy ranges from 47.9% ± 3.9 (DeepAgents) to 64.1% ± 5.1 (Terminus-2). Teacher-based ceiling alone gives ρ = 0.7, mismatch alone ρ = −0.7, and ceiling minus mismatch ρ = 1.0. Terminus-2 wins through the smallest mismatch despite a middle ceiling; DeepSeek Harness and mini-SWE-agent have the two highest ceilings but large mismatch — the batched-click profile before adaptation.

• Mismatch scales with the number of decisions. DeepAgents has a per-decision mismatch close to Terminus-2's, but the evaluated model makes nearly three times as many model calls under it, accumulating the largest total mismatch; DeepSeek Harness has the largest mismatch per decision.

• The same pattern holds with exact quantities. On MiniWoB++ with a frozen Qwen3.5-9B as its own reference under H0–H5, the ceiling alone ranks the most successful harness last and the mismatch alone ranks H0 below two less successful harnesses; only their difference orders all six correctly.

Methodology in Plain English

The authors first formalize what a harness does: it encodes a decision state into a context, decodes the model's response into an executed operation with arguments (a "semantic action"), and executes it. Because different text outputs can map to the same action, the same frozen weights express different semantic policies under different interfaces — that is the formal root of the problem.

They then set up the standard KL-regularized control problem each harness induces, where the reward is discounted and a penalty keeps the policy near a reference. In this setting the optimal policy has a known softmax form over action scores, which lets them prove that behavior identifies exactly the relative scores and nothing more.

To test this with real models they build a fully computable testbed: six harnesses (H0 does one native action per decision; H1–H5 add batched clicks, automatic checks, verification tools, or extra instructions) over three MiniWoB++ task families, compiled into finite graphs. Each prompt lists the complete action menu with one token per action, so the model's next-token distribution is the semantic policy exactly. They post-train Qwen3.5-2B with exact policy gradients against the frozen base model as reference, solve for each harness's optimum with a soft Bellman solve, and score checkpoints on held-out tasks.

The theoretical core is applying the regularized performance-gap identity to this harness-induced problem: return = ceiling − mismatch, where mismatch is the KL divergence from the optimal policy accumulated along the policy's own trajectories. This cleanly separates what a harness offers (weight-independent) from how well a checkpoint uses it. A corollary turns this into a concrete threshold for when adaptation to a harmful revision pays off.

Finally they move to deployment conditions where distributions and optima are unavailable: mismatch is estimated from action samples at states drawn from the discounted occupancy (rollouts truncated at a random time with probability (1−γ)γ^t), and for coding agents a stronger model (GPT-5.6-luna) stands in for the optimum, with actions mapped to categories shared across harnesses and the evaluated model used as its own reference.

Why This Matters

Impact on research. The paper reframes "prompt/interface sensitivity" from an empirical annoyance into a structural property of the control problem: the optimal policy is provably unchanged, so all the damage is attributable to the policy's readout. It also gives automated harness-search a principled objective — ceiling minus mismatch rather than frozen-model score — and supplies an exact, checkable decomposition of deployed return that connects agent engineering to established regularized-RL theory.

Real-world applications:

• Agent platform and harness engineering: deciding whether a new tool set, action-batching scheme, or prompt layout is actually better, rather than better for one particular checkpoint.
• Post-training pipelines: knowing that a model's learned preferences are harness-specific argues for training and deploying under matching interfaces, or re-adapting after interface changes.
• Evaluation and benchmarking: frozen-model harness leaderboards systematically penalize harnesses with high ceilings but high initial mismatch, and the paper shows how to correct for that.
• Deployment monitoring: mismatch can be estimated from sampled actions at runtime, using a teacher model in place of an unavailable optimum, giving a signal for when to re-adapt an agent.

Industry relevance. Teams shipping browser agents, coding agents, and tool-using assistants change prompts and tools constantly. The result that a purely cosmetic field reordering can drop success from 85.0% to 19.2% — while leaving every optimal decision intact — is a direct operational warning, and the ceiling-minus-mismatch criterion is a cheap way to avoid discarding harnesses that only look bad.

Future Directions

• Better stand-ins for the optimum. Coding-agent estimates rely on a stronger teacher (GPT-5.6-luna) that is not itself optimal, and the authors only test whether the estimates order harnesses; a principled treatment of teacher suboptimality, and tighter bounds than Proposition 6.1, remain open.
• Scaling beyond finite, solvable control problems. Exact ceilings and optima were computed only on MiniWoB++ finite graphs; extending exact ceiling-mismatch accounting to open-ended, long-horizon agent tasks is unresolved.
• Harness search with the new objective. Since ceiling and pre-adaptation mismatch predict post-adaptation gains (ρ = 0.95 across eight recorded swaps), using them as the search objective for automated harness design — instead of frozen-model scores — is a natural next step.
• Joint adaptation policies. Three of the seven beneficial swaps change no action at all (H5 only adds instructions) yet raise the ceiling through the induced reference, suggesting that understanding and exploiting reference-induced ceiling gains, and co-adapting harness and weights, deserves direct study.

Target Audience

Researchers and advanced practitioners in LLM agent post-training, reinforcement learning theory, and agent evaluation — particularly those who design or benchmark agent harnesses (prompts, tools, control logic) and want a formal account of why interface changes break trained models. It will also interest engineers building automated harness-optimization systems, who need an evaluation criterion that does not conflate a harness's potential with a single checkpoint's fit to it. Readers without comfort in KL-regularized control and soft Bellman equations will find the empirical results (85.0% → 19.2%, the nine-swap study, the five coding-agent harnesses) accessible, but the derivations require an advanced background.

Authors’ abstract

Language model agents are post-trained and deployed through a harness, the prompts, tools, and control logic that present a task to the model and execute its outputs. Harnesses are often revised and judged by how an already trained model performs under them, which assumes that what the model learned under one harness carries over to the next. We ask what a policy learns from its harness and what survives when the harness changes. Under KL-regularized control, we show that the log odds of an optimal policy recover how strongly its harness favors each action over the others, so a trained policy carries the preferences of its harness. Deployed return then equals the ceiling of the harness, the best return it offers, minus the mismatch of the policy, the return lost to its own suboptimal decisions. Training changes only the mismatch, while changing the harness moves both terms. Comparing harnesses with a fixed model therefore mixes what each harness offers with how well that model uses it. In MiniWoB++, post-training makes a model's preferences specific to its training harness. Reversing the order of the observation fields leaves every optimal decision unchanged yet lowers a trained model's success from 85.0% to 19.2%. Conversely, every new harness that a fixed-model comparison rejects but that raises the ceiling outperforms the original agent after further training under it, and ceiling and mismatch measured beforehand predict these gains. For coding-agent harnesses on Terminal-Bench Lite, ceiling minus mismatch estimated with a stronger model as a stand-in for the optimum reproduces the accuracy ranking, while either term alone does not. Code is available at https://github.com/2020-qqtcg/HRP.