The Pulse
Study Finds No Clear Coding Gain From Vendor Agent Harnesses
A private benchmark finds no statistically clear average coding advantage for vendor-native agent harnesses over a neutral alternative, while documenting configuration differences that limit the comparison.
AI.info Team ·
A new study finds no clear average coding advantage for vendor-native agent harnesses over a neutral alternative, challenging a common assumption in evaluations of autonomous coding systems.
The paper, “Harness or Model? Isolating the Harness Effect in Agentic Coding with a Contamination-Controlled Private Suite”, was revised and published on arXiv on September 8, 2026. Mohsen Arjmandi compares harnesses while holding the underlying model constant, separating the software that manages tools, prompts, context and control flow from the model that generates the code.
For Claude Opus 4.8, the vendor-native Claude Agent SDK solved 48.8% of tasks, compared with 50.0% for the neutral DeepAgents harness. For GPT-5.5, the OpenAI Codex SDK reached 55.6%, against 54.4% for DeepAgents. Neither difference is statistically distinguishable from zero on the study’s selected task pool.
Same Models, Different Harnesses
The experiment uses a private suite of 256 tasks designed to limit contamination from public benchmark data. The suite contains 179 repository tasks drawn from four production codebases and 77 programming-contest tasks published after the study’s eligibility cutoff.
The main comparison uses the same 80 tasks for each model-harness pair. Every run takes place in its own KVM microVM, with identical repository seeds and prompts. The paper discloses configuration differences that remain: the Codex/GPT-5.5 cell ran in 4 GiB VMs, while DeepAgents ran in 8 GiB VMs. Inference budgets and tool surfaces also differed. An append-only event ledger records the sessions, while a separate Docker-isolated grading system evaluates the resulting patches against hidden tests.
The study planned 800 main-run keys across six experimental cells. The grading oracle produced 792 verdicts: 380 passes, 404 failures, seven timeouts and one apply error. Eight planned runs ended as infrastructure failures and were not graded.
Opus Results Split by Workload
The overall Opus result hides a sharp difference between task categories. On 61 repository tasks, the native harness trails DeepAgents by 9.0 percentage points. On 19 contest tasks, the native harness leads by 23.7 points.
A permutation test gives that interaction a p-value of 0.003, but the paper treats the pattern as exploratory rather than conclusive. The repository-versus-contest partition was chosen after the data had been inspected, so the author says a study designed around that distinction would be needed to test it properly.
The selected pool also does not represent a random sample of the full 256-task suite. Twenty-four tasks on which the two vendor-native systems disagreed during screening account for 30% of the final 80-task pool. The design therefore concentrates on tasks that already appeared sensitive to harness choice.
Correct Patches Do Not Always Mean Completion
The results separate two outcomes that coding-agent benchmarks often combine: whether an agent produces a correct patch and whether it finishes autonomously before a time limit.
The runs had a 1,200-second wall-clock ceiling. Of 81 runs cancelled at that limit, 22 had already produced patches that passed the grading oracle. A timeout therefore did not always mean that the generated code was wrong; in some cases, the harness failed to complete the session despite leaving behind a correct result.
That distinction matters for teams choosing systems based on operational behavior rather than pass rate alone. Latency, cancellation policy and the handling of partially completed work can change the practical value of a harness even when final task accuracy is similar.
The Cost Result Was Rewritten
The revised paper also retracts part of its earlier cost analysis. A telemetry normalizer in the August 2026 manuscript treated cache-token fields as if all SDKs used the same accounting convention. The author says that mistake distorted the original cost figures.
After recalculating from raw per-turn usage at frozen list prices, the neutral harness costs between 1.3 and 1.6 times as much per solved task as the native harness for Opus 4.8. The corresponding estimate for GPT-5.5 is 1.2 times as much. Those figures are observed-usage estimates rather than confirmed billed totals.
On the Anthropic account, 58 runs ended without a usage record. Assigning the missing spend entirely to either harness would move the Opus cost ratio from 0.7 to 2.3, leaving the billed ordering unresolved. The paper releases the orchestrator, grading oracle, reanalysis code and derived aggregates, but keeps the task suite private.
What the Study Can—and Cannot—Show
The findings do not establish that harness engineering has no effect on coding agents. They show that, under this design, a vendor’s own harness does not produce a reliable average advantage when compared with DeepAgents on the two tested models.
They also show why model-only rankings can be incomplete. A harness determines how an agent receives tools, manages context, handles truncation, responds to failures and decides when to stop. Those choices can affect cost and completion behavior even when paired solve rates remain close.
The strongest result is narrower: for the selected 80-task pool, the native Claude harness scores 1.25 points below DeepAgents on Opus 4.8, while the native OpenAI harness scores 1.25 points above DeepAgents on GPT-5.5. Both confidence intervals include zero.