Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents

AI summary

Overview

Research area: Software engineering benchmarks for large language model (LLM) coding agents, specifically the evaluation of "harness engineering" — the construction of the software layer that mediates between an LLM and its execution environment.

Technical level: Intermediate. The concepts (agents, tool use, containers, unit-test-style assertions) are accessible, though the paper assumes familiarity with LLM agent architectures and benchmark design.

Scope: The paper introduces Doc2Harness, a two-stage deterministic benchmark that measures how well nineteen frontier LLMs can construct a complete, working coding-agent harness from a written specification and an interface-only skeleton.

What This Paper Is About

LLM-driven coding agents are increasingly used on long-horizon tasks, and their performance depends as much on the "harness" around the model — the component that manages the interaction loop, tool execution, context, permissions, session state, failure recovery, and result verification — as on the model itself. Existing benchmarks measure downstream task success under a fixed harness, so the act of engineering the harness itself has gone unmeasured. Doc2Harness addresses this gap by directly evaluating whether a model can build a harness from documentation, then testing that harness as a black box.

Key Contributions

1. Doc2Harness, a two-stage benchmark protocol. In Stage 1, the build model under evaluation is placed inside a fixed builder harness (OpenCode) with a development document and a contract skeleton that declares interfaces only, and must construct a complete candidate harness inside an isolated container. In Stage 2, the frozen candidate harness is evaluated separately.

2. A black-box test suite with positive controls. The frozen candidate harness is driven by a scripted mock model, and each test case is scored through four observable channels: the requests sent to the model, the files on disk, the process exit code, and the command output. Positive controls are used to rule out "vacuous passes."

3. A test suite of 328 test cases built by three complementary synthesis methods — public issue mining, cross-harness comparison, and single-variable expansion — partitioned into nine mutually disjoint categories.

4. An evaluation of nineteen frontier models plus a source-code analysis of all fifty-seven scored candidate harnesses (three builds per model), linking harness architecture (module count, lines per module, largest-module share) to pass rate.

Main Findings

• The benchmark is far from saturated. Across nineteen models, each evaluated with three independent builds over 328 test cases, GPT-6 Astra achieves the highest pass rate at 53.05%, while the weakest model, Hy3, reaches 27.44% — a gap of roughly twenty-six percentage points. Even the strongest model fails nearly half of the test cases. The overall mean across the nineteen models is 38.30%.

• Top two models diverge on boundary scenarios. On ordinary scenarios, Opus 5 leads with 80.68% against 72.46% for GPT-6 Astra. On boundary scenarios, the order reverses: GPT-6 Astra leads at 48.97% while Opus 5 falls to 37.02%.

• Categories differ sharply in discriminative power. Permissions provides the greatest separation: average pass rate 43.7%, a strong-weak gap of 50.8 points, with 51 of its 61 test cases discriminating. File operations has a comparable average pass rate (39.6%) but a gap of only 20.9 points. Interactive terminal averages 79.9% and is approaching saturation. Artifact checks achieves 100.0% average pass rate with a 0.0 gap, and extension discovery is 0.0% with a 0.0 gap.

• Fifty-six test cases are failed by every candidate harness, providing no discrimination within this cohort, and sixteen test cases are passed by every candidate harness. The remaining 256 are discriminating. The floor cases are concentrated in direct drive, session state, permissions, and headless.

• Model ranks vary widely across categories. Among the six main categories, a model's rank varies by as many as sixteen positions. Grok 4.7 ranks first on file operations but eleventh on session state; Opus 4.6 ranks third on direct drive but nineteenth on permissions; Hy4 preview ranks fifth on session state but nineteenth on direct drive. GPT-6 Astra leads four of the six categories, while Grok 4.7 and Qwen3.8 Max lead the remaining two.

• Categories are positively associated rather than trading off. All fifteen pairwise correlations between the six main categories are positive, with a mean of +0.521, and the overall pass rate correlates with all six (five between 0.69 and 0.94). This is consistent with a largely one-dimensional capability.

• The test suite is internally consistent. The KR-20 coefficient computed over the fifty-seven candidate harnesses across all test cases is 0.913.

• Harnesses with more modules score lower. Controlling for code lines, the partial correlation between module count and pass rate is −0.423, holding between −0.484 and −0.309 under a leave-one-model-out check. Lines per module shows a partial correlation of +0.504, holding between 0.312 and 0.649. The paper states that failures cluster around how cross-cutting functionality is split across modules.

• Implementation scale is set mainly by the model. Per-model mean code size ranges from 1,271 to 16,681 lines; across the fifty-seven builds the range is 1,062 to 19,208. 95.9% of the variance in code lines is between models, and the coefficient of variation across a model's three builds averages 12.7%, compared with 60.1% across the nineteen model means. Between-model variance shares are 87.9% for module count, 80.6% for lines per module, and 95.7% for largest-module share. The correlation between code lines and pass rate is 0.614 across the fifty-seven builds and 0.629 between model means.

• Outlier in architecture: GPT-5.5 produced the smallest harnesses in the table at 1,271 code lines across 2.7 modules, with its largest module holding 78% of all code lines, and scored 34.65%.

Methodology in Plain English

Stage 1 — Construction. The model being tested is plugged into a fixed, off-the-shelf coding-agent framework (OpenCode 1.17.0) so that every model works under the same building conditions. That framework is given two things: a "development document" written in ordinary language, and a "contract skeleton" — a minimal scaffold of five Python source files (roughly seventy lines total) plus an install script, an entry script, and a project config, stripped down from mini-swe-agent so that only interfaces remain and no implementation is included. The development document was assembled from the user documentation of twelve frontier harnesses (Claude Code, Kimi Code, Gemini CLI, Qwen Code, OpenCode, OpenClaw, Codex CLI, Aider, OpenHands, Cline, Crush, and Cursor), with overlapping commands merged and conflicting names split apart. The model works in a fresh empty container with no external network access except the designated model gateway, and submits its work for a "contract check" that verifies form rather than behavior — that the deliverable compiles and installs, that all required commands are present, that no placeholder stubs remain, and that the three contract interfaces are in place. If the check fails, the deliverable is returned for revision. Each build allows at most three submissions and a two-hour wall-clock limit.

Stage 2 — Evaluation. The frozen candidate harness is copied into a separate container for each test case, with its own fixture (the files and initial state prepared in the workspace before a run). A mock model — not a real LLM — drives it following a preset script, so the evaluation is deterministic and free of the instability a real model would introduce. Each test case runs one complete session under a 900-second limit. Effects are observed through four channels: the requests sent to the model, the files on disk, the process exit code, and the command output.

Test case construction. Three methods were used in alternation. (1) Public issue mining: every issue from the public repositories of Claude Code (9,868), OpenCode (5,438), Gemini CLI (4,831), and Kimi Code (946) — 21,083 in total after filtering out pull requests — ranked by relevance, reviewed manually from the top, and clustered into candidate test cases. (2) Cross-harness comparison: partitioning a candidate harness's source and a mature reference harness's source onto a common set of functional boundaries and enumerating implementation differences. (3) Single-variable expansion: taking a test case already shown to discriminate and changing exactly one scenario variable — the tool, the input form, the erroneous parameter, or the site where the result is returned.

Positive controls. A test case can pass simply because the feature under test never ran — a "vacuous pass." For example, a path-escape test asserting that an out-of-bounds file's content never appears in model requests may pass because the implementation blocked the read, or because its read tool cannot read any file at all. Positive controls share the scenario and fixture but change only the input or access route, asserting an outcome that must occur when the observation precondition holds. Of the 328 test cases, 130 are gated by at least one positive control: 114 by one, 15 by two, and one by three. The widest-scope control gates six test cases, and the next two gate five each. If a positive control fails, its test cases are recorded as FAIL regardless of their own status. Positive controls do not count toward the pass rate.

Why This Matters

Impact on research. Prior work on harness quality — including Claw-SWE-Bench and Harness-Bench — infers harness quality from downstream task success, which entangles the harness with the ability of the model driving it and, in the case of self-improving harness work, with the confound of increased test-time search. Doc2Harness isolates the construction of the harness itself, judges it clause by clause against a specification, and uses a scripted mock model so the measurement is deterministic. The authors state it is the first deterministic benchmark for measuring long-horizon harness engineering, and note that output-only scoring cannot detect failures where a harness returns a correct answer along a trajectory that reached unauthorized resources.

Real-world applications:

• Agent framework development. Teams building or selecting coding agents can use harness-engineering scores as a signal for which models can build and maintain agent infrastructure, not just solve tasks inside it.
• Recursive self-improvement research. An agent that modifies the infrastructure governing its own behavior — execute, observe, modify the harness, reuse it — depends on the generalization Doc2Harness tests.
• Model and vendor evaluation. Vendors can compare models on construction capability separately from downstream task success, and the paper's note that model rank moves by up to sixteen positions across categories argues for category-level reporting.
• Specification and documentation engineering. The development document's design (clauses with syntax, meaning, and expected behavior, disambiguated across twelve sources) is a reusable template for specifying harness behavior in a verifiable, implementation-independent way.

Industry relevance. Because harness choice has been shown to shift scores substantially (Claw-SWE-Bench reports a 27.4-point swing for harness choice versus 29.4 for model choice), knowing which models can build harnesses well matters for teams deciding whether to adopt, fork, or generate their own agent infrastructure.

Future Directions

• From single build to build–evaluate–repair loops. The paper's own limitation section notes that each build allows up to three submissions but is not an iterative build–evaluate–repair loop, and that the contract check tests form rather than behavior. The pass rate of an early submitter is therefore a lower bound rather than an upper bound. A protocol that feeds behavioral feedback back into construction remains unmeasured.

• Reducing correlated failures across test cases. The authors note that a single defect can fail several test cases at once, because single-variable expansion derives several variants from one seed test case. Whether weighting or restructuring the suite around independent defects would change the rankings is an open question.

• Explaining the fragmentation penalty. The finding that more modules correlate with lower pass rate (partial correlation −0.423) and more lines per module with higher pass rate (+0.504) is correlational. Whether the relationship is causal, and what architecture guidance follows from it, is not established.

• Closing the coverage gaps. One test case (extension discovery) is failed by every candidate harness and 56 test cases provide no discrimination within this cohort. The paper notes that a clause with no anchored test case cannot serve as a seed for single-variable expansion, so such gaps can only be filled by the other two synthesis methods.

• Category-specific capability. Given that all fifteen pairwise category correlations are positive with a mean of +0.521 but ranks still shift by up to sixteen positions, whether the remaining variation reflects genuine specialization or noise is unresolved.

Target Audience

This paper benefits researchers and engineers working on LLM agents and coding assistants, benchmark designers interested in deterministic and black-box evaluation of software artifacts, and practitioners who build or maintain agent harnesses and need to decide which models to use for that work. It is also relevant to teams studying recursive self-improvement, since harness engineering is the mechanism by which an agent can modify the infrastructure governing its own behavior. Readers should be comfortable with LLM agent architectures, containerized execution, and benchmark scoring, though the paper's core ideas are explained without requiring deep implementation expertise.

Authors’ abstract

Agents driven by large language models are increasingly deployed on long-horizon tasks, where they use tools and modify real workspaces. How well they perform depends as much on the harness around the model as on the model itself. Existing benchmarks measure downstream task success under a fixed harness, leaving harness engineering itself unexplored. We propose Doc2Harness, a two-stage benchmark that evaluates long-horizon harness engineering directly. In Stage 1, a build model operating inside a fixed builder harness receives an interface-only contract skeleton and a development document that consolidates behavior across twelve frontier harnesses, and then constructs a complete candidate harness in an isolated container. In Stage 2, the frozen candidate harness runs in containers separate from the construction environment, one per test case, and is driven end to end by a scripted mock model. We score each test case using four observable channels, namely the requests sent to the model, the files on disk, the process exit code, and the command output; positive controls rule out vacuous passes. Across nineteen frontier models, each evaluated with three independent builds on 328 test cases, GPT-6 Astra achieves the highest pass rate at 53.05%, while the weakest model reaches 27.44%, a gap of roughly twenty-six percentage points. Even the strongest model fails nearly half of the test cases, and failures cluster around how cross-cutting functionality is split across modules. To our knowledge, Doc2Harness provides the first deterministic benchmark for measuring long-horizon harness engineering.