HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

AI summary

Overview

Research area: Software engineering for LLM agents — specifically the agent harness (the execution infrastructure surrounding a model: execution loop, tool use, context management, persistent state, lifecycle control, and verification). Published under cs.SE (arXiv:2609.01437v1).

Technical level: Advanced. The paper is a benchmark and empirical study requiring familiarity with agent scaffolding, held-out evaluation, and token-cost accounting, though its central question is stated plainly.

Scope in one sentence: The paper introduces HarnessDev, a benchmark that asks whether LLMs can build a complete agent harness from a deliberately weak seed and then improve it using downstream execution feedback, measuring both task capability and execution-token cost.

What This Paper Is About

Most agent evaluations fix the harness and report how well a model does inside it, treating the surrounding execution system as part of the experimental setup rather than as something to be developed. HarnessDev flips that: it evaluates the runnable infrastructure a model produces, not the answer it gives on a single task. The goal is to measure whether a model can both create a persistent, reusable harness from a weak starting point and evolve it from execution feedback, while checking transfer to unseen tasks and to a different executor model.

Key Contributions

1. A benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages — Creation (building a harness from a weak seed plus 1–3 development cases) and Evolution (iteratively revising the creator's own frozen harness using downstream execution feedback) — with scores reported on two axes: capability (task success) and efficiency (executor-model tokens consumed).

2. Creation results spanning six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. The suite is SWE-bench Pro public split (731 tasks, task success), Terminal-Bench 2.1 (89 tasks, task success), MLE-bench (75 tasks, medal score), EQ-Bench3 (46 tasks, rubric score), and BrowseComp (1,266 tasks, accuracy). MLE-bench results cover 33 physical cells and 2,475 results.

3. An Evolution protocol with explicit budget and separation of feedback from generalization. The creator starts from its own frozen RQ1 code harness and receives results from a fixed 100-task SWE-Pro feedback set and all 89 Terminal-Bench tasks; the budget is ten post-H₀ full-evaluation pairs, with at most two fixed-subset probes (the same first five tasks of both benchmarks) between charged pairs. Every official version is later evaluated on a 630-instance SWE-Pro held-out split never shown to the creator.

4. A behavioral analysis of the produced artifacts — edit statistics, which mechanisms actually trigger during runs, dead code, response to feedback, and transfer across executors — plus a constraint-compliance audit of every run.

Main Findings

• Model-built harnesses trail mature human-engineered references on code and on search and research, but match or exceed them on writing and machine-learning experimentation. Under Self-Eval, Opus 4.8 has the highest overall score (67.8) but stays below the human-engineered reference average (86.2).
• The gaps are domain-specific. On BrowseComp the human harness + paired model reaches 92.2 accuracy versus Opus 4.8's 52.4; on SWE-bench Pro the reference is 80.0 versus Opus 4.8's 69.3; on Terminal-Bench 2.1 the reference is 88.8 versus Gemini 3.1 Pro's 68.8 (best generated result). The paper notes these human rows are system-level references from different harness–model combinations, not paired controls under one executor, and that SWE-Pro 80.0, Terminal-Bench 88.8, and BrowseComp 92.2 are external results not re-run in this experiment.
• Writing and MLE-bench are the reverse case. Opus 4.8 scores 84.6 on EQ-Bench3 against a human reference of 83.7, and Opus 4.8 and Gemini 3.1 Pro lead MLE-bench with medal rates of 32.9 and 32.4 against a human reference of 24.0.
• The seed harness scores zero everywhere. It is a runnable, policy-free compatibility layer with no execution loop, task decomposition, tool policy, context management, persistent state, verifier, retry/recovery logic, or stopping rule, so any nonzero Creation score comes from logic the creator added.
• Execution cost varies enormously and does not reliably buy quality. MLE-bench token use varies by about nineteen-fold, yet higher cost does not reliably produce a higher score; DeepSeek V4 Pro spends a mean of 1,449.9 million execution tokens on BrowseComp for 40.9 accuracy, while Opus 4.8 spends 593.9 for 52.4.
• A single generated harness is not representative. An Opus Code harness that does well under Self-Eval nearly collapses under Gemini because it hard-codes a 120-step limit around the original executor — its SWE-Pro score falls from 69.3 to 33.0. This motivates reporting avg@3 (three independent harnesses per creator–benchmark pair).
• Fixing the executor changes the ranking substantially. Qwen, Seed, and DeepSeek improve in several Data and Search settings under Gemini; Qwen gains 17.6 points on BrowseComp and 12.9 on MLE-bench. Opus shows the opposite: its Writing score falls from 84.6 to 74.2, and its Search harness's duplicate-query rate rises from 10.1% to 88.2%.
• Edit size does not predict performance. The 18 Code artifacts add 17,111 net lines in total, but Gemini adds the fewest lines (1,006) and obtains the best Terminal-Bench score (68.8); median per-artifact net LOC ranges from 324 (Gemini) to 1,339 (Qwen).
• State and memory are the clearest structural gap. All 18 Code harnesses implement an explicit execution loop, and tools, lifecycle control, and verification are complete in 13/18, 13/18, and 15/18 artifacts respectively. 11/18 define a State class, but only one exposes a state-saving interface and only one implements periodic checkpointing — and no checkpoint event appears in 26,679 recorded task trajectories. All 18 unobserved component instances (out of 108) concern state and memory; of 587 Writing features, 124 are confirmed dead code, and 36 Data mechanisms sit on dead paths.
• Validation is mostly syntactic. 441 of 2,325 executed Data tasks produce degenerate submissions that no harness detects. 77.8% of failed Data tasks are attributed to harness defects.
• Self-testing volume alone is a weak signal; acting on failures is stronger. Self-test count correlates with downstream score at only 0.13–0.26 (not significant), whereas revision calls reach 0.57 (p ≤ .0005).
• Evolution produces gains on the visible feedback pair for all five self-runtime creators, but these shrink on held-out tasks. Opus 4.8 has the largest held-out improvement at +4.44 points. Qwen

Authors’ abstract

As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.