Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement

AI summary

Overview

• Research area: Autonomous software development with LLM-based coding agents; agent harness design, multi-agent planning–coding–testing loops, and long-horizon software engineering.
• Technical level: Advanced.
• Scope: The paper introduces Harness-of-Harness (HoH), a framework that wraps existing coding-agent harnesses in iterative planning–coding–independent-testing loops so that agents can build complete software from high-level requirements over many iterations, evaluated on three benchmarks and a multi-day game-development deployment.

What This Paper Is About

Most LLM coding agents still work with a human in the loop: a developer defines tasks, reviews changes, and intervenes when something fails. This paper asks whether agents can instead take only high-level requirements and autonomously produce complete, functional, deployable software, without further human guidance. The core problem is that such development spans very long trajectories, where agents forget earlier requirements and decisions, apply local fixes that break other parts, repeat inspection and repair without progress, or declare completion while functionality is still missing or wrong. HoH addresses this by organizing development into repeated bounded increments with independent verification and carried-forward project state.

Key Contributions

1. Harness-of-Harness (HoH) framework. A framework that operates on existing coding-agent harnesses and organizes their executions into iterative planning–coding–testing loops, without prescribing a rigid agent workflow.
2. Cross-loop state management. HoH maintains two complementary states across loop boundaries: the software artifact state A_t (code, configuration, resources, metadata) and the execution evidence state E_t (verified behaviors, unsupported claims, unresolved failures), so development is both incremental and evidence-guided.
3. Separation of roles under a deterministic Runtime. Objective selection is assigned to a Project Planner, artifact modification to a Developer (single-writer boundary), and independent acceptance to a QA Tester that receives a frozen, read-only candidate and performs complementary white-box and black-box tests.
4. Empirical validation across three benchmarks and three harness–model pairs, plus a multi-day open-ended deployment. HoH is tested on GameCraft-Bench, FrontierSWE, and ProgramBench with Codex + GPT-5.5 (high), OpenCode + DeepSeek-V4-Pro, and Pi + MiniMax-M3, and additionally used to autonomously build a first-person-shooter game over more than 70 iterations.

Main Findings

• Consistent gains over standalone harnesses. After three iterations, HoH outperforms Vanilla on all three benchmarks under all three harness–model configurations. Reported absolute gains after three iterations are 16.62–22.08 points on GameCraft-Bench, 19–29 points on FrontierSWE, and 6.09–16.85 points on ProgramBench, with an average relative gain of 52.25% and a maximum gain of 82.86%.

• GameCraft-Bench Overall scores. Codex with GPT-5.5 (high) rises from 49.58 (Vanilla) to 71.52 (HoH@3, +21.93); OpenCode with DeepSeek-V4-Pro from 26.90 to 48.98 (+22.08); Pi with MiniMax-M3 from 42.16 to 58.78 (+16.62).

• FrontierSWE results. Reported mean reward increases from 0.31 to 0.54 (Codex), 0.23 to 0.31 (OpenCode), and 0.26 to 0.55 (Pi). Cross-configuration Dominance for Codex rises from 44% under Vanilla to 58%, 60%, and 71% at HoH@1–3, a 27 percentage-point gain.

• ProgramBench results. Average Test Pass Rate rises from 60.41 to 66.50 for Codex (+6.09), 45.27 to 57.56 for OpenCode (+12.29), and 35.83 to 52.68 for Pi (+16.85).

• Gains hold for both strong and weak starting configurations. Codex has the strongest Vanilla scores and still gains 21.93 points on GameCraft-Bench; OpenCode starts from the lowest Vanilla scores on GameCraft-Bench and FrontierSWE yet reaches 48.98 and 0.31, and even exceeds Codex Vanilla in the Action and Simulation groups on GameCraft-Bench and in Performance on FrontierSWE.

• Improvement accumulates across loops. GameCraft-Bench Overall increases monotonically from HoH@1 to HoH@3 for all three harness–model pairs; for OpenCode the gain over Vanilla grows from 1.71 points at HoH@1 to 13.42 at HoH@2 and 22.08 at HoH@3.

• Extended ten-loop run on FrontierSWE. Continuing Codex with GPT-5.5 (high) to HoH@10, the paper reports improvement from 22% to 72.67% over ten iterations. In a separate analysis using a fixed 11-checkpoint comparison pool of Vanilla and HoH@1–10, Dominance rises from 39.33% at HoH@3 to 72.67% at HoH@10 and reaches 76.00% at HoH@9, while Vanilla obtains 27.33% in that pool.

• All four GameCraft-Bench rubric components improve. HoH@3 improves Core Mechanics, Content Depth, Functional Visuals, and Art and Presentation under every configuration, with gains of 20.00–25.56 points (Codex), 19.25–34.63 (OpenCode), and 11.32–25.38 (Pi). For Codex, Functional Visuals rises from 48.67 to 74.23 and Art and Presentation from 45.28 to 65.28.

• Gains are not just "more passes." Against Vanilla Continuation with the same model and inference settings on GameCraft-Bench, HoH scores 59.71, 64.84, and 71.52 at one, two, and three development passes, versus 49.58, 54.99, and 58.24 for Vanilla — gains of 10.13, 9.85, and 13.28 points. HoH@2 reaches 64.84 using 5.67M mean cumulative tokens, exceeding the 58.24 obtained by three-pass Vanilla Continuation with 6.33M tokens.

• Ablation results (GameCraft-Bench, Codex + GPT-5.5 high, full HoH@3 = 71.52, 8.41M tokens). Removing plan update drops the score to 63.39 (−8.13, 7.56M tokens); removing evidence feedback to 65.23 (−6.28, 7.46M); removing warm-start to 63.67 (−7.85, 11.12M).

• Qualitative game improvements. For Momentum Lab, Kitchen Rush, and Ant Empire, Overall scores increase from 34.05 to 70.61, 42.63 to 73.38, and 65.52 to 87.88 respectively between Vanilla and HoH@3.

• Multi-day autonomous deployment. Over a multi-day deployment with more than 70 iterations, HoH autonomously develops a first-person-shooter game with a coherent storyline, fully implemented core mechanics, a human-playable experience, polished visuals, and integrated audio. Code changes and role-specific artifacts are committed to a public GitHub repository after each agent stage. The provided content does not report a formal human user study of the resulting game.

Methodology in Plain English

The researchers treat an existing coding agent (the "harness" plus its model) as a fixed component and wrap a repeatable loop around it. Each loop has three steps performed by the same harness–model configuration invoked in three different roles:

1. Project Planner reads the high-level specification, the accumulated evidence from previous loops, and the current code (read-only), then writes a development document defining one small but locally complete increment, plus the functionality that must be preserved and observable conditions for validating it. This keeps changes bounded and verifiable instead of collapsing into endless local repair.
2. Developer is the only role allowed to write to the project, warm-starting from the existing artifact and using a baseline–change–retest cycle (shift-left testing) so failures surface close to the changes that caused them.
3. QA Tester evaluates a frozen, read-only copy of the candidate using black-box tests (user-visible behavior, flows, state transitions) and white-box tests (source, configuration, resources, logs). Criteria are derived from the specification and the current plan rather than being fixed. Observed failures and insufficient evidence are recorded as gaps, not assumed successes, and form the evidence state passed to the next loop.

A deterministic Runtime controls each role's inputs, permissions, write access, and required structured output schema; outputs that violate the schema trigger a retry. Roles are constrained by required artifacts and outcomes, not by prescribed tool sequences. To manage context, HoH uses progressive disclosure instead of a memory module: plans, reports, and histories are persisted to the file system and surfaced through a concise categorized index, with details retrieved on demand. Tools (such as MCP servers, expert models, domain algorithms) are organized by role, with lightweight Markdown skills for on-demand guidance, and agents are encouraged to reuse existing resources. Versioned project histories let HoH return to previously verified states after regressions and reuse earlier evidence when similar failures recur.

Evaluation setup. Three benchmarks were used: GameCraft-Bench (140 tasks across 15 game families; 45 tasks sampled via stratified random sampling with three tasks per family and a fixed random seed; families grouped into Action, Timing, Strategy, Simulation, and Adventure), FrontierSWE (15 of 17 tasks: 4 Implementation, 9 Performance, 2 Research), and ProgramBench (cleanroom program reconstruction from a compiled executable and documentation). Vanilla performs one standard development pass; HoH@T performs T planning–coding–testing iterations with artifact and evidence carried forward; the main experiments use T = 3. Artifacts are evaluated only after the complete run, and evaluator outputs are never returned to the development loop. For the open-ended setting, HoH was additionally equipped with role-specific tools and skills for engine interaction, asset acquisition and generation, reference retrieval, testing, and project-state management.

Why This Matters

Impact on research. The paper reframes autonomous software development as a problem of sustaining coherent progress over long horizons rather than simply extending execution length. It positions the harness, and specifically the structure of repeated loops with independent verification, as the object of improvement — a distinct angle from work that optimizes prompts, memory, or multi-agent orchestration, and from harness-synthesis approaches such as AutoHarness, Meta-Harness, and Self-Harness. The paper also argues that existing coding harnesses organize development within a bounded episode with limited support for preserving decisions, verified functionality, and evaluation evidence across revisions, and it supplies a concrete alternative with ablation evidence for each mechanism.

Real-world applications:

• Greenfield product development from product requirements, such as building a complete game or application without a human developer driving each change.
• Long-running maintenance and extension of evolving codebases, where preserving previously validated behavior while adding capability is the central challenge.
• Automated regression-sensitive development pipelines, where an independent QA role that cannot silently repair the artifact it evaluates provides auditable evidence for each accepted increment.
• Traceable, version-controlled agentic workflows in regulated or review-heavy settings, since every role stage commits code and artifacts so the full development trajectory can be inspected.

Industry relevance. HoH operates on off-the-shelf harnesses (Codex CLI 0.142.5, OpenCode 1.14.30, Pi Coding Agent 0.80.10) with different frontier models, so it does not require replacing existing tooling. The pass-controlled comparison addresses the most obvious objection to agentic frameworks — that they simply spend more compute — by showing higher scores at matched pass budgets and, in one case, a higher score at a comparable token budget. The reported multi-day run with more than 70 iterations also illustrates a practical operating mode for unattended development, with all changes committed to a public repository for traceability.

Future Directions

• Scaling loop count and budget trade-offs. The paper runs HoH to three iterations on all benchmarks and to ten iterations for one configuration on FrontierSWE; how far gains continue for other harness–model pairs and other task types is not established.
• Extending to the full benchmark suites. FrontierSWE evaluation used 15 of 17 tasks and GameCraft-Bench used 45 of 140 tasks, sampled with a fixed seed; results on the complete suites are not reported.
• Richer open-ended evaluation. The multi-day game deployment is assessed by its playable features and committed trajectory; the paper does not report a formal human user study or standardized quality metric for that artifact.
• Broadening beyond the evaluated domains. The three benchmarks cover game development, repository-level software engineering with performance and research objectives, and cleanroom program reconstruction; whether the planning–development–testing loop generalizes to other software categories is an open question.

Target Audience

Researchers and engineers working on LLM coding agents, agent harness design, and autonomous software engineering; practitioners evaluating multi-agent or iterative agent architectures for real development pipelines; and readers interested in long-horizon agent behavior, verification, and state management across extended development trajectories. The paper is technical and assumes familiarity with coding-agent harnesses, benchmark evaluation, and software engineering practices such as shift-left testing and iterative incremental development.

Authors’ abstract

This paper studies autonomous software development, in which LLM-based coding agents transform high-level requirements into complete, functional, and usable software systems without human intervention. We introduce Harness-of-Harness (HoH), a framework that enables coding agents to continually improve software during autonomous development. HoH operates on existing coding-agent harnesses, and organizes their executions into iterative planning-coding-testing loops. To sustain improvement across loops, HoH balances repair with capability growth, scopes development into small and verifiable increments, separates implementation-time testing from independent evaluation, and constrains verifiable outputs rather than prescribing agent workflows. It progressively exposes deliverables, role-specific tools, and skills, encourages reuse rather than recreation, and maintains versioned project histories. On GameCraft-Bench, FrontierSWE, and ProgramBench, three harness-model pairs (Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3), HoH consistently outperforms the corresponding standalone harnesses, achieving an average relative gain of 52.25 percent and a maximum gain of 82.86 percent after three iterations. In a multi-day deployment with more than 70 iterations, HoH autonomously develops a first-person-shooter game, featuring a coherent storyline, fully implemented core mechanics, human-playable experience, polished visuals and integrated audio. Github: https://github.com/Flesymeb/HarnessOfHarness Project Page: https://flesymeb.github.io/HarnessOfHarness/