Rethinking Verification in Agent Harnesses

AI summary

Overview

Research area: Large language model (LLM) coding agents and the "harnesses" that surround them — the instructions, tool interfaces, and control logic that govern how a model works through a task.

Technical level: Intermediate. The ideas are conceptual rather than mathematically heavy, but readers benefit from familiarity with LLM agents, test-time compute, and coding benchmarks such as Terminal-Bench and SWE-bench.

Scope: An empirical comparison of 17 verification designs across four stages of the agent lifecycle, evaluated on seven frontier coding models with two coding benchmarks plus a new 18-task expert-judged visual benchmark.

What This Paper Is About

Coding agents rarely get things right on the first try, and the surrounding harness — not the model itself — decides when and how the agent's work gets checked and revised. Practitioners currently pick verification strategies by intuition and engineering folklore, with little evidence about which designs actually help, for which models, and at what cost. This paper treats verification as a tunable axis of test-time scaling and systematically measures when to intervene, who should do the checking, and how findings should feed back into revision.

Key Contributions

1. A four-layer framework for verification in agent harnesses, organized along the agent lifecycle: L0 initial instructions (guidance given before execution), L1 execution-time reminders (nudges during work), L2 completion gates (reopening work when the agent tries to finish), and L3 independent review (a separate critic model checks before acceptance).

2. A large controlled comparison of 17 verification designs across seven frontier coding models — Qwen3.7-Max, Qwen3.8-Max, GLM-5.2, DeepSeek V4-Pro, DeepSeek V4-Flash, Kimi-K2.7-Code, and MiniMax-M2.5 — on Terminal-Bench 2.1 (a 31-task study set drawn from an 89-task registry) and SWE-bench Verified, measuring both accuracy and dollar cost.

3. VisualBench, a new benchmark of 18 visual and interactive tasks (playable games, interactive 3D configurators, sales data explorers), scored through blinded expert comparisons of functionality and visual quality, with wins, losses, and ties recorded.

4. Evidence that independent review scales as a test-time compute axis, showing that additional review rounds (depth) or additional independent critics (width) further improve accuracy, and that consolidating feedback from multiple critics before revision is more cost-efficient than repeated review rounds.

Main Findings

• Self-verification prompts help only modestly and unevenly. L0 instructions in the system prompt gained +3.3 points on TB and +3.9 on SWE on average, versus +1.2 and −1.1 for the user-role variant. Some models regress: Qwen3.8-Max loses −2.3 points on SWE under both L0 variants, and Kimi-K2.7-Code loses −8.3 points on TB with the user-role instruction.

• Agents rarely check their own work spontaneously. Only 6.2–13.6% of detected editing turns were followed by a recognized checking command before the next edit or the end of the run — the motivation for L1 reminders.

• Execution-time reminders improve accuracy while using fewer tokens. L1 designs reduced average output tokens by 14–25% on TB and 7–10% on SWE relative to baseline, with cost at 0.90–1.12× (TB) and 0.85–0.89× (SWE). Qwen3.7-Max gained +7.1/+14.3 points on TB with after-edit system/user reminders while using 33%/28% fewer output tokens.

• Completion gates work but are expensive. The strongest L2 design (early user rejection) gained +6.0 points on TB and +6.2 on SWE — the largest average L2 gains — but rejection-based designs cost 1.60–1.76× baseline on TB and 1.48–1.56× on SWE. Early user rechecking was cheaper at 1.13/1.20× baseline. The paper recommends avoiding completion-time self-checking.

• Independent review offers the best accuracy–cost tradeoff. L3 designs improved average accuracy by 5.9–9.5 points on TB and 3.9–6.6 on SWE at 1.13–1.25× and 1.00–1.04× baseline cost respectively. The strongest L3 design (+9.5 TB / +6.6 SWE) beat the strongest L2 design (+6.0 TB / +6.2 SWE) at lower cost on both benchmarks.

• Independent review dominates on visual work. All four L3 designs achieved positive net preference on VisualBench, from +13.2 to +23.8 points, exceeding all evaluated L0–L2 designs. L0 and L1 each averaged −3.2 points net preference, and L2 early rechecking reached only +2.7.

• Changes-based critic context beats full-trajectory context. Changes-based review scored +21.2 versus +13.2 for trajectory-based without required visual inspection, and +23.8 versus +21.6 with it. Trajectory-based designs also incurred 29–43% higher critic cost on TB and SWE, and only the changes-based designs were positive for all seven models.

• Review scales with depth and width. On TB, one critic and one round reached 62.7% accuracy at 38.2k mean critic tokens; two rounds reached 65.6% at 79.5k, and two critics 64.4% at 69.7k. Four rounds reached 66.1% and four critics 65.8%. Four critics improved average accuracy by 3.1 points over one critic at 1.33× total cost, and approached four-round accuracy at 19% lower total cost — because the savings come from the agent side ($0.48 vs $0.63 per trial), not the critic side (both about $0.14).

• Checks can be shallow even when frequent. In case studies, Kimi-K2.7-Code verified compilation but missed invalid word substitutions on overfull-hbox, and DeepSeek V4-Flash verified output equality but missed a runtime criterion on query-optimize.

Methodology in Plain English

The researchers built a common harness and swapped in different verification behaviors at four points in a task's life. Before the agent starts, they either append a short verification instruction to the system prompt or to the user's request (L0). During work, they inject reminders — either periodically or right after the agent edits a file — varying whether the message looks like an automated system checkpoint or a user message (L1). When the agent tries to declare itself finished, they either ask it to inspect changes, ask it to run and check, recheck before finishing, or outright reject the completion and force more work (L2). Finally, before acceptance, a separate critic model — Qwen3.7-Plus for every coding model — starts in a fresh context, gets access to the same sandbox, runs its own checks, and returns findings for the agent to repair (L3). The critic cannot modify the deliverable.

Every trial ran in a fresh E2B cloud sandbox orchestrated with Harbor. Coding conditions ran three trials per model–task pair; VisualBench ran two trials per task, model, and condition. Cost was computed from Alibaba Cloud Model Studio rates and includes agent execution, critic checks, and revisions. For VisualBench, raters from an internal company data team compared artifacts blind, with sides randomized, judging functionality and visual quality against the task brief.

Why This Matters

For research, the paper reframes verification as an allocation-of-compute problem rather than a modeling problem, and provides a controlled comparison that separates four intervention points that prior work tends to conflate. It also supplies a rare accuracy-versus-cost view: many prior self-correction results report gains without pricing the extra work, and this paper shows that some popular designs (completion gates) buy modest gains at 1.5–1.8× cost while a cheaper separate critic buys more.

Real-world applications:

• Coding agent products (CLI agents, IDE assistants) that must decide whether to spend extra tokens re-checking a diff, running tests, or invoking a second reviewer before returning a patch to a user.
• Automated software maintenance pipelines, where PR-generating agents need a reliable gate before a change is proposed for human review, with predictable cost per attempt.
• Visual and interactive artifact generation — dashboards, configurators, small games — where the deliverable's quality cannot be judged from code alone and an independent critic that actually renders and exercises the result catches defects the author missed.
• Cost-sensitive deployment, where operators must configure verification effort per task difficulty; the paper's scaling results give concrete guidance on choosing breadth (multiple critics) versus depth (multiple rounds).

Industry relevance is direct: the harnesses studied (Claude Code, Codex CLI, Gemini CLI are cited as motivating examples) are shipping products, and the finding that a low-cost critic beats an expensive self-review loop is immediately actionable for anyone tuning an agent's spend.

Future Directions

• Which critic model, and how cheap can it be? The study fixes Qwen3.7-Plus as the critic for all coding models. Whether smaller or same-family critics preserve the gains — and where the tradeoff breaks — is untested.
• Adaptive verification. The results show high variance across models and tasks (MiniMax-M2.5 gains +12.6 on SWE from an L0 system instruction while Qwen3.8-Max loses −2.3). Predicting when to escalate to a critic, and when to skip it, remains open.
• Better feedback consolidation for multi-critic setups. Four critics nearly matched four rounds at 19% lower total cost, which suggests feedback aggregation is itself a design lever worth optimizing.
• Verification for artifact-rich domains beyond coding and visuals. VisualBench is 18 tasks with expert raters; extending lifecycle-based verification to other tool-using domains, and understanding when full trajectories help versus hurt, are natural next steps. The paper notes the trajectory context did not consistently improve outcomes and cost 29–43% more.

Target Audience

Agent framework and harness engineers who choose prompts, intervention points, and review loops in production systems; researchers studying test-time compute, self-correction, and LLM-as-judge verification; and evaluation practitioners interested in the design of expert-judged benchmarks for visual and interactive artifacts. Readers seeking a first introduction to agent architectures will find the taxonomy useful but should expect assumed familiarity with coding benchmarks and token-cost accounting.

Authors’ abstract

Agent harnesses enable large language models to act through tools, observe the results, and iteratively work toward a task goal. Verification checks and revises agents' work to improve output quality. Yet there is limited guidance on how harnesses should organize this effort. We study verification as an axis of test-time scaling by examining when to intervene, who checks the work, and how checking leads to revision. Our framework spans four layers of the agent lifecycle: initial instructions, mid-turn reminders, completion gates, and independent review. We compare 17 designs across seven frontier coding models on Terminal-Bench (TB) and SWE-bench Verified (SWE), and introduce VisualBench for expert evaluation of visual and interactive artifacts. Initial instructions and execution-time reminders yield modest, model-dependent average gains on the coding benchmarks, without substantially increasing average output token use. Completion gates can yield larger gains but may substantially increase cost. Independent review by a critic offers a more favorable accuracy-cost tradeoff, improving average accuracy by 5.9-9.5 percentage points on TB and 3.9-6.6 on SWE at 1.13-1.25 and 1.00-1.04 times baseline cost, respectively. On VisualBench, independent review achieves net preferences of 13.2-23.8 points over the baseline across its four designs, exceeding the evaluated L0-L2 designs. Additional review rounds or independent critics further improve average TB accuracy by up to 3.5 percentage points. Together, these results provide empirical guidance for verification design in agent harnesses and support independent review as a practical axis of test-time scaling.