VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents

AI summary

Overview

Research area: Large language model (LLM) agents, automated software verification, and synthetic training-data generation (cs.SE).

Technical level: Intermediate. The paper is readable without deep verification background, but it assumes familiarity with agent benchmarks, test oracles, and basic classification metrics (true positive / true negative rates).

Scope: The paper introduces and analyzes VerifierBench, a benchmark of 222 agent tasks with 4,993 hidden test cases that measures whether an LLM agent can write an executable verifier which accepts every valid solution to a task and rejects invalid ones.

What This Paper Is About

Agents are increasingly asked not just to solve tasks but to write the checker that decides whether a solution is correct — a job that matters when verifiers supply reinforcement-learning rewards or filter synthetic training data. A solver only needs to find one valid solution, whereas a verifier must accept all valid solutions while rejecting invalid ones, so a verifier must separate genuine task requirements from choices that can legitimately vary. The paper builds a benchmark to measure how well current models do at this verifier-generation task, audits the benchmark's own labels for correctness, and then investigates why generated verifiers fail.

Key Contributions

1. VerifierBench, a benchmark of 222 tasks drawn from six open-source agent benchmarks (BFCL v4, ACEBench, τ²-bench, AppWorld, AgentDojo, SWE-bench Verified), covering tool-call, state-based, and code verification, with 4,993 hidden test cases that include reference solutions, valid variants, ordinary errors, and deliberate attacks.

2. Verifier4Verifier (V4V), a six-check audit — replay, repeatability, provenance, input-label consistency, trivial baselines, and mutation testing — applied both to the benchmark's source verifiers and to its constructed test cases. It excluded 45 of 267 initial tasks, leaving 222 tasks whose source verifiers become the reference verifiers.

3. A large evaluation: 19 models run in a common harness (Pi) plus 17 further model–harness configurations (36 total), scored with normalized balanced accuracy, true positive rate, true negative rate, and an attack false-positive rate.

4. A failure analysis based on reading generated verifier code and generation traces, identifying unjustified checks and two distinct self-testing failure modes.

Main Findings

• Scores are far from perfect. Across the 19 models in the Pi harness, normalized balanced accuracy ranges from 10.6 to 62.1. GPT-6-astra ranks first (62.1), followed by Claude Opus 4.8 (54.0) and Kimi K3 (53.4); Doubao Seed 2.0 Mini scores lowest at 10.6. Constant or random baselines score 0.0.

• The shared weakness is accepting valid solutions. Every model has a lower true positive rate than true negative rate: TPR ranges from 21.1% to 70.2%, while TNR ranges from 79.2% to 96.3%. Explicit FAIL decisions, not missing decisions, account for 57.1%–100% of each model's unaccepted positive cases.

• The gap is worst on state and code tasks. Averaged over models, TPR is 27.7% on state tasks and 40.2% on code tasks, versus 79.2% on tool-call tasks.

• Low attack acceptance can be misleading. On state tasks, GPT-5.6-sol has an attack false-positive rate of 0.0% but accepts only 23.0% of positive cases. On code tasks, 17 models have an attack FPR of 0.0% at the reported precision while their normalized balanced accuracy ranges from −25.9 to 76.9.

• False rejections concentrate in "all-reject" verifiers — verifiers that return FAIL on every test case of their task. In archived records they produce 87.9–96.0% of each model's false rejections. In the separate Astra trace batch, 68–75 of the 222 verifiers are all-reject in each condition and contribute 93.3–95.5% of false rejections.

• Harnesses move scores in both directions. Claude Code raises Claude Opus 4.8 by 12.2 points to 66.2, the best of all 36 configurations, and Gemini CLI raises Gemini 3.5 Flash by 15.0 points. In contrast, Codex lowers all four GPT models it runs by 4.1–26.6 points, and Gemini CLI lowers Gemini 2.5 Pro by 9.2 points. No harness improves every model it runs.

• Reasoning effort barely changes aggregate scores but shifts the error profile. With Pi, HIGH instead of LOW effort raises TPR by 2.8 points, lowers TNR by 2.3, and raises attack FPR by 2.2, for a net change of +0.5. Code tasks improve (+3.8 with Pi, +1.4 with Claude Code) while state tasks worsen (−5.4 and −3.6). HIGH effort also exhausts the 20-execution generation budget far more often: among 127 state and code tasks, exhausting runs rise from 4 to 77 with Pi and from 21 to 98 with Claude Code.

• Unjustified checks drive selected failures. In 16 verifiers from 12 tasks, the authors attribute 119 positive FAIL decisions (covering 90 distinct test cases) to conditions the task never requires — overbroad source-file protection, omitted permitted read-only tools, incomplete category vocabularies, and over-strict representations of values.

• Self-tests fail in two distinct ways. In 8 of those 16 verifiers, self-tests used only solutions the unjustified check already allowed ("confirmatory self-testing"); in the other 8, the agent never obtained a complete positive PASS at all, yet submitted the verifier anyway.

Methodology in Plain English

The researchers collected tasks from six existing agent benchmarks that already ship with an instruction, an environment, and some executable checking procedure. For each task, they wrote hidden test cases in four groups: the designated reference solution, valid variants that change presentation or implementation while still satisfying the task, ordinary errors such as partial or near-miss solutions, and attacks that exploit gaps in checking. Labels were assigned from the task requirements before any verifier ran.

Because both the original benchmark checkers and the newly written test cases could contain bugs, they built a six-part audit. The most important checks: rerun the original verifier on every case and compare with the intended label; rerun it from the same reset state to confirm stability; check that "always accept," "always reject," and "accept only the reference solution" are all caught; and mutate one check inside the original verifier to confirm that some test case notices the change. Tasks failing the audit were dropped.

In the experiment, an agent receives exactly the instruction and environment a normal solver would get — the reference verifier and hidden tests are withheld — and must deliver an executable verifier returning PASS or FAIL. During generation the agent can write its own self-tests, and for state and code tasks it can execute code in the environment through a run_public interface, capped at 20 executions per task; evaluation executions are budgeted separately. The delivered verifier is then frozen and run on every hidden case, and its decisions are compared against the reference labels.

Scoring uses task-averaged true positive and true negative rates combined into normalized balanced accuracy, TPR + TNR − 100, so constant decisions score 0 and perfect agreement scores 100; missing or malformed decisions stay in the denominator without credit. A separate attack false-positive rate tracks how often attack cases are accepted. Finally, for a selected set of failing verifiers, the authors read the generated code and generation traces and attributed each rejection to a specific check only when the public task requirements permitted the rejected property.

Why This Matters

Impact on research. The paper shifts attention from "can an agent solve a task" to "can an agent write the checker," and shows that the two abilities diverge sharply. It also argues that standard self-testing — confirming your verifier accepts your own solution — is insufficient evidence of correctness, and it provides an audited benchmark plus a reusable screening procedure for anyone measuring generated verifiers.

Real-world applications:

• Reinforcement learning and recursive self-improvement: verifiers assign rewards, so an over-strict verifier can repeatedly discard legitimate solution strategies or reward violations.
• Synthetic training-data filtering: generated verifiers decide which candidate solutions enter a training corpus; overly restrictive ones silently narrow the data distribution.
• Automated grading and code review: task-completion verifiers that check tool calls, environment state, and patches resemble automated acceptance tests in CI and evaluation pipelines.
• Agent safety and attack resistance: the attack test cases probe whether verifiers can be fooled by fabricated success signals or interference with the checking process.

Industry relevance. Teams deploying agents that generate their own evaluation code need to know that a verifier passing its own self-tests may still reject nearly every valid solution. The finding that harness choice alone can swing scores by up to 26.6 points also matters for anyone benchmarking agent capability, since harness effects are not a fixed acceptance/rejection trade-off.

Future Directions

• Requirement-traced verification: require each acceptance condition in a generated verifier to be justified by the instruction, tool definitions, or environment rules, and test it with cases derived from those requirements rather than from solutions the check already permits.
• Reserving budget for final-verifier testing: generation currently shares one execution budget between exploration and self-testing, and most HIGH-effort state and code runs exhaust it. Separating these budgets, and distinguishing an execution failure from a failed requirement, is proposed but explicitly not evaluated.
• Intervention experiments on prevalence: the code and trace analysis attributes rejections by reading source code rather than by controlled intervention, and covers mainly GPT-6-astra verifiers. Measuring how widespread unjustified checks are would require intervention-based studies across models.
• Stronger statistical grounding: the authors note that rankings are point estimates without repeated generations or significance tests, and that the concentration analysis uses archived per-task records for nine models plus one trace batch rather than final per-task records for all 19 models.

Target Audience

Researchers working on LLM agents, agentic software engineering, and automated test or specification generation; practitioners who build self-improving or self-training agent pipelines where generated verifiers gate rewards or data; and benchmark designers who need to know that upstream benchmark checkers themselves can be buggy — the audit found 35 Terminal-Bench tasks whose test-output parser counted skipped tests as passes.

Authors’ abstract

Agents need reliable ways to validate their outputs, both when completing tasks and when filtering synthetic training data. We study verifier generation: from the instruction and environment available for solving a task, an agent writes an executable verifier that must accept every valid solution and reject invalid ones. We introduce VerifierBench, a benchmark of 222 tasks from six agent benchmarks spanning tool-call, state-based, and code verification, with 4,993 hidden test cases that include alternative valid solutions and attacks; Verifier4Verifier (V4V), a six-check audit, screens its source verifiers and test suites. Across 19 models in a common agent harness, normalized balanced accuracy ranges from 10.6 to 62.1, with generated verifiers consistently rejecting invalid solutions more reliably than they accept valid ones. This apparent strength can mask overly restrictive verification: most false rejections in the analyzed records come from verifiers that accept no tested solution to a task. Case studies reveal unsupported assumptions about solution formats, tool use, and code changes that exclude valid solutions. These errors survive self-testing because agents test solutions that share the verifier's mistaken assumptions, or submit verifiers without confirming that any correct solution passes. These findings suggest testing both whether a verifier accepts a correct solution and whether it accepts other solutions permitted by the task. The project page and dataset are publicly available.