CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?

AI summary

Overview

Research area: Software engineering / static analysis — specifically benchmarking LLM-based coding agents on the task of synthesizing static-analysis checkers.

Technical level: Intermediate. Some familiarity with static analysis (Clang Static Analyzer, CodeQL) and coding agents helps, but the paper's core ideas are explained in plain terms.

Scope in one sentence: The paper introduces CheckerBench, an executable benchmark of 300 real-world checker-synthesis tasks built from 297 CVEs, plus CheckerLab, an evaluation framework, and shows that 21 model–harness configurations average only 32.30% Pass@1, with a best result of 45.33%.

What This Paper Is About

Static analyzers like Clang Static Analyzer (CSA) and CodeQL only detect defects if someone writes a checker — analyzer-specific rules encoding what a bug looks like. Writing these checkers is slow expert work. This paper asks whether modern coding agents can do it end-to-end: read a defect specification, explore a repository, implement checker logic inside an analyzer framework, and refine it through compile-and-scan feedback.

The authors argue that existing benchmarks do not test this, because they are not checker-centric, allow only limited interaction, differ in environment and tooling, and measure patch correctness rather than checker properties like vulnerability coverage, false-positive control, and analyzer compatibility. CheckerBench is built to fill that gap.

Key Contributions

1. CheckerBench, described as the first comprehensive executable benchmark for repository-level, long-horizon static-analysis checker synthesis by coding agents, containing 300 real-world tasks across multiple language ecosystems and analyzer environments.

2. A scalable construction pipeline (six stages) that automatically reconstructs analyzer environments, packages synthesis tasks, and validates checker behavior — intended to support reproducible task creation beyond manually curated checker examples.

3. CheckerLab, a unified evaluation framework with a shared task specification and scoring protocol, supporting independent repeated generation and frozen-candidate verification, with fine-grained measurement of checker correctness, patch localization, false-positive control, and computational cost.

4. Empirical results across 21 model–harness configurations and three independent repeats per configuration, showing mean Pass@1 of 32.30% and a best of 45.33%, plus comparisons against KNighter and ablations of skills, execution feedback, and harness use.

Main Findings

• Low end-to-end success: Across 21 model–harness configurations, mean Pass@1 is 32.30%. The best configuration, Claude Opus 4.8 with OpenCode, reaches 45.33%.

• Build and differential verification far exceed full acceptance: Overall Build is 78.25% and Diff. SR is 73.65%. The gap between Diff. SR and Pass@1 is 41.35 percentage points, much larger than the 4.60-point gap between Build and Diff. SR. GPT-5.6 with Hermes Agent reaches 100.00% Build, 64.00% Diff. SR, and only 23.00% Pass@1.

• Harness choice matters: OpenCode has the highest mean Pass@1 across models (36.95%), followed by Claude Code (33.81%) and Hermes Agent (26.14%) — a 10.81-percentage-point spread. Model rankings also shift: Claude Opus 4.8 leads with OpenCode and Hermes Agent, while DeepSeek-V4-Pro leads with Claude Code (38.33% vs 35.00% for Claude Opus 4.8). Kimi-K3 drops from 38.67% (OpenCode) to 24.33% (Hermes Agent), a 14.34-point gap.

• Quality scores do not guarantee success: Qwen3.8-Max with Hermes Agent has the highest Process (90.92/100) and Semantic (8.53/10) scores of all configurations, yet only 25.00% Pass@1 and 44.00% Build.

• False positives vary by harness and model: OpenCode's mean FP_pass (6.63%) exceeds Claude Code's (1.97%). Within OpenCode, Claude Opus 4.8 records 13.79% FP_pass with #WN = 174, compared with 0.00% and #WN = 7 for GPT-5.6.

• C/C++ is the shared weak spot: Every model has its lowest language-specific Pass@1 in C/C++; even the leading model, DeepSeek-V4-Pro, reaches only 24.53%. This stratum contains 159 of the 300 tasks. Elsewhere, Claude Opus 4.8 leads Python (65.36%), DeepSeek-V4-Pro leads Go (53.33%), and Qwen3.8-Max leads Java (54.44%) and JavaScript (53.33%).

• Efficiency can accompany accuracy: Kimi-K3 uses the fewest average tool calls on successful runs in all three harnesses. With Claude Code it matches DeepSeek-V4-Pro closely on Pass@1 (38.00% vs 38.33%) while using fewer turns (31.81 vs 47.17) and tool calls (38.61 vs 62.50) — a 38.2% reduction in tool calls.

• Skill-guided method beats KNighter on KNighter's own tasks: On KNighter's 61 Linux-kernel tasks (all methods using DeepSeek-V4-Pro), the authors' method reaches a 80.3% valid-checker rate (49/61) versus KNighter's 67.2% (41/61). Removing the skill library drops the rate to 29.5% (18/61).

• Ablations show large dependencies: For DeepSeek-V4-Pro on the same 300 tasks with OpenCode, Pass@1 is 39.67% (Full), 8.33% without skills and strategy references, 13.33% with execution feedback hidden, and 0.33% for one-shot generation without tools or harness. Artifact pass is 48.67%, 18.67%, 20.33%, and 1.33% respectively. One-shot runs produced invalid or incomplete submissions in 237/300 tasks.

• Most gains arrive well before the time limit: Aggregate Pass@1 rises from 7.24% at 10 minutes to 26.05% at 30 minutes and 32.30% at 50 minutes. The 40-minute cutoff captures 93.02% of the successes observed by 50 minutes, and the final 10 minutes add 2.25 percentage points.

• Acceptance is stable across false-positive thresholds: The 25% threshold reproduces Table 2's acceptance rates. Aggregate acceptance rises from 32.05% at 10% to 32.30% at 25% and 32.83% at 50% — a net increase of 0.78 percentage points. The largest individual increase is 1.67 points (GLM-5.2 with Claude Code).

• Dominant failure modes: From 4,360 selected failed runs across seven models and three harnesses, the leading categories are budget exhaustion (25.32%), inadequate synthesis process (22.96%), inadequate semantic modeling (21.15%), and excess false positives (15.96%); remaining failures are differential detection or localization (8.17%), patch-specific overfitting or anchoring (3.56%), missing or invalid artifacts (1.83%), and compilation or registration (1.05%).

• Contamination appears limited: CVE publication postdates model release for 3/300 tasks with Claude Opus 4.8, one with GLM-5.2, and none with other models. On 100 language-stratified tasks, Paraphrase–Identifier-substitution Pass@1 gaps are +3.00 and +2.00 percentage points for Claude Opus 4.8 and DeepSeek-V4-Pro, while exact CVE recall falls from 10.00% to 1.00% and from 8.00% to 1.00% respectively — a 7–9 point recall decline paired with only a 2–3 point Pass@1 drop.

Methodology in Plain English

The authors built the benchmark through a six-stage pipeline:

1. Collect vulnerability sources. Public CVE records were linked to fixing commits in open-source C/C++, Java, Python, JavaScript, and Go repositories. Each candidate needed a retrievable vulnerable–fixed revision pair and a patch exposing a reusable static-analysis property.

2. Reconstruct executable environments. Both revisions were rebuilt in isolated workspaces with pinned toolchains. Clang Static Analyzer (CSA) covers C/C++ and CodeQL covers the other four languages. Pairs that could not be reconstructed or did not support stable differential analysis were rejected.

3. Package synthesis tasks. Each task exposes the patch, pre- and post-fix functions, build metadata, compile/scan commands, and a minimal checker scaffold. Random workspace identifiers and omitted CVE/CWE labels reduce provenance shortcuts. A shared skill library provides workflow skills, read-only analyzer documentation, example checker implementations, and semantic utilities — but no task-specific checker solutions.

4. Freeze an independent verifier before collecting any rollouts. It rebuilds each submitted checker from source in a clean workspace and checks vulnerable–fixed diagnostic contrast and patch relevance. For CSA, a passing checker requires a report that disappears in a patch-modified function, plus process and semantic quality, mainline false-positive control, and mechanism and anti-hardcoding checks.

5. Collect rollouts with human-guided calibration. After a failed attempt, a human diagnoses the verification evidence and revises prompt guidance for a fresh rollout, while the task, verifier, and thresholds stay fixed. Each task permits one initial attempt and at most two rerolls, stopping at the first pass; tasks with no pass after three attempts are excluded. Checker–trace pairs enter sampling only if the checker passes the frozen verifier and the trace contains at least 40 interaction steps.

6. Quality control, deduplication, and splits. Automated checks verify artifact integrity, reproducible builds and scans, verifier isolation, and package completeness; manual review confirms defect semantics. Tasks connected by repository, CVE, shared revisions, or similar patches stay in one split.

For evaluation, each CheckerLab run gets an isolated E2B sandbox and workspace with the instruction, fixing patch, code context, and analyzer tools; the reference checker and held-out verifier stay hidden. Agents work within a common 50-minute active generation budget and can inspect code, implement checkers, and use compilation and analysis feedback. At termination, the submitted CSA checker or CodeQL query source is frozen. The verifier rebuilds it and checks differential behavior; a broader mainline scan assesses false positives without checker edits. Claude Opus 5 serves as the LLM judge for process quality, vulnerability semantics, and sampled mainline alerts, with human validation giving a pooled Cohen's κ = 0.85.

The seven models are GPT-5.6-Sol, Claude Opus 4.8, Qwen3.8-Max, DeepSeek-V4-Pro-0813, GLM-5.2, Kimi-K3, and HY4-Preview, run under three harnesses: OpenCode, Claude Code, and Hermes Agent.

Why This Matters

Impact on research. The paper argues that checker synthesis is a distinct capability from patch generation or vulnerability detection, and that it needs its own benchmark with checker-specific signals: whether the checker compiles, whether diagnostics disappear on the fixed revision, whether reports localize to the patch, and whether the checker floods the codebase with false positives. CheckerBench and CheckerLab provide a shared task specification and scoring protocol so that results become comparable across agent systems rather than tied to one system-specific scaffold.

Real-world applications:

• Enterprise SAST adoption: Teams that maintain CodeQL or Clang Static Analyzer rule sets could use agents to draft and refine checkers for newly disclosed defect classes, rather than waiting for expert-authored rules.
• Security patch triage: A checker that reliably suppresses its report on the fixed revision gives a machine-checkable signal that a defect class has been addressed at a specific location.
• Coverage of smaller language ecosystems: Because the benchmark spans Java, Python, JavaScript, and Go via CodeQL alongside C/C++ via CSA, the approach applies to repositories that often lack dedicated analyzer expertise.
• CI integration and false-positive control: The framework's explicit false-positive evaluation and reporting-volume measures target the practical reason static analysis rules get disabled in CI pipelines.

Industry relevance. The results are a reality check for anyone planning to deploy coding agents on analysis-infrastructure work. More than half of tasks remain unsolved even for the best configuration, and no evaluated configuration leads simultaneously in synthesis success, process and semantic quality, and observed diagnostic precision. The ablation showing Pass@1 falling from 39.67% to 0.33% without tools or a harness, and to 8.33% without skills, indicates that agent scaffolding and reusable guidance — not just model capability — drive outcomes.

Future Directions

• Improve C/C++ and CSA performance: Every model scores lowest on C/C++ tasks, which account for 159 of 300 tasks and strongly influence aggregate success. Whether this reflects CSA's complexity, the repository scale of projects like Linux (107 tasks), or the defect families involved is left open.

• Test under separately imposed shorter deadlines: The time-budget analysis is retrospective, applying cutoffs to existing trajectories. The paper states that performance under separately imposed shorter deadlines remains untested.

• Extend beyond the current analyzer coverage: The benchmark uses CSA and CodeQL only. Whether skill-guided checker synthesis transfers to other analyzers and domains is not established here.

• Address budget exhaustion and plan-to-implementation drift: Budget exhaustion (25.32% of failures) and inadequate synthesis process (22.96%) are the top failure categories. The paper gives an example where a planned path-sensitive analysis degraded into AST matching that omitted locking and control-flow conditions — a concrete target for future agent designs.

• Broaden the false-positive adjudication cohort: The threshold-sensitivity analysis keeps adjudication labels fixed and retains all 300 tasks in the denominator; the authors report no ablation of the judge's alert sampling beyond the reported agreement level.

Target Audience

Researchers and practitioners working on coding agents, automated program analysis, and security tooling. It is most useful for:

• Benchmark and evaluation researchers who need a checker-centric alternative to patch- or detection-oriented benchmarks.
• Static analysis engineers evaluating whether agents can author or maintain CSA checkers and CodeQL queries.
• Agent framework developers comparing harness designs, since the paper shows harness choice changes both success rates and model rankings.
• Security teams weighing the feasibility of agent-assisted custom rule development, and who need to know the measured Pass@1 ceiling (45.33%) and the leading failure modes before committing engineering effort.

Readers without a static-analysis background can still follow the framing, but the verification rules — differential contrast on vulnerable versus fixed revisions, patch localization, and false-positive gates — are where the benchmark's distinctiveness actually lives.

Authors’ abstract

Static-analysis checker synthesis requires agents to interpret a defect specification, inspect a repository, implement analyzer-specific logic, and refine the checker through repeated compilation and analysis feedback. Existing coding-agent benchmarks focus on tasks such as patch generation or vulnerability detection and rarely assess whether an agent can develop a working checker in a repository from start to finish. We introduce CheckerBench, an executable benchmark of 300 tasks derived from 297 CVEs across 167 repositories, 85 CWEs, and five language ecosystems. Each task includes vulnerable and fixed revisions, a pinned analysis environment, and a checker scaffold. We further introduce CheckerLab, a common evaluation framework that independently rebuilds submitted checkers and measures vulnerable-fixed diagnostic contrast, patch localization, false positives, and tool use. Across 21 model-harness configurations and three independent repeats per configuration, mean Pass@1 is 32.30%, while the best reaches 45.33%. These results show that reliable, reusable checker development remains challenging for current coding agents.