The Pulse
CWE-bench v2 Leaves 20 Security Tasks Unsolved
Collinear AI’s CWE-bench v2 reports that 20 of 250 security tasks remain unsolved by every model tested. Claude Opus 5.5 leads the leaderboard, while the benchmark measures agents’ ability to audit and repair vulnerabilities in real codebas
AI.info Team · Written with AI · fact-checked ·
Collinear AI’s November 2 update to CWE-bench v2 reports that 20 security tasks remain unsolved by every coding model tested. Five are in the public set and 15 in the held-out set. The results measure agents’ ability to find and fix vulnerabilities in real codebases.
“An attacker needs to exploit only one vulnerability; a defender must find and fix all of them.”
— The authors of the CWE-bench paper write
Claude Opus 5.5 leads, but leaves 52 tasks
Claude Opus 5.5 ranks first among 11 models, with 67.1% pass@1 and 79.2% pass@4 across the 250-task benchmark. The site says the leader fails 52 tasks on every attempt. Muse Wonder follows at 66.1% pass@1, while Gemini 4 Argon records 65.0%.
The leaderboard reports results as of November 2, 2026, and lists average cost per rollout alongside the scores. Claude Opus 5.5 averages $1.85 per rollout; DeepSeek-V4.1-Flash (1017) averages $0.04 and scores 57.6% at pass@1. Each model receives four rollouts per task at high reasoning, with a one-hour limit for each run.
Agents must audit code without a list of flaws
CWE-bench gives an agent a checkout of a real open-source repository and asks it to audit the code and fix what it finds. The task does not disclose the vulnerabilities or how many there are. Agents are tested on defense: the benchmark does not ask them to create exploits.
A deterministic verifier checks whether the exploit is blocked and existing behavior still works; the benchmark awards no partial credit. Collinear AI says every task passed seven checks and received a manual review. The public set contains 100 tasks across 91 CWEs, while 150 more remain held out; the combined benchmark spans 185 distinct CWEs.
A Rust decoder shows how a plausible fix can fall short
One example asks an agent to harden blurhash-rs, an open-source Rust library that decodes BlurHash strings into pixels. The task is tied to CVE-2023-42447. Malformed input could make the decoder panic, so a valid repair must return a typed error for bad input while preserving the output for valid hashes.
The verifier tests eight inputs in three groups: character boundaries, decoded-value bounds and normal behavior. That design checks two distinct failure paths: unexpected multibyte characters and values that parse successfully but fall outside a safe lookup range. A patch that rejects unusual characters alone still fails the numeric cases; the benchmark requires both checks and unchanged valid output.
The public set is open; the evaluation set is not
Anyone can run the public tasks with the open grader and harness, while Collinear AI keeps the held-out tasks closed to track performance on cases models have not seen. The company says its separate training corpus contains more than 4,000 tasks and that evaluation tasks never appear in training deliveries.
The site’s roadmap calls for regularly refreshing the held-out set with newly disclosed vulnerabilities and adding multi-turn audits. Those audits would let an agent investigate, ask questions and refine a fix over several turns. For now, the published leaderboard is based on four one-hour-limited rollouts per model on each of the 250 tasks.