The Pulse
Claude Opus 5.5 (xhigh) Scores 18.5% on ProgramBench With mini-SWE-agent
Claude Opus 5.5 (xhigh), evaluated with mini-SWE-agent, fully solves 18.5% of ProgramBench’s 200 software-reconstruction tasks and passes at least 95% of behavioral tests on 70.5% of them. The latter figure includes fully solved tasks, and
AI.info Team · Written with AI · fact-checked ·
Claude Opus 5.5 (xhigh), evaluated with mini-SWE-agent, fully solves 18.5% of ProgramBench’s software-reconstruction tasks and passes at least 95% of behavioral tests on 70.5% of them. The second figure includes fully solved tasks; it is not a separate count of tasks that came close but failed at least one test.
ProgramBench’s leaderboard lists the results across 200 tasks. At those rates, 37 tasks were fully solved, while 141 met the benchmark’s “almost” threshold of at least 95% of tests passed.
From zero fully resolved tasks to 18.5%
ProgramBench measures whether an AI agent can rebuild software from a compiled program and its documentation. Its 200 tasks range from small command-line tools to major projects such as SQLite, FFmpeg and the PHP interpreter. Agents must reproduce the reference program’s behavior; they are not given a prescribed implementation to copy.
When the benchmark paper appeared in May 2026, none of the nine models evaluated fully resolved a task. The strongest result reached the paper’s “almost” threshold—passing at least 95% of tests—on 3% of tasks. The paper reports that the best model passed 95% of tests on only 3% of tasks; the current leaderboard separately lists Claude Opus 4.7 at 0% resolved and 3% almost.
“Almost” includes fully resolved tasks
The current figures put two overlapping measures side by side: Opus 5.5 fully resolves 18.5% of tasks and reaches the “almost” threshold on 70.5%. The first figure counts tasks where every test passed. The second counts tasks where at least 95% passed, including the tasks that passed all tests.
That higher “almost” score signals progress, but it is not interchangeable with the benchmark’s strict resolved measure. A task counts as resolved only when all tests pass; a result below 100% can still leave behavior unmatched. The leaderboard does not specify which tests account for the gap on any individual task.
A benchmark built around behavior, not code similarity
The ProgramBench authors designed the evaluation to test broad software development rather than a single bug fix or requested feature. Agents receive a program and documentation, then build a codebase that matches the reference executable’s behavior. The paper’s authors describe the task this way:
In ProgramBench, given only a program and its documentation, agents must architect and implement a codebase that matches the reference executable's behavior.
The paper says its 200 tasks include compact command-line tools as well as widely used software such as FFmpeg and SQLite. The agent must choose a language, design the architecture, write the source code and produce a build script. ProgramBench compares the submitted program’s behavior with the original using its test suite.
That design makes the score a measure of whether an agent can reconstruct program behavior across a broad test suite. It does not show that Opus 5.5 can autonomously rebuild arbitrary production systems, nor does one benchmark settle how well the model performs on ordinary programming work.
Clark expects more benchmark gains
Jack Clark, author of the Import AI research newsletter, expects ProgramBench to reach a high score by the end of 2027, while acknowledging that the benchmark may have a ceiling below 100%. That is his forecast, not a result established by the current evaluation. For now, the ProgramBench leaderboard lists Opus 5.5 (xhigh) with mini-SWE-agent at 18.5% fully resolved and 70.5% almost across the 200 tasks.