# Claude Opus 5.5 (xhigh) Scores 18.5% on ProgramBench With mini-SWE-agent

Published 2026-10-26 · AI.info Team · AI.info

> Claude Opus 5.5 (xhigh), evaluated with mini-SWE-agent, fully solves 18.5% of ProgramBench’s 200 software-reconstruction tasks and passes at least 95% of behavioral tests on 70.5% of them. The latter figure includes fully solved tasks, and the benchmark paper describes how the evaluation tests agents’ ability to recreate software behavior.

Claude Opus 5.5 (xhigh), evaluated with mini-SWE-agent, fully solves 18.5% of ProgramBench’s software-reconstruction tasks and passes at least 95% of behavioral tests on 70.5% of them. The second figure includes fully solved tasks; it is not a separate count of tasks that came close but failed at least one test.

Source [arxiv.org](https://arxiv.org/abs/2605.03546)

ProgramBench’s leaderboard lists the results across 200 tasks. At those rates, 37 tasks were fully solved, while 141 met the benchmark’s “almost” threshold of at least 95% of tests passed.

## From zero fully resolved tasks to 18.5%

ProgramBench measures whether an AI agent can rebuild software from a compiled program and its documentation. Its 200 tasks range from small command-line tools to major projects such as SQLite, FFmpeg and the PHP interpreter. Agents must reproduce the reference program’s behavior; they are not given a prescribed implementation to copy.

When the benchmark paper appeared in May 2026, none of the nine models evaluated fully resolved a task. The strongest result reached the paper’s “almost” threshold—passing at least 95% of tests—on 3% of tasks. The paper reports that the best model passed 95% of tests on only 3% of tasks; the current leaderboard separately lists Claude Opus 4.7 at 0% resolved and 3% almost.

## “Almost” includes fully resolved tasks

The current figures put two overlapping measures side by side: Opus 5.5 fully resolves 18.5% of tasks and reaches the “almost” threshold on 70.5%. The first figure counts tasks where every test passed. The second counts tasks where at least 95% passed, including the tasks that passed all tests.

That higher “almost” score signals progress, but it is not interchangeable with the benchmark’s strict resolved measure. A task counts as resolved only when all tests pass; a result below 100% can still leave behavior unmatched. The leaderboard does not specify which tests account for the gap on any individual task.

## A benchmark built around behavior, not code similarity

The ProgramBench authors designed the evaluation to test broad software development rather than a single bug fix or requested feature. Agents receive a program and documentation, then build a codebase that matches the reference executable’s behavior. The paper’s authors describe the task this way:

> In ProgramBench, given only a program and its documentation, agents must architect and implement a codebase that matches the reference executable's behavior.

The paper says its 200 tasks include compact command-line tools as well as widely used software such as FFmpeg and SQLite. The agent must choose a language, design the architecture, write the source code and produce a build script. ProgramBench compares the submitted program’s behavior with the original using its test suite.

That design makes the score a measure of whether an agent can reconstruct program behavior across a broad test suite. It does not show that Opus 5.5 can autonomously rebuild arbitrary production systems, nor does one benchmark settle how well the model performs on ordinary programming work.

## Clark expects more benchmark gains

Jack Clark, author of the Import AI research newsletter, expects ProgramBench to reach a high score by the end of 2027, while acknowledging that the benchmark may have a ceiling below 100%. That is his forecast, not a result established by the current evaluation. For now, the ProgramBench leaderboard lists Opus 5.5 (xhigh) with mini-SWE-agent at 18.5% fully resolved and 70.5% almost across the 200 tasks.

## Sources

1. Original source · Preprint [arXiv](https://arxiv.org/abs/2605.03546) arXiv:2605.03546

Other pages read

2. [ProgramBench](https://programbench.com/) programbench.com
3. [Import AI](https://importai.substack.com/p/import-ai-478-math-for-alignment) importai.substack.com

## Related stories

- [Claude Chained Exploits Across Five Platforms, Anthropic Says](https://ai.info/news/anthropic-claude-exploit-chains-five-platforms)
- [Anthropic Uses Claude to Prove 239 Tests Across 123 Libraries](https://ai.info/news/anthropic-claude-formal-verification-123-libraries)
- [Anthropic Used Claude to Find Gaps in Its Training Monitor](https://ai.info/news/anthropic-claude-training-monitor-misalignment)
- [Anthropic Tests 211 Claude-Designed Scorpion Toxin Binders](https://ai.info/news/anthropic-claude-scorpion-toxin-binders)
- [Anthropic Tests Dreaming to Correct Claude’s Memory](https://ai.info/news/anthropic-tests-dreaming-to-correct-claude-memory)

## Explore

[More articles](https://ai.info/news)

---

Canonical: https://ai.info/news/claude-opus-55-programbench-18-5-percent
