The Pulse
METR Finds Many SWE-bench-Passing Pull Requests Would Not Be Merged
A METR study found that maintainer acceptance rates were 24.2 percentage points lower than automated SWE-bench scores after four maintainers reviewed 296 AI-generated pull requests from three repositories.
AI.info Team ·
Automated tests did not predict maintainer acceptance
A study from the research nonprofit METR found a substantial gap between AI-generated pull requests that passed the SWE-bench Verified automated grader and those that repository maintainers said they would merge. The researchers asked four active maintainers from three open-source repositories to review 296 AI-generated pull requests and assess them as they would in ordinary code review.
The maintainers covered scikit-learn, Sphinx and pytest. Together, the repositories represented three of the 12 projects in SWE-bench Verified and 95 of the benchmark’s 500 issues. Reviewers were not told whether a pull request had been written by a person or an AI system, and they evaluated the patches in GitHub-based reviews.
“We find that roughly half of test-passing SWE-bench Verified PRs written by mid-2024 to mid/late-2025 agents would not be merged into main by repo maintainers, even after adjusting for noise in maintainer merge decisions.”
— Parker Whitfill, Cheryl Wu, Joel Becker and Nate Rush, METR contributors
The 24.2-point gap
METR’s main comparison was between the automated grader’s pass rate and the rate at which maintainers accepted the same type of work. On average, the automated grader’s results were 24.2 percentage points higher than maintainer merge decisions. The reported standard error for that difference was 2.7 percentage points.
The study used a human baseline to account for the fact that code review itself can involve subjective decisions. The researchers had maintainers review 47 original human-written pull requests that had already been merged into the repositories. About 68% of those “golden patches” were merged by the study’s maintainers, and the researchers normalized their results against that baseline.
METR said the difference shows why a benchmark score should not automatically be read as a measure of real-world usefulness. SWE-bench tasks have an automated, verifiable scoring procedure. A maintainer reviewing a pull request also considers whether the patch follows repository standards, introduces problems elsewhere or actually solves the underlying issue.
Why passing was not enough
Maintainers categorized rejected patches by their primary problems. The categories included code quality, undocumented or other failures, breaking other code and core functionality failures. Some patches passed the automated tests but were still judged unsuitable for merging because they were too verbose, did not conform to project conventions or failed to resolve the issue correctly.
For the main analysis, METR reviewed AI-generated patches that passed the automated grader. The study also treated patches that failed the grader as maintainer failures, while testing that assumption against a smaller sample. In a review of 31 patches covering 27 issues, the researchers found one issue in which maintainers considered a patch valid even though the automated grader rejected it.
The agent runs came from Epoch’s benchmarking hub and included Claude 3.5 Sonnet, Claude 3.7 Sonnet, Claude 4 Opus, Claude 4.5 Sonnet and GPT-5. The researchers used patches in their original state, apart from removing miscellaneous debugging files that agents had not been instructed to remove before submission.
Limits of the comparison
METR cautioned that the results do not establish a fundamental limitation preventing AI systems from passing maintainer review. The agents generally had only one opportunity to submit a solution, while human developers commonly receive feedback and revise their work. Better prompting, agent design or an iterative review process could address some of the problems identified in the study.
The researchers also said the review setting was not fully identical to normal development. Continuous integration tools were unavailable because the patches were uploaded to repositories in historical states, and maintainers were asked to ignore testing requirements because the agents had not been prompted to include proper tests.
The study’s central finding is narrower than a claim that automated benchmarks are useless. METR said SWE-bench Verified still provides information for comparing systems, but mapping its scores to performance in human workflows is difficult. The organization’s results suggest that benchmark scores are one piece of evidence and should not be treated as equivalent to a maintainer’s decision to approve and merge a pull request.