SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring

AI summary

Overview

Research area: AI coding agents and software engineering benchmarks, specifically repository-level code refactoring. Technical level: Advanced (assumes familiarity with agent scaffolds, Docker-based test evaluation, and resolve-rate metrics). Scope: The paper introduces SWE-Bench ProMax, an expert-curated multilingual refactoring benchmark of 170 instances across seven programming languages, and reports the results of evaluating six frontier models on it under two agent scaffolds.

What This Paper Is About

Existing coding benchmarks are saturating (frontier agents exceed 75% resolve rate on SWE-bench Verified) and an audit found that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests—35.5% overly narrow and 18.8% overly broad. Refactoring, which requires coordinated, behavior-preserving changes across many files, is described as a harder and more realistic test of agent capability, yet is underserved by current benchmarks. The paper's goal is to build a large-scale, multilingual, expert-curated refactoring benchmark that is both high quality and unsaturated.

Key Contributions

1. A new multilingual refactoring benchmark. SWE-Bench ProMax contains 170 instances mined from real commits across seven languages (Python, Java, TypeScript, Go, C, C++, and Rust) and 70 repositories, retained from 29,782 initial candidates.
2. A multi-stage curation pipeline targeting known evaluation defects. Issue descriptions are rewritten from scratch to be precise and unambiguous, test suites are manually reviewed to remove overly narrow tests (which reject correct solutions) and overly broad tests (which check unstated requirements), and each description is verified to be both a necessary and sufficient condition for the gold patch.
3. A deliberately large-scale, filtered task set. Tasks confined to a single file, with too few lines of code modified, or involving overly simplistic patterns were discarded. The remaining instances average 11.4 modified files and 261.6 lines of code, with 30% modifying more than 10 files and 32% changing over 200 lines of code—compared to SWE-bench Verified, where 86% of instances modify only a single file.
4. An evaluation and failure-mode analysis of frontier models. Six models (Gemini-3-Pro, Claude Sonnet 4.6, GPT-5.2, GLM-5, Kimi-K2.5, Qwen3.5) are evaluated under two scaffolds (mini-swe-agent and OpenHands), with analysis of language-level differences, cost efficiency, and agent trajectories.

Main Findings

• The benchmark is far from saturated. The best model, GPT-5.2 under OpenHands, achieves a 41.2% resolve rate, well below the 75%+ frontier agents reach on SWE-bench Verified. Gemini-3-Pro is the only model that does not improve when moved from mini-swe-agent to OpenHands (26.5% to 19.4%).
• Scaffold choice matters substantially. Every model except Gemini-3-Pro improves markedly under OpenHands; GPT-5.2 rises from 21.8% to 41.2%. The authors attribute this to richer runtime tooling being particularly beneficial for large-scale refactoring.
• Open-weight models are competitive at far lower cost. Under OpenHands, GLM-5 and Qwen3.5 both reach 36.5% and Kimi-K2.5 reaches 32.9%, versus GPT-5.2 at 41.2% and Claude Sonnet 4.6 at 38.8%. Their per-instance costs are $0.24, $0.78, and $0.72 respectively, compared with $3.60 and $4.77.
• No single model dominates across languages. Claude Sonnet 4.6 leads on TypeScript (53.6%) and Rust (63.6%), GLM-5 leads on Java (34.6%), GPT-5.2 leads on Python (48.3%) and C (75.0%), Kimi-K2.5's best result is on Go (43.5%), and Qwen3.5's best is on C++ (54.5%).
• TypeScript and Rust show surprising variance. Claude Sonnet 4.6 reaches 53.6% on TypeScript while Gemini-3-Pro reaches 0.0%; GPT-5.2 reaches 54.5% on Rust while Kimi-K2.5 reaches 18.2%. The authors suggest this reflects language-specific training data differences rather than inherent language difficulty.
• The dominant failure mode is incomplete refactoring. Trajectory analysis of Claude Sonnet 4.6 and Kimi-K2.5 shows agents track the gold patch distribution for small changes (up to about 5 files) but diverge sharply for larger ones: the gold patch cumulative distribution reaches 90% only around 20 files, while both agents reach 90% by approximately 10 files. Agents often find and edit the core files but stop short of peripheral call sites, documentation, configuration files, and test fixtures.
• Failed attempts consume more interaction rounds. Successful resolutions complete in markedly fewer rounds, while unresolved instances show rightward-shifted, more gradual curves—indicating agents entering unproductive cycles of reading, editing, test failures, and reverting.
• Higher cost does not guarantee better results. Claude Sonnet 4.6 is the most expensive model ($4.77 per instance, 117.9 steps on average) yet trails GPT-5.2 (41.2% at $3.60); Gemini-3-Pro spends $1.49 for 19.4%. GLM-5 resolves 36.5% at $0.24, roughly one-twentieth of Claude Sonnet 4.6's cost. Qwen3.5 takes the most steps under both scaffolds (155.4 and 141.2 on average) yet achieves the lowest mini-swe-agent resolve rate (20.6%).
• Patch scale varies by language. Average modified files per instance are 21.4 for C++, 20.8 for Java, 17.9 for C, 16.0 for Go, 14.5 for Rust, 11.9 for TypeScript, and 10.6 for Python. C has the highest average lines of code (424.1). The overall average is 15.9 files and 261.6 lines of code per instance.
• Repository coverage is uneven by language. Go draws 23 instances from 16 distinct repositories, while TypeScript's 28 instances come from only 2 repositories, with Angular alone contributing 25.
• Tasks are multi-faceted. Multi-label classification into ten categories shows Refactoring Cleanup in 66.5% of instances, API Interface Change in 65.3%, New Feature in 43.5%, Bug Fix in 41.2%, and Documentation in 21.2%. 46.5% of instances span three or more categories, and no instance involves fewer than two. API Interface Change and Refactoring Cleanup co-occur in 79 instances; Bug Fix accompanies them in 33 and 48 instances respectively.
• Nearly all instances require cross-file reasoning. Required-skill classification reports cross-file reasoning in 99.4% of instances, API semantics in 98.8%, interface contract reasoning in 97.1%, pattern matching in 91.8%, data flow in 88.8%, domain knowledge in 79.4%, and type system reasoning in 50.6%.

Methodology in Plain English

The benchmark was built through a three-stage pipeline. Stage 1 (collection): the GitHub API was used to find repositories with at least 500 stars, an approved open-source license, and a primary language comprising at least 80% of the codebase among the seven target languages. From these, commits submitted after January 2025 whose messages contain "refactor" but not "bug fix," and which modify both test and non-test files, were extracted. Stage 2 (environment construction): for each candidate commit, an isolated Docker container was built holding the repository at the pre-refactoring state with all dependencies installed, using automated environment construction tools including SWE-Factory. The gold patch was applied and the full test suite run; instances where an environment could not be established or the gold patch failed the tests were discarded. Stage 3 (filtering and rewriting): human experts working with LLM assistance analyzed each commit diff, removed instances with insufficient complexity (single-file tasks, too few lines changed, overly simplistic patterns), reviewed test suites to delete overly narrow and overly broad tests, and rewrote issue descriptions from scratch as precise, self-contained specifications. A final human review verified alignment among description, tests, and gold patch.

Each benchmark instance has four parts: a pre-configured Docker environment at the pre-refactoring commit, an issue description, a test suite, and a gold patch. An instance counts as resolved only if the agent's modifications pass every test in the suite—an outcome-driven formulation that ignores intermediate commands. Evaluation used mini-swe-agent and OpenHands scaffolds with a step limit of 300 and a cost limit of $10 per instance applied uniformly to all models. The primary metric is resolve rate (Pass@1) over all 170 instances, reported overall and per language, alongside average cost per instance.

Why This Matters

Impact on research. The paper provides an unsaturated, execution-based refactoring benchmark with human-verified tests, addressing both the difficulty ceiling of existing benchmarks and the test-quality defects (nearly 60% of unsolved SWE-bench Verified instances according to the cited audit) that led OpenAI to deprecate that benchmark. It also responds to evidence of data contamination and the lack of a standardized evaluation for refactoring, where prior studies used incompatible measures such as code smell counts, compilability, and alignment scores.

Real-world applications.

• Large-scale framework migrations, such as unifying header includes across NASA's F'Prime flight software framework, which the paper reports touches 244 files while preserving identical runtime behavior.
• Systematic renaming and reorganization of configuration fields, as in the betaflight motor protocol configuration refactor (62 files, 846 lines of code).
• Coordinated API and interface changes across component libraries, such as unifying the destroyOnHidden prop across ant-design components (27 files, 97 lines of code).
• Centralizing cross-cutting behavior in application code, such as provider output-format handling in google/langextract (30 files, 1,960 lines of code).

Industry relevance. The paper cites OpenAI highlighting "project-scale refactors" as a primary use case for sustained, multi-context-window agents and Cursor reporting that real-world developer tasks increasingly span many files and tools. Cost analysis showing open-weight models within a few points of proprietary ones at a fraction of the cost is directly relevant to teams deploying coding agents, and the finding that frontier agents score well below 20% on long

Authors’ abstract

As AI coding agents take on increasingly complex, long-horizon software engineering tasks, existing benchmarks are rapidly saturating and their evaluation quality has come under serious scrutiny: a recent audit found that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests -- either overly narrow tests that reject correct solutions or overly broad tests that check unstated requirements -- and that frontier models can verbatim reproduce gold patches from training data. Code refactoring, which requires coordinated, behavior-preserving changes across many files, offers a substantially harder and more realistic test of agent capability, yet remains underserved by current benchmarks. We introduce SWE-Bench ProMax, an expert-curated, multilingual code refactoring benchmark of 170 instances drawn from real commits across seven programming languages (Python, Java, TypeScript, Go, C, C++, and Rust). Every instance undergoes rigorous, multi-stage curation that directly addresses the quality problems identified in prior benchmarks: issue descriptions are rewritten from scratch to provide precise, unambiguous specifications, and test suites are manually reviewed to remove overly narrow and overly broad tests. Tasks with insufficient complexity or limited cross-file scope are filtered out, yielding a benchmark of challenging, large-scale refactoring tasks that average 11.4 modified files and 261.6 lines of code per instance, substantially exceeding the scale of existing benchmarks. Experiments with frontier models under two agent scaffolds show that the best model achieves only 41.2% resolve rate, confirming that SWE-Bench ProMax presents a meaningful and unsaturated challenge for current AI coding agents. Our benchmark is available at https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax.