SWE-Duel: Towards a Self-Scaling Arena for Coding Agents

AI summary

Overview

• Research area: Evaluation methodology for LLM-based coding agents; adversarial/arena-style benchmarking; automated bug generation and program repair.
• Technical level: Intermediate. The framework is conceptually simple, but reading it assumes familiarity with repository-level coding benchmarks (SWE-bench and successors), agent harnesses, and Bradley–Terry rating.
• Scope: The paper proposes SWE-Duel, a self-scaling coding arena in which two agents author and solve bug-hiding tasks against each other at evaluation time, and reports a 1,500-turn tournament among 11 model–harness participants over 15 repositories.

What This Paper Is About

Static coding benchmarks are fixed at release: their tasks eventually leak into training data, and stronger agents eventually saturate them. The authors ask whether a benchmark can instead be self-scaling — with tasks and oracles created at evaluation time so nothing exists in advance to leak, and with difficulty set by the participating agents rather than by a fixed human-written pool. SWE-Duel answers this with a mirrored Red–Blue duel in which one agent hides a bug inside a new feature and the other must find and fix it under blind review.

Key Contributions

1. A self-scaling arena protocol. SWE-Duel evaluates coding agents against each other over mirrored Red–Blue turns. Each Red task is a pull request that adds a new feature to a real repository and hides a bug inside that feature; Blue reviews the pull request without being told a bug exists and wins only if its revision passes the feature tests, the repository's existing tests, and the sealed bug tests.
2. Executable validation gates and a blind self-repair admission rule. Tasks are admitted only after three validation gates pass, the third being a fresh session of the same agent repairing the bug blind from exactly the public bundle Blue will receive. This rules out unsolvable tasks.
3. An open arena toolkit and leaderboard. The authors release a toolkit so the community can start a tournament from scratch or add a new agent to the existing leaderboard.
4. A 1,500-turn tournament across 11 (model, harness) pairs over 15 repositories in five languages, including a demonstration that a late entrant can be placed with only five matches without changing the existing ordering.

Main Findings

• Two leaders, two different strengths. GPT-5.5 with Codex CLI ranks first overall in Bradley–Terry rating (log-strength +2.82), winning 8 of 10 matches with 1 draw and 1 loss. The abstract describes it as repairing 98.3% of the defects it reviews; Table 1 lists its Blue win rate as 98.7% and its Red win rate as 11.3%. Claude Opus 4.8 with mini-swe-agent ranks second (+2.46) and is the strongest task author, with a 27.3% Red win rate — more than twice that of either GPT-5.5 participant (11.3% and 10.4%).
• The rest of the ordering. Below the top two: GPT-5.5 with mini-swe-agent (+2.13, 10.4% Red / 97.0% Blue), Kimi-K2.7-Code (+1.82, 15.6% / 90.4%), DeepSeek-V4-Pro (+1.50, 18.5% / 91.9%), GLM-5.2 (+1.19, 14.8% / 89.6%), Qwen3.7-Max (+0.49, 13.3% / 91.1%), Claude Opus 4.8 with Claude Code (−1.41, 7.3% / 87.3%), Nemotron-3-Ultra (−2.60, 6.7% / 79.3%), MiniMax-M3 (−3.86, 2.0% / 74.7%), and Gemini-3.1-Pro (−4.55, 2.7% / 72.0%).
• A seemingly worse Red/Blue record can still rank lower. DeepSeek-V4-Pro's 18.5%–91.9% record looks better than Kimi-K2.7-Code's 15.6%–90.4%, yet Kimi ranks above DeepSeek because Kimi has one fewer loss; their 95% confidence intervals overlap.
• Overall solving rate. Across all 1,500 recorded turns, Blue succeeds 87.9% of the time (95% CI 84.6–90.9%). Across the 1,219 contested turns, Blue succeeds 85.1% (95% CI 81.0–88.6%), comprising 1,037 clean repairs, 158 cases where the latent bug shipped, and 24 collapses that broke the feature or existing behavior.
• Difficulty varies by language and bug type. Contested Blue success is 89.2% for Java, 85.7% for Go, 83.9% for C/C++, 83.5% for Python, and 82.9% for Node. By self-reported bug category, logic errors are hardest to repair (81.9%), followed by security vulnerabilities (84.9%), off-by-one errors (88.9%), type confusion (90.6%), and input validation (95.5%).
• Task difficulty tracks authoring capability within one family. Holding the solver fixed at DeepSeek-V4-Pro with mini-swe-agent and changing only the task author among GLM-5.1, GLM-5.2, and GLM-5.3 (all with mini-swe-agent), the solver fails on 8.3% (GLM-5.1), 0.0% (GLM-5.2), and 42.9% (GLM-5.3) of admitted tasks (exact trend test p = .015). The abstract frames this as the newest version defeating a fixed-reference solver far more often than older versions, 42.9% vs. 4.0%. In head-to-head matches, GLM-5.3's tasks defeat GLM-5.1 in 7/14 (50.0%) and GLM-5.2 in 4/14 (28.6%) contested turns, while the two older versions each defeat GLM-5.3 only once (1/12 for GLM-5.1, 1/13 for GLM-5.2).
• A single fixed reference puts all task creators on one scale. GLM-5.2 with mini-swe-agent solved all 134 tournament tasks and failed on 15 of them (11.2%). Against this reference, Claude Opus 4.8 with mini-swe-agent remains the strongest task creator (26.7% against the reference versus 27.3% in the tournament), while Kimi-K2.7-Code drops from 17.9% to 0 of 13, suggesting its tournament wins came from a few weaker defenders. Reference and tournament rates correlate at Spearman ρ = 0.66.
• Late entry works. Gemini-3.1-Pro joined after all 45 incumbent matches, created its own bank over the same 15 repositories, and played ⌈log₂10⌉+1 = 5 active-sampled matches. It finished 0–1–4 (W/D/L), ranked last, and had the lowest contested Blue success in the field (61.8%). Refitting Bradley–Terry on the 45 incumbent-only matches reproduced the incumbent ordering of the full 50-match fit exactly (Kendall's τ = 1.0).
• Admission is mostly stable. Re-running the blind self-repair check three more times on a stratified sample of 44 tournament tasks (4 per participant), 35 tasks were self-repaired in all three runs, 8 in two, and one in none — a pooled self-repair rate of 91.7% (95% Wilson CI 85.7–95.3%).
• Bug tests are largely sound. From a seeded random sample of 50 of the 158 turns Blue lost only on bug tests, manual inspection found 45 genuine misses, three valid repairs rejected by the test, and two ambiguous cases — a 6.0% false-negative rate (95% Wilson CI 2.1–16.2%). All three false negatives come from two Go JWT tasks whose tests require one specific error type or option.
• Harness choice matters, and models do not transfer equally. In a harness-ablation tournament where the same model (DeepSeek-V4-Pro, GLM-5.2, or Kimi-K2.7-Code) plays against itself under mini-swe-agent, Codex CLI, and Claude Code, mini-swe-agent leads with BT +0.73 [−0.25, +5.56], followed by Codex CLI at −0.25 [−3.50, +1.34] and Claude Code at −0.49 [−3.83, +0.65].
• Costs. The tournament consumed 2,109 USD of API spend: $670 for Red task creation (including failed attempts), $158 for blind self-review validation, and $1,281 for Blue evaluation — about $26 per match. Onboarding the newcomer cost $243 ($126 to author its bank once, plus $117 for its five matches, split into $61.81 for the entrant solving incumbent tasks and $55.64 for incumbents solving the entrant's tasks); had Gemini played all ten incumbents, the authors estimate the entrance at about $379.

Methodology in Plain English

A participant is a (model, harness) pair; the same model under two harnesses counts as two participants. Each duel runs in a Docker container built for one open-source repository pinned to a specific commit, and the repository's existing test suite is re-run on every artifact to detect regressions.

Red creates a task in three context-isolated sessions, each followed by an executable gate. In Session A, Red explores the repository, picks a target module it has not used before, implements a new feature, and writes the feature specification and feature tests; Gate A requires the existing tests and the new feature tests to pass on the feature-only diff and the diff and tests to meet minimum size thresholds. In Session B, a fresh session receives the specification and feature-only diff, edits the feature to embed a bug, and writes bug tests; Gate B requires the final diff to apply and build, all visible tests and size checks to still pass, and the bug tests to pass on the feature-only diff but fail on the final diff — so the bug leaves no visible failing test. In Session C, a fresh instance of the same participant receives exactly the public bundle Blue will later receive and must repair the bug blind; the task is admitted only if the existing, feature, and bug suites all pass on its revision.

Each participant requests N tasks per repository, one per slot, with at most five attempts per slot; the first attempt passing all three gates is admitted, and a slot that exhausts five attempts stays empty and is never replaced, because task creation is part of the ability being measured.

Blue receives the repository, feature specification, pull-request diff, and feature tests, but never the bug tests or any bug metadata, and is not told that a bug exists. Blue submits a final diff, and the framework scores it with a hard conjunction: Blue wins the turn only if all three suites pass; otherwise Red wins. If Blue submits the pull request unchanged, Gate B guarantees the bug tests fail, so Red wins. Reverting the pull request also fails, because the feature tests would fail.

The tournament is a round robin in which every pair of participants plays one match. A match over |R| repositories with N tasks each has 2N|R| turns, and each participant plays Red and Blue equally often. Missing tasks (from exhausted slots) are scored as automatic Blue wins, which avoids survivor bias and counts task creation as part of Red's capability. The paper uses 15 repositories and 2 × 15 = 30 turns per match; 50 matches × 30 turns gives 1,500 turns. Rankings use Bradley–Terry log-strength ratings fitted by maximum likelihood, with draws counted as half a win for each side, and 1,000 bootstrap resamples of matches for rank distributions and 95% confidence intervals.

Why This Matters

• Impact on research: The paper reframes benchmark design from "publish a fixed, harder task set" to "publish a protocol whose tasks do not exist until evaluation time." It argues that continuously refreshed benchmarks such as SWE-bench-Live and SWE-rebench mitigate leakage but do not escape the ceiling imposed by human-written task difficulty, and that model-generated-task approaches such as BugPilot (fixed task set) and SwingArena (task pool still drawn from human GitHub issues) address only part of the problem. It also notes that OpenAI stopped reporting results on SWE-bench Verified after finding frontier models could reproduce its gold patches.
• Real-world applications:
  • Continuously ranking coding agents without rebuilding a benchmark each time a new model ships, by adding entrants through a small number of active-sampled matches and reusing existing task banks.
  • Measuring whether a model can find defects that pass CI, which is close to what code review is actually for.
  • Comparing agent harnesses independently of the underlying model, as the paper does in its harness-ablation tournament.
  • Profiling agents across a wide slice of the software engineering lifecycle — requirement specification, feature implementation, test generation, code review, fault localization, and program repair — rather than only patch generation.
• Industry relevance: The paper's own ethics statement flags the pipeline as an automated generator of CI-evading defects and describes the 27.3% attack success rate of the strongest Red participant against frontier-model reviewers as a disclosure-relevant security finding, demonstrating that plausible, test-passing contributions can retain defects despite automated review. It states that all generation and evaluation ran in isolated local containers at pinned revisions, that no generated artifact was submitted to a real repository, and that if the pipeline later reveals an independently existing vulnerability, coordinated disclosure to the maintainer is required before publication. The cost accounting ($26 per match, $243 to onboard a newcomer versus an estimated $379 for a full round robin) is the kind of number teams need when deciding whether to run such an evaluation continuously.

Future Directions

• Does difficulty really scale with authoring capability? The authors present the GLM-5.1/5.2/5.3 result as preliminary evidence, noting the tournament cannot test the claim directly because participants differ in more than strength.
• The self-repair gate caps task difficulty. Because a task is admitted only if its own author can repair the bug blind, a task can be no harder than its author's bug-finding ability; the paper frames this as a deliberate trade-off between solvability and difficulty.
• Task banks are retired, not reusable forever. Appendix A argues a bank is static within a tournament but should be regenerated across tournaments, since full publication enables exact replay while making the tasks unsuitable for future unseen-task claims.
• Where the oracle can be wrong. The measured 6.0% false-negative rate on bug tests, concentrated in two Go JWT tasks requiring a specific error type or option, points to test-generation robustness as an open problem.
• Stochastic admission. The 91.7% pooled self-repair rate across repeated runs indicates some tasks may be admitted by luck; reducing that variance is unresolved.

Target Audience

Researchers and engineers working on LLM-based coding agents, benchmark and evaluation design, and automated program repair will get the most from this paper, particularly those who need a benchmark that does not decay as models improve. It is also relevant to teams evaluating agent harnesses and to security-minded readers interested in defects that survive automated review. Readers without background in repository-level coding benchmarks or arena rating methods will need to consult the cited prior work first.

Authors’ abstract

Static coding benchmarks are fixed at release. Over time their tasks leak into training data, and stronger agents eventually saturate them. We argue that a benchmark should instead aim to be self-scaling: its tasks and oracles are created at evaluation time, so no content exists in advance to leak, and its difficulty is set by the participating agents rather than fixed at release. We introduce SWE-Duel, an arena in which two coding agents evaluate each other over mirrored Red-Blue turns. In each turn, the Red agent works on a real repository across three context-isolated sessions. It proposes and implements a new feature with feature tests, then embeds a subtle bug in that feature along with sealed bug-triggering tests. The resulting pull request is admitted only if (1) it passes the feature tests and the repository's existing tests while failing exactly the bug tests, and (2) a fresh session of the same agent can repair the bug blind, which rules out unsolvable tasks. The Blue agent then reviews the pull request without being told a bug exists. It wins only if its revision passes all three test suites, and then the two agents swap roles. We run a tournament among 11 model-harness participants on 15 repositories in five languages for 1,500 turns. The leaders win in different ways. GPT-5.5 with Codex CLI ranks first as the strongest solver, repairing 98.3% of the defects it reviews. Claude Opus 4.8 with mini-swe-agent ranks second as the strongest task author, with a 27.3% Red win rate. Finally, within one model family, tasks written by the newest version defeat a fixed-reference solver far more often than those of older versions (42.9% vs. 4.0%), giving preliminary evidence that task difficulty scales with task authoring capability.