SWEeper-Bench: Can Agents Discover Bugs in Interactive Software?

AI summary

Overview

• Research area: Software engineering agents and agent evaluation benchmarks (cs.SE), specifically whether autonomous coding agents can discover interaction-dependent bugs in running web applications rather than only fix pre-reported ones.
• Technical level: Intermediate. The paper is readable without deep ML background, but assumes familiarity with coding agents, harnesses, and benchmark design.
• Scope in one sentence: The paper introduces SWEeper-Bench, a 200-task benchmark built from real bug fixes in 200 open-source web applications, and measures how well 15 frontier agents find, diagnose, and fix bugs when given only an open-ended request naming a product area.

What This Paper Is About

Coding agents increasingly build interactive software on the fly, but bugs in such software often surface only after a specific sequence of user actions, so users end up finding them and waiting for fixes. Existing benchmarks like SWE-bench assume the bug has already been reported, which leaves the agent's ability to find interaction-dependent failures untested. SWEeper-Bench closes that gap by giving agents only a buggy codebase, a running app with a browser, and a one-sentence request such as "Find and fix issues in Focalboard's board sharing and membership," then grading repairs by what a user would observe in the browser.

Key Contributions

1. A discovery-first benchmark. 200 tasks, each drawn from a merged pull request fixing a user-facing bug in a different open-source web application (200 applications total, across seven domains, median repository 4.6k GitHub stars). The agent never sees the original issue, pull request, reference patch, repository history, or behavior tests, and outbound network access is blocked.
2. An open-ended task format. Each task supplies only a one-sentence request naming a product area, with no statement of what is broken, where the code lives, or how to reproduce the problem, mirroring a request to sweep a feature before release.
3. A browser-based agentic verifier. Every task has two behavior tests written as step-by-step user instructions: a target test that reproduces the failing interaction and a preservation test that checks nearby features still work. A task passes only if both pass, so any correct implementation is accepted regardless of approach.
4. A stage-by-stage diagnostic of agent trajectories. The authors decompose each trajectory into cumulative stages (identifying the target feature, identifying the buggy behavior, diagnosing the root cause, fixing while preserving behavior) to locate exactly where agents fail.

Main Findings

• The benchmark is far from solved. The best of 15 agents, Grok 4.6 with Cursor, passes 59.0% of the 200 tasks; Qwen 3.8 Flash follows at 57.5%, and three agents tie at 56.5% (Claude Fable 5.1, Claude Opus 5, GPT-6 Astra). Nine of the fifteen pass fewer than half, with Qwen 3.8 Max lowest at 28.0%.
• Failures are about finding, not breaking. Preservation pass rates stay between 97.5% and 100% for every agent, while target pass rates track overall pass rates almost exactly. Agents rarely regress working features; they simply miss the bug.
• The bottleneck is recognizing buggy behavior. Averaged over 14 agents, 75.4% of trajectories reach the feature containing the bug (63.5% to 91.4% per agent), but only 50.0% identify the buggy behavior, 43.8% diagnose the root cause, 41.0% pass the target test, and 40.8% pass both. The largest single drop is at identifying the buggy behavior.
• Seeing a failure is not the same as calling it a bug. On the Focalboard task, all four inspected agents reach the empty recipient sidebar, but only Claude Fable 5.1 and GLM 5.3 treat it as a bug and fix it; Claude Opus 5 concludes the behavior is intended because the existing unit tests expect no sidebar update, and GPT-6 Astra moves on to other bugs.
• Fixing is easy once the bug is named. Replacing the open-ended request with a detailed bug description raises all five tested agents to between 96.0% and 96.5% pass rates, up from 35.0% to 51.5%, and nearly erases the gap between agents. For GPT-5.6 Luna, median browser launches fall from 26 to 14 and median changed lines from 49 to 13.
• Agents do not test in the browser unless told to. Without the browser instruction, GPT-5.6 Luna drives a browser on 2 of 200 tasks (pass rate 39.5% versus 42.0% with the instruction) and GLM 5.3 on 6 of 200 (31.5% versus 43.5%). They fall back on reading code, shell commands, and HTTP requests.
• More time and more browser use do not predict success. Work time varies by nearly an order of magnitude (Grok 4.6 at 12.3 minutes and GPT-6 Astra at 13.7 minutes versus Qwen 3.8 Flash at 101.8 minutes and Kimi K3 at 90.1 minutes) without tracking pass rate. Browser actions make up 17% to 62% of turns, or 371 to 1,082 actions per task, and neither figure tracks pass rate; Qwen 3.8 Max issues 517 browser actions per task and passes 28.0%, while Claude Fable 5.1 uses the browser in 25% of turns and passes 56.5%.
• Harness choice matters modestly; the model matters more. GPT-5.6 Luna run under five harnesses ranges from 35.0% to 42.5%, doing best in its native Codex harness.
• Extra time has diminishing returns. On the same 40 tasks with budgets of 20, 40, 60, and 80 minutes, pass rates rise from 20 to 40 minutes and then flatten, peaking at 50.0% for GPT-5.6 Luna, 52.5% for GPT-5.6 Terra, and 62.0% for GLM 5.3; the authors note these differences are not statistically significant at 40 tasks.
• The verifier is reliable. Across three configurations run three times on all 200 tasks, target pass rates on the buggy version range from 0.2% to 0.8% and on the reference fix from 99.8% to 99.8%, with standard deviation at most 0.8 points. The default is the cheapest configuration, GPT-5.6 Luna with Codex at $0.160 per task.

Methodology in Plain English

The authors built each task from a merged pull request that fixed a user-facing bug in an open-source web app, using the parent commit as the buggy version and the merge as the reference repair, with one task per application. They hand-reproduced every bug in a browser, confirmed the reference fix resolves it, and walked through both behavior tests; they chose bugs that are hard to spot but obvious once seen, reproducible through normal use, and that any user would agree need fixing.

For each task, an agent works in an isolated sandbox containing the buggy source, a running instance that reflects its edits, a command interface for tests and builds, and a Chromium browser with Playwright. The prompt names only a product area, tells the agent to drive the UI like a real user, and forbids network access, git history, and searching for known fixes. Agents run with an 8-hour limit per task.

Grading happens outside the agent's control. The agent's code changes are applied to a fresh copy of the buggy app, and a verifier agent uses that app through the browser without seeing the source, following verbatim step-by-step instructions with explicit pass and fail criteria. Setup problems such as a failed login are reported separately from product failures, and the verifier is barred from using fetch, XHR, DevTools Network, API JSON, or database reads to decide pass or fail. To calibrate, the authors ran the tests on both the buggy version and the reference fix three times per configuration, checking that the target test fails on the buggy build and passes on the fix while the preservation test passes on both.

To find where agents go wrong, a judge model (GPT-5.6 Luna) read each trajectory against a task-specific description written from the reference fix and marked which cumulative stages the agent reached. The authors also ran controlled ablations: removing the browser instruction, swapping in a detailed bug description, holding the model fixed across five harnesses, and varying the time budget on a 40-task subset.

Why This Matters

Impact on research. SWE-bench and its successors start from an issue a user already filed, so they measure diagnosis and repair but not discovery, and their hidden unit tests check internal state rather than user-visible behavior, sometimes rejecting valid alternative fixes. SWEeper-Bench shifts the evaluation target to the step that produces the bug report, and its behavior-test grading rewards any fix that restores what a user observes.

Real-world applications:

• Pre-release sweeping of agent-generated apps. An assistant asked to organize a team offsite might build a sign-up form, shared schedule, and budget tracker for twenty colleagues within minutes; no testing team ever sees such an app, so the agent itself must catch silently vanishing sign-ups.
• Code review tooling. Tools such as Cursor Bugbot, GitHub Copilot code review, CodeRabbit, Greptile, Codex code review, Claude Code Review, and Gemini Code Assist already look for unreported problems, but mostly by analyzing diffs and surrounding code. SWEeper-Bench targets the harder case where a bug appears only after a sequence of user actions in the running app.
• Regression safety for multi-user features. The Focalboard example — a board shared with a colleague never appearing in the colleague's sidebar, visible only after nine steps across two accounts — mirrors the permission- and sharing-related failures that teams typically catch late.
• Training signal design. The authors suggest behavior-based verifiers like theirs could reward user-like testing during training.

Industry relevance. The evaluated agents are current commercial products and harnesses (Codex, Claude Code, Cursor, Kimi Code, DeepSeek, Muse Code), and the results quantify a practical gap: agents can act on a bug report at roughly 96% success but produce the report themselves only 59.0% of the time at best. Cost spreads are also stark — agents with similar pass rates differ in cost by more than 30 times, and Claude Opus 5 matches Claude Fable 5.1's pass rate at more than twice the cost ($25.01 versus $11.76 per task).

Future Directions

• Teaching agents to reach for interactive testing unprompted. Without an explicit instruction, agents drove a browser on 2 and 6 of 200 tasks respectively, so making user-like testing a default behavior rather than a prompted one is an open problem.
• Improving recognition, not just exploration. Since the largest drop is at identifying buggy behavior (75.4% reach the feature, 50.0% recognize the bug), future work must address agents that see a failure and talk themselves out of it, including cases where existing unit tests reinforce the wrong expectation.
• Broadening beyond web apps and functional correctness. The authors name extending the benchmark to other interactive software and to visual design, ease of use, and responsiveness as a natural next step.
• Moving from recall to precision. The benchmark scores one known bug per task, so fixes to other real issues earn no credit and unnecessary changes go unpenalized unless they break a preservation test — a multi-bug or precision-aware formulation is left open.

Target Audience

Researchers and engineers working on coding agents, agent harnesses, and software-engineering benchmarks; teams building code review or automated QA tools; and practitioners who deploy agents that generate interactive software for end users. It is also useful for readers who want a concrete, well-instrumented case study in why "the agent can fix it once you tell it what's wrong" is a different claim from "the agent can find what's wrong."

Authors’ abstract

Coding agents increasingly interact with users by generating interactive software on the fly rather than replying in text, yet many bugs in such software reveal themselves only after multi-step interactions. Today, users must discover these bugs and wait for fixes, making the development loop slow and expensive. Existing benchmarks typically assume the bugs have already been identified, leaving agents' ability to find interaction-dependent failures largely unevaluated. We introduce SWEeper-Bench, 200 tasks built from real bug fixes in 200 widely used open-source web applications. Each task gives the agent only the codebase and an open-ended request such as "Find and fix issues in Focalboard's board sharing and membership." Because these bugs are hard to detect from code alone, agents must design interactive tests to uncover them. An agentic verifier then judges each repair by what a user would observe in the browser. The best of 15 frontier agents passes only 59.0% of tasks. Through stage-by-stage analysis of agent trajectories, we show that the biggest bottleneck is identifying the buggy behavior: agents often reach the relevant feature but never trigger the failure or fail to recognize it. SWEeper-Bench measures progress toward agents that close this loop themselves, delivering software that works without waiting for user feedback.