BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?

AI summary

Overview

Research area: Natural Language Processing / code agents and software-engineering benchmarking.

Technical level: Advanced. The paper assumes familiarity with SWE-bench-style evaluation, agent scaffolds, and executable test-based scoring, though the headline claims are readable without that background.

Scope: The paper introduces BeyondSWE, a 500-instance benchmark drawn from 246 real-world GitHub repositories, and uses it to measure how well current code agents handle software-engineering tasks that go beyond single-repository bug fixing, plus a controlled SearchSWE baseline for studying search-augmented coding.

What This Paper Is About

Most code-agent benchmarks, led by SWE-bench Verified, evaluate localized issue resolution inside a single target repository, where the needed context is assumed to be recoverable from the issue text and the codebase. In practice, many engineering tasks require information that lives outside the repository, or require a change that spans a much larger portion of the system, so those capabilities stay under-tested. The paper builds BeyondSWE to stress-test exactly those cases, and adds SearchSWE as a controlled diagnostic to ask whether giving agents web search actually closes the gap.

Key Contributions

1. BeyondSWE benchmark. A 500-instance benchmark spanning 246 real-world GitHub repositories, organized around four settings that vary in knowledge scope (repository-local versus external software, domain, documentation, or specification knowledge) and resolution scope (localized fix versus repository-wide transformation versus full repository construction).

2. Four representative stress-test task families. Cross-repository issue resolution (CrossRepo), domain-specific issue resolution (DomainFix), dependency-driven migration (DepMigrate), and document-to-repository generation (Doc2Repo), each with its own construction pipeline, human verification, and executable test criteria.

3. Broad empirical evaluation. A comparison of frontier and code-specialized agents under OpenHands, the Codex harness (v0.118.0), and SearchSWE, showing that the benchmark remains far from saturated and that agents are brittle across all four task families.

4. SearchSWE as a controlled search-augmented baseline. A minimal extension of a standard code-agent workflow with web search and browser fetch tools, under a blocklist that prevents direct access to solution-revealing target-repository artifacts, used to diagnose why search helps but does not fully close the gap.

Main Findings

• BeyondSWE is far from saturated. Under the OpenHands scaffold, the best model, DeepSeek-V4-Pro (Max), reaches 46.12 average score. The Codex harness with GPT-5.4 (xhigh) reaches 48.48 with the default prompt and 56.65 under a SearchSWE-style prompt, leaving substantial headroom across all four task families.

• Target solutions are much larger than existing SWE benchmarks. BeyondSWE instances affect an average of 10.9 files and 1039.6 lines per instance, compared with 1.3 files and 11.6 lines for SWE-bench-Verified, 2.7 files and 65.1 lines for SWE-bench-Live, and 4.1 files and 107.4 lines for SWE-bench Pro.

• The benchmark is not difficult for one single reason. CrossRepo stresses external software knowledge, DomainFix requires domain-specific reasoning, DepMigrate demands coordinated repository-level edits, and Doc2Repo exposes a different limitation: pass rate can overstate success, since even the best configuration produces only 2 fully correct repositories out of 50.

• Search access helps, but unevenly. Compared with OpenHands, SearchSWE improves seven of eight evaluated models, with the strongest SearchSWE configuration reaching 48.73. At the task level, 20 of 32 paired comparisons improve while 31.2% regress. DomainFix benefits most; CrossRepo, DepMigrate, and Doc2Repo retain substantial regressions.

• Prompt design matters for frontier search-code agents. Using the same GPT-5.4 (xhigh) model and Codex harness, the SearchSWE-style prompt raises AVG from 48.48 to 56.65, with gains concentrated on DomainFix (+19.4) and CrossRepo (+8.9), and increases average web-search calls by about 2.8 times while keeping the local coding-tool profile similar.

• More search calls do not monotonically produce larger gains. On DomainFix (largest average gain) the points are mostly positive; on Doc2Repo (weakest average gain) they cluster around small or negative gains. The paper's interpretation is that the bottleneck is not search frequency but whether retrieved evidence is relevant and grounded in the local task context.

• Three recurring failure modes in the search-to-code pipeline. Evidence retrieved at the wrong granularity (unidata_siphon_pr234, where search returned a user-facing help page instead of backend source logic from akrherz/iem/raob.py, and the patch failed test_no_future_data_with_pressure_iastate); external knowledge not grounded in local dependency versions (behave_behave-django_pr162, where a modern @classmethod pattern was applied to a legacy instance-method _pre_setup lifecycle in a repository pinned to Django 2.2/3.x); and keyword-matched but semantically unrelated results contaminating context (abravalheri_validate-pyproject_pr105, where "family" and "review" pulled in results about Autodesk Revit and RelativityOne, and a generic entry_points plugin-registration fallback caused a duplicate registration failing with AssertionError assert 2 == 1).

• Cost and token volume are not the bottleneck. Higher API cost does not reliably correspond to higher performance, and the token-performance view shows a LOWESS trend that rises in the low-token region then flattens or declines. Many high-token runs spend budget on repeated exploration, noisy search, or ineffective repair loops.

Methodology in Plain English

The authors first define two axes for task selection: whether a task needs information outside the repository, and how large the required change is. They then build one task family per combination of interest.

For CrossRepo, they scan Python-dominant GitHub repositories for merged pull requests containing external links, collecting about 3,000 candidates, filtering to about 800 after environment construction and stability checks, and manually verifying relevance, yielding 200 issues across 67 repositories with an average of 1.3 external links each. For DomainFix, they work with domain experts across 11 research fields (including quantum physics, molecular dynamics, geospatial analysis, bioinformatics, and materials science) to pick 21 repositories, narrow roughly 800 candidate pull requests to around 200 stable ones, and require unanimous acceptance by three domain experts, leaving 72 issues across 12 repositories. For DepMigrate, they target 23 widely used packages with significant version upgrades, use LLM-based filtering to cut about 7,000 candidates to about 1,000 after stability inspection, then have four software engineering experts validate real migrations, yielding 178 issues across 120 repositories. For Doc2Repo, they collect Python repositories created between January and November 2025 with continued activity after August 2025, at least three contributors and more than 20 stars; Gemini 3 Pro explores each codebase and writes a specification that omits implementation details and directory structure, repository names are masked as target_repo, and tests are adapted with LLM assistance and human review, leaving 50 selected instances from 60 candidates.

Environments are built with an agent-assisted process: the agent starts from a base Ubuntu container, clones the repository, checks out the pre-PR commit, and iteratively fixes setup failures until tests run, after which the successful command history is distilled into a reproducible Dockerfile. Each environment is then validated by running tests five times, requiring pass-to-pass tests to pass and fail-to-pass tests to fail before the reference patch and both sets to pass afterwards.

Evaluation deliberately separates agent execution from scoring: the agent works in its own container, its changes are extracted and applied to a fresh container, git commits and metadata after the target commit are removed, and test files are restored to their original state before running. Scores are Resolved Rate for CrossRepo, DomainFix, and DepMigrate, and Pass Rate plus (Almost) Correct Count for Doc2Repo. SearchSWE adds a search tool and a browser tool to a standard agent workflow, with a blocklist over search results and shell commands that filters target-repository URLs and operations across GitHub, GitLab, raw-content endpoints, API endpoints, and direct git operations.

Why This Matters

Impact on research. The paper argues that the dominant SWE-bench-style paradigm tests a narrow slice of software engineering, and that progress on it may not transfer to tasks requiring external knowledge or broader coordination. By releasing a benchmark with executable, test-based evaluation and documented environment construction, it gives the community a way to measure capabilities that the existing benchmark family largely misses, and it reframes "search for coding" as an open research problem rather than a solved add-on.

Real-world applications (drawn from the four task settings):

• Maintaining a library when a bug or API behavior can only be understood by consulting a related upstream project or linked artifact (CrossRepo).
• Repairing and evolving scientific and engineering code that depends on specialized knowledge in fields such as quantum physics, molecular dynamics, geospatial analysis, bioinformatics, or materials science (DomainFix).
• Adapting a codebase consistently when an upstream dependency introduces breaking changes, including finding affected call sites and updating them together (DepMigrate).
• Building a working repository from a natural-language specification rather than editing an existing one (Doc2Repo).

Industry relevance. Dependency upgrades and cross-project breakages are routine maintenance costs for engineering teams, and scaffolding a new project from a written spec is a common starting point. The paper's finding that higher API cost and longer trajectories do not reliably buy better results also matters for teams budgeting agent deployments: token productivity, source discrimination, version grounding, and local verification appear to be the limiting factors rather than raw budget.

Future Directions

• Build agents that discriminate sources, not just retrieve them. The failure cases show agents need to reject high-ranking but out-of-domain or wrong-granularity evidence before it contaminates the coding context.
• Ground retrieved knowledge in local version constraints. Agents should treat installed dependency versions, APIs, and inherited code structure as filters on external knowledge, rather than defaulting to the newest pattern.
• Improve the search-to-code conversion step. The paper concludes that progress requires agents that combine external evidence with repository-local reasoning and execution-based verification, since search access alone is useful but insufficient.
• Raise token productivity rather than token volume. The flattening token-performance trend suggests research into reducing repeated exploration, noisy search, and ineffective repair loops, and into scaffolds that convert additional budget into grounded edits.

Target Audience

Researchers and engineers working on code agents, agent scaffolds, and software-engineering benchmarks; benchmark designers interested in task construction, environment reproduction, and contamination safeguards; and practitioners evaluating whether search-augmented coding agents are ready for repository-level maintenance work such as dependency migration, cross-project bug fixing, domain-specific repair, and specification-driven repository generation.

Authors’ abstract

Current benchmarks for code agents primarily assess narrow, repository-specific fixes, overlooking critical real-world challenges such as cross-repository reasoning, domain-specialized problem solving, dependency-driven migration, and full-repository generation. To address this gap, we introduce BeyondSWE, a comprehensive benchmark that broadens existing evaluations along two axes - resolution scope and knowledge scope - using 500 real-world instances across four distinct settings. Experimental results reveal a significant capability gap: even frontier models plateau below 45% success, and no single model performs consistently across task types. To systematically investigate the role of external knowledge, we develop SearchSWE, a framework that integrates deep search with coding abilities. Our experiments show that search augmentation yields inconsistent gains and can in some cases degrade performance, highlighting the difficulty of emulating developer-like workflows that interleave search and reasoning during coding tasks. This work offers both a realistic, challenging evaluation benchmark and a flexible framework to advance research toward more capable code agents.