CrossSWE: Benchmarking Cross-Repository Joint Development for Coding Agents

AI summary

Overview

Research area: Software engineering (cs.SE) — evaluation benchmarks for LLM-based coding agents, specifically repository-level issue resolution extended to multi-repository settings.

Technical level: Advanced. The paper assumes familiarity with coding-agent scaffolds, SWE-bench-style evaluation (FAIL_TO_PASS tests, gold patches, Docker-based execution environments), and Pass@1 metrics.

Scope (one sentence): The paper introduces CrossSWE, a 171-instance benchmark of real-world tasks in which a single software change must be implemented jointly across multiple interdependent repositories, and uses it to evaluate five frontier LLMs.

What This Paper Is About

Existing coding-agent benchmarks such as SWE-bench, SWE-bench Pro, and SWE-Bench ProMax increase difficulty by making tasks longer or larger, but the required patch and the validating tests still live inside one repository. Some newer benchmarks (BeyondSWE, DepBench) include other repositories only as context the agent reads, not as places the agent must also edit. The authors argue that real development often requires coordinated edits across repository boundaries — when an API, protocol, or dependency changes in one repository, dependent repositories must change compatibly — and they build a benchmark that explicitly verifies this joint necessity.

Key Contributions

1. A cross-repository benchmark. CrossSWE contains 171 real-world instances spanning 165 repositories, 20 technical domains, and 13 programming languages, with 165 instances involving two repositories, five involving three, and one involving five.

2. An executable joint-necessity criterion. For an instance with N repositories, the authors enumerate all 2^N mixed-version configurations of base and gold-patched repositories and require that the target tests fail in the all-base configuration and in every partially patched configuration, and pass only when all required patches are applied together. This separates tasks that merely touch multiple repositories from tasks whose changes are genuinely jointly necessary.

3. An evaluation of five frontier LLMs (GPT-6.1 Sol, Claude Sonnet 5.5, DeepSeek V4.1 Flash, GLM-5.3 Flash, Qwen3.8-Flash-Next) using a fixed mini-SWE-agent v2.4.6 scaffold, reporting Pass@1 plus token, step, and cost statistics.

4. A controlled joint-versus-single-side comparison and a failure taxonomy. The authors derive 330 single-side tasks from the 165 two-repository instances and classify unsuccessful trajectories into five primary failure categories, identifying cross-repository inconsistency as the dominant mode.

Main Findings

• The strongest model reaches only 58.48% Pass@1. GPT-6.1 Sol leads the five evaluated models at 58.48%, which the authors contrast with its reported 75.2% on DeepSWE v1.1. The remaining results are DeepSeek V4.1 Flash at 52.63%, Claude Sonnet 5.5 at 50.29%, Qwen3.8-Flash-Next at 45.03%, and GLM-5.3 Flash at 43.86%.

• Language boundaries matter for some models. GPT-6.1 Sol scores 60.34% on single-language tasks versus 54.55% on multi-language tasks, and Claude Sonnet 5.5 scores 52.59% versus 43.64%. DeepSeek V4.1 Flash is nearly flat across the two settings (52.59% single-language, 52.73% multi-language).

• Joint development is harder than single-side modification. On the 330 derived single-side tasks, GPT-6.1 Sol rises from 58.48% to 64.55% (+6.07 points) and its average steps drop from 32.90 to 18.54; DeepSeek V4.1 Flash rises from 52.63% to 61.81% (+9.18 points) with average steps dropping from 152.70 to 126.94.

• Closed-source models are cost-competitive through shorter trajectories. GPT-6.1 Sol averages $0.3496 per instance, close to GLM-5.3 Flash at $0.3411 and moderately above DeepSeek V4.1 Flash at $0.2881 and Qwen3.8-Flash-Next at $0.2939, despite higher per-token pricing. GPT-6.1 Sol and Claude Sonnet 5.5 need 32.9 and 29.2 steps on average, versus 152.7, 141.7, and 182.1 for DeepSeek V4.1 Flash, GLM-5.3 Flash, and Qwen3.8-Flash-Next.

• Fewer steps do not mean proportionally smaller edits. The authors report that differences in final source lines changed are much smaller than differences in interaction length, and that GPT-6.1 Sol and Claude Sonnet 5.5 tend to locate code and start editing earlier with larger, more concentrated edits, while DeepSeek V4.1 Flash and GLM-5.3 Flash spend more interactions exploring.

• Cross-repository inconsistency is the top failure mode. It accounts for 44.2% of GPT-6.1 Sol's unsuccessful trajectories, 41.2% of Claude Sonnet 5.5's, 29.2% of DeepSeek V4.1 Flash's, and 40.7% of GLM-5.3 Flash's. Incomplete implementation is the next major category for Claude (29.4%), DeepSeek (25.0%), and GLM (25.4%); DeepSeek shows more distributed failures, including compatibility regressions (18.8%) and boundary-handling errors (14.6%). The text does not report per-category percentages for Qwen3.8-Flash-Next.

• Reference patches are broader than in prior benchmarks. 56.1% of instances modify more than ten files, compared with 6% in SWE-bench Pro and 30% in SWE-Bench ProMax; 66.7% change more than 200 lines, compared with 13% and 32% respectively.

• Task-intent distribution is uneven. Of the eight categories, feature development covers 54 instances, API/schema/protocol evolution 43, bug fixing 22, security/identity/observability coordination 21, deployment/platform/runtime coordination 14, migration/upgrade/refactoring 9, test/benchmark/conformance adaptation 7, and transitive change propagation 1.

Methodology in Plain English

The authors built the benchmark in four stages.

First, they searched GitHub development histories from January 2025 through September 2026 for merged pull requests, issues, commits, and release discussions that showed cross-repository cues — API evolution, dependency migration, provider–consumer or client–server relations, and explicit references to issues, PRs, or repositories elsewhere. This stage was deliberately high-recall.

Second, humans screened candidates, keeping only cases where at least two repositories contain substantive implementation changes tied to the same task, and discarding cases where the second repository only supplied documentation, release notes, or changelogs.

Third, each surviving candidate was packaged into a Docker image containing all participating repositories at verified base commits with dependencies installed, kept in a clean Git state with gold patches and hidden tests excluded from the agent-facing image. The authors then reconstructed per-repository gold patches and enumerated all 2^N base/patched combinations, running the FAIL_TO_PASS tests in each. A task survives only if the tests fail in every configuration except the fully patched one. For a two-repository instance A and B, that means (A0,B0), (A1,B0), and (A0,B1) must fail while (A1,B1) passes.

Fourth, following SWE-Bench ProMax, they rewrote each verified case into a self-contained problem statement describing the observable failure and expected cross-repository behavior from a pre-fix perspective, stripping solution details and checker metadata.

For evaluation, all five models ran under the same mini-SWE-agent v2.4.6 scaffold with high reasoning effort, capped at 300 interaction steps and $10 per instance. An instance counts as resolved only if the complete evaluation suite passes.

Why This Matters

Impact on research. CrossSWE moves benchmark difficulty from "bigger single-repository tasks" to a different axis: coordinated change across repository boundaries, with an executable test (the 2^N configuration sweep) that a task truly requires joint edits. This gives future work a way to distinguish cross-repository reasoning from cross-repository modification, and it identifies consistency maintenance, not code-writing ability alone, as the bottleneck.

Real-world applications.

• API and protocol evolution, where a shared interface change in one repository must be mirrored in clients, servers, or consumers.
• Dependency migration and upgrade, where a downstream consumer must be repaired in step with an upstream change.
• Security, identity, and telemetry coordination, such as the paper's motivating case where oauth2-proxy must preserve cookies as a list while Higress must process and emit each cookie separately — fixing either alone leaves the shared interface inconsistent.
• Deployment, platform, and runtime alignment across repositories that ship together.

Industry relevance. Teams that maintain multi-repository products (SDKs plus services, plugins plus hosts, control planes plus gateways) face exactly these change-propagation problems. The results suggest current agents should not be trusted to land a coordinated multi-repo change unsupervised, and the failure taxonomy gives a concrete checklist — interface compatibility, completeness, regression safety, state logic, boundary cases — for reviewing agent-generated cross-repo patches.

Future Directions

• Scale to larger repository networks. Only six of the 171 instances involve three or more repositories (five with three, one with five); the authors call for extending to larger webs of interdependent repositories.
• Add repository discovery. CrossSWE hands the agent the participating repositories. A harder variant would add irrelevant repositories and evaluate whether an agent can identify which repositories must change.
• Validate beyond the mini-SWE-agent scaffold. Because mini-SWE-agent is Bash-centric, the authors note their findings may transfer to other shell-based harnesses but that direct validation remains necessary.
• Attack the dominant failure mode directly. With cross-repository inconsistency accounting for the largest share of failures, methods that explicitly check interface compatibility across repositories before and after editing are an open problem.

Target Audience

Researchers and engineers building or evaluating coding agents, benchmark designers working on repository-level software engineering tasks, and platform teams that maintain interdependent multi-repository systems. Readers need some background in agent evaluation and CI-style test harnesses to get full value, though the core finding — that coordinated multi-repository change is substantially harder than single-repository work — is accessible to a general technical audience.

Authors’ abstract

As coding agents achieve increasingly high success rates on software engineering benchmarks, recent efforts have sought to increase task complexity through longer horizons and more extensive code modifications. However, real-world software development often spans multiple interdependent repositories, introducing additional challenges in dependency reasoning, change propagation, and interface consistency. Rather than further increasing complexity within individual repositories, we introduce CrossSWE, a benchmark for cross-repository joint development comprising 171 real-world instances spanning 165 repositories, 20 technical domains, and 13 programming languages. CrossSWE verifies the necessity of coordinated changes by testing whether partial repository modifications are insufficient to achieve the intended behavior. Our evaluation of five LLMs reveals substantial challenges, with the strongest model, GPT-6.1 Sol, achieving only 58.48% Pass@1. Furthermore, cross-repository inconsistencies emerge as the most prevalent failure mode (44.2%), highlighting the difficulty of maintaining consistency across interdependent repositories.