SWE-at-Scale: Scalable Issue Synthesis from Historical Bug Fixes

AI summary

Overview

Research area: Software engineering (cs.SE), specifically automated synthesis of repository-level SWE benchmark tasks and the evaluation of LLM-based coding agents.

Technical level: Advanced — the paper assumes familiarity with supervised fine-tuning, LoRA, unified diffs, execution-based test validation (Pass-to-Fail), and information-theoretic diversity metrics such as entropy.

Scope: The paper presents SWE-at-Scale, a framework that reverse-engineers historical bug fixes into training supervision so that a fine-tuned model can inject realistic, diverse bugs into new code contexts and thereby mass-produce executable SWE task instances.

What This Paper Is About

Real-world SWE benchmarks such as SWE-bench Verified are hard to scale because new tasks appear only as fast as real software development, and each one is expensive to curate into an executable environment. Existing synthetic approaches (SWE-smith, SWE-Synth, BugPilot) rely on prompting general-purpose LLMs, so repeated generation tends to revisit familiar bug patterns rather than expanding diversity. The paper's goal is to convert a finite collection of real fixes into reusable supervision for a dedicated bug-injection model that can synthesize many realistic, challenging, and diverse SWE tasks from the same code context.

Key Contributions

1. A reverse-learning paradigm for SWE task synthesis. Instead of treating a historical fix as a single one-off task, SWE-at-Scale inverts fix patches into "correct-to-buggy" transformations, learning how bugs are introduced rather than only where they were fixed.

2. A large-scale training corpus and two model variants. The authors build 13,407 real-fix transformations from 688 repositories (drawn from SWE-Fixer, SWE-rebench, SWE-bench-Live, and SWE-Gym, which together contain more than 134K issue–fix instances). SWE-at-Scale learns to emit bug-inducing patches directly; SWE-at-Scale + learns to emit natural-language bug patterns that an executor then realizes in code, using DeepSeek-V4-Flash-0731 as the expert teacher.

3. Controlled comparison against real and synthetic tasks. Using the same target files and golden repository states as SWE-bench Verified instances, the authors compare against SWE-smith's LLM-Modify and LLM-Rewrite strategies and BugPilot's FeatAdd, with SWE-bench Verified and DeepSWE as real-world references, measuring realism, usability, and difficulty.

4. Evidence that bug diversity scales with training coverage and inference budget. Diversity is analyzed along three axes — repository coverage, sampling temperature, and sampling budget — showing that bug-type diversity keeps increasing as sampling grows, up to 200 generations per file.

Main Findings

• Higher human-judged realism. In pairwise comparisons against real SWE-bench Verified issues, human evaluators chose SWE-at-Scale- and SWE-at-Scale +-generated bugs as more likely to occur in real development in 55.9% and 61.2% of cases respectively. Judgments came from three software engineers with at least three years of experience, decided by majority vote.

• Higher usability. SWE-at-Scale + reaches a 94.0% Usability Rate with 61.2% Human Preference Rate; SWE-at-Scale reaches 72.6% and 55.9%. The best LLM-Rewrite configuration reaches 72.2% usability and 51.8% HPR, and LLM-Modify shows substantially lower human preference across generation models.

• Harder tasks than synthetic baselines. Across four resolution models (DS-V4-Flash, GPT-5.4-mini, GLM-5.3-Flash, Qwen3.8-Flash), SWE-at-Scale achieves Pass@1 between 31.68% and 48.48%, and SWE-at-Scale + between 16.81% and 37.02%, both lower than SWE-bench Verified (69.60%–97.00%) and the synthetic baselines. SWE-at-Scale + approaches the difficulty of human-expert-designed DeepSWE (24.30%–63.40%).

• More balanced bug-type coverage. Figure 1 shows SWE-at-Scale covering a more balanced range of bug types than SWE-smith, which the authors attribute to learning from real fixes rather than LLM prior knowledge alone.

• Broader repository coverage improves diversity. With a fixed 2,000 training datapoints, Mean Bug Type Entropy increases as the training pool grows from 5 to 100 repositories, while usability stays relatively stable and improves modestly for the largest pools.

• Moderate sampling temperature is the sweet spot. Over six temperatures on 50 target files (3,000 candidates), MBTE rises from 0.527 at 0.0 to 1.153 at 0.4 and peaks at 1.173 at 0.6, but usability falls from 69.0% at temperature 0.0 to 23.4% at 1.0, with entropy declining to 0.714 at the highest temperature.

• Diversity keeps scaling with sampling budget. At temperature 0.6 over 50 target files, MBTE increases from 0.583 bits at K=5 to 2.063 bits at K=200, with usability stabilizing around 47% beyond K=50. At K=200, 4,192 of 4,749 usable candidates (88.27%) remain unique after normalized-diff deduplication.

• Much less repetition than a prompted LLM. LLM-Modify with DS-V4-Flash reaches only 1.22 bits of MBTE and 44.1% usability at K=200, with 61.74% of usable candidates unique.

• Larger bug-inducing patches at moderate cost. SWE-at-Scale and SWE-at-Scale + modify 18.793 and 18.536 lines on average versus 14.334 for SWE-bench Verified. The direct SWE-at-Scale path consumes 17.45k tokens per instance, more than LLM-Modify (4.46–10.23k) but less than LLM-Rewrite (34.92–62.82k); FeatAdd is an outlier at 2.29M tokens per instance.

Methodology in Plain English

The approach starts from a large pool of real bug-fix commits. For each one, the authors reconstruct the file as it looked before the fix and after the fix, then compute a diff running in the opposite direction — from the fixed (correct) code back to the buggy code. That reversed diff is exactly the "bug-inducing patch" the model should learn to produce.

Quality control is heavy: they drop unparseable or non-code patches, changes dominated by imports, comments, whitespace or formatting, structurally incomplete patches, and examples exceeding a 64K-token limit. They verify that forward and reverse patches both apply cleanly and reproduce the original states through round-trip application, annotate bug types following BugPilot, sample in a distribution-aware way to balance bug types, repositories and context lengths, and manually inspect the result. Instances from repositories appearing in SWE-bench Verified are excluded to avoid overlap with evaluation data.

A Qwen3.5-9B backbone is fine-tuned with LoRA supervised fine-tuning for one epoch on a single NVIDIA RTX PRO 6000 GPU, learning to emit a unified diff that turns a complete, line-numbered golden file into its buggy counterpart. Because the whole file is provided, the model must learn both where to edit and how. For the SWE-at-Scale + variant, an expert teacher model (DeepSeek-V4-Flash-0731) writes an executor-ready natural-language bug pattern — task category, target location, current behavior, specific modification, constraints, expected observable effect — and the student learns to predict that pattern instead of the diff. An executor then applies the pattern, presented as ordinary code-modification instructions without revealing that a bug is intended.

At synthesis time, generated patches are applied with git apply, the repository's test suite is run, and only candidates that cause at least one previously passing test to fail (a Pass-to-Fail test) are kept. An LLM then writes a GitHub-style user-facing issue description from the diff, a P2F test, its execution output, and two demonstrations. Difficulty is measured by having coding agents (run through mini-SWE-agent at temperature 0 with at most 250 actions) attempt resolution, using model-disjoint evaluation so no model resolves tasks whose issue statements it generated, with each configuration run three times and averaged.

Why This Matters

Impact on research. The work reframes benchmark construction as a learning problem: rather than asking a general LLM to invent bugs, it distills the distribution of real bug patterns into a reusable model. This offers a path around the throughput ceiling of naturally occurring issues and gives the community a way to generate evaluation data whose diversity grows with compute rather than saturating.

Real-world applications:

• Coding-agent evaluation at scale — continuously producing fresh, contamination-controlled, executable tasks to benchmark agents without waiting for new GitHub issues.
• Training data for repair models — the validated buggy repositories plus P2F tests yield bug-fix pairs usable as supervised repair trajectories.
• Regression-test and test-suite auditing — the Pass-to-Fail filter exposes repository states where existing tests actually catch behavioral change, highlighting areas of weak coverage.
• Difficulty-calibrated curriculum construction — the ability to tune temperature and sampling budget lets practitioners dial the difficulty and diversity of generated tasks.

Industry relevance. Organizations running coding agents need held-out, non-leaking evaluation sets; a model that can be pointed at an internal repository and produce realistic issues on demand addresses that need directly. The reported cost profile (17.45k tokens per instance for the direct path versus 34.92–62.82k for LLM-Rewrite and 2.29M for FeatAdd) also matters for teams paying per-token to build evaluation suites.

Future Directions

• Cross-language generalization. The study covers only Python repositories; whether reversed-fix supervision transfers to languages with different syntax, semantics, and development practices is untested. The authors propose multi-language bug-inducing models as future work.
• Repositories with weak test coverage. Bugs that the available test suite cannot detect are discarded by the Pass-to-Fail filter, which limits how many tasks can be constructed from poorly tested repositories.
• Closing the gap to expert-designed tasks. SWE-at-Scale + approaches but does not match DeepSWE difficulty, leaving open how to synthesize tasks that are reliably as hard as human-expert-authored ones.
• Further exploitation of the learned bug distribution. MBTE was still rising at K=200 rather than saturating, so larger budgets, alternative decoding strategies, or better executor design remain unexplored levers for diversity.

Target Audience

Researchers and engineers working on LLM-based coding agents, benchmark and evaluation-set construction, and synthetic data generation for software engineering. It is most useful to readers already comfortable with supervised fine-tuning, repository-level test execution, and agent evaluation harnesses. Practitioners building internal evaluation suites for code agents will find the pipeline design and cost figures directly applicable, while those new to the area should treat the methodology section as the entry point and expect the diversity analysis (entropy-based metrics, temperature and budget sweeps) to require careful reading.

Authors’ abstract

As coding agents become increasingly capable, robust evaluation requires a broader and more diverse supply of verifiable software engineering (SWE) tasks. However, real-world tasks are constrained by the natural frequency of issue reporting and the substantial cost of manual curation. Furthermore, existing synthetic approaches rely primarily on the pretrained knowledge of large models coupled with heuristic prompting, often failing to capture the authentic, complex mechanics of software failures. To exponentially scale SWE task production, we introduce SWE-at-Scale, a framework that automatically synthesizes realistic issues by reverse-engineering real-world software fixes. Our key insight is that historical fixes encapsulate deep mechanistic knowledge of bug formation: inverting a fix transforms repair supervision into bug-induction supervision grounded in actual development scenarios. SWE-at-Scale transfers these learned bug patterns to novel code contexts, repeatedly sampling diverse bug typologies to construct executable SWE tasks. Extensive experiments show that SWE-at-Scale generates more diverse and realistic SWE tasks than current synthesis baselines while posing greater challenges to state-of-the-art coding agents. Notably, SWE-at-Scale can generate distinct bug types from a single code context, with bug diversity scaling robustly even up to 200 generations per file. Overall, SWE-at-Scale turns finite real-world fixes into reusable supervision for scalable synthesis of realistic, diverse, and challenging SWE tasks.