Schrödinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It?

AI summary

Overview

Research area: Software engineering (cs.SE) — evaluation methodology for repository-level coding agents, specifically the validity of SWE-bench-style benchmarks.

Technical level: Advanced. The paper assumes familiarity with SWE-bench, coding-agent scaffolds, AST-based code transformation, and terms such as Pass@1 and data leakage.

Scope (one sentence): The paper introduces SchrodingerRepo, an evaluation framework that dynamically transforms a test repository into semantically equivalent but superficially unfamiliar views, and uses it to test whether strong SWE-bench performance reflects genuine repository reasoning or memorization of canonical repository cues.

What This Paper Is About

Repository-level coding benchmarks such as SWE-bench are built on popular open-source repositories that repeatedly appear in model training data, so a high score may reflect memorization of a repository's naming conventions, file layouts, and implementation patterns rather than real code reasoning. Because the benchmark presents one fixed, canonical view of each repository, it is hard to tell these two explanations apart. The authors address this by making the repository representation an evaluation-time variable: the agent sees a transformed but behaviorally identical version of the repository that only materializes when the agent enters the environment, and the authors measure how much performance and interaction behavior change.

Key Contributions

1. SchrodingerRepo, a four-level transformation framework. It combines problem statement reconstruction (Level 1), namespace remapping (Level 2), intra-file layout reordering (Level 3), and functionality-preserving code rewriting (Level 4). The transformations are seeded, invertible, and preserve the original executable behavior and test-defined correctness criteria. Levels can be applied jointly or individually.

2. A motivation experiment quantifying leakage on SWE-bench Verified. Human experts revealed issue-description semantic units round by round to models that could not access repository files, the gold patch, or tests. For each evaluated model, more than 65% of instances showed clear data-leakage evidence, and more than 18% of instances could be recalled at the patch/test level.

3. The first systematic study of repository-representation sensitivity in coding-agent evaluation, spanning four model backends (GPT-5.4-mini, GPT 5.1, DeepSeek-v4-Flash, Gemini-3.1-Flash-Lite) on SWE-bench Verified, SWE-QA, and the March 2026 SWE-rebench Leaderboard split.

4. A temporally held-out control. By evaluating on SWE-rebench instances created after the release of the evaluated LLMs, the authors separate "the task got harder" from "familiar cues were removed."

Main Findings

• Transformation degrades Pass@1 on SWE-bench Verified. With all four levels enabled, GPT-5.4-mini drops from 46.8% to 35.6% (−11.2%), GPT 5.1 from 44.6% to 36.2% (−8.4%), DeepSeek-v4-Flash from 72.8% to 66.8% (−6.0%), and Gemini-3.1-Flash-Lite from 56.7% to 42.3% (−14.4%). The abstract summarizes this as a 6.0–14.4 percentage-point reduction. Drops are statistically significant at p < 0.05.

• Namespace mapping (Level 2) is the dominant contributor. It reduces Pass@1 by 7.4, 6.4, and 6.0 percentage points for GPT 5.1, GPT-5.4-mini, and DeepSeek-v4-Flash respectively (p < 0.01), and drives the largest interaction-cost increases: average actions rise by 63.3%, 32.6%, and 112.4%, while input tokens rise by 162.3%, 108.2%, and 218.3%. Input-token usage increases by more than 2.5× for the strongest affected configurations.

• Problem statement reconstruction (Level 1) alone barely matters. GPT 5.1 and GPT-5.4-mini were unchanged in Pass@1; DeepSeek-v4-Flash decreased by 2.0 percentage points. Rephrasing the natural-language statement is not enough to remove familiarity advantages.

• Layout reordering and rewriting (Levels 3 and 4) have milder effects. Pass@1 changes ranged from 0.8 to 3.4 percentage points for Level 3 and from 1.2 to 2.8 percentage points for Level 4, though interaction costs still rose. The authors note these levels matter because they remove cues beyond those affected by Level 2, and the full combined setting produces the largest overall degradation.

• Extra effort goes into exploration, not editing. Under the full setting, 83.6% of DeepSeek-v4-Flash's additional actions fell into navigate, search, read, and probe, with only 16.5% in edit and test. For GPT-5.4-mini, 81.6% of additional actions were again in navigate, search, read, and probe.

• Models differ in exploration strategy. DeepSeek-v4-Flash allocated 31.4% of additional actions to probe, 19.4% to read, and 17.9% to search. GPT-5.4-mini allocated 41.8% to read and 33.1% to search, with probe at 3.7%. More exploratory agents kept more of their Pass@1 but paid higher interaction cost; more conservative agents degraded more.

• Effects transfer to repository-level question answering. On SWE-QA (Conan, Reflex, Streamlink; 48 instances each, 144 total), GPT-5.4-mini's average score fell from 70.35 to 65.71 (a drop of 4.64) while actions rose from 6.28 to 7.42 (↑18.15%) and input tokens rose 34.65%. DeepSeek-v4-Flash fell only marginally from 72.97 to 72.42 (↓0.75) but actions rose from 24.49 to 35.02 (↑43.02%) and input tokens rose 59.10%. Again Level 2 caused the largest change.

• On temporally held-out instances, solvability is preserved. On the March 2026 SWE-rebench Leaderboard split (110 instances created after the release of GPT-5.4-mini), GPT-5.4-mini's Pass@1 stayed at 17.27% under the full setting, while average actions rose from 15.82 to 17.11 (↑8.15%), input tokens rose 22.01%, and output tokens rose 15.58%. The authors state the cost increase was more pronounced in widely used, long-standing repositories.

Methodology in Plain English

The authors treat the repository as a "latent variable": instead of always showing a model the same canonical copy of a project, they generate a fresh, semantically equivalent version at evaluation time, seeded randomly so each run gets a distinct but deterministic view. The mapping is invertible, so the agent's actions are translated back into the real repository before execution and final patches are translated back into original coordinates for standard scoring.

The four transformation levels operate at different layers:

• Level 1 uses two LLM passes over the issue text — a generator that reorders information, paraphrases, and removes non-essential details, and a verifier that checks all task-defining constraints survived, triggering refinement if they did not.
• Level 2 builds an offline mapping bundle from identifiers extracted from the codebase's abstract syntax tree. It filters out Python built-ins, reserved keywords, and third-party library symbols, then decomposes the remaining repository-internal identifiers into subword tokens and maps them to plausible alternatives while preserving CamelCase, snake_case, UPPER_CASE, and dotted module-path conventions and keeping shared sub-tokens consistent (for example, QuerySet decomposes into Query + Set and becomes LedgerSuite). During the run, a bidirectional translator rewrites observations into the virtual namespace and agent actions back into the original one.
• Level 3 reorders only within contiguous runs of reorderable definitions (top-level functions and classes, and methods inside class bodies), using a definition-time dependency graph and randomized topological sorting over the induced partial order. If no valid alternative ordering exists, the original sequence is kept.
• Level 4 has a constrained rewriting agent produce a unified diff that restates issue-relevant code in a behaviorally equivalent but substantially different form — the goal is representation change, not bug fixing.

Validity is enforced per level. Human reviewers inspected 100 randomly sampled instances per level and confirmed the transformed and original versions refer to the same task. For Level 2, the translator preserves the command head and only rewrites namespace-bearing arguments or embedded payloads, with reverse translation restricted to substitutions actually observed in a session notebook. For Levels 3 and 4, transformed repositories are accepted only if the official SWE-bench test suite yields Pass_to_Pass = 1 and Fail_to_Pass = 0, preserving already-correct behavior while leaving the target bug unfixed. Final submissions are recovered by diffing the original repository state against the final repository state over files whose contents changed.

Setup details: the agent scaffold is mini-swe-agent with default parameters, decoding temperature 0, and a maximum of 250 actions per instance. Three transformed views are generated per instance with different random seeds and results are averaged across the three runs. Levels 3 and 4 are restricted to golden-patch-related files or code regions for computational feasibility. Gemini-3.1-Flash-Lite was evaluated on the 300 instances identified by the motivation experiment as having the strongest data-leakage evidence.

Why This Matters

Impact on research. The paper reframes a benchmark score as a joint outcome of reasoning ability and repository-cue familiarity, and offers a reusable instrument for separating the two. It also provides quantitative leakage evidence on SWE-bench Verified — more than 65% of instances showing clear leakage evidence and more than 18% at the patch/test level — which supports the concern that static, widely used benchmarks no longer measure what they claim. Importantly, the held-out SWE-rebench result shows the framework does not simply make tasks harder, which strengthens the interpretation that the degradation on SWE-bench Verified comes from removing memorized cues.

Real-world applications.

• Evaluating coding agents before deployment with representation-diverse views, so a vendor's benchmark number is not inflated by repository familiarity.
• Cost planning for agent deployment: the paper shows Pass@1 can hold steady while token consumption and action counts rise substantially, which changes the economics of running agents at scale.
• Regression testing and model selection: comparing agents not only on success rate but on how much exploration budget they need when codebases are refactored, renamed, or reorganized.
• Repository migration and refactoring work, where the paper's Levels 2–4 (namespace remapping, reordering, behavior-preserving rewriting) are essentially the transformations a real codebase undergoes.

Industry relevance. Teams that ship or procure coding agents rely on leaderboards built from public repositories. This work indicates those leaderboards may overstate capability for the repos most present in training data, and that agents with conservative, read-and-search-heavy strategies are the most fragile when a codebase looks unfamiliar — a common situation in private or recently restructured codebases.

Future Directions

• Extending the transformations beyond the golden-patch-related files and code regions, which the authors restricted here for computational feasibility, to measure robustness under repository-wide representation change.
• Applying the framework to additional repository-level task types beyond issue resolution and question answering, to test how broadly the representation-sensitivity effect generalizes.
• Investigating the exploration-versus-robustness trade-off the paper identifies — more exploratory agents preserve more Pass@1 at higher token cost — and whether exploration policies can be improved without paying that cost.
• Building on the observation that interaction-cost inflation was more pronounced in widely used, long-standing repositories, to characterize which repository properties predict maximal memorization benefit.
• Deriving practical guidance for benchmark designers, such as whether dynamic representation should become standard practice rather than a post-hoc robustness check.

Target Audience

Researchers and practitioners in software engineering and LLM evaluation — especially those who build or use repository-level coding benchmarks, evaluate coding agents, or study data contamination and benchmark validity. It is also relevant to engineering teams choosing coding agents for private codebases, and to benchmark maintainers who need to decide whether static repository presentations remain defensible. Readers should already understand what SWE-bench is and how agent scaffolds interact with executable environments; the paper is not an introductory survey.

Authors’ abstract

Repository-level coding benchmarks have become the standard for evaluating coding agents, yet they inherently suffer from data leakage because they are built upon popular open-source repositories repeatedly used for training. Consequently, strong performance may reflect memorization of canonical repository cues rather than robust repository reasoning. We propose SchrodingerRepo (Schrödinger's Repository), an evaluation framework for testing coding agents under dynamically instantiated repository representations. Instead of repeatedly using a static representation of the test repository, SchrodingerRepo treats the test repository as an evaluation-time latent variable that is dynamically instantiated only when the agent enters the evaluation environment. The instantiated repository preserves the original executable behavior while eroding familiar cues such as naming conventions, file layouts, and implementation patterns through four transformation levels: problem statement reconstruction, namespace remapping, intra-file layout reordering, and functionality-preserving code rewriting. We evaluate popular LLMs on SWE-bench Verified and SWE-QA. Results show that removing familiar repository cues consistently degrades agent performance and substantially increases interaction costs across models. Further analysis reveals that the additional cost is primarily caused by increased difficulty in repository exploration and localization. These findings suggest that current coding agents may partially rely on memorized repository-side cues, highlighting the need for evaluation under dynamically instantiated repository representations.