SWE-Shift: Evaluating Coding Agents in Evolving Workspaces

AI summary

Overview

Research area: Software engineering agents / evaluation benchmarks for LLM-based coding systems (repository-level issue resolution, agent reliability).

Technical level: Intermediate. Readers should be comfortable with the idea of an LLM agent editing a repository and running tests, but the paper's core idea — paired continuations from a saved state — is easy to grasp without specialized background.

Scope: The paper introduces SWE-Shift, a benchmark of 120 scenarios across 12 change families and nine intervention positions that tests whether coding agents can keep succeeding when the workspace changes mid-execution.

What This Paper Is About

Existing repository-level benchmarks such as SWE-bench assume the task and environment stay fixed once an agent starts working. In real software development, code, dependencies, runtime state, and requirements can change while work is still in progress. The paper's goal is to measure whether an agent's success survives such a change, while separating "could not solve the task" from "was disrupted by the change" — which it does by forking the same saved execution state into a Clean continuation and a matched Shifted continuation.

Key Contributions

1. A new evaluation target: robustness to mid-execution change. The paper distinguishes failures on the underlying task from losses of continuations that would otherwise have succeeded, an axis that static task success does not capture.

2. The SWE-Shift benchmark. 120 scenarios across 12 change families, built on 23 task-root snapshots from 17 repositories drawn from SWE-bench Verified (60 scenarios), Multi-SWE-bench (35), and SWE-bench Multilingual (25), with nine intervention positions (T10–T90) per scenario, yielding 1,080 scheduled evaluation cells per system.

3. A matched-continuation protocol. The runner saves agent-visible history, workspace, controller state, tools, settings, and counters at a checkpoint, then restores that state twice: once for a Clean continuation and once for a Shifted continuation with a task-relevant update applied at the fork.

4. An empirical study of 17 coding-agent systems, showing that every system has lower Shifted than Clean success, plus family-level and timing analyses, a qualitative audit of failing trajectories, and diagnostic controls on state access and token budget.

Main Findings

• Every system degrades after a change. All 17 Clean–Shifted comparisons favor Clean. Clean success ranges from 10.71% to 67.35%, Shifted from 1.16% to 24.79%, and the Paired Success Rate (PSR) from 0.90% to 20.34%.

• GPT-6 Astra shows the largest absolute drop among top systems. It goes from 67.35% Clean to 24.79% Shifted, with 20.34% PSR and a Conditional Retention Rate (CRR) of 30.20% — fewer than one-third of its weighted Clean successes also succeed under change.

• Higher Clean success does not imply higher retention. Kimi-K3 reaches 31.41% Clean, 8.20% PSR, 25.12% CRR, while DeepSeek V4-Pro has lower Clean success (29.37%) but higher PSR (8.83%) and CRR (29.15%). Kimi-K2.6 reaches the highest reported CRR (34.63%) despite a Clean score of 20.50%.

• Tool/session changes and stale evidence are universal failure points. Family F06 (tool and session changes) and F11 (stale evidence) both show 0.00% PSR for 17 of 17 systems.

• Runtime/resource changes (F05) are near-universal failures. MiMo V2.5-Pro obtains 0.60% PSR; 16 of 17 systems score zero.

• Disclosure changes (F08) are broadly but not universally failed. 13 of 17 systems score 0.00% PSR, while Claude Fable 5.1 reaches 16.20% and GPT-6 Astra 14.90%.

• Some families show selective strengths. Kimi-K3 reaches 63.50% on F01 (user implementation) and Claude Fable 5.1 60.80%; on F04 (indirect collaborator) GPT-6 Astra scores 57.30% and GPT-5.6 Sol 39.30%; on F07 (external effects) GPT-6 Astra reaches 22.10% and GLM-5.3-Flash 18.50%, with 13 of 17 systems at zero.

• Timing cuts both ways. Comparing mean PSR at T70–T90 versus T10–T30, GPT-6 Astra loses 20.05 percentage points, Claude Fable 5.1 loses 15.43, GPT-5.6 Sol loses 12.53, and Claude Opus 5 loses 11.93 — but GLM-5.3-Flash gains 2.97, Kimi-K2.6 gains 2.20, and DeepSeek V4-Flash gains 1.60.

• Passing the original tests can hide broken work. In the audited Django case, all 39 original tests pass even though the agent removed a collaborator's API that a preservation check flags as missing. In the Astropy case, compensation for an external effect succeeds by the tool record while the workspace-preservation check still fails. In the scikit-learn case, the agent edits code without re-querying restored evidence, and the patch also exceeds the allowed edit scope.

• State handling helps only modestly. Cumulatively adding a change notice, a read-only state refresh, and removal of pre-change dialogue adds 5.98–8.71 PSR points, but final PSR remains 10.89–17.54%. Adding history reset gives the largest single increment in four of the five controlled systems, contributing 5.50 points for MiniMax-M2.7; Qwen3.8-27B gains more from state refresh (2.89) than history reset (2.63).

• More post-fork tokens stop helping past 4B. With B = 1,000,000 cumulative input-plus-output tokens per continuation (tested from B/2 to 16B, i.e. 500,000–16,000,000 tokens), each doubling up to 4B adds 0.79–2.67 points, while 4B→8B adds only 0.12–0.28 and 8B→16B changes PSR by −0.09 to +0.15. Quadrupling beyond 4B adds just 0.19–0.28 points in total, and PSR at 16B remains 7.15–11.98%.

Methodology in Plain English

The design rests on a fork. An agent is allowed to work on a repository task normally. At a scheduled checkpoint, the harness freezes everything the agent can see and use — its conversation history, the workspace, controller state, available tools, system settings, and counters. That frozen snapshot is then replayed twice. In the Clean run, the agent simply continues, with no change. In the Shifted run, a task-relevant update is injected through the scenario's declared interface: a workspace edit, a delivered message, a changed tool or session response, or new information from a permitted state query. Because both runs share identical history up to the fork, Clean answers "could this agent have finished from here?" while Shifted answers "does that success survive the update?" — something independent runs cannot isolate.

Changes must be both relevant (ignoring them can invalidate an otherwise successful continuation) and recoverable (at least one valid path to completion still exists using only agent-accessible tools). Scenarios are admitted only after passing a battery of construction checks: the untouched root must fail its task tests, a reference repair must pass, an unadapted continuation must fail Shifted checks, a pre-update-state repair must fail strict success, and a valid post-update recovery must pass, alongside replay and restore consistency.

Scoring is conjunctive. Success requires task checks, preservation of still-valid behavior, absence of prohibited effects, current goal and verifier versions, budget compliance, and — for Shifted — recovery and fresh-observation checks. The primary metric, PSR, gives credit for joint Clean-and-Shifted success, or for verified completion before the intervention could occur. CRR reports the fraction of Clean success retained under change. Scenarios are weighted equally by family, then cluster, then scenario.

Timing is handled by running each scenario at nine positions, T10–T90, derived from an eligible prefix of a reference solution (the prefix ends before the first action whose correctness depends on the update). These are scheduled positions, not normalized semantic progress — agents follow their own trajectories, so the same nominal checkpoint can land at different task stages across systems. Experiments used the same 120 scenarios and nine positions for all systems, with a shared post-fork budget of 1,000,000 cumulative input-plus-output tokens; provider-hosted systems ran in their native apps at xhigh reasoning, and open-weight systems were served on four nodes of eight NVIDIA B300 GPUs each (32 GPUs total).

Why This Matters

Research impact. The paper reframes coding-agent evaluation away from single-shot task success toward continuation reliability. The matched-fork design is a methodological contribution in its own right: it converts an otherwise confounded question ("did the change cause the failure?") into a controlled comparison. Table 1 positions SWE-Shift as the only benchmark in its comparison group checking all seven design properties: mid-episode change, same prefix, cross-plane changes, preserve/harm, multi-position, trajectory oracle, and fresh evidence.

Real-world applications:

• IDE and CLI coding assistants, where a developer edits the same file the agent is working on, or changes requirements mid-task.
• CI/CD and automated repair pipelines, where dependencies, runtime state, or locks change while an automated patch is being produced.
• Multi-agent software teams, where a collaborator's commit or API change lands while another agent is mid-edit.
• Agent platform engineering, where tool schemas, authorizations, and session protocols get rotated during long-running sessions.

Industry relevance. The failure families that are universal — tool/session invalidation (F06) and stale evidence (F11), both 0.00% PSR across 17 of 17 systems — are exactly the conditions produced by production infrastructure: token expiry, schema migrations, and cached or eventual-consistency data. The finding that quadrupling the token budget beyond 4B buys almost nothing (0.19–0.28 points total) is directly actionable for teams deciding where to spend inference compute.

Future Directions

1. Evaluating repeated changes. The paper explicitly names this as a next step: SWE-Shift applies one update per run, whereas natural development involves cascading changes.

2. Closing the tool/session and stale-evidence gap. Since no system achieves nonzero paired success on F06 or F11, the open question is what architectural change — explicit provenance tracking, forced re-observation before mutation, interface discovery — could move these off zero.

3. Reducing dependence on position effects. Late-position PSR falls for some systems and rises for others, and intervention position co-varies with execution stage, accumulated context, and remaining work. Separating those factors is unresolved.

4. Broadening the diagnostic controls. Only five systems were used in the state-access and token-budget controls, and model randomness remains uncontrolled even though pairing fixes history and workspace state; extending both is a natural follow-up.

Target Audience

Researchers building and benchmarking LLM coding agents; agent-platform and developer-tooling engineers who need reliability metrics beyond pass rates; evaluation scientists interested in controlled forking methodology; and teams deploying coding agents in environments where dependencies, tooling, or requirements change during execution. Readers looking for a leaderboard of raw coding ability will find the paper's value is instead in its diagnostic framing — the family-level and trajectory analyses are more informative than the headline numbers.

Authors’ abstract

Repository-level coding benchmarks largely assume that tasks and environments remain unchanged throughout execution. In software development, code, dependencies, runtime state, and requirements can change while an agent is still working. We introduce SWE-Shift, a benchmark for evaluating coding agents under such mid-execution changes. From the same saved history and workspace, a Clean continuation proceeds without intervention, while a matched Shifted continuation receives a task-relevant update. Success requires completing the current task, preserving still-valid work, and satisfying the applicable state and evidence checks. The benchmark contains 120 scenarios across 12 change families, each evaluated at nine scheduled intervention positions. We evaluate 17 coding-agent systems, all of which have lower Shifted than Clean success. GPT-6 Astra drops from 67.35% to 24.79%, with 20.34% paired success. Tool/session changes and stale evidence yield zero paired success across all systems. Late-position paired success is lower for some systems and higher for others. These results show that success on the unchanged task can coexist with failures to preserve work, recover state, or refresh evidence after an update.