Switch-SWE: Do Coding Agents Hold Focus When Switching Between Unfinished Tasks?

AI summary

Overview

Research area: AI for software engineering — evaluating and improving LLM-based coding agents that work on repository-level development tasks.

Technical level: Intermediate (readers should be comfortable with how coding agents, benchmarks, and test-based patch scoring work; the paper supplies its own definitions for the rest).

Scope: The paper introduces Switch-SWE, a 200-task benchmark with matched uninterrupted and switched sessions plus 400 session specifications, measures how interleaved unfinished work degrades coding agents' ability to resume a task, and proposes a plug-in context-management module (RCM) that narrows the resulting gap.

What This Paper Is About

Real development sessions rarely consist of one issue carried straight through to a patch; a request often sits unfinished while the developer or agent handles something else in the same repository. Existing repository-level benchmarks score a single issue-to-patch episode, so they do not isolate whether an agent can return to an earlier unfinished task and recover both its requirements and its partial progress. The paper builds a paired benchmark that holds the focus task's requirements and verifier constant while inserting other unfinished same-repository tasks, and then asks whether context management can recover the lost performance.

Key Contributions

1. Evaluation setting. The authors formulate same-repository task resumption as a benchmark setting with matched uninterrupted and interleaved sessions, where the switched schedule projects onto the uninterrupted schedule for the focus task (same payloads, same order, finalization last) and only the final focus patch is scored.

2. Benchmark. Switch-SWE is constructed from executable multi-stage repair tasks whose requirement stages are reconstructed from real issue, pull-request, and review histories, yielding 200 focus tasks from 19 Python repositories and 400 matched session specifications (100 near and 100 far locality tasks).

3. Paired evaluation. The paper reports paired measures of focus-task resolution, source-directory locality (near vs. far interruptions), and resumption behavior, including recovery cost in tokens, model calls, and tool calls, plus a failure breakdown into implementation, submission, and incomplete execution.

4. Method. A plug-in Resumption Context Manager (RCM) combines a Task History Router with task-state memory, and improves switched focus-task completion on OpenCode by 5.5 to 15.5 percentage points across the evaluated models while keeping uninterrupted resolution within 0.5 points of native performance.

Main Findings

• Consistent resumption gap. All seven model–agent configurations show a positive gap (uninterrupted minus switched resolution), averaging 11.5 percentage points and ranging from 6.0 to 15.0 percentage points.

• Largest absolute drop. DeepSeek-V4.1-Flash with OpenCode falls from 46.5% uninterrupted to 31.5% switched, a gap of 15.0 points and a relative reduction of 32.3%.

• A small gap is not the same as strong switching. Gemini 3.7 Flash has a gap of 6.5 points but the lowest switched resolution at 23.0% (29.5% uninterrupted), so gaps must be read next to absolute resolution.

• Switching reorders the leaderboard. Claude Opus 5 (medium) with Claude Code has the highest uninterrupted resolution at 53.5%, while GLM-5.3 with OpenCode leads under switching at 43.0% and also has the smallest gap at 6.0%.

• Similar controls, divergent switched performance. Kimi K3 and GLM-5.3 differ by only 1.0 point uninterrupted (49.0% vs. 48.0%) but by 8.5 points switched (43.0% vs. 34.5%).

• Far interruptions hurt more. All seven configurations show larger gaps in the far group; the average is 8.1% for near and 14.9% for far, with within-configuration differences from 3.0 to 10.0 points. GLM-5.3 shows the largest shift, from a 1.0-point gap near to 11.0 points far.

• Locality effects are not explained by control performance. All seven configurations achieve higher switched resolution in the near group, and for five configurations this reverses the uninterrupted ordering. The authors describe this as an association, not an isolated causal effect, because the groups contain different focus tasks.

• Switching inflates computation. Full switched sessions consume 4.5–11.1 times as many tokens as uninterrupted sessions versus 2.4–4.3 times as many model calls, so average context volume per call grows.

• Interruption history keeps costing after the return. Resumed focus stages consume 1.41–2.95 times the tokens of a full uninterrupted session, without uniformly higher model-call counts, since interruption history is not excluded from subsequent focus inputs.

• Recurrent failure behaviors. Trajectory analysis identifies cross-task requirement confusion and loss of task state as the repeated resumption behaviors, illustrated by a Moto trajectory where the agent plans to collect changes across checkouts instead of submitting only the focus patch, and a Click trajectory where a later extension leaves an earlier requirement unresolved.

• Routing helps under switching. Adding the Task History Router raises switched resolution by 11.0 points for DeepSeek-V4.1-Flash and 3.5 points for GLM-5.3, while uninterrupted resolution changes by only −1.0 and +0.5 points. For GLM-5.3 the gain appears entirely in the far group, from 40.0% to 47.0%.

• Memory adds further gains. Adding task-state memory on top of routing improves switched resolution by a further 4.5 points (DeepSeek-V4.1-Flash) and 2.0 points (GLM-5.3). Full RCM brings overall gaps to −1.0 and 1.0 points respectively, with switched resolution of 47.0% and 48.5%.

Methodology in Plain English

The authors start from real repository repairs. They collect candidates from the SWE-bench family and additional merged Python pull requests, require a reproducible base revision, a traceable issue or PR record, a reference fix, and a pinned environment, and keep only tasks whose designated tests behave stably (at least one fail-to-pass test and pass-to-pass tests with no regression).

Each task is then turned into a multi-stage request sequence. The first stage comes from the issue title and body, or from the PR description when there is no issue; later stages follow substantive pre-merge review rounds, with automated messages, duplicate comments, and approvals without new requirements removed. The authors audit the request sequence against the scoring tests so every stage corresponds to a requirement or public behavior, and they withhold reference code and any instructions that depend on unavailable intermediate patches. The final step is a requirement-free finalization request asking the agent to complete and submit.

For evaluation, every focus task is run twice. In the uninterrupted control, the focus requests arrive in order and then finalization. In the switched condition, the same focus requests arrive in the same order with the same wording, but ordered chunks from four other unfinished tasks in the same repository are inserted between them, with finalization still last. Each of the five tasks has its own persistent checkout at its own base revision, so code changes persist while the conversation is shared; both conditions start from clean files, clean conversation state, and empty agent memory. Interruptions are classified as "near" if the focus and interruption reference fixes modify files whose immediate parent directories overlap, and "far" if they do not, with the constraint that the two tasks never touch the same source file. The benchmark selects 100 near and 100 far focus tasks and samples four distinct interruption tasks per focus task.

Scoring uses only the final focus patch: the submitted non-test changes and a held-out test patch are applied to the focus base revision in a clean environment, and the task counts as resolved only if the artifact applies and every fail-to-pass and pass-to-pass test passes. Runs without a final submission or scoring result count as unresolved, and intermediate interruption implementations never enter the primary score. All runs go through Harbor in one continuous agent session, with a 900-second budget per invocation and at most three timeout continuations across the session, on 4 vCPU, 16 GiB RAM, and 30 GiB disk.

The proposed RCM module hooks into OpenCode's plug-in interface without changing the agent's native loop or the checkouts. An auxiliary model assigns each incoming request to an existing or new task and routes history so that user requests, assistant actions, and tool results for related tasks travel together; when routing confidence falls below 0.8, the full history is retained. A second auxiliary model maintains per-task records of goal, constraints, progress, pending work, relevant files, and uncertainties, in bounded batches with source references for constraints and progress. On resumption, the selected history plus the memory record are assembled into the context; older assistant and tool exchanges are replaced only when covered by a memory record, and current user instructions take precedence.

Why This Matters

Impact on research. The paper shifts the unit of evaluation from a single issue-to-patch episode to a session in which tasks are suspended and resumed, and its matched-pair design holds requirements, verifier, and workspaces fixed so that the measured difference is attributable to interleaving rather than to task difficulty. The authors position this against SWE-bench, SWE-bench-Live, SWE-ContextBench, InterruptBench, SWE-Touch, ContinuityBench, Handoff Debt, and CORPGEN, and their comparison table shows Switch-SWE as the only listed benchmark marked as supporting interleaved unfinished tasks, repeated resumption, and persistent per-task workspaces together.

Real-world applications:

• Multi-request developer assistants: tools where a user keeps several feature requests and bug reports open in one long session, and expects each to be picked back up correctly.
• Agent handoff and on-call workflows: engineering settings where a session must recover an earlier task's constraints after unrelated incidents or review requests interrupt it.
• CI and automated maintenance agents: bots that juggle several repository issues in parallel and must avoid submitting changes belonging to a different task.
• Harness and context-engineering tooling: plug-in layers that decide which conversation history and which state summaries to feed a model at each turn.

Industry relevance. The paper reports that interleaving raises token consumption by 4.5–11.1 times and model calls by 2.4–4.3 times while lowering completion, which directly affects cost and reliability for anyone deploying long-running coding agents. It also shows that leaderboard order changes under switching (Claude Opus 5 leads uninterrupted, GLM-5.3 leads switched), which matters for model and harness selection.

Future Directions

• Separate history from state more precisely. The authors note that aggregate token usage cannot separate repeated processing of interruption history from differences in outputs and native context management, so finer-grained attribution is an open question.
• Convert the locality association into a causal test. The paper explicitly states that near/far comparisons reflect an association rather than an isolated causal effect, since the groups contain different focus tasks.
• Extend beyond OpenCode and the evaluated models. RCM was only tested as an OpenCode plug-in on DeepSeek-V4.1-Flash and GLM-5.3; generalization to other harnesses is not reported.
• Broaden the benchmark. Switch-SWE covers 19 Python repositories and 200 focus tasks, and the authors present the released code and planned data release as support for independent reproduction and further analysis.

Target Audience

Researchers and engineers working on coding agents, agent benchmarks, and long-horizon or multi-task agent sessions will get the most from this paper, as will practitioners building harnesses or context-management layers who need evidence about how agents behave when a session interleaves unfinished work. Readers who build evaluation suites will find the matched-session design and the locality labeling most directly reusable; readers focused on product reliability will find the resolution gaps and the token and call multipliers most immediately relevant.

Authors’ abstract

Software development sessions often involve several unfinished tasks rather than one issue carried through to completion. As coding agents take on repository-level development, they need to recover an earlier task's requirements and partial progress after intervening work. Existing repository-level benchmarks typically score one issue-to-patch episode and do not directly measure whether an agent can resume a task after advancing other unfinished work in the same repository. We introduce \textbf{\ourmethod}, a benchmark for focus-task resumption in sessions that interleave unfinished tasks drawn from the same source repository, with a persistent checkout for each task. Each focus task is evaluated in matched uninterrupted and switched sessions with identical focus requirements and executable verifiers. The switched session interleaves ordered stages from other unfinished tasks in the same repository, while only the final focus patch is scored. \ourmethod~contains 200 focus tasks from 19 Python repositories, reconstructed from issue, pull-request, and review histories and rendered as 400 matched session specifications. Across 7 coding agents, interleaved unfinished work produces an average focus-task resolution gap of 11.5 percentage points, ranging from 6.0 to 15.0 percentage points. Trajectory analysis identifies cross-task requirement confusion and loss of task state as recurrent resumption behaviors. We introduce a plug-in context-management module that selects task-relevant history and maintains concise task-state summaries. On OpenCode, the module improves focus-task completion under same-repository task switching by 5.5 to 15.5 percentage points across the evaluated models. Code is available at github.com/Miraclecsc/Switch-SWE.