SWE-Trail: History-Guided Trajectory Collection for Coding Agent Training

AI summary

Overview

Research area: Artificial intelligence — specifically, post-training data collection for LLM-based coding agents that resolve repository-level software issues.

Technical level: Intermediate. Readers need familiarity with coding agents, supervised fine-tuning, and benchmark evaluation, but the core idea (using prior attempts to decide where to spend extra collection effort) is explained largely in plain language.

Scope: The paper proposes and evaluates SWE-Trail, a method that uses earlier trajectories on the same issue to guide where additional collection effort is spent during new rollouts, while keeping that history out of the coding agent's own context.

What This Paper Is About

Coding agents fix repository issues through long sequences of exploring, editing, and testing, and these recorded trajectories are increasingly used as training data — so how they are collected matters. The common approach samples many trajectories per issue independently, which becomes increasingly redundant: later samples revisit the same repair targets and behaviors. The authors show that earlier trajectories contain signals about where extra collection effort would be useful, and they design a collector that exploits those signals without ever showing the history to the agent, avoiding a train–test mismatch.

Key Contributions

1. An empirical diagnosis of redundant independent sampling. The paper quantifies diminishing returns across five independently sampled attempts per issue and shows substantial patch-target overlap across all issue types, including issues where every attempt fails.

2. A separation of "where" from "what." The central design principle is that history should guide where collection effort is spent, while the coding agent decides what to generate from its ordinary context. This is positioned against prior work that injects history as advice into the agent's context.

3. SWE-Trail, a two-stage trajectory collector. An offline "compiler" turns issue history plus bounded read-only repository checks into frozen acquisition records — a detector that recognizes states, a local comparison objective, and a sampling temperature schedule. Online, at detected states, the agent proposes candidate next actions, SWE-Trail executes them briefly from the same checkpoint in isolated workspaces, and an identity-blind selector keeps the better-supported route.

4. A controlled downstream evaluation. Rather than measuring collection success, the authors train student models on the collected trajectories and measure resolution on SWE-bench Multilingual and SWE-bench Pro, with ablations that randomize the detection location or the retention decision, plus cost and history-scale analyses.

Main Findings

• Independent sampling saturates. With one attempt per issue, 58.4 issues are solved in expectation (over the 300 collection issues); going from 1 to 2 attempts adds 13.1 solved issues, while reaching 3, 4, and 5 attempts adds only 8.0, 5.1, and 3.4 more. Expected solved issues are 58.4, 71.5, 79.5, 84.6, and 88.0 for k = 1 to 5.

• Heavy patch-target overlap even in failure. A single attempt already covers 90.0% of the five-attempt file union on stable-solved issues, 73.2% on mixed issues, and 80.1% on never-solved issues. On never-solved issues the mean pairwise Jaccard overlap of modified files is 0.841, with 93 of the 198 issues with a defined score at exactly 1.0.

• Mixed histories reveal decision points. An audit of 43 of the 51 mixed issues found, using GPT-5.6-Sol, 26 issues with a comparable success/failure decision point, and using GLM-5.1, 31 issues; 20 were identified by both.

• Main downstream result (124B student). Training on SWE-Trail-H5 trajectories improves mean resolution over independent sampling (SeedBank) by 2.33 percentage points on SWE-bench Multilingual (65.67 vs 63.33) and 1.60 points on SWE-bench Pro (42.82 vs 41.22). Every H5 run exceeds every SeedBank run on both benchmarks; an exact one-sided permutation test gives p = 0.05 on each.

• Collection success does not explain the gain. The no-history control H0 produces more resolved trajectories than SeedBank (329 vs 292) but improves the student by only 0.11 points on Multilingual (63.44) and 0.73 on SWE-Pro (41.95). H5 adds only 16 more resolved trajectories than H0 yet gains a further 2.22 and 0.87 points.

• Advice-in-context hurts. The adapted SWE-Exp condition, where the agent reads an experience message compiled from prior attempts, falls below SeedBank by 1.33 points on Multilingual (62.00) and 2.78 on SWE-Pro (38.44).

• Both collection decisions matter. Randomizing where to compare (random detection) lowers mean resolution by 1.44 and 1.28 points; randomizing which executed route continues (random retention) lowers it by 0.89 and 1.19 points. Both variants still sit above SeedBank, and their ranges overlap H5's.

• Gains transfer to a smaller student. With Qwen3-14B-Base, H5 improves over SeedBank by 3.11 points on Multilingual and 2.42 on SWE-Pro, and over H0 by 1.67 and 1.60.

• More history brings smaller gains at higher cost. H10 (ten prior attempts) reaches 65.89 on Multilingual and 42.95 on SWE-Pro — only 0.22 and 0.14 points above H5 — while acquisition-only tokens rise from 307M to 510M (a 66.0% increase) and total collection tokens by 12.9%. H5 itself uses at most 3.7% more tokens than H0, excluding the cost of collecting the prior attempts, and its ordinary rollouts use 4.4% fewer tokens.

• Selection is not a substitute, but is complementary. Selecting 5 of 10 independent trajectories (Top-5-of-10) reaches 64.11 and 41.86, still 1.56 and 0.96 points below H5. Applying the same verifier to five SeedBank plus five H5 trajectories (Top-5-hybrid) reaches 66.22 and 43.14, exceeding H5 by 0.56 and 0.32 points.

• Gains concentrate on mixed histories. Mixed issues are only 51 of the 300 collection issues but contribute 13 of H5's 16 additional resolved trajectories and five of its six additional solved issues; stable-solved histories show no difference, and never-solved histories account for the remaining three trajectories and one issue.

• Mechanism check. Candidate sampling occurs in 1,147 of 1,500 H5 rollouts versus 664 under H0, and both suffixes are executed in 968 versus 560. Candidate diversity is similar (at least two distinct actions in about 90% of sampling cases), and alternative-selection rates among completed comparisons are close (37.4% vs 39.0%), while clear local progress on one route appears in 40.7% of H5 rollouts versus 28.7% of H0.

• Process differences. H5 reduces mean actions from 40.89 to 39.51 and repeated file reads from 7.59 to 7.01 relative to H0, while the share of edited trajectories with a check after the final edit moves only from 88.91% to 89.53%.

Methodology in Plain English

The setup is deliberately controlled. Every condition collects five trajectories for each of the same 300 SWE-rebench V2 issues, giving 1,500 trajectories, using GLM-5.1 with the OpenHands scaffold in an offline sandbox with commit histories removed and no access to reference patches. A 124B Ling-3.0-flash-base student is then trained with full-parameter supervised fine-tuning on 128 H800 GPUs, three independent runs per dataset, and evaluated on SWE-bench Multilingual (300 tasks) and SWE-bench Pro (731 tasks). Trajectories are used whether or not they resolve the issue. Qwen3-14B-Base serves as a second student.

SWE-Trail works in two stages. Offline, a compiler reads the issue statement, repository snapshot, and prior attempts, extracts signals (decisions backed by successful attempts, differences between repair routes, recurring failures), runs bounded read-only repository checks to settle open questions, and freezes an acquisition record: a detector whose rules match states rather than fixed step numbers, a comparison objective for that state, and a temperature schedule — higher for mixed and never-solved histories. Proposed detector states are validated against evidence and replayed on recorded attempts, and unsupported ones are discarded.

Online, when a rollout hits a detected state, the agent samples several candidate next actions using only its ordinary context; history, the acquisition record, and the objective are never added. Equivalent proposals are grouped, the default candidate is kept, and at most one executable alternative is shortlisted. Two isolated workspaces are restored from the same checkpoint, and both candidates run for a short, equal horizon. The selector sees the two resulting suffixes in shuffled order, blind to which is the default, and keeps the alternative only if the executed evidence clearly favors it. Each trajectory performs at most one such comparison, and only the selected route is exported — the agent's own reasoning and actions are preserved, and private collector events and the unselected suffix are removed.

Ablations isolate each ingredient: H0 runs the same pipeline with no history; random detection keeps H5's procedure but picks a random eligible state; random retention keeps H5's detector but picks the continuing route at random. H10 tests history scale, and Top-5-of-10 and Top-5-hybrid test verifier-based selection.

Why This Matters

The paper reframes trajectory collection as a data-quality problem rather than a solve-rate problem: a collector can produce more successful-looking trajectories while adding little training value, and conversely, a modest number of extra resolved trajectories can carry large downstream gains.

Impact on research: It provides a concrete alternative to the dominant post-hoc curation paradigm (filtering, masking tool errors, pruning steps) and to experience-in-context methods, with a randomized ablation showing that both the location of extra effort and the choice of route contribute. The H0 vs H5 contrast is a useful methodological warning for anyone evaluating data-collection pipelines by collection yield.

Real-world applications:

• Building post-training datasets for enterprise coding assistants without the cost of brute-force resampling per issue.
• Automated issue resolution in large codebases, where redundant exploration wastes compute on already-covered repair targets.
• Data flywheels where a model's own failed attempts on hard issues inform where to spend further collection.
• Extension to other tool-using agents that support state restoration, such as web navigation.

Industry relevance: The work comes from a collaboration between the University of Connecticut and Ant Group, addresses a production-scale bottleneck (data collection cost versus training value), and reports token overhead explicitly — H5 stays within a few percent of the no-history control, which matters for anyone budgeting large-scale rollout collection.

Future Directions

• Generalize beyond a single collection model and scaffold. The authors note that experiments use two student models but one collection model (GLM-5.1) and one agent scaffold (OpenHands), leaving other combinations untested.

• Update history across collection rounds and allocate per-issue budgets. The current setting freezes history within each round and spends the same collection budget on every issue, which the authors identify as a limitation given that gains concentrate on mixed histories.

• Improve candidate proposal diversity. The exploratory replay audit in Appendix G.1 motivates testing more varied candidate actions at detected states.

• Extend the framework beyond coding. Applying SWE-Trail elsewhere requires environments that support state restoration and feedback for comparing alternatives, with web navigation named as one candidate direction.

Target Audience

Researchers and engineers working on agentic post-training data, synthetic trajectory generation, and coding-agent systems — particularly those who build data-collection pipelines and care about downstream utility rather than collection success rate. It is also relevant to practitioners designing test-time search or experience-reuse methods for agents, since the paper's ablation directly contrasts collector-side history with agent-visible advice. Readers seeking a gentle introduction to agent training will need background in supervised fine-tuning and repository-level benchmarks such as SWE-bench.

Authors’ abstract

Coding agents solve repository issues through long sequences of exploration, editing, and validation. These trajectories are increasingly used as post-training data, so how they are collected shapes what the trained model learns. A common approach is to sample multiple trajectories independently for each issue and then select or curate them. However, as more trajectories are sampled independently, later trajectories become increasingly redundant, even though earlier trajectories could guide where subsequent collection effort should be spent. A natural way to use this history is to expose it to the coding agent as advice. But training on such trajectories can make the model rely on advice that is unavailable at test time, creating a train-test mismatch. Instead, we argue that history should guide where collection effort is spent, while the coding agent decides what to generate. We introduce SWE-Trail, a trajectory collector that uses earlier trajectories to guide subsequent collection while keeping that history outside the coding agent's context. During a new rollout, SWE-Trail uses history to identify a state where additional collection effort may be useful. There, the coding agent proposes candidate next actions from its ordinary context, and SWE-Trail briefly executes them from the same checkpoint to decide which route continues. We evaluate collection approaches by training student models on the trajectories they produce. For a 124B student initialized from Ling-3.0-flash-base, SWE-Trail improves mean resolution over independent sampling by 2.33 and 1.60 percentage points on SWE-bench Multilingual and SWE-bench Pro, respectively. Without history, the same pipeline produces more resolved trajectories than independent sampling, but improves by only 0.11 points on Multilingual. These results show the value of using history to guide collection without exposing it to the coding agent.