Selecting Long-Horizon Trajectories for Reliable and Efficient Terminal-Agent Training

AI summary

Overview

Research area: Machine learning for agent training — specifically imitation learning of "terminal agents" (agents that operate in command-line/terminal environments) from long teacher trajectories.

Technical level: Advanced. The paper combines empirical agent-training results with a learning-theoretic bias–complexity argument.

Scope: The paper argues that the number of trajectory tokens used to supervise training — the supervision horizon — is an underexplored design axis that controls both agent reliability and training cost, and proposes a selection strategy that beats training on every long-horizon token.

What This Paper Is About

Terminal agents are typically trained by copying long teacher trajectories from start to finish, and it is usually assumed that more supervision is better. This paper asks a question that has largely gone unexamined: how much of each trajectory should actually be kept for training? The authors define this quantity as the supervision horizon, show that reliability does not simply increase with it, and design a trajectory-selection method that gets better results at lower training cost.

Key Contributions

1. Framing the supervision horizon as a design axis. The paper isolates the number of retained trajectory tokens as a first-class training decision affecting both reliability and cost, rather than a fixed preprocessing constant.

2. Demonstrating that reliability saturates rather than monotonically increasing. On Terminal-Bench, a 12K-token horizon solves more tasks than a 16K-token horizon (29±0.7 vs. 26±0.8) while using 30% less training time.

3. A theoretical account of saturation. A bias–complexity bound explains the trade-off: longer supervision reduces temporal supervision bias but increases finite-sample estimation error because late-stage histories are more heterogeneous.

4. Selective long-horizon refinement. A training procedure that learns from short prefixes first, then refines only on the continuations most likely under the resulting warm-start model, outperforming full long-horizon training at equal or reduced data and compute.

Main Findings

• Longer is not always better. On Terminal-Bench, the 12K-token horizon solved more tasks than the 16K-token horizon (29±0.7 vs. 26±0.8) and required 30% less training time, showing reliability saturates and then declines.

• Horizon length changes agent behavior qualitatively. Short horizons produce premature termination, intermediate horizons produce productive error recovery, and long horizons induce over-persistence.

• A principled reason for saturation. The bias–complexity analysis attributes the trade-off to two opposing effects: less temporal supervision bias with longer horizons, but more finite-sample estimation error from the greater heterogeneity of late-stage histories.

• Selective refinement improves success at fixed horizon. At a 16K horizon, the method raised successful attempts from 110±2.7 to 126±2.1 and raised tasks solved in at least six of eight attempts from 9±0.7 to 14±0.6.

• Selective refinement improves efficiency. Using half of the long-horizon data, it still reached 122±2.4 successful attempts while cutting training time by 23%.

• Gains transfer across benchmarks. The approach improved performance from 64±2.6 to 73±2.1 on Terminal-Bench v2.0 and from 137±2.7 to 155±2.2 on OpenThoughts-TBLite.

• Selection beats volume. The paper's headline claim for long-horizon supervision is that choosing the right trajectories matters more than training on all of them.

Methodology in Plain English

The researchers treat the length of the training trajectory as a tunable setting. They train terminal agents with varying numbers of retained trajectory tokens and measure how many tasks get solved, how much training time is consumed, and how the agents fail — for example by stopping too early or by refusing to stop.

To explain the resulting pattern, they use a bias–complexity argument. Keeping more of a trajectory means the training signal covers more of the true task, which lowers bias. But the later portions of trajectories are more varied across examples, so learning from them with a finite amount of data introduces more estimation error. The two effects push in opposite directions, producing the observed saturation point.

Their proposed fix, selective long-horizon refinement, follows from that analysis. Rather than training on full trajectories or on short ones alone, it does two stages: first train on short prefixes to get a reasonable starting model, then continue training only on the long-horizon continuations that the warm-start model finds most likely. This filters out the late-stage data that would add the most estimation error, so the extra supervision is spent where it helps.

Why This Matters

Impact on research. The paper reframes a routine preprocessing decision — how much of a trajectory to keep — as a source of both reliability and cost improvements, and backs the reframing with a theoretical explanation rather than only empirical curves. It also suggests that trajectory selection, not trajectory volume, is the more promising lever for long-horizon agent training.

Potential applications:

• Training agents that automate software and system administration work in terminal environments.
• Building developer tooling agents that run commands, inspect output, and recover from errors.
• Reducing the compute budget for fine-tuning agentic models on long interaction logs.
• Curating large-scale agent training corpora by filtering late-stage trajectory data instead of using it wholesale.

Industry relevance. Training on long trajectories is expensive, and the abstract reports both a lower-cost operating point (30% less training time at 12K) and a data-reduction result (half the long-horizon data, 23% less training time). Teams that fine-tune agents on logged terminal sessions have a direct lever for cutting cost while improving task success and consistency.

Future Directions

• Generalizing the saturation point. The reported behavior is tied to Terminal-Bench and related benchmarks; whether the same horizon trade-off appears in other agentic domains and environments is an open question.
• Theory-guided selection criteria. The bias–complexity account motivates selecting likely continuations, but the abstract leaves open how broadly that criterion should be defined and whether other selection signals work as well.
• Interaction with model and data scale. The abstract does not describe how the optimal horizon shifts with model size, trajectory length, or the amount of available trajectory data.
• Adaptive horizon schedules. Instead of a fixed horizon or a two-stage prefix-then-refine procedure, the findings suggest value in schedules that change the supervision horizon during training.
• Connecting horizon choice to failure modes. Since short, intermediate, and long horizons produce premature termination, productive recovery, and over-persistence respectively, a natural next step is diagnosis tools that pick a horizon for a specific observed failure mode.

Target Audience

Researchers working on agent training, imitation learning from demonstrations, and long-horizon decision-making will get the most from this paper, particularly those interested in the theory of when more supervision data helps. It is also relevant to practitioners who fine-tune terminal or tool-using agents on logged interaction traces and need to balance task reliability against training cost. Readers without background in learning theory can still follow the empirical results, but the bias–complexity section requires familiarity with generalization bounds.

Authors’ abstract

Terminal agents are commonly trained by imitating long teacher trajectories, yet how much of each trajectory to supervise remains unexplored. We study the \emph{supervision horizon}, the number of trajectory tokens retained for training, and show that it is a key design axis for reliability and cost. Reliability improves with longer horizons but saturates: on Terminal-Bench, a 12K-token horizon solves more tasks than 16K ($29\pm0.7$ vs.\ $26\pm0.8$) while requiring 30\% less training time. The horizon also shapes agent behavior: short horizons cause premature termination, intermediate horizons yield productive error recovery, and long horizons induce over-persistence. We analyze this saturation through a bias--complexity bound, in which longer supervision reduces temporal supervision bias but increases finite-sample estimation error from more heterogeneous late-stage histories. Guided by this analysis, we propose \emph{selective long-horizon refinement}, which first trains on short prefixes and then refines only on continuations that are most likely under the warm-start model. It consistently outperforms full long-horizon training. At 16K, it raises successful attempts from $110\pm2.7$ to $126\pm2.1$ and tasks solved in at least six of eight attempts from $9\pm0.7$ to $14\pm0.6$; with half of the long-horizon data, it still reaches $122\pm2.4$ while cutting training time by 23\%. The gains transfer across benchmarks, from $64\pm2.6$ to $73\pm2.1$ on Terminal-Bench v2.0 and from $137\pm2.7$ to $155\pm2.2$ on OpenThoughts-TBLite. For long-horizon supervision, selecting the right trajectories matters more than training on all of them.