Scaling Agentic Data for Long-Horizon Terminal Intelligence

AI summary

Overview

Research area: LLM agents for terminal/command-line environments; synthetic training-data generation and supervised fine-tuning for long-horizon autonomous work.

Technical level: Intermediate — the paper is readable without deep RL background, but assumes familiarity with LLM agents, supervised fine-tuning (SFT), and agent benchmarks.

Scope: This paper introduces TerminalHorizon, an automated engine that builds long-horizon terminal task environments and the trajectories to train on, then shows that fine-tuning Qwen3.5-27B on roughly 3K such trajectories (TerminalHorizon-3K) improves performance on 12 benchmarks.

What This Paper Is About

Existing terminal-agent training data produces mostly short trajectories: agents learn to run commands and call tools, but rarely learn to explore, accumulate evidence, revise strategy from feedback, and carry intermediate results into a final deliverable. The authors argue that even complex generated tasks can accidentally hand the agent the answer — if the environment already contains part of the implementation, the agent never has to do the exploration, debugging, or verification the task was meant to teach. The goal is a fully automated pipeline that builds environments where that substantive work is genuinely required, and then extends tasks progressively so those behaviors recur across stages of one problem-solving process.

Key Contributions

1. A framing of long-horizon terminal data around "agentic behavioral patterns" and their cross-stage connections, rather than around task count, difficulty, or interaction turns alone. The authors organize these into 27 patterns across five families (information exploration and judgment; diagnosis and error correction; construction and work progress; verification and assurance; strategy and execution coordination).

2. TerminalHorizon, a two-stage automated synthesis engine: Challenge-Guided Long-Horizon Synthesis (an independent Challenger agent solves each candidate task without access to construction records, and the resulting gaps drive environment reconstruction) plus Progressive Horizon Expansion (accepted tasks are extended under the same goal with new requirements that depend on earlier results and feedback).

3. TerminalHorizon-3K, a dataset of 1,535 task environments and 2,852 training trajectories spanning 25 professional domains, with trajectories averaging 52.8 interaction turns (P90 of 98 turns) and 8.6 core behavior patterns on average.

4. TerminalHorizon-27B, an SFT model from Qwen3.5-27B that improves on its base model across all 12 evaluated benchmarks, plus multiple recursive self-improvement (RSI) case studies with a 12-hour research budget each.

Main Findings

• Terminal-Bench gains: 55.0 on Terminal-Bench 1.0 (base 46.3, +8.7), 55.8 on Terminal-Bench 2.0 (base 41.6†, +14.2), and 60.67 on Terminal-Bench 2.1 (base 46.82, +13.85).

• Data efficiency: TerminalHorizon-27B beats Terminal-Universe-27B by 3.0% on Terminal-Bench 2.0 using the same base model and harness but fewer than one-tenth as many training trajectories (about 3k versus 32.0k).

• Transfer to real-world productivity tasks: AutomationBench Pass Rate rises from 5.5 to 17.2 (+11.7) and Partial score from 26.4 to 49.7 (+23.3); VitaBench Cross-domain from 20.5 to 31.8 (+11.3); τ²-bench Airline from 79.0 to 84.5 (+5.5) and Retail from 84.9 to 87.5 (+2.6).

• General reasoning is preserved, not traded away: MATH-500 98.9 (base 96.9, +2.0), GSM8K 97.3 (+1.0), AIME 2024 96.7 (+3.4), AIME 2025 93.3 (+6.6), KOR-Bench (Cipher) 91.2 (+1.1).

• Trajectories are substantially longer than prior synthesis methods: 52.8 mean turns and 98 turns at P90, versus the compared methods sampled at 300 tasks each under the same teacher protocol; the median longest continuous feedback-linked action chain is 25 turns.

• Gains hold across model sizes: +13.85% for Qwen3.5-27B, +13.11% for Qwen3.5-35B-A3B, and +14.98% for Qwen3.5-122B-A10B over their respective base models.

• Training-data scaling has not saturated: scaling from 1,000 to approximately 3k trajectories raises Terminal-Bench 2.1 to 60.67%, with no observed saturation.

• Teacher model matters for interaction quality: on a 100-task sample, GLM-5.2 produced 46.1 mean turns, a 19.2 mean feedback chain, 15.8% exact repeats, and a 44.0% pass rate; DeepSeek-V4-Pro-0813 produced 53.5 mean turns, a 30.3 mean feedback chain, 13.2% exact repeats, and a 50.0% pass rate, motivating its use as the teacher.

• Category-level gains on Terminal-Bench 2.1: TerminalHorizon-27B matches or outperforms the base model in all 16 categories, improving in 11, with the largest gains in debugging (+46.7 points), data science (+25.0), and machine learning (+22.2).

• Long-horizon RSI behavior: in a multi-view point cloud fusion case, the base model stopped after 12.1 minutes on a self-designed proxy metric, while the fine-tuned model introduced a leave-one-view-out diagnostic, improved cross-view alignment, performed global refinement, and used confidence-based point selection; its hidden evaluation score exceeded the base model's by 27.03%.

Methodology in Plain English

The pipeline starts from real professional work rather than from invented puzzles. A Planner Agent looks at what existing tasks already cover and writes research briefs for new directions; a Deep Research Agent then digs through documentation, code repositories, and issue reports to ground each brief in real technical constraints, producing a task seed. A Construction Agent turns the seed into a full task package — public instruction, executable environment, reference solution, private verifier, and execution configuration — under a rule that the verifier may only check requirements the agent can actually discover from the public materials.

The key check is that a separate Challenger agent, which never sees the construction records, reference solution, or private tests, solves the task from scratch. Its trajectory is analyzed to find places where the environment gave away work: if a core function the task asks for is already implemented, reconstruction restores the need to implement and debug it while keeping the specs the agent needs. The revised task goes back through the Challenger, and records from the whole batch feed the Planner's next round.

For tasks that pass, an Evolution Agent proposes extensions under the same goal along four directions — broader roles, deeper mechanisms, richer evidence, and method upgrades — and wires them in so later work depends on earlier findings, tool effects, or instance-specific state. Trajectories are then collected with DeepSeek-V4-Pro-0813 as the teacher and Terminus2 as the harness (two per task), filtered to 2,852, decontaminated against benchmarks, and used for full-parameter SFT of Qwen3.5-27B (3 epochs, learning rate 1×10⁻⁵, max sequence length 262,144, DeepSpeed ZeRO-3 on 144 H20-141GB GPUs across 18 nodes).

Why This Matters

The paper's central practical claim is that what the environment leaves undone determines what the agent learns — a design principle that applies well beyond terminals. It also shows that data quality can substitute for data volume at this scale: about 3k trajectories outperforming a 32.0k-trajectory baseline on the same base model.

Real-world applications:

• Software engineering and debugging assistants that must diagnose failures from execution feedback rather than apply a known fix — the category with the largest measured gain (+46.7 points on debugging in Terminal-Bench 2.1).
• Enterprise back-office automation, such as the multi-step workflows measured by AutomationBench, where partial credit and full pass rate both improved.
• Customer-service and transactional agents, as measured by the Airline and Retail domains of τ²-bench.
• Scientific and engineering optimization, where agents must keep experimenting past a first plausible solution — the RSI case studies with a 12-hour research budget are the illustration.

Industry relevance: the two-stage recipe (independent challenger to detect leaked work, then dependent expansion) is a general template for anyone generating agent training data, and the reported runtime-defect, information-leakage, and scoring-inconsistency checks address failure modes that plague synthetic environment pipelines.

Future Directions

• How far does the scaling curve extend? The authors observe no saturation from 1,000 to approximately 3k trajectories but do not report results beyond that range, leaving the ceiling open.

• Beyond the terminal. The paper evaluates coding/terminal tasks, agentic productivity benchmarks, and general reasoning; whether the same behavioral-pattern framing transfers to other interfaces (GUIs, APIs, robotics) is not tested.

• From SFT to other training signals. The work uses supervised fine-tuning only; whether these long-horizon trajectories could be used for preference optimization or reinforcement learning is not reported.

• Automatic detection of "leaked work" at scale. The Challenger-based gap analysis is the mechanism that makes the data distinctive; how reliably it catches subtle shortcuts across 25 domains, and where it fails, is a natural question for follow-up work.

Target Audience

Researchers and engineers building LLM agents or agent training data — particularly those working on terminal/command-line agents, synthetic environment generation, or long-horizon task training — will get the most from this paper. It is also useful for practitioners who need to fine-tune a mid-sized open model for autonomous multi-step work and want evidence on data efficiency, plus evaluation teams interested in how agentic training affects general reasoning. Readers without background in LLM agent training will find the introduction and the analysis section accessible but should expect the method section's formalism to be dense.

Authors’ abstract

Terminal agents are moving beyond local operations toward completing real-world professional tasks. Short trajectories can demonstrate commands and tool calls, but they do not fully show how agents accumulate evidence through exploration, use it to inform judgments, continually adjust strategies based on feedback, and turn intermediate results into final deliverables. High-quality long-horizon trajectories capture these agentic behavioral patterns and their connections within a single problem-solving process, providing richer experience for sustained autonomous work. However, environment synthesis may complete or simplify the key work underlying these patterns in advance and thus remove the need for exploration, implementation, and verification that the agent was meant to perform. To address this, we introduce TerminalHorizon, a fully automated data synthesis engine for long-horizon terminal agents. Starting from real-world professional work, it uses independent execution to identify key work omitted during construction and uses the resulting evidence to guide environment reconstruction. To extend these patterns across stages, it progressively expands tasks under the same goal, introducing new requirements that build on prior results and call for further exploration and improvement. Using this engine, we construct 1.5K task environments and perform supervised fine-tuning of Qwen3.5-27B on the resulting dataset to obtain TerminalHorizon-27B. We achieve substantial performance gains on all 12 benchmarks spanning coding-agent tasks, agentic tasks in real-world production settings, and general reasoning. Multiple RSI case studies, each with a 12-hour research budget, further demonstrate the model's ability to continue experimenting beyond an initial solution, improve its methods based on feedback, and retain effective improvements in its final deliverables.