Large-Scale Terminal Agentic Trajectory Generation from Dockerized Environments

AI summary

Overview

• Research area: Natural Language Processing / machine learning for AI agents — specifically post-training data construction for terminal-based (command-line) agents.
• Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, agent scaffolds, Docker, pass@k sampling, and repository-level code benchmarks.
• Scope in one sentence: The paper describes a pipeline that turns real GitHub repositories into Dockerized execution environments, generates terminal task instances and agent trajectories inside them, filters those trajectories with executable validation code, and trains Qwen2.5-Coder models on the result.

What This Paper Is About

Training agents to work in a terminal requires large numbers of long, realistic interaction traces, but two things make such data hard to build at scale: every task needs its own runnable environment, and there is no single standard way to check whether a task was actually completed. The authors build TerminalTraj, a pipeline that scores repositories with a trained model to pick ones that can be Dockerized, automatically writes task queries plus executable validation code (pytest suites) for each instance, and rolls out agents to produce trajectories that are kept only if the validation code passes.

Key Contributions

1. A scalable environment-construction method. Instead of filtering repositories by stars or commit counts, the authors train a reward model called ScoreModel that assigns each code file a score in [0, 1], and define a repository quality score as the mean of its files' scores. Repositories below τ = 0.2 are discarded. This yields 32,325 Docker images across eight programming languages.
2. Docker-aligned instance generation. The pipeline pairs documentation (.md) with execution logic (.sh) from the same repository and prompts an LLM (Qwen3-Coder-480B) to reverse-engineer the user intent, producing both a task query and executable validation logic. Domain-specific instances are additionally grounded in eight specialized domains using extra context such as data files, images, videos, model checkpoints, and web links.
3. Executable, state-based verification. Rather than accepting LLM judgments of success, the pipeline runs generated pytest validation code against the final environment state after the agent finishes, keeping only trajectories that pass. Sampling is run four times per query because Qwen3-Coder-480B's pass@k was observed to saturate at k = 4.
4. A released dataset and trained models. The result is 50,733 verified trajectories (from 899,741 collected repositories, 2,010,113 code files, and 20,000 additional input links, and 1,030,695 generated instances), plus TerminalTraj-7B, TerminalTraj-14B, and TerminalTraj-32B models trained on it.

Main Findings

• Large gains over backbones: On TerminalBench (TB), the Qwen2.5-Coder backbones score low — 6.25 / 0.00 (7B), 6.25 / 1.18 (14B), and 5.00 / 4.49 (32B) on TB 1.0 / TB 2.0 — while TerminalTraj models reach 23.01 / 10.10 (7B), 28.91 / 19.10 (14B), and 35.30 / 22.00 (32B), i.e. absolute gains exceeding +20 points on TB 1.0 and +10 points on TB 2.0.
• Strong results under 100B parameters: TerminalTraj-32B scores 35.30 on TB 1.0 and 22.00 on TB 2.0, described as state-of-the-art among models with fewer than 100B parameters, and outperforms the 30B-class baseline Qwen3-32B-Nex-N1 (28.75 / 16.70) by 6.55 and 5.3 points.
• Beats several much larger models: On TB 1.0, TerminalTraj-32B surpasses Qwen3-235B-A22B-Instruct (15.00 / 13.50) and DeepSeek-V3.1-Nex-N1 (31.56 / 31.80) by 20.30 and 3.74 points, and is comparable to Qwen3-Coder-480B-A35B-Instruct (37.50 / 23.90) with far fewer parameters. Closed-source models remain the strongest overall, e.g. GPT-5.2 (54.38 / 54.00), Gemini-3-Pro-Preview (46.35 / 56.90), Claude-Opus-4.5 (47.50 / 57.80).
• Better test-time scaling: For k ∈ {1, 2, 4, 8, 16}, TerminalTraj models show pass@k increases of 20%–30% from k = 1 to 16, and TerminalTraj-32B reaches pass@16 of 63% (stated as 63.75% in the conclusion). All Qwen2.5-Coder backbones stay below 35% at pass@16, and Qwen3-Coder-480B's gains diminish beyond k = 4 while TerminalTraj-32B keeps improving.
• Code verification beats LLM judging: Fine-tuning Qwen2.5-Coder-32B-Instruct on 1K, 2K, 4K, and 8K trajectories selected by executable validation consistently beats trajectories judged successful by GPT-5.1 without execution, with the largest advantage at 1K–4K instances.
• More efficient than the closest dataset: Compared with Nex-N1 (69,008 trajectories), TerminalTraj (50,733 trajectories) produces gains 1.7× and 1.3× higher on TB 1.0 and TB 2.0 respectively, despite fewer trajectories and a weaker backbone.
• Environment diversity aids scaling: The related SETA work is limited to 400 curated environments; TerminalTraj reports pass@8 of 45.00% versus 12.5% for SETA.
• Repository scoring works: Building from repositories scoring above 0.5 gives a 19.5% Docker build success rate, nearly double random Dockerfile-based selection (9%); the more inclusive 0.2 threshold still gives 17%. The overall verified-trajectory rate is about 4%.
• Instance quality tracks repository score: Stratifying instances into five groups by 0.2-wide intervals over [0, 1] and sampling 20 per stratum, both human raters and LLM judges found quality increasing with repository score; with the 0.2 cutoff, most instances score above 5.5 on a 0–8 scale across Task Soundness, Evaluability, Information Sufficiency, and Behavioral Constrainedness.
• Domain data matters: Removing any single domain hurts; removing WS costs 8.9% / 7.4% on TB 1.0 / TB 2.0 and removing QEMU costs 9.7% / 8.7%. A size-matched control (Full-domain 9k = 29) still beats domain-removed variants such as w/o WS (13) and w/o QEMU (21).
• Performance keeps rising with data: In an appendix scaling study, TB 1.0 performance goes from 28 at 8K verified trajectories to 32 at 32K and 35 at the full dataset.
• Retention is stable across languages: Per-language verified-trajectory rates range from 3.04% (C) to 8.21% (GO) while the overall figure is 4.92%, and Docker build success from selected repositories is about 17% throughout.

Methodology in Plain English

The authors treat open-source repositories as raw material. They crawl roughly 900K GitHub repositories in eight languages (Python, C++, C, Java, JavaScript, PHP, HTML, GO), preferring ones that already contain Dockerfiles and build configuration, excluding repositories created after the TerminalBench release to avoid leakage, and filtering files that match test reference solutions. A trained reward model then scores every file for completeness and executability, and repositories are averaged into a single quality score, with anything below 0.2 dropped; dependency injection fills gaps for domain-specific tasks that lack Dockerfiles. From the surviving environments, an LLM reads paired Markdown and Shell files and writes both a plausible user request and validation code in the form of pytest suites. An agent running on the Terminus-2 scaffold then works inside each container through a normal terminal, seeing only stdout and stderr, and its trajectory is kept only if the validation code passes against the resulting environment state. Because agent runs vary, each query is attempted four times. Finally the collected trajectories are used for multi-turn supervised fine-tuning with Megatron-LM on Qwen2.5-Coder backbones (chosen because they predate TerminalBench, unlike Qwen3-Coder), and evaluated on TB 1.0 and TB 2.0 with Terminus-2, using four independent runs and confidence intervals for models without published results. The pipeline also enforces license filtering (the final 32,325 repositories are MIT 66.30%, Apache-2.0 24.03%, BSD-3-Clause 8.62%, BSD-2-Clause 1.05%), removes 1,184 trajectories containing copied repository code via N-gram duplicate detection, and removes 2,580 trajectories with potentially sensitive content using regex rules.

Why This Matters

The paper argues that the field's bottleneck is not model architecture but the supply of executable, verifiable training environments. It shows that grounding supervision in actual container execution rather than LLM judgment changes both accuracy and how well models convert extra inference compute into results at test time.

Real-world applications:

• Software engineering automation: agents that localize and fix issues inside real repository environments, in the style of SWE-bench-style workflows.
• System administration: tasks involving tools such as nginx or QEMU-related operations, which the ablation identifies as especially valuable for general terminal competence.
• Automated debugging and environment setup: agents that resolve dependencies, inspect filesystem state, and repair failing builds.
• Domain-specific pipelines: SQL, data processing, security, model training and evaluation, and multimodal tasks that require external files, images, videos, or checkpoints.

Industry relevance: the data recipe matters commercially because it produces competitive agentic performance from a 32B model, with the paper reporting parity to Qwen3-Coder-480B while using more than an order of magnitude fewer parameters. For teams training coding agents, that is a direct cost and latency argument, and the released code and data lower the barrier to replicating the pipeline.

Future Directions

• Reinforcement learning on the same environments: the authors state they prioritized SFT but note that the interactive tasks naturally support online RL, which they expect could improve generalization further.
• Scaling to larger backbones: training was capped at Qwen2.5-Coder-32B-Instruct because of compute; the monotonic 7B→32B improvement suggests larger backbones may yield more.
• Extending beyond the current coverage: the pipeline is instantiated for eight programming languages and eight specialized domains, leaving room to add languages and domains.
• Safety and deployment constraints: the paper notes that automated privacy and security filtering may not achieve perfect recall, and calls for rigorous sandboxing (such as gVisor or virtual machines) and user confirmation for consequential operations when such agents are deployed outside controlled environments.

Target Audience

Researchers and engineers working on agentic post-training, agent data synthesis, or repository-level code models; practitioners building terminal or DevOps agents who need executable training environments; and benchmark-oriented readers interested in TerminalBench results and test-time scaling behavior. Readers without a background in supervised fine-tuning, agent scaffolds, or Docker-based evaluation will find the pipeline details harder to follow.

Authors’ abstract

Training agentic models for terminal-based tasks critically depends on high-quality terminal trajectories that capture realistic long-horizon interactions across diverse domains. However, constructing such data at scale remains challenging due to two key requirements: \textbf{\emph{Executability}}, since each instance requires a suitable and often distinct Docker environment; and \textbf{\emph{Verifiability}}, because heterogeneous task outputs preclude unified, standardized verification. To address these challenges, we propose \textbf{TerminalTraj}, a scalable pipeline that (i) filters high-quality repositories to construct Dockerized execution environments, (ii) generates Docker-aligned task instances, and (iii) synthesizes agent trajectories with executable validation code. Using TerminalTraj, we curate 32K Docker images and generate 50,733 verified terminal trajectories across eight domains. Models trained on this data with the Qwen2.5-Coder backbone achieve consistent performance improvements on TerminalBench (TB), with gains of up to 20\% on TB~1.0 and 10\% on TB~2.0 over their respective backbones. Notably, \textbf{TerminalTraj-32B} achieves strong performance among models with fewer than 100B parameters, reaching 35.30\% on TB~1.0 and 22.00\% on TB~2.0, and demonstrates improved test-time scaling behavior. All code and data are available at https://github.com/Wusiwei0410/TerminalTraj.