T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks

AI summary

Overview

Research area: Reinforcement learning for large language model agents — specifically post-training a 122B-parameter Mixture-of-Experts model to operate a real Linux shell across long-horizon tasks, with reward derived from executing each task's own verifier.

Technical level: Advanced. The paper is aimed at researchers and engineers working on large-scale RL post-training, MoE training-inference consistency, and agentic evaluation.

Scope: The paper presents T1, a recipe (TITO token stitching, R³ rollout routing replay, a warm-started critic, and a dense per-assertion execution reward) for stable reinforcement learning of a 122B MoE terminal agent, evaluated on Terminal-Bench 2.1, Long-Horizon Terminal-Bench, and Terminal-Bench Hard.

What This Paper Is About

Training an agent to work in a Linux terminal is unforgiving: actions have persistent side effects, and success is judged by running a verifier rather than by matching a preference rubric. The authors' central problem is that reinforcement learning at this scale breaks in two ways — the system that generates a trajectory and the system that scores it disagree about which tokens and which experts were used, and a binary pass/fail reward gives almost no signal when tasks are hard. T1 is their attempt to fix both at once, post-training Qwen3.5-122B-A10B on terminal tasks using only executed outcomes.

Key Contributions

1. T1 itself: a 122B Mixture-of-Experts terminal agent trained purely by reinforcement learning on executed outcomes, operating a real shell in a cloud sandbox for up to 300+ tool-call turns per task, rewarded by running each task's own verifier.

2. A stabilization stack for large-scale sparse agentic RL: Token-In-Token-Out (TITO) construction, which trains on the exact sampled token identifiers with drift repair at turn boundaries; R³ (rollout routing replay), which records the sampler's per-token expert choices at every MoE layer and replays them during training; and a scheduled critic, plus the infrastructure that keeps a co-resident 122B actor–critic pair alive for days.

3. A dense execution reward scored by the absolute number of passing assertions on a fixed global scale, together with its measured behaviour and a candid record of two shaping variants that failed.

4. A fully out-of-distribution training corpus: isolated seeds and synthesized tasks disjoint from Terminal-Bench 2.1, so that measured gains reflect capability transfer rather than benchmark overfitting.

Main Findings

• The two fidelity conditions are independent and both must be enforced. The authors separate token fidelity (the trainer must condition on the identifiers the sampler emitted) from routing fidelity (the trainer's TopK expert selection must match the sampler's). Dense models cannot violate routing fidelity at all, and short-horizon tasks make token fidelity nearly automatic, but a model emitting tens of tool-calling turns violates both across roughly 10⁴ loss-bearing positions.

• TITO and R³ cut the train–inference gap from 0.021 to 0.013. Measured as the mask-weighted mean of |d_j| = |log π_t − log π_{t−1}| over the loss region, the per-step series shows this reduction with an exponential moving average at α = 0.25. The residual is expected, since R³ aligns expert selection but not kernel numerics.

• Zero token drift in the loss region. A production audit over 1,402 samples puts the drift rate inside the loss region at exactly 0.0000%. The routing placeholder set from turn-boundary repair is bounded at 0.003% and is disjoint from the loss region.

• Replaying routing is cheap. A routing record costs Lk · 4 B = 1536 B/token; at (L, k) = (48, 8) this is 1.5 KiB per position, or roughly 48 MiB for a 33k-token trajectory, holding rollout overhead below 3% with zero additional arithmetic.

• Terminal-Bench 2.1 results. Three epochs of PPO on the quality-filtered T1-15k lift the supervised checkpoint from 49.4% to 64.0% resolved — a 28.5% relative gain from RL alone. The abstract states the pipeline raises the initial base model from 43.8% to 64.0%. Under an identical harness, T1 achieves 64.0%, above GPT-5.4 (54.8%) and DeepSeek-V4-Flash (56.9%), close to Claude Opus 4.7 (66.1%), and above Claude Opus 4.6 (63.8%); the authors call it the best model in its size band.

• Gains concentrate where terminal agency is tested. T1 scores 100.0 on debugging and 88.9 on system administration, both surpassing a stronger general-purpose model.

• Longer-horizon and harder benchmarks. On Long-Horizon Terminal Bench, T1 reaches 27.9%, surpassing GPT-5.4 and GLM-5.1 and matching Gemini-3.1-Pro. On Terminal-Bench Hard it reaches 38.0%, ahead of DeepSeek-V4-Pro and well above both the supervised and the base checkpoint.

• Binary reward alone never beat the supervised baseline. The authors report that their first binary-reward campaign never exceeded its supervised baseline, which motivated scoring by the absolute number of passing assertions instead.

• Critic warm-up fixes a badly initialized value function. A cold-started critic opens at EV = −33.6 and is negative for 30 of 58 logged steps, whereas the Critic Warm-Up run never goes negative and plateaus between 0.71 and 0.86. Warm-up trains the value network for one epoch over TMax-15k before any policy step; the dense-reward campaign loads those weights only, needing just N = 2 re-calibration rollouts.

• Critic-to-actor learning-rate ratio matters. The critic runs at 1.5 × 10⁻⁵ against 1.0 × 10⁻⁶ for the actor. On 27B pathfinding runs, moving this ratio from 10× to 20× lifted EV from −39 to +0.11, while 30× shortened Critic Warm-Up further — yet reward stayed flat across these settings, which redirected attention to the data.

• Deliberately spare objective. Surrogate clipping is symmetric at ε = 0.2; both KL terms are disabled because the frozen reference routes with its own selection; the MoE load-balancing coefficient is set to zero because balancing pressure asks the router to redistribute exactly the choices replay asks it to reproduce; optimization uses Adam with β = (0.9, 0.98), weight decay 0.1, and a constant learning rate; γ = λ = 1 and ε_v = 0.2.

• Throughput choices. Training runs at a batch of 560 with rollout oversampling that admits the first 512 trajectories to complete and truncates the tail, using data parallel at 8 — bounding step time at the price of a mild bias against the longest trials.

• Dataset composition skews the failure profile. T1-15k is 15,000 tasks selected from synthesis rounds. Scripting and automation (17.9%), software development (16.5%), system administration (13.8%), environment and package setup (10.4%), and version control (9.3%) together make up two-thirds of the pool (top five = 67.9%), while data science (3.7%), debugging (1.3%), and performance work (1.0%) are thin.

• Audit and selection. The audit weights sum to 1.00: Verifier 45%, Solution 25%, Instruction 20%, Task Value 10%, with instruction–verifier alignment carrying the largest single weight at 20% because a mismatch is a hidden requirement. Selection ran in five stages over 15 rewrite rounds, including a semantic pass yielding 5,902 accepted, 3,251 borderline and 5,847 rejected, and instruction-only repair of 6,875 tasks.

• Other pools. TMax-15k contains 14,601 tasks converted into terminal-bench layout whose verifiers emit only binary outcomes, so only binary rewards are possible there. RST-38k contains 37,484 synthesized tasks. T1-15k's pre-flight confirmed per-assertion records in 93% of sampled tasks, and the pool is materialized in quality-rank order, making per-epoch shuffling mandatory.

• Model anatomy. T1 derives from Qwen3.5-122B-A10B; expert weights account for 116.0B of 121.4B parameters, with each token engaging 8 of 256 experts per layer, and (L, k, E) = (48, 8, 256). The paper states separately that experts hold 95.5% of parameters. Each of the actor and critic must fit alone in 95 GiB.

Methodology in Plain English

The authors take an existing 122B Mixture-of-Experts language model and train it with PPO (the slime framework) to drive a real shell inside a cloud sandbox. Each task ships with its own held-out verifier and a reference solution the agent never sees; the verifier reports each assertion separately, which turns reward from a modeled judgment into an executed one.

Because generation runs on throughput-optimized inference replicas while training runs on an exact-gradient backend, the two disagree in small numeric ways. Two mechanisms close that gap. TITO has the trainer consume the token identifiers the sampler actually emitted, stitching multi-turn interaction logs into one training stream with loss only on sampled tokens; observations and glue enter as masked context. At each turn boundary the assembler tries progressively weaker prefix relations — strict, normalized (a bounded search over a 97 × 17 grid with the 96-token shared template suffix), retokenized, and split — and the first admissible case decides the repair, with a hard rule that the re-tokenized copy never enters the training stream. R³ records the expert selection at every MoE layer during generation and replays that mask in the training forward pass, keeping the softmax normalization on the live logits so the router remains trainable and the computation graph is untouched. Captures are cached alongside the key-value cache and reused on prefix hits; a missing capture aborts the rollout rather than silently dropping the sample.

For reward, each trial's per-assertion outcomes are converted into a dense process reward scored by the absolute number of assertions satisfied on a scale fixed once for the whole run, so harder tasks carry proportionally more signal. The scalar lands on the final response token and the critic distributes credit across the horizon. The critic is trained first each step to anchor the advantage estimator, and it is warm-started on TMax-15k before any policy step. Tasks themselves come from recursive task synthesis: each round extends accepted seeds with additional executable steps, then realigns the environment, verifier, and instruction to the longer workflow, with candidates validated in a fresh sandbox where the reference solution must genuinely pass its verifier.

Why This Matters

Impact on research. The paper's central claim is that reward derived from real execution, rather than a learned preference model, can be optimized stably at frontier scale — and it supplies the two specific mechanisms (token fidelity and routing fidelity) that make the on-policy ratio meaningful for sparse MoE policies. The zero-drift measurement and the explicit separation of version skew from cross-system discrepancy give the field a way to audit rather than assume training-inference consistency. The reported negative results — a binary-reward campaign that never beat its supervised baseline, and two failed shaping variants — are as useful to the community as the successes.

Real-world applications:

• Autonomous software engineering: bisecting hundreds of commits to locate a defect, patching it, rebuilding to a named target, and proving the repair.
• System administration on live infrastructure, where the paper reports an 88.9 score.
• Debugging workflows, where the paper reports 100.0.
• Environment and package setup and version-control operations, the third- and fifth-largest categories in the training pool.

Industry relevance. The work was produced across Tencent Hy Foundation Model Frontier, the National University of Singapore, the University of Georgia, Indiana University, and the University of Maryland, College Park, and the paper frames terminal performance as a step toward agents acting on consequential infrastructure. The engineering focus on keeping a co-resident 122B actor–critic pair alive for days, bounding step time against heavy-tailed completions, and serving 300+ tool-call turns per task is directly relevant to teams attempting production-scale agentic RL. The result that a 122B model outperforms GPT-5.4 and DeepSeek-V4-Flash is framed by the authors as an order-of-magnitude advantage in active parameters.

Future Directions

• Close the residual training–inference gap. The gap after TITO and R³ is 0.013, not zero; R³ aligns expert selection but not the kernel numerics that remain. What further mechanism removes the remainder is left open.
• Address the dataset's category skew. Debugging at 1.3% and performance work at 1.0% of T1-15k are thin relative to scripting and automation at 17.9%; the authors note this skew predicts where residual failures land, but the paper does not report a remedial run.
• Recover from the reward-shaping failures. Two shaping variants that failed carry no separate alternative in what is shown here; whether a different shaping scheme succeeds for the tasks T1 still cannot complete is unresolved.
• Extend beyond terminal tasks. The introduction positions terminal performance as a step rather than an end goal, toward agents acting on consequential infrastructure; how this recipe generalizes to other stateful, verifier-scored environments is not reported.

Target Audience

Researchers and engineers working on reinforcement learning for large language models, particularly those post-training Mixture-of-Experts models at the 100B-plus scale or building terminal and software-engineering agents. It is also relevant to practitioners who need concrete engineering guidance on training-inference consistency, critic initialization, and dense reward design from executed verifiers. Readers without background in PPO, MoE routing, or agentic RL will find the level advanced; the many equations and the truncated numerical results assume familiarity with the underlying optimization machinery.

Authors’ abstract

Agent usage is shifting toward long-horizon tasks such as coding and scientific discovery, among which terminal tasks are especially important. We introduce T1, a Mixture-of-Experts model of 122B total trained with reinforcement learning, operating a real shell in a cloud sandbox for up to 300+ tool-call turns per task, rewarded by executing each task's own verifier. We provide a comprehensive recipe: First, an aggressively warm-started to stabilize actor-critic training, with a dense process reward scoring trajectories by the absolute number of passing verifiers. Second, stable optimization through TITO construction, training on the exact sampled token identifiers with drift repair at turn boundaries, and rollout routing replay, recording the sampler's per-token expert choices at every MoE layer and replaying them during training. Third, fully out-of-distribution training corpus: isolated seeds and synthesized tasks disjoint from Terminal-Bench 2.1 ensures gains reflect genuine capability transfer over benchmark overfitting. Together, TITO and R3 cut the training-to-inference log-probability difference from 0.021 to 0.013, with exactly aligned zero token drift in the loss region. On Terminal-Bench 2.1, our post-train pipeline raises initial base model from 43.8% to T1 with 64.0% resolved. On Long-Horizon Terminal Bench, T1 reaches 27.9% and surpasses GPT-5.4 and GLM-5.1.