ReTerm: Recovery Training for Self-Improving Terminal Agents

AI summary

Overview

Research area: AI agents for terminal/command-line environments, specifically post-training of large language models (supervised fine-tuning plus reinforcement learning) for error recovery in long-horizon coding tasks.

Technical level: Advanced. The paper assumes familiarity with SFT, GRPO-style reinforcement learning, rollout/verifier pipelines, and agent harnesses.

Scope: The paper introduces ReTerm, a training framework (dataset + RL algorithm) that teaches a terminal agent to recover from failure states produced by its own execution, evaluated on Terminal-Bench 2.0, OpenThoughts-TBLite, SWE-bench Verified, and four out-of-domain benchmarks.

What This Paper Is About

Terminal agents can handle complex coding tasks, but when errors occur during long execution — command errors, faulty implementations — they often fail to diagnose the cause and keep acting in the wrong direction, which eventually causes task failure. The authors build an agent that learns specifically how to recover from its own failed attempts, rather than from clean environments or from hand-injected errors. The goal is a self-improving agent whose recovery ability co-evolves with its task-solving ability during training.

Key Contributions

1. ReTerm, a recovery-oriented post-training framework for terminal agents, supported by a dedicated data pipeline and the ReTerm-10K dataset — roughly 10K high-quality recovery trajectories that supply demonstrations for SFT cold start. Unlike conventional trajectory synthesis that starts from a clean environment, ReTerm-10K teaches continuing from a faulty environment state, identifying what went wrong, and repairing the remaining problems.

2. ReTerm-GRPO, an online RL algorithm that forks from the policy's own failure states and generates multiple independent self-recovery attempts from the same state, using verifier outcomes as group-relative learning signals so that self-generated failures become training opportunities.

3. A recovery task queue that stores recovery tasks across training steps (new tasks inserted at the front and sampled first; oldest removed when capacity is exceeded), so recovery training can proceed even when a batch yields too few failed rollouts, while reducing mismatch with the current policy's failure distribution.

4. Extensive evaluation showing strong results on terminal and coding benchmarks, generalization across out-of-domain tasks and across four agent harnesses, and higher recovery rates than standard GRPO across five error categories.

Main Findings

• Core benchmark gains: Built on Qwen3.5-9B, ReTerm-9B reaches 34.8% Pass@1 on Terminal-Bench 2.0, 60.6% on OpenThoughts-TBLite, and 58.4% on SWE-bench Verified — improvements of +14.6, +16.4, and +13.2 points over the Qwen3.5-9B base model (base values 20.2, 44.2, and 45.2 respectively).

• Comparison to same-scale and larger agents: ReTerm-9B outperforms TMax-9B on all three benchmarks (TMax-9B: 57.2 OpenThoughts-TBLite, 27.2 Terminal-Bench 2.0, 53.5 SWE-bench Verified), surpasses Socratic-SWE (9B, 50.4 on SWE-bench Verified) on SWE-bench Verified, and outperforms several 32B specialized agents including Terminal-World-32B (31.5 Terminal-Bench 2.0) and OpenThinkerAgent-32B (54.0 SWE-bench Verified). It also substantially outperforms TermiGen-32B (19.3 Terminal-Bench 2.0, 21.4 SWE-bench Verified), which is trained on synthesized error-correction trajectories.

• Cross-harness generalization: Keeping Terminal-Bench 2.0 tasks fixed, ReTerm-9B improves over the base model under the training harness (ReAct, 20.2 → 34.8, +14.6), Terminus-2 (22.5 → 32.6, +10.1), mini-SWE-agent (19.1 → 25.8, +6.7), and OpenHands (15.7 → 28.1, +12.4). Gains are largest when training and evaluation harnesses match but remain substantial under all alternatives.

• Out-of-domain generalization: ReTerm-9B improves on all four out-of-domain benchmarks: AIME 2024 (76.7 → 86.7, +10.0), AIME 2025 (73.3 → 93.3, +20.0), Aider Polyglot (17.3 → 20.4, +3.1), and SpreadsheetBench Verified (37.8 → 48.8, +11.0).

• Ablation on Terminal-Bench 2.0: Base 20.2, SFT 25.8, GRPO 28.0, ReTerm-GRPO 34.8 — every stage helps, and ReTerm-GRPO substantially outperforms standard GRPO.

• Error-category recovery: Across five error types (Planning, Tool Use, Implementation, Verification, Resource Management), ReTerm-GRPO shows higher recovery rates than both the base model and the GRPO baseline. However, except for Tool Use Error, ReTerm-GRPO's recovery rates remain at or below 32% across the other categories, indicating that most errors still go unrecovered.

• Training dynamics: During ReTerm-GRPO, both original-task and recovery-task reward groups improve rapidly early and then stabilize; the recovery group shows the larger improvement from its initial performance, while original-task reward also improves and remains stable.

• Dataset composition (ReTerm-10K): Nine domains with relatively even shares (8.4–14.2%), with file operations the largest share. For recovery continuations, the 4–8k output-token range is most common (37.1%) and 69.6% contain 2–8k output tokens; 9–16 assistant turns is the most common range (38.0%) and 87.8% finish within 32 turns.

• Not reported: The paper does not state the numerical capacity K of the recovery task queue, the number of attempts sampled per task in the data pipeline, or per-category recovery-rate values (these appear only in Figure 5).

Methodology in Plain English

The authors split their approach into a data stage and an online learning stage.

Building recovery data (ReTerm-10K). They start from tasks sampled from TMax-15K. The base model (Qwen3.5-9B) attempts each task multiple times, executing in task containers hosted inside virtual machines. At the end of an attempt, the virtual machine is paused, a snapshot of its running state is taken, and the verification is run afterwards so the verifier's actions do not contaminate the state. Failed attempts — their snapshots and full interaction histories — are retained. Then a stronger teacher model, DeepSeek-V4-Pro, resumes from each saved failure state and continues the task, conditioned on the original instruction and the student's history. The teacher is told only that the previous attempt failed and receives no detailed feedback from hidden tests, so it must infer the unresolved problems from the execution history and restored environment; a structured recovery workflow guides it to audit inherited state, reconstruct the task contract, diagnose the failure, and decide what state to keep, repair, isolate, or discard. Candidates then pass four filters: verifier filtering (must pass the task verifier), format filtering (malformed messages/tool calls removed), compliance filtering (actions violating task constraints), and efficiency filtering (meaningless action loops repeating unsuccessful commands). Verifier and format filtering use deterministic rule-based checks; compliance and efficiency first use rule-based flagging and then DeepSeek-V4.1-Flash to judge each flagged case in context.

SFT cold start. Training examples take the task instruction, the student's failed interaction history, and a minimal recovery prompt as input, with loss computed only on teacher-generated tokens; the detailed recovery workflow used during data generation is removed at training time to reduce dependence on that guidance.

ReTerm-GRPO. For each original task, the policy generates G independent rollouts from the initial state and scores them with the task verifier. For each task group with at least one failure, one failed rollout is sampled and its interaction history and resulting environment state are preserved as a recovery task. Recovery tasks are stored in a queue, sampled in later iterations, and each is rolled out G times from the failure state with a recovery prompt. Group-relative advantages are computed separately for original-task groups and recovery-task groups and the two objectives are combined for joint optimization, using a clipped GRPO-style objective with a normalized advantage. Training used 16 NVIDIA A100 (80GB) GPUs; ReTerm-10K was split into 6K trajectories for SFT and 4K task instances for RL; full-parameter SFT used AdamW with global batch size 16 and learning rate 5×10⁻⁶; ReTerm-GRPO was initialized from the SFT checkpoint and used AdamW with learning rate 5×10⁻⁷, sampling 8 original task groups and 2 recovery task groups per update with 16 rollouts per group (global batch size 160), a maximum generation length of 8,192 tokens, a 65k context window, and at most 32 interaction turns.

Evaluation. A lightweight ReAct-style harness exposing a Bash interface was used, with a 256k-token budget per task, maximum generation length 16k tokens, temperature 1.0, top-p 1.0, and task seed fixed at 0. Qwen3.5-9B was evaluated under exactly the same harness and configuration; other models' results are taken from their original papers or official releases when available.

Why This Matters

Impact on research. Prior terminal-agent work focuses mainly on improving task execution, and prior recovery work (TermiGen) injects hand-crafted errors and relies solely on SFT. This paper shows that errors the policy actually produces can be turned into RL training signal, and that recovery-oriented training does not destabilize task-solving reward. It also provides an open dataset and algorithm, plus evidence that these gains transfer across harnesses and out-of-domain tasks rather than being tied to one interaction setup.

Real-world applications (from the paper's framing of terminal tasks and the benchmarks used):

• Long-running software engineering workflows: resolving real-world GitHub issues, as measured by SWE-bench Verified.
• Repository and repository-state maintenance tasks where prior agent actions have left broken or partial state (the "inherited state audit" and repair protocol).
• Automated shell-based system administration and environment configuration work, where invalid commands, wrong arguments, or polluted environments must be diagnosed and repaired.
• Out-of-domain agentic work such as spreadsheet manipulation (SpreadsheetBench Verified) and multi-language coding (Aider Polyglot), suggesting recovery training transfers beyond terminal-specific tasks.

Industry relevance. The results place a 9B open model above several 32B specialized agents on the reported terminal and coding benchmarks, and the gains under harnesses like Terminus-2, mini-SWE-agent, and OpenHands matter because real deployments use whichever harness is available. The finding that recovery rates stay at or below 32% for four of five error categories also gives a clear signal that current agents remain far from reliable in long-horizon execution.

Future Directions

• Push recovery rates beyond the current ceiling. Except for Tool Use Error, ReTerm-GRPO's recovery rates remain at or below 32% across the other four categories, which the authors interpret as substantial room for improvement.
• Reduce reliance on a queue and better match the policy's current failure distribution. The queue favors recent tasks but still mixes in older failures; the paper does not report the queue capacity, so the sensitivity of results to that design choice is an open question.
• Extend recovery training beyond the terminal and coding benchmarks tested. The paper observes positive transfer to AIME 2024/2025, Aider Polyglot, and SpreadsheetBench Verified, but does not study other domain families.
• Close the gap between matched and mismatched interaction setups. The gain under the training harness (14.6 points) exceeds the gains under Terminus-2, mini-SWE-agent, and OpenHands (10.1, 6.7, 12.4 points), raising the question of how to obtain the full benefit regardless of harness.

Target Audience

Researchers and engineers working on LLM agents for terminal, coding, and software-engineering tasks; practitioners building post-training pipelines (SFT plus verifier-based RL) who care about rollout infrastructure, data curation, and failure-state management; and anyone evaluating agent reliability in long-horizon execution, especially readers interested in self-improvement from self-generated failures rather than injected or synthetic errors.

Authors’ abstract

Terminal agents have shown strong abilities in complex coding tasks. However, they often struggle to recover from errors during long-running execution (e.g., command errors, faulty implementations), eventually leading to task failures. To this end, we present ReTerm, a self-improving terminal agent trained by learning how to recover from its own failed terminal tasks. To support its training, we construct a customized data pipeline and produce ReTerm-10K, a high-quality dataset that provides effective recovery demonstrations for SFT cold start. Furthermore, we propose ReTerm-GRPO, an online RL algorithm that forks from the agent's own failure states to generate multiple independent self-recovery attempts. By learning from whether these attempts successfully complete the task, the agent self-improves through its own recovery experience. Built on Qwen3.5-9B, our ReTerm-9B achieves 34.8% Pass@1 on Terminal-Bench 2.0, improving over the base model by 14.6 points. Further analysis shows that our method achieves consistently higher recovery rates across diverse error categories compared with standard GRPO training. All code, data and model are released at https://github.com/twothsnd/ReTerm.