Terminal-RSI: Recursive Self-Improving Terminal Agents via Recursive Self-Rewrite and Policy-Refine

AI summary

Overview

Research area: Reinforcement learning post-training for large language model (LLM) agents that operate a real terminal — an area sitting at the intersection of agentic RL, credit assignment, and reward design.

Technical level: Advanced. The paper assumes familiarity with PPO, GAE, actor–critic training, importance ratios, and value-function calibration.

Scope: The paper introduces Terminal-RSI, a five-stage recursive training recipe that alternates verified trajectory rewriting (actor-only) with critic-inheriting policy refinement, plus two credit-assignment and reward-shaping components (TB-GAE, FDR/SPR), and evaluates the resulting 27B-parameter agent on Terminal-Bench 2.1, 3.0, 4.0, Hard, LHTB, and SWR100.

What This Paper Is About

Terminal agents are judged by whether they leave an executable environment in a state that passes a protected verifier, not by whether their explanations sound plausible. Training such agents is hard for two reasons: credit for a final success gets diluted across thousands of observation tokens that involve no decisions, and the value network goes stale every time the policy is updated or the reward is recalibrated. The paper's goal is to convert verified successes into better policies and to convert partial progress into strict task completion, by repeatedly rewriting verified trajectories and then refining the policy while carrying the critic forward instead of resetting it.

Key Contributions

1. A recursive actor–critic training framework. The paper specifies five retained stages — Base, RSR-1, RPR-1, RSR-2, and RPR-2 — that alternate actor-only Recursive Self-Rewrite (RSR) with critic-inheriting Recursive Policy-Refine (RPR). Critic weights survive the supervised transitions, a Warm-up Critic Model is prepared with one SWR epoch on the frozen RSR-1 actor, and each refinement phase begins with four critic-only calibration updates before joint optimization.

2. Turn-Boundary Skip-Observation GAE (TB-GAE). Standard GAE attenuates credit by textual length, so two runs with identical decisions and outcomes can receive different credit if one logs more verbosely. TB-GAE keeps token-specific baselines but applies the decay factors γ_turn, λ_turn only when the interaction turn changes, so crossing B turn boundaries gives attenuation (γ_turn λ_turn)^B regardless of observation length or within-turn token count. The paper also analyzes fixed-input continuation, the truncation error from resetting traces at chunk boundaries, and conditional variance under critic error.

3. Reward evolution: FDR and SPR. Family Dense Reward (FDR) normalizes passed checks by a frozen per-family 0.90 quantile D_f of total non-hygiene calibration checks (fallback 82). Success-Prioritized Reward (SPR) replaces this with S + P/(P + D'_f), so every failure scores below one and every success at least one. Independent rescoring reduces the failed share of reward from 50.09% to 28.72% while preserving outcome labels across reward variants.

4. A staged evaluation across six terminal benchmarks. Each retained checkpoint is scored with Mean@3 on Terminal-Bench 2.1, 3.0, 4.0, and Hard, plus LHTB and SWR100, with complete stage trajectories, discarded alternative branches, and difficulty-stratified and checked-level analyses that separate measured completion, shaped reward, and checker coverage.

Main Findings

• Final benchmark scores: Terminal-RSI-27B reaches 19.84% on Terminal-Bench 4.0, 86.52% on Terminal-Bench 2.1, 16.95% on Terminal-Bench 3.0, and 79.00% on Terminal-Bench Hard.

• Four-suite mean success: The equal-weight mean over Terminal-Bench 4.0, 3.0, 2.1, and Hard rises from 20.68% at Base through 34.25%, 44.50%, 48.75%, to 50.58%. That is 29.90 and 16.33 percentage points over Base and RSR-1, or 37.70% and 24.83% relative to initial failure. The failure complement falls from 79.33% at Base to 49.42%.

• Both stage types contribute: RPR-1 improves over RSR-1 by 7.43, 7.49, 10.57, and 15.50 points on Terminal-Bench 2.1, 3.0, 4.0, and Hard. RSR-2 then adds 8.99, 0.71, 0.79, and 6.50 points on those suites.

• Long-horizon progress becomes strict completion: On LHTB, process reward climbs 0.21 → 0.29 → 0.30 → 0.31 → 0.37, a gain of 0.16 or 76.19% relative to Base. Strictly completed tasks go 0/46 → 0/46 → 2/46 → 4/46 → 5/46 (10.87% at the final stage). RSR-1 improves partial progress without any strict completion.

• Scientific workflow reconstruction doubles, then plateaus: SWR100 success moves 3.00% → 5.00% → 6.00% → 6.00% → 6.00%. The final three stages tie, indicating a plateau rather than a second-cycle gain; the paper notes these tasks have no overlap with the training pool.

• The reward audit exposes a proxy–outcome mismatch: Across 4,352 audited trajectories, failures receive 50.09% of FDR reward mass; 158 failures score at least one, and 48 pass every executed scenario check but fail the terminal gate. SPR rescoring cuts the failed share to 28.72%.

• RSR-first beats direct PPO in this setting: At local step 20, RPR-1 exceeds Direct PPO by 29.22, 12.86, 16.67, and 30.50 points on Terminal-Bench 2.1, 3.0, 4.0, and Hard. Direct PPO records no Terminal-Bench 4.0 success at steps 5, 10, 15, or 20. The paper states this is not a compute-matched comparison because rewriting adds discovery and supervision cost.

• Frontier comparison is benchmark-specific: On Terminal-Bench 4.0, 19.84% is compared against 19.70% for Gemini 3.8 Flash and 14.60% for DeepSeek V4 Pro. On Terminal-Bench 2.1, 86.52% is compared against 85.40% for GPT-6 Sol and 85.00% for Kimi K3. GPT-6 Sol is still higher on Terminal-Bench 4.0 and Gemini 3.8 Flash higher on Terminal-Bench 2.1; the paper notes external scores are reference values whose protocol reconciliation remains incomplete.

• Continuation branches are nonmonotonic: Continuing RPR-1 past step 20 to step 35 drops Terminal-Bench 4.0 from 16.67% to 11.11% while Terminal-Bench 2.1 rises to 78.95%. Ordinary FDR RPR-2 falls from 17.46% at step 6 to 11.11% at step 10 on Terminal-Bench 4.0. The retained SPR branch reaches 19.84% (up 5.55 points from its step 6); at step 18 it beats ordinary RPR-2 on three suites but is 0.19 points lower on Terminal-Bench 3.0. Pushing Long-Reward to step 20 preserves Terminal-Bench 2.1 but loses 2.66, 2.38, and 3.00 points on Terminal-Bench 3.0, 4.0, and Hard.

• Critic calibration holds up under recursive handoff: Explained variance starts at −1.15 during Warm-Up and ends at 0.01; RPR-1 starts at 0.03, peaks at 0.33, ends at 0.27, and includes a transient −0.04. All 16 RPR-2 observations stay positive, in the range 0.01–0.52, across 18 Warm-Up, 17 RPR-1, and 16 RPR-2 logged observations.

• Difficulty-stratified outcomes are uneven: Across 322 benchmark–task pairs in the four-suite subset (118 Easy, 106 Medium, 57 Hard, 41 Extreme), Easy rises from 31.78% to 72.88%, Medium from 23.58% to 58.96%, and Hard from 14.91% to 49.12%. Extreme falls from 10.98% at RPR-1 to 6.10% at RSR-2 before recovering to 10.98% at RPR-2.

• Training signals run on separate clocks: In RPR-1, five-observation means move from 0.11 to 0.16 for dense reward, 0.29 to 0.41 for binary reward, and 50.00 to 84.80 turns. In RPR-2, FDR moves from 0.45 to 0.48 with turns falling 75.80 to 61.40, while SPR moves 0.50 to 0.53 with turns rising 83.80 to 116.80. The absolute train–inference log-probability gap ranges from 4.66 × 10⁻³ to 4.98 × 10⁻³.

Methodology in Plain English

The starting point is Qwen3.8-27B, a 27B-parameter model with a 262,144-token context capacity, trained and served with T1's infrastructure stack (slime for coordination, Megatron for optimization, SGLang for sampling, Harbor for execution).

Step one, rewriting. The team only learns from runs that actually passed the hidden verifier. A planner model is shown the public task plus a cleaned transcript of a verified run and

Authors’ abstract

Converting verifier outcomes into useful policy updates and partial progress into complete task resolution is crucial for terminal agents. We present Terminal-RSI, which alternates Recursive Self-Rewrite (RSR) of verified trajectories with critic-inheriting Recursive Policy-Refine (RPR). We formulate Turn-Boundary Skip-Observation GAE (TB-GAE) to retain token-specific baselines while applying decay across interaction turns. Family Dense Reward (FDR) calibrates semantic progress across task families; Success-Prioritized Reward (SPR) separates strict success from bounded partial credit. A Warm-up Critic Model, recursive critic handoff, and stage-specific learning rates coordinate the two refinement phases. The five-stage sequence from Qwen3.8-27B base model through RSR-1, RPR-1, RSR-2, and RPR-2 yields Terminal-RSI-27B: 19.84% on Terminal-Bench 4.0, 86.52% on Terminal-Bench 2.1, 16.95% on Terminal-Bench 3.0, and 79.00% on Terminal-Bench Hard. On LHTB, process reward rises from 0.21 to 0.37 and strict completion from 0/46 to 5/46; SWR100 success doubles from 3.00% to 6.00%. By recursively coupling verified trajectory rewriting with critic-inheriting policy refinement, each retained stage consolidates discovered behaviors and progressively advances the agent from partial local repairs to strict terminal verification.