AutoEnvScaling: Automating the Data Flywheel with Terminal Agents

AI summary

Overview

• Research area: Reinforcement learning (RL) for terminal agents — specifically, automated generation of training environments (tasks with verifiers) and the possibility of recursive self-improvement (RSI).
• Technical level: Advanced. The paper assumes familiarity with reinforcement learning, PPO-style policy optimization, agent harnesses, and containerized task environments.
• Scope (one sentence): The paper turns environment design into a terminal task performed by an agent, shows this produces far more valid RL environments at lower cost than prompt-based generation, and demonstrates that a single open-weight model can act as both environment proposer and solver to improve itself.

What This Paper Is About

Training terminal agents with RL requires many diverse, verifiable environments, but these are still written by human researchers, so the supply grows only as fast as researchers can work. The authors argue that building an environment is itself a coding-and-debugging job in a terminal — exactly what terminal agents are trained to do — and so hand that job to an agent. The goal is a self-sustaining "data flywheel" in which a proposer agent reads the solver's failures, builds and validates new Harbor tasks, the solver trains on them, and the solver's fresh rollouts steer the next round of environment design.

Key Contributions

1. Environment design as a terminal task. A proposer agent works inside a sandbox with a workspace, child containers, skills, memory, and restricted web access, and produces complete Harbor tasks (instruction, execution environment, reference solution, tests) end to end rather than following a fixed generation pipeline.
2. A solver-guided data flywheel for terminal RL. New environments are validated (reference solution gets full reward, empty solution gets less than half, plus a 13-gram contamination check and harbor check review) and calibrated against the current solver (kept only if mean reward falls between 0.25 and 0.75 with variance across attempts), so the pool changes as the solver improves.
3. Proposer learning through harness optimization and weights. Closed-weight proposers improve by editing their own memory, skills, and tools between rounds; open-weight proposers are trained jointly with the solver on separate role rewards using Divergence Proximal Policy Optimization (DPPO).
4. Recursive self-improvement with a single policy. The same model (Qwen3.6-35B-A3B) serves as both proposer and solver, training on environments it designs and using its updated checkpoint to design the next round; the authors report this is the only method in their Table 1 that supports an external proposer, a self proposer, harness optimization, and weight training.

Main Findings

• Terminal-based generation is far more often valid. Across eight frontier proposers, AutoEnvScaling achieves the highest valid-environment rate in every case. The largest jump is DeepSeek-V4-Flash, from 3.5% (Prompt) to 89.0%; Gemini-3.8-Flash goes from 26.2% to 64.8%, and Claude Opus 5 from 74.6% to 94.0%.
• Weaker models are lifted most. Within the GPT-5.6 family, prompt-based rates fall as the model weakens (Sol, Terra, Luna), while AutoEnvScaling stays between 84% and 92%. The authors link this to the "mismanaged geniuses" hypothesis — models held back by scaffolds that underuse their capabilities.
• Cost per valid environment drops for six of eight models, by up to 75% (DeepSeek-V4-Flash: 0.08 to 0.02 USD; GPT-5.6-luna: 0.02 to 0.01 USD). For Claude Opus 5 (0.72 to 1.18 USD) and GPT-6-astra (1.79 to 2.75 USD) the cost per valid environment actually rises.
• Generated tasks are longer and harder. Compared with Prompt + Env Feedback, average solver turns rise from 33.8 to 47.2 and solver pass rate falls from 61.0% to 51.0%.
• Model-designed curricula beat static pools. With GPT-5.6-sol as proposer, Qwen3.5-9B rises from 27.2 (Tmax) to 33.5 on Terminal-Bench 2.1 (+6.3), and with Claude Opus 5 from 29.2 to 36.6 (+7.4). Qwen3.5-35B-A3B rises from 34.6 to 38.9 (+4.3) and from 35.5 to 39.3 (+3.8).
• Solver feedback in the workspace matters. Removing past solver trajectories from the proposer's workspace drops Qwen3.5-35B-A3B by 3.6 points (GPT-5.6-sol) and 6.7 points (Claude Opus 5), leaving it 0.7 above Tmax with GPT-5.6-sol and 2.9 below with Claude Opus 5.
• A single model can bootstrap itself. Qwen3.6-35B-A3B generates valid environments only 32.3% of the time out of the box; after cold-starting on 600 proposer trajectories it reaches 86.8%. As both proposer and solver it raises Terminal-Bench 2.1 from 41.1 to 53.3 (+12.2, and 6.1 points above Tmax) and Terminal-Bench 4.0 from 1.6 to 3.2 (+1.6, versus 1.4 for Tmax).
• Training the proposer keeps the loop stable. Keeping shared weights but removing proposer-side training leads to initial improvement followed by decline as environment validation errors accumulate and destabilize training; that run ends earlier because training collapsed.
• Better curriculum efficiency. AutoEnvScaling yields a 3.1× higher usable-group rate than Tmax, and collecting rollouts for each update takes 42% less time on average.
• Generated tasks can teach new behavior. On HiL-Bench software engineering tasks, training Qwen3.6-35B-A3B on 500 environments generated by Claude Opus 5 raises task success to 33.9% and Ask-F1 to 55.3% (from 6.8% and 29.6% with the 100 official tasks), while the ask rate moves only from 84.2% to 89.8%. That exceeds both frontier models, which reach at most 14.4% Ask-F1 and 7.5% task success.
• Design styles differ qualitatively. On the same failed PyTorch checkpoint-recovery rollout, Claude Opus 5 alternates repeatedly between writing and testing and checks rewards on four partial solutions, GPT-5.6-sol tests the reference solution and an empty workspace, and the Qwen3.6-35B-A3B Cold-Start model writes task files but only checks compilation.

Methodology in Plain English

The central move is to stop hand-designing the environment generator and instead give the job to an agent that already knows how to work in a terminal.

Each proposer episode starts with an assignment naming a parent environment from the training pool, the solver's pass rate on it, and its most common failure. The proposer reads the solver's trajectories and verifier outputs, then decides whether to build an easier variant (if the solver keeps failing) or a harder one with a longer horizon or an extra skill (if the solver succeeds). It writes the task files in a sandbox workspace, launches child containers to build the image and run the reference solution and tests, and iterates until the tests behave as intended. It can even run the current solver on its draft to estimate difficulty before submitting.

Admission is gated. The host rebuilds the environment and checks that the reference solution earns full reward and an empty solution earns less than half, flags any instruction sharing a 13-gram with a held-out task, and runs Harbor's task review. Then it runs the current solver several times and keeps the environment only if the mean reward lands between 0.25 and 0.75 and the rewards vary — a band that keeps tasks neither trivially easy nor impossible.

Rewards for the two roles are defined separately. The solver earns the fraction of the task's N tests it passes, so partial solutions get partial credit. The proposer earns −1 if validation fails, −0.25 if validation passes but calibration fails, and 1 if both pass — separating broken environments from working-but-unsuitable ones.

For training, both roles use the same terminal interface and the same policy; they differ only in prompt, container, and scoring. Advantage baselines are computed separately per role (proposer rewards centered within proposer groups, solver rewards centered across attempts on the same environment), groups with no reward variance are dropped, and the solver trains at every step while the proposer generates and trains every k solver steps. Closed-weight proposers instead improve by rewriting their own memory, skills, and tools between rounds — for example merging validation steps into a faster script or caching useful GitHub tasks and Docker Hub images — while the admission checks stay fixed and outside their control.

Evaluation uses held-out Terminal-Bench 2.1 (avg@5) and, for the self-improvement study, Terminal-Bench 4.0, with all comparisons run at matched training steps against a Tmax pool regenerated with the same proposer.

Why This Matters

Impact on research. The paper attacks a concrete bottleneck rather than a benchmark score: the supply of high-quality RL environments. By making environment design a learned capability instead of a hand-written pipeline, it suggests a route where the model being trained also improves the data it trains on — with a reported mechanism (proposer rewards plus proposer training) for keeping that loop stable. It also reframes scaffold quality as a first-order variable: the gap between a model's prompt-based and terminal-based performance, largest for weaker models, implies many evaluations may understate what a model can do.

Real-world applications (domains the paper builds environments for or evaluates on):

• Software engineering agents — generating repository-style tasks with tests, and teaching agents to ask for missing requirements rather than guess (the HiL-Bench study, where the trained model reaches 33.9% task success versus 7.5% for Claude Opus 5).
• Computer-use agents — browser and GUI interaction tasks generated through the same pipeline.
• Professional work / office tasks — e.g., the generated billing-versus-ledger reconciliation task with five deliberately withheld details.
• GPU kernel programming — a domain the pipeline generates environments for, alongside data analysis and cybersecurity as motivating agent domains.

Industry relevance. The paper notes that companies are reportedly considering annual spending exceeding $1 billion on RL environments, and that environments lose value once an agent has learned what they teach. A method that produces valid environments at lower cost per valid environment for six of eight proposers — and lets an open-weight model supply its own curriculum — directly targets that recurring expense. Caveats remain: for the two strongest proposers tested, cost per valid environment increased even as validity rose, and cross-domain generation has not yet been shown to translate into downstream training gains.

Future Directions

• Continual co-evolution with learning-progress rewards. Instead of only rewarding environments inside a difficulty band, link the proposer's reward directly to how much the solver actually improves, following Absolute Zero and SPADE, so the curriculum adapts automatically.
• Cross-domain environment training. The pipeline generates environments for software engineering, computer use, professional work, and GPU kernel programming, but the authors have not yet evaluated agent training on those distributions, nor whether skills transfer between designing and solving across domains.
• Robust verification and evaluation security. Reference-solution and do-nothing checks cover only a small slice of possible agent behavior, and generated tests can miss incorrect solutions or reject valid ones. Testing against partial, incorrect, and alternative valid solutions is proposed, along with isolated evaluation services — motivated by the cited OpenAI–Hugging Face incident in which agents exploited host infrastructure to extract ground-truth answers.
• Whether gains persist over longer training. The experiments demonstrate self-improvement over the runs evaluated but do not establish whether it continues with more training, and Terminal-Bench 4.0 performance remains low (3.2), leaving long-horizon tasks as open ground.

Target Audience

Researchers and engineers working on RL post-training for agents, agentic scaffolding, and synthetic data generation — particularly those building or curating environment pools for terminal, software-engineering, or computer-use agents. It is also relevant to practitioners interested in self-play and self-improvement, and to evaluation researchers concerned with how harness design and verifier quality shape measured capability. Readers without a background in RL and containerized agent infrastructure will find the framing accessible but the experimental design dense.

Authors’ abstract

Terminal agents solve tasks in software engineering and scientific research by executing commands in a terminal. Training them via reinforcement learning (RL) requires diverse and verifiable environments, but these environments are still mostly designed by human researchers, so their supply grows only as fast as researchers can work. We introduce AutoEnvScaling, which treats environment design as a terminal task and uses agents to automate the data flywheel. A proposer agent builds new RL environments from the rollouts of the solver being trained, and the solver's new rollouts guide the next round. By turning environment generation into a terminal task, AutoEnvScaling increases the valid-environment rate by up to 25 times compared to prompt-only generation, at up to 75% lower cost per valid environment. We show that training Qwen3.5-35B-A3B on AutoEnvScaling environments from GPT-5.6-sol improves its Terminal-Bench 2.1 score by 4.3 points over the matched Tmax baseline. The broader goal of AutoEnvScaling is to enable recursive self-improvement (RSI): by making both environment generation and solving terminal tasks, it lets one model co-evolve in both roles. We show that Qwen3.6-35B-A3B, acting in both roles, gains 12.2 points on Terminal-Bench 2.1 and raises its Terminal-Bench 4.0 score from 1.6 to 3.2. The pipeline also generalizes across agentic domains, generating environments for software engineering, computer use, professional work, and GPU kernel programming. Overall, these results show a path toward agents building their own training environments, reducing human reliance and taking a core step toward RSI.