SPADE: Self-Play in Adaptive Synthetic Executable Environments

AI summary

Overview

Research area: Reinforcement learning for large language model (LLM) agents, specifically self-play, automatic environment synthesis, and unsupervised environment design for post-training.

Technical level: Advanced. The paper assumes familiarity with Markov decision processes, Gym-style reset()/step() interfaces, GRPO, minimax regret, and unsupervised environment design.

Scope: The paper introduces SPADE, a single-model self-play framework in which one LLM both writes executable training environments as Python code and learns to solve them, and evaluates it on eight held-out math, science, code, and reasoning benchmarks plus three tool-use benchmarks at three Qwen3 backbone scales.

What This Paper Is About

Training language agents through experience requires a supply of interactive tasks with verifiable rewards, but existing environment pools — hand-curated, synthetically generated, or produced by frozen verifiers — stay fixed while the learner improves, so the agent stops improving once it exhausts them. SPADE's goal is to make environment design itself a learnable part of training: a single LLM plays both an Environment Designer that writes complete multi-turn environments as executable code, and a Reasoning Agent that solves them, so the environment distribution co-evolves with the agent's capability frontier rather than remaining static.

Key Contributions

1. A general co-evolution framework for environment synthesis and agentic capability via self-play. The same LLM generates executable environments as Python programs with a Gym-style reset()/step() interface and learns to solve them, unifying single-turn reasoning and multi-turn agentic tasks under one interface and turning environment design into an RL-trained component of post-training.

2. A hint-based regret reward for the Environment Designer. The designer is rewarded by the gap between the Reasoning Agent's return with and without a privileged hint the designer itself writes (Equation 3), a lightweight estimate of minimax regret. The paper argues this yields constrained competitive dynamics — unlike a pure adversary (which can make environments unsolvable) or a cooperative designer (which can inflate reward without teaching anything) — and presents an equilibrium analysis in Appendix E.

3. A corpus-grounded, memory-augmented design pipeline spanning two domains. Prior environment-generation work typically targets a single domain; SPADE demonstrates gains in both cognitive-skill games and tool-use tasks, with environments grounded in pretraining corpus documents and an accumulated memory of past environments.

4. Validation of self-play at 30B+ scale. The authors report a complete recipe covering environment validation, reward-hacking avoidance, and curriculum design across three Qwen3 backbones.

Main Findings

• Games setting beats fixed-environment baselines at every backbone scale. On eight held-out benchmarks (AIME 2025 and 2026 at Avg@32, GPQA-Diamond accuracy, LiveCodeBench-v6 Pass@1, and Reasoning-Gym win rate at hard across RG-Math, RG-Algo, RG-Cog, and RG-Logic), SPADE reaches an unweighted mean of 44.1 (+5.2 over base) at Qwen3-4B-Instruct-2507, 55.5 (+5.7) at Qwen3-8B, and 58.3 (+8.1) at Qwen3-30B-A3B-Instruct-2507. The corresponding fixed-environment baselines are 42.5 (+3.6) for Fixed-env RLVE and 39.9 (+1.0) for Fixed-env GRPO at 4B; 53.8 (+3.9) and 51.0 (+1.2) at 8B; and 53.0 (+2.8) and 51.4 (+1.2) at 30B-A3B.

• The margin over the strongest baseline grows with model scale. The paper reports that SPADE improves over the strongest fixed-environment baseline by +5.3 points on average and up to +7.5 on individual benchmarks at 30B-A3B in the games setting, and that the margin over the strongest baseline grows with model scale on games. Across eight held-out benchmarks at 30B scale the abstract reports +5.3 on average.

• Gains are largest on procedural reasoning skills. At 30B-A3B, SPADE's per-benchmark gains over the same-model base are +18.3 on Reasoning-Gym Math (63.3 vs. 45.0), +14.7 on RG-Cog (37.7 vs. 23.0), +14.1 on RG-Algo (32.1 vs. 18.0), +5.8 on RG-Logic (72.8 vs. 67.0), +5.4 on GPQA-Diamond (75.8 vs. 70.4), +4.1 on LiveCodeBench-v6 (47.3 vs. 43.2), +1.3 on AIME'25 (62.8 vs. 61.5), and +0.9 on AIME'26 (74.4 vs. 73.5).

• Tool-use setting lifts multi-turn interaction benchmarks. SPADE improves BFCL v4 multi-turn by +10.3 at 4B and +5.7 at 30B-A3B, and ACEBench-Agent by +13.9 at 30B-A3B. At 4B, BFCL v4 multi-turn average rises from 22.0 to 32.3, τ²-bench average from 31.0 to 34.8 (+3.8), ACEBench-Agent average from 48.4 to 57.3 (+8.9), and the combined average from 33.8 to 41.4 (+7.7). At 8B, BFCL v4 rises from 35.4 to 41.8 (+6.4), τ²-bench from 26.2 to 29.5 (+3.3), ACEBench-Agent from 60.0 to 69.0 (+9.0), and the combined average from 40.5 to 46.8 (+6.2).

• Corpus grounding and environment memory are critical components. The paper reports that grounding the Environment Designer on documents sampled from a large pretraining corpus and giving it an accumulated environment memory are components critical to success. In the games setting grounding uses 10k mathematics and 5k science documents drawn from DCLM and MegaScience; in tool use, 15k documents from the Nemotron pretraining code corpus.

• Hint-based regret outperforms an EMA learning-potential signal. Ablations compare hint-based regret against an exponential-moving-average learning-potential signal and report an advantage for hint-based regret; the full adaptive, corpus-grounded, memory-augmented configuration also outperforms partial and non-adaptive controls. The detailed ablation numbers appear in Section 7.2 and Table 3, which are not included in the provided excerpt.

• Emergent curriculum from simple to complex. Qualitative analysis shows SPADE progresses from simple single-skill tasks to complex multi-constraint environments requiring long-horizon interaction. Figure 2 shows four environments the Environment Designer produced over one 30B-A3B run, from step 0 (early) to step 384 (late), with tasks shifting toward state-gated, multi-turn interaction.

• Hints work by shortening search. Figure 5 shows two positive-regret examples from the canonical 30B games run: the hint hands the fiber-task agent the probe pattern almost outright (arm means 0.00 to 1.00), and narrows the audio-task agent's search to a parameter range it would otherwise reach only after repeated failed attempts (0.30 to 0.65). The paper notes expected regret can dip below zero for smaller backbones when hints mislead the policy.

Methodology in Plain English

A single pretrained LLM is used in two roles, distinguished by system prompts. In the Environment Designer role it writes a self-contained Python program implementing a Gym-style reset()/step() interface — the environment's state transitions, reward function, and verification code are all inside the program — and then writes a privileged hint for that environment, a few sentences carrying the key insight or strategy without revealing the exact answer. In the Reasoning Agent role the same model plays the environment, both with the hint in context and without it.

The designer's reward is the gap between the Reasoning Agent's average return with the hint and its average return without it. A large gap means the environment is at the learning frontier: solvable with help, not yet solvable alone. A small gap either means mastery (the agent solves it both ways) or intractability (it fails even with the hint), and neither is rewarded. In training, the raw regret is floored at zero and normalized to [0, 1] with weight 0.4, and blended with a flat-top difficulty anchor with weight 0.6 that pays environments whose Reasoning Agent win rate falls in the band [0.4, 0.6] and decays linearly outside it. The Reasoning Agent's reward is task correctness from the environment's reward function. Both rewards update the same shared weights using GRPO.

Two inputs keep environment generation from collapsing toward what the model already knows. First, corpus grounding: every round the designer conditions on freshly sampled human documents, which supply breadth from outside the loop. Second, environment memory: a buffer of previously generated environments annotated with regret scores and skill tags gives the designer high-regret seeds to vary, so the difficulty held at the frontier moves as the frontier moves. The corpus shapes what environments are about; memory shapes how hard they are.

Stability techniques for training one policy on two coupled objectives include per-role advantage normalization, upweighting the less frequent Environment Designer trajectories, a delay of k rollouts in the designer's update (corrected with truncated importance sampling), asymmetric clipping at ε_low = 0.2 and ε_high = 0.28, and the regret floor. Each candidate environment must pass syntax and executability checks before entering the training pool.

Training uses GRPO for 400 rollouts of 24 environments each on three Qwen3 backbones: Qwen3-4B-Instruct-2507, Qwen3-8B, and Qwen3-30B-A3B-Instruct-2507 (the primary model). The 4B and 30B runs use instruct models; the 8B run enables thinking for both roles. The Reasoning Agent plays each environment 16×k times without the hint and 16 times with it, and regret subtracts hint and no-hint averages measured at the same regeneration step (16 plays each). The RL backbone is slime. Both fixed-environment baselines were retrained per backbone from the same base model for 400 training iterations, with Fixed-env RLVE following the official RLVE sampling and curriculum settings; RLVE itself hand-engineers 400 verifiable environments with adaptive difficulty levels.

Why This Matters

Research impact. The paper reframes the environment-pool bottleneck — the limited supply of interactive tasks with verifiable rewards — as a learnable RL component rather than a fixed dataset or a frozen generator. Because the designer emits full multi-turn MDPs as code rather than tasks with sparse terminal rewards, and because it is trained online alongside the solver, the framework connects classical unsupervised environment design (POET, PAIRED) to LLM post-training in an unbounded, code-defined environment space. The paper also reports validating the approach beyond small models, at 30B scale.

Real-world applications:

• Tool-using assistants that must chain multiple API calls across turns, as probed by BFCL v4 multi-turn, τ²-bench, and ACEBench-Agent.
• Coding assistants that learn from procedurally generated verifiable programming tasks grounded in real documentation and algorithm implementations.
• Science and technical reasoning tutors built on environments seeded from physics forums and university-level textbooks.
• Game-playing and procedural-reasoning agents trained in automatically generated cognitive-skill environments that can be re-targeted as the agent improves.

Industry relevance. The paper frames training environments as a major industrial investment: it cites major labs weighing environment budgets exceeding $1 billion a year and a new class of startups raising nine figures to supply them, alongside the argument that high-quality human text is a finite resource. If environment design can be learned by the same model that is being trained, the cost of scaling agentic post-training is no longer bounded solely by how fast humans can write environments.

Future Directions

• Broadening beyond the two demonstrated domains. The paper covers games and tool use; whether the same self-play loop transfers to computer-use sandboxes, robotics, or other agentic settings is left open.
• Extending the equilibrium and stability analysis. The minimax-regret equilibrium analysis appears in Appendix E under idealized assumptions; the observed negative-regret regime for smaller backbones (when hints mislead the policy) indicates that the theoretical story does not yet fully cover current policies.
• Scaling further and studying the scale trend. The paper reports that the games-setting margin over the strongest baseline grows with model scale, motivating tests above 30B and study of how the curriculum's complexity evolves over longer runs than the reported 400 rollouts / step 384.
• Improving the designer reward signal. The ablations contrast hint-based regret with an EMA learning-potential signal; the design of better frontier-targeting signals, and further safeguards against reward hacking in generated environments, remains an active question.

Target Audience

Researchers and engineers working on LLM post-training, reinforcement learning for agents, self-play, curriculum learning, and unsupervised environment design. It is most useful to readers who already understand RLVR and GRPO, since the paper assumes fluency in MDP formalism and policy-gradient objectives. Practitioners building agentic training pipelines — especially those who need multi-turn tool-use or verifiable-reward environments at scale — will find the recipe details and baseline comparisons directly relevant, while readers interested in open-endedness and continual self-improvement will find the framing valuable even without the full technical background.

Authors’ abstract

Continuous self-improvement requires an ever-expanding pool of self-generated, diverse, adaptive goals. For language agents, existing training environment pools (hand-curated, statically synthesized, or frozen-verifier) keep the goal distribution fixed as the learner scales. We introduce SPADE (Self-Play in Adaptive Synthetic Executable Environments), a self-play RL framework in which a single LLM plays two roles: an Environment Designer that writes complete, long-horizon training environments as executable code with an OpenAI Gym-style reset()/step() interface, and a Reasoning Agent that learns to act in them. Each is a stateful, multi-turn environment (state transitions, reward functions, and verification code), so one interface spans reasoning problems and multi-step agentic tool use. The Reasoning Agent's regret is estimated using the gap between its reward with and without privileged hints; in optimizing this regret signal the Environment Designer learns to target environments at the edge of the agent's capabilities while keeping them feasible. Through extensive experimentation, we find several components critical to success: grounding the Environment Designer on documents sampled from a large pretraining corpus, and giving it an accumulated environment memory. Scaling to 30B-parameter models, SPADE improves over the strongest fixed-environment baseline by +5.3 on average across eight held-out math, science, code, and reasoning benchmarks, and lifts the tool-use setting by +5.7 on BFCL-v4 multi-turn and +13.9 on ACEBench-Agent; on the games setting, the margin over the strongest baseline grows with model scale. By making environment design itself a learnable component, SPADE takes a concrete step toward open-ended self-improvement.