Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning

AI summary

Overview

Research area: Agentic reinforcement learning (RL) for LLM-based tool-use agents, specifically the automated synthesis of executable training environments.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning (GRPO, POMDPs, reward shaping), LLM agent frameworks, the Model Context Protocol (MCP), and database-backed state management.

Scope: The paper introduces Agent World Model (AWM), an open-source pipeline that automatically generates database-backed, code-executable environments at scale, and demonstrates through large-scale RL that agents trained purely in these synthetic environments generalize to three unseen tool-use benchmarks.

What This Paper Is About

Training LLM agents to use tools well requires large-scale reinforcement learning across many diverse environments, but real-world environments are expensive, hard to scale, and often lack public APIs. Existing synthetic approaches mostly generate tasks or trajectories rather than environments, and the alternatives that do simulate environments with an LLM suffer from hallucinated state transitions and high inference cost. AWM's goal is to synthesize 1,000 fully executable, database-backed tool-use environments from scratch — using only 100 seed scenario names and no human-written API documentation — so that agents can be trained with reliable, cheap, resettable online RL.

Key Contributions

1. AWM, an open-source environment synthesis pipeline that generates executable tool-use environments in five stages (scenario, task, database, interface, verification), with execution-based self-correction that feeds runtime errors back to the LLM for repair.
2. 1,000 ready-to-use environments, 35,062 tools, and 10,000 tasks paired with synthesized verification code — described by the authors as the largest open-source environment set to date, requiring minimal human input beyond 100 scenario names.
3. A code-augmented LLM-as-a-Judge reward design that combines structured database-state verification signals with an LLM judge, plus an Environment Error category to handle imperfect synthetic environments.
4. Empirical evidence of out-of-distribution generalization: large-scale RL (1,024 isolated environment instances per optimization step) on a 526-environment, 3,315-task subset of AWM improves over untrained baselines on BFCLv3, τ²-bench, and MCP-Universe — benchmarks the environments were not tailored for.

Main Findings

• Environment scale and structure: AWM produces 1,000 environments with 35,062 tools and 10,000 tasks, seeded from 100 popular domain names, with 10 tasks generated per scenario. Environments use SQLite as the state backend rather than NoSQL or key-value stores.
• Pipeline cost and reliability: On a per-100-sample basis using GPT-5 as the generator, total cost is $57.09. Stage success rates are 88.3% for database synthesis (1.12 trials), 88.2% for sample data (1.12 trials), and 86.8% for environment code (1.13 trials). Toolset schema costs $23.74, sample data $13.75, environment code $12.81, verification $2.21, task $0.56, and scenario $0.43. The pipeline achieves over 85% first-attempt success.
• Environment complexity is non-trivial: Mean values are 18.5 database tables, 129.3 sample data records, 35.1 exposed tools, and 1,984.7 lines of environment code. With Claude-4.5-Sonnet as the agent backbone and a step budget of 20, the task completion rate is 62.6%, with 13.7% of tasks exceeding the step budget; mean agent steps per task is 8.5 and mean unique tools used per task is 7.1.
• Largest scale versus comparable sets: AWM has 5× more environments than the closest concurrent work, EnvScaler (191 environments, 18.6 tools, 662.1 code lines). τ-bench has 2 environments/12.5 tools, τ²-bench 3/22.7, MCP-Universe 11/12.1, and AutoForge 10 environments.
• Generalization on BFCLv3: Overall scores rise from 54.92 to 64.50 for 4B, 53.83 to 65.94 for 8B, and 61.25 to 70.18 for 14B. EnvScaler's 8B model drops to 36.83. AWM shows a modest weakness on the hallucination category (70.80 for 8B versus 76.42 for Base), attributed to the format-correctness reward always encouraging tool use.
• Generalization on τ²-bench: Pass@1 improves from 15.83 to 22.57 (4B), 26.44 to 33.45 (8B), and 36.69 to 39.03 (14B). Pass@4 improves from 34.89 to 43.89 (4B), 50.72 to 55.40 (8B), and 55.40 to 57.19 (14B).
• Generalization on MCP-Universe: Overall scores rise from 6.15 to 6.70 (4B), 6.70 to 11.17 (8B), and 8.38 to 12.29 (14B), with large gains in Financial and Location.
• Executable beats simulated: The Simulator baseline, trained with RL in environments where GPT-5 generates each state transition, consistently underperforms AWM. EnvScaler regresses on BFCLv3 (−8.93) and MCP-Universe (−1.39) on average, while AWM improves over Base across all benchmarks.
• Gains shrink with task difficulty but remain positive: For the 8B agent on BFCLv3, gains are +26.7 (Simple), +15.3 (Medium), and +1.1 (Hard). On τ²-bench, +9.3, +6.2, and +4.5 respectively.
• Code-augmented verification is the strongest strategy: At 8B, LLM-only verification yields BFCLv3 55.46, τ² Pass@1 26.44, τ² Pass@4 52.52, MCP 10.62; code-only yields 60.00, 29.59, 52.88, 5.59; code-augmented yields 65.94, 33.45, 55.40, 11.17. The same ordering holds at 4B and 14B.
• Judge reliability: On 100 sampled trajectories evaluated 5 times each, self-consistency is 90.8% (GPT-5.1), 82.0% (Sonnet-4.5), and 76.3% (Qwen3.5); pairwise agreement is 95.5%, 91.8%, and 88.1%; Fleiss' κ is 0.891, 0.826, and 0.728; reward flip rate is 9.2%, 18.0%, and 23.7%.
• Synthesized environment quality: Across 100 sampled environments judged by GPT-5.1, AWM scores 3.68±1.02 (Task Feasibility), 4.04±0.91 (Data Alignment), and 3.65±0.87 (Toolset Completeness) versus EnvScaler's 2.94±1.25, 3.73±0.89, and 2.89±0.79. Blocked tasks are 14.0% versus 57.1% under GPT-5.1, and 11.5% versus 46.8% under Claude-4.5. During RL training, environment error rate stays around 4%.
• Bug profile: 44% of manually inspected bugs come from unhandled edge input cases and 14% from operations conflicting with database constraints.
• Tasks are genuinely hard: With bucketing by required tool calls (1-3, 4-6, 7-10, 11+), tasks comprise 35.1%, 31.5%, 15.7%, and 17.7% respectively, with average steps of 3.3, 5.8, 9.1, and 17.9. GPT-5.1 achieves 61.7/27.0/11.9/3.0 Pass@1 and Sonnet-4.5 68.4/71.8/63.3/31.0; 69.0% of Very-Hard tasks are solved by neither. 27.1%, 23.7%, and 28.2% of tasks are never solved even once during training by Qwen3-4B/8B/14B.
• Pipeline is largely model-agnostic: With Claude-4.5 as generator, database and sample data synthesis reach 100.0% and environment code 99.0%; Qwen3.5 reaches 79.0%, 97.0%, and 77.0%. Mean pairwise diversity distance is nearly constant across generators (0.34, 0.31, 0.35).
• Diversity holds at the implementation level: Cross-environment AST function duplicate rate is 0.0% and endpoint name Jaccard similarity is 0.004.

Methodology in Plain English

The pipeline decomposes environment creation into stages that mirror how real software is built, letting an LLM generate each part while keeping the pieces consistent.

1. Scenario synthesis. Starting from 100 seed domain names, an LLM proposes scenario descriptions for stateful applications (shopping platforms, CRMs, and so on) rather than static content sites. An LLM classifier keeps only scenarios requiring core create/read/update/delete operations, embedding-based deduplication removes near-duplicates, and over-represented categories are capped.
2. Task synthesis. For each scenario, the LLM writes 10 user tasks that act as functional requirements. Tasks must be solvable through an API (no clicking or page navigation) and assume the user is already logged in.
3. Database synthesis. The LLM infers the tables, attributes, and relations needed to make every task feasible, generating a SQLite schema and populating it with sample records that satisfy each task's preconditions.
4. Interface synthesis. The LLM first designs a toolset schema, then generates Python code where each endpoint becomes an MCP tool. Calls to these tools perform database reads and writes, which is what actually drives the environment's state transitions.
5. Verification synthesis. For each task, a module inspects the database state before and after the agent runs, extracting signals about success or failure. A final LLM-as-a-Judge combines those structured signals with the agent trajectory and returns one of four labels: Completed, Partially Completed, Agent Error, or Environment Error.

At every stage, generated code is executed in an isolated sandbox; if it fails, the error message and the offending snippet are fed back to the LLM for correction, for up to five iterations per component. The pipeline tolerates up to 10% errors per stage to control cost.

For training, the authors use GRPO on top of AgentFly and verl. Rewards combine step-level format checking (an invalid tool call triggers early termination with a reward of −1.0) with task-level outcome rewards (1.0 for Completed, 0.1 for Partially Completed, 0.0 otherwise, broadcast to all action steps). Because deployed agents often truncate long interaction histories, the training objective also conditions each action on a truncated history (sliding window w = 3, maximum 20 turns) so training matches inference. Agents were trained on a 526-environment, 3,315-task subset with batch size 64 and 16 rollouts — 1,024 isolated environment instances per step — for up to 96 optimization steps at a learning rate of 7×10⁻⁷, using Qwen3 thinking models at 4B, 8B, and 14B.

Why This Matters

Impact on research. The paper reframes the bottleneck in agentic RL as environment scarcity rather than task or trajectory scarcity, and shows that code-driven, database-backed state transitions provide a more stable learning signal than LLM-simulated transitions. It also introduces verification design as a first-class variable, demonstrating empirically that a code-augmented LLM judge outperforms either LLM-only or code-only verification across every benchmark and model size tested.

Real-world applications.

• Training enterprise tool-use agents in domains such as CRM, workflow automation, user management, and finance, where real APIs are unavailable or unsafe for thousands of RL interactions.
• Reinforcement-learning data factories: automated environment generation that costs $57.09 per 100 samples versus the cost of building or licensing equivalent real systems.
• Benchmark construction and stress-testing, giving researchers thousands of held-out environments to test generalization instead of a handful.
• Agent safety and robustness testing against environment imperfections such as timeouts, partial execution, and constraint violations, via the Environment Error reward category.

Industry relevance. The work originates from a collaboration between Snowflake and UNC Chapel Hill, and the code is released at github.com/Snowflake-Labs/agent-world-model. Its relevance to industry is that it lowers the cost of training generalist agents for internal tools and SaaS workflows, where database-backed state is the norm and public APIs are not always available.

Future Directions

• Closing the modality gaps. AWM does not target conversational interaction (needed by τ²-bench), refusal scenarios (needed for BFCLv3 hallucination tests), or browser automation and information retrieval (central to MCP-Universe). Extending synthesis to cover these is a natural next step.
• Reducing environment bugs. Even with high quality scores, 74-83% of sampled environments contain bugs, concentrated in unhandled edge inputs (44%) and database-constraint conflicts (14%). The authors suggest a stricter error threshold or coding-agent harnesses such as Claude Code could improve quality at higher correction cost.
• Scaling training beyond the current subset. Training used only 526 of the 1,000 environments and 3,315 of the 10,000 tasks, and the 14B model was optimized for only 32 of the intended 96 steps; the effect of full-scale training is unreported.
• Understanding why difficulty limits gains. Absolute improvements shrink on Hard tasks on both benchmarks, and a substantial share of tasks is never solved during RL. Whether harder tasks need more environment diversity, better exploration, or stronger base models is an open question.

Target Audience

Researchers and engineers working on agentic RL, LLM tool-use, and synthetic data generation — particularly those building training infrastructure where environment diversity and state consistency are the limiting factors. It is also useful for practitioners who need executable sandboxes for evaluating agents across many domains, and for those interested in reward design and verification robustness for long-horizon multi-turn agents. Readers without a background in reinforcement learning or agent frameworks will find the systems-engineering pipeline accessible, but the experimental analysis requires familiarity with GRPO, POMDP formalization, and tool-calling benchmarks.

Authors’ abstract

Recent advances in large language model (LLM) have empowered autonomous agents to perform multi-turn interactions with tools and environments. However, scaling such agent training is limited by the lack of diverse and reliable environments. In this paper, we propose Agent World Model (AWM), a fully synthetic environment generation pipeline. Using this pipeline, we scale to 1,000 environments covering everyday scenarios, in which agents can interact with rich toolsets and obtain high-quality observations. Notably, these environments are code-driven and backed by databases, providing more reliable and consistent state transitions than environments simulated by LLMs. Moreover, they enable more efficient agent interaction compared with collecting trajectories from realistic environments. To demonstrate the effectiveness of this resource, we perform large-scale reinforcement learning for multi-turn tool-use agents. Thanks to the fully executable environments and accessible database states, we can also design reliable reward functions. Experiments on three benchmarks show that training exclusively in synthetic environments, rather than benchmark-specific ones, yields strong out-of-distribution generalization. The code is available at https://github.com/Snowflake-Labs/agent-world-model.