Agent World Model in the Wild: Computer-Use Agent Environment Synthesis

AI summary

Overview

Research area: AI agents / computer-use agents (CUAs), specifically synthetic environment and task generation for agentic reinforcement learning.

Technical level: Advanced — the paper assumes familiarity with reinforcement learning (GRPO), agent harnesses, verifier design, and benchmark evaluation.

Scope: The paper presents WildAWM, an all-open-source pipeline that synthesizes executable personal-computer environments and long-horizon tasks spanning GUI, MCP, and CLI interfaces, then trains general computer-use agents inside those synthetic worlds.

What This Paper Is About

Training computer-use agents by letting them interact with real computer systems is expensive and risky, and hand-built environments do not scale. The authors build a pipeline that automatically generates realistic, resettable "personal computer" worlds — websites, model-context-protocol services, and local files — together with tasks that require an agent to move between all three interface types, and with executable verifiers that score whether the task was actually completed. The goal is to show that high-quality agent training data and rewards can be produced using only open-source models, and that agents trained purely in these synthetic worlds transfer to unseen real benchmarks.

Key Contributions

1. A fully open-source synthesis pipeline (WildAWM) that builds executable personal-computer environments jointly supporting graphical user interfaces, MCP services, and command-line interfaces, rather than specializing in a single interaction mode.
2. Execution-based quality gates for task synthesis that address solvability, difficulty, and verifier reliability using heterogeneous agent rollouts, cross-judge consistency checks, and counterfactual probes of judgment robustness (exploiting LLM sycophancy).
3. A released corpus of 341 environment–task pairs, 273 labeled hard and 68 medium, with long-horizon workflows that can require over 100 agent steps.
4. An empirical demonstration that training exclusively in these synthetic worlds generalizes, with two Qwen models improving on all eight out-of-distribution benchmarks spanning web, claw, and computer-use agents.

Main Findings

• Frontier models still struggle: On the synthesized tasks, GPT-6-Astra solves only about 60% (pass@1), and even after eight attempts it still fails on 27.3% of tasks.
• Synthesis rollouts are hard: Mean success rate without hints during synthesis is 21.3%, rising to 45.6% with hints. Every accepted task is solved by at least one agent, and none is solved by all agents.
• Tasks genuinely cross interfaces: Tasks average 4.2 sub-tasks and 3.1 interface switches; 74.5% revisit a previously used interface, so state must be preserved across GUI/CLI/MCP transitions.
• Broad OOD transfer at two scales: The 35B-A3B model rises from 42.84 to 53.42 average (+10.58), and the 4B model from 21.83 to 30.04 (+8.21), across Online-Mind2Web, WebArena, QwenClawBench, ClawEval, CUA-Gym, OSWorld-Verified, OSWorld 2.0, and Agents' Last Exam.
• Larger models benefit more: The 35B-A3B gains are largest on ClawEval (+16.67), OSWorld-Verified (+16.06), WebArena (+13.72), and CUA-Gym (+12.22), suggesting long-horizon cross-interface supervision scales with capacity.
• Gap to frontier narrows: The trained 35B-A3B model matches GLM-5.2 on WebArena (67.26) and exceeds GPT-5.5 on Online-Mind2Web (80.77 vs. 72.22), against a GPT-6-Astra average of 73.11.
• Environments are larger and more diverse than prior work: WildAWM world states have a median size of 52.4K tokens versus 1.1K for CUA-Gym's task-specific injected states, with a higher Vendi score and a nearest-neighbor cosine distance of 0.25 vs. 0.20.
• Tasks are provably solvable: Across 24 model–harness configurations, every task is solved by at least one agent; the three open-source models used during synthesis alone solve over 95% of tasks, and with hints all three solve 51.5% of tasks.
• Verifiers are reliable: On 100 randomly sampled rollouts the verifier agrees with the judge majority 95% of the time with a Cohen's kappa of 0.90.
• The pipeline keeps scaling: Coverage reaches 134 domains and 91 task types at a 53% acceptance rate without clear saturation, while mean pairwise cosine distance stays stable.
• Every quality gate matters: Removing any gate in the 4B ablation reduces average performance versus the full SFT model (31.86); the largest drops come from removing judge consistency (22.56) and initial unsatisfaction (24.08).

Methodology in Plain English

The pipeline works in two stages.

Stage 1 — building worlds. The system first invents personas (an occupation, named contacts, frequently used sites, ongoing multi-day storylines), checking new candidates against accepted ones to avoid duplicates. Each persona's storylines become day-by-day plans. A coding agent then materializes each day on top of the previous day's state, reusing a growing library of 83 "event skills" and application implementations, and writing deterministic, replayable programs that update databases, MCP service backends, and local files. A state is kept only if two independent replays produce identical results, all services stay healthy, and a second coding agent confirms the state matches the plan. Fifteen mirrored websites from WebHarbor supply rich visual content. The result is a cumulative, resettable world state per persona per day.

Stage 2 — building and vetting tasks. From a chosen world state, the generator explores the live environment and proposes tasks that require all three interfaces, with a diversity gate filtering out near-duplicates. Each proposal ships with a task spec, an executable verifier, a rubric, a hint, and a sub-task breakdown. The verifier checks N binary criteria and requires all of them to pass. Six rollouts per task (half with hints) are then run by 18 model–harness pairs formed from GLM-5.2, DeepSeek-V4-Flash, and MiniMax-M3 crossed with three coding agents (OpenCode, Codex, Claude Code) and three claw agents (Nanobot, OpenClaw, Hermes Agent). Three judges score each rollout: the programmatic verifier, an LLM judge, and an agent judge that actively inspects the environments.

Five gates then decide acceptance: the verifier and both judges must agree on all six rollouts; the verifier must reject the untouched initial state; at least one rollout must succeed and must actually use every declared interface; difficulty is graded by how many of the three unhinted rollouts succeed (k ≤ 1 is hard, k = 2 medium, k = 3 too easy and sent back); and a counterfactual probe tells the agent judge the opposite verdict and checks whether it holds its ground. Failures go back to the generator with diagnostics for up to two revision rounds.

Training. The 341 tasks are split into 100 for SFT, 208 for RL, and 33 for in-domain evaluation. Qwen base models are warm-started with SFT for 3 epochs on 100 GLM-5.2 trajectories, then trained with GRPO for up to 64 steps using a batch of 8 tasks and 4 rollouts per task. The reward is 1 for full success plus a small partial-credit term (0.1/N times the fraction of checklist criteria passed), capped at 1, with −1 for format failures. A DAPO-inspired per-task rollout multiplier (initialized at 1, capped at 3, updated with weight 0.8) resamples groups to avoid zero-variance reward groups. All synthesis and training used 8 NVIDIA B200 GPUs.

Why This Matters

Impact on research. Environment supply is described as a key bottleneck for agentic RL. WildAWM shows that the environment-authoring step — not just task authoring — can be automated with open-source models, and it introduces verification methodology (cross-judge consistency, counterfactual probing against sycophancy) that applies well beyond computer use. Its ablations give the community direct evidence about which quality controls actually pay off.

Real-world applications:

• Enterprise workflow automation: training agents that move between a web app, a ticketing/tool API, and local scripts while preserving state across each hop.
• Software QA and RPA: generating resettable sandbox environments instead of testing automations against brittle live systems.
• Safe agent pretraining: giving agents a place to explore with risky actions (purchases, messages, deletions) without exposing real accounts.
• Benchmark construction: producing challenging, verifiable evaluation suites that resist saturation even for frontier models.

Industry relevance. The pipeline is entirely open-source, which matters for reproducibility and for organizations that cannot route training data through proprietary frontier APIs. The demonstration that a 35B-A3B model can match GLM-5.2 on WebArena and beat GPT-5.5 on Online-Mind2Web after training only in synthetic worlds is a strong argument that environment synthesis is a viable substitute for expensive human demonstrations.

Future Directions

• Scaling the corpus well beyond 341 tasks: the authors report a 53% acceptance rate with no clear saturation at 134 domains, leaving open how far coverage and difficulty can be pushed.
• Closing the frontier gap further: trained models remain well below GPT-6-Astra's 73.11 average, and it is unresolved whether more or harder synthetic worlds, rather than more capacity, is the binding constraint.
• Hardening verifiers against reward hacking: the counterfactual probe addresses sycophancy in the judge, but the paper does not establish how robust programmatic verifiers stay under adversarial optimization during long-horizon RL.
• Broadening beyond the "all three interfaces" requirement: every synthesized task mandates GUI, MCP, and CLI use, so it is an open question whether single-interface or partial-interface tasks are worth generating, and whether gains reported on claw and web benchmarks would extend to other agent families.

Target Audience

Researchers and engineers working on agentic reinforcement learning, computer-use/GUI agents, and synthetic data generation for agents; benchmark designers who need verifiable, non-saturating task suites; and infrastructure teams in industry building sandboxed training or evaluation environments for autonomous computer-use systems. Readers without background in RL or agent harnesses will find the task-synthesis and quality-gate sections accessible, but the training and ablation sections assume familiarity with GRPO and benchmark protocols.

Authors’ abstract

Agent learning is shifting from learning from human data toward learning through interaction with the world. However, scaling such learning directly in real-world computer systems is challenging and can introduce substantial security risks. We therefore propose Agent World Model in the Wild (WildAWM), a scalable pipeline for synthesizing controllable environments and tasks for training general computer-use agents (CUAs). Whereas existing approaches typically specialize in a single agent interaction interface, we construct executable environments that jointly support graphical user interfaces, model context protocol services, and command-line interfaces, allowing agents to flexibly combine visual interaction, tool use, and code execution. WildAWM is entirely powered by open-source models, demonstrating that high-quality computer-use environments can be generated without proprietary frontier models. Beyond environments, we address three recurring challenges in task synthesis: ensuring solvability, maintaining sufficient difficulty, and producing reliable verifiers, by introducing execution-based quality gates that combine heterogeneous agent rollouts, cross-judge consistency checks, and counterfactual probes of judgment robustness. The resulting pipeline synthesizes 341 long-horizon environment-task pairs that can require over 100 agent steps to solve and remain challenging even for frontier models, with GPT-6-Astra solving only 60% of them. We train two Qwen models of different sizes exclusively in these synthetic worlds. Both models achieve substantial gains across 8 out-of-distribution benchmarks spanning web, claw, and computer-use agents, demonstrating that learning in WildAWM's synthesized worlds generalizes broadly to unseen computer-use tasks. The code is publicly available at https://github.com/MSR-Orchard/WildAWM.