RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents

AI summary

Overview

• Research area: Computer-use agents (CUAs), specifically "hybrid" agents that interleave graphical-interface operation with code and command-line work; also agent training data generation and agentic benchmarking.
• Technical level: Advanced.
• Scope: The paper introduces RecreationWorld, a five-platform framework of reproducible "recreation" task environments, a scaled training pipeline built on those environments, and RecreationBench, a 250-task held-out benchmark with frozen reference-grounded test suites.

The paper content provided is truncated; the Analysis section ends mid-discussion of "final-loop closure," so the reported findings here stop at that point.

What This Paper Is About

Computer-use agents have developed along two separate tracks: one operates real applications through graphical interfaces, the other writes and runs software through code and a terminal. Each is blind to what the other does best — a GUI agent can observe how a system behaves but cannot build the software behind it, while a terminal agent can author programs but cannot see the interface its own actions produce. This paper studies hybrid CUAs that fuse the two, and builds the environments needed both to measure that fused capability under controlled conditions and to generate verified experience for training it at scale.

Key Contributions

1. RecreationWorld framework. A five-platform (Ubuntu, macOS, Windows, Android, Web) environment built around recreation: given a running reference application, an agent must discover its behavior and construct a faithful implementation with no prescribed workflow. It standardizes reference execution, candidate delivery, and experiment logging while preserving each platform's native interaction semantics, plus a unified harness exposing platform-native GUI control and coding tools through a common execution contract.

2. A verifiable, scalable task formulation. Because the running reference acts as an oracle, hidden behavioral tests can be derived from it and replayed against any candidate, yielding an objective, implementation-agnostic, execution-grounded reward. The supply of open-source applications turns this into an open-ended stream of tasks and verified trajectories rather than a fixed hand-authored set.

3. RecreationBench. A held-out benchmark of 250 tasks — 50 each on Ubuntu, macOS, Windows, Android, and Web — spanning heterogeneous UI frameworks, build systems, and accessibility and automation stacks, with reference implementations withheld where the platform permits.

4. Training data and OOD transfer evidence. 7,000 selected trajectories per platform yield a balanced 35,000-trajectory SFT mixture used to fine-tune two model initializations (Qwen3.7-Plus and Qwen-Flash-CPT), with transfer measured on five out-of-distribution benchmarks.

Main Findings

• Benchmark leaderboard: GPT-6 Astra obtains the highest overall score at 58.06% (stated as 58.1% overall in the abstract), followed by Claude Opus 5 at 44.16% and GPT-5.6 Sol at 42.06%.

• Full-suite programmatic passes are rare: GPT-6 Astra is the only model with full-suite passes on multiple platforms, with 90% and 100% Prog coverage of 17.6% and 2.8% of tasks, versus at most 5.5% and 0.8% for any other model. It passes all programmatic tests on just 2.8% of tasks.

• Recreation training transfers out of distribution: On ProgramBench, GameCraft-Bench, Vision2Web, OSWorld 2.0, and WeaveBench, both training runs finish above their first evaluated checkpoints, with gains of up to 17.9 percentage points. The two sweeps are not uniformly monotonic.

• Behavioral shifts accompany score gains: Averaged equally across the two arms, both increase all three tracked metric groups, own-output image reads, and GUI observation and interaction; Qwen3.7-Plus also increases reference/asset reads. These are descriptive shifts and do not establish causation.

• Recreation trajectories are long and genuinely hybrid: RecreationBench trajectories have a median of 282.5 top-level calls and 9.08 GUI–code-edit transitions per 100 calls. ProgramBench is longer but has no GUI calls; OSWorld 2.0 records only computer-tool calls and thus no separately observable code-edit calls; WeaveBench uses both but is shorter and switches less often, at 178 calls and 1.56 transitions per 100 calls. This is an interface-level structural comparison using each benchmark's native scaffold and Claude Opus 4.8, not a controlled model-performance comparison.

• Tests reach beyond surface presence: Across the frozen suites, 94.2% of cases check an interaction outcome rather than mere presence, and 40.7% require an exact expected result. Macro-averaged navigation depth is 22.8% start-surface, 53.2% one hop, and 24.1% two or more hops; about 77% of cases leave the start surface and 24% traverse two or more surfaces.

• Broader exploration co-occurs with higher scores: GPT-6 Astra covers 58.1% of platform-supported reference-side GUI interaction types, versus 41.8% for Qwen3.8-Max-0902, and observes 5.23 distinct reference-window identities per trajectory versus 2.21. GPT-6 Astra's lead over runner-up Claude Opus 5 is 13.9 percentage points.

• Controlled input variation is uncommon and similar across models: Entering at least two distinct nonempty values in the same reference input field appears in 17.0% of GPT-6 Astra trajectories, close to Claude Opus 5 at 16.8% and only modestly above GLM-5.3 and Qwen3.8-Max-0902 at 13.4%.

• Visual input is routine: Claude Opus 5, Qwen3.8-Max-0902, and GPT-6 Astra receive a median of 53 images per trajectory, and 96.8% of their trajectories include at least one image. GLM-5.3 receives structured GUI representations but no screenshot pixels.

• Final-loop closure remains low across all models under the strict final-loop criterion (the passage is truncated at this point).

• A programmable runtime lowers interaction overhead without materially changing observed quality: On the same 50 Windows applications with Claude Opus 4.8, output tokens increase by 15.7% while tool-result text and input tokens fall by 65.5% and 40.7%; wall-clock time falls from 4.12 to 3.04 hours per task and estimated model cost from $90.50 to $41.58 per task. This is a one-rollout comparison of complete configurations, showing lower observed interaction overhead rather than an isolated causal effect.

• Delivered artifacts fall short of references in shape as well as behavior: Agents reproduce static interface structure more reliably than interactions and computed outputs, while generated applications remain substantially smaller and more monolithic than their references.

• Evaluation integrity is enforced: Agent-owned processes are terminated and the candidate tree frozen at handoff, with a failed isolation check invalidating the run. For Web, detected direct repackaging or replay of reference implementation artifacts caps the task aggregate at 0.10, while independently authored reconstructions remain permitted. Android does not yet attest a packet-level egress filter.

Methodology in Plain English

The researchers picked a task that cannot be solved either by clicking or by coding alone. An agent is shown a running reference application and a high-level prompt, and must produce source code that builds and launches into a working imitation of it. Nothing prescribes the order of work, so agents move back and forth: explore the reference, write code, build and launch their own version, look at what it renders, and revise. This loop is what the authors call recreation.

Verification comes free from the reference. Because a reference can be executed, an orchestrator can exercise it, capture exact text, widget state, navigation, persistence, and computed values through each platform's automation substrate — AT-SPI on Ubuntu, AXUIElement on macOS, UI Automation on Windows, UiAutomator on Android, DOM/ARIA on Web — and turn those into hidden test cases. Visual assertions judge layout, color, canvas content, and other rendered properties the structured substrate does not expose, using a shared Qwen3.7-Plus judge at temperature zero returning a binary verdict. Each proposed assertion must pass on the reference and be reviewed by humans before the suite is frozen and replayed unchanged against candidates. A candidate that cannot build or launch scores zero; otherwise it earns a graded fraction of assertions passed.

Two channel scores are reported separately: Prog for structured automation behavior and VLM for rendered behavior. Applications are macro-averaged within each platform and platforms are weighted equally, so a candidate cannot change its own denominator. The construction pipeline for each task runs from reference selection and pinned upstream revisions, through behavior inventory and test authoring by specialized subagents, to validation, human review, and freezing. For training, rollouts run in task-isolated workers on a horizontally scalable virtual-machine pool with a 20-hour per-rollout wall-clock timeout, and high-scoring trajectories selected by the task-specific verifiers are used for rejection-sampling SFT.

Why This Matters

• Research impact: The paper argues that hybrid computer use is not only broader than either modality alone but naturally self-grounding: because the agent builds something it can run and inspect, every cycle produces objective, executable feedback. That makes recreation a substrate for self-improvement and for automatic, implementation-agnostic scoring, rather than relying on human preference or source-code similarity. It also reframes verification as the tractable side of the asymmetry between generating and checking.

• Real-world applications:

  • GUI migration and modernization, where an agent must recover an application's behavior and reimplement it on a new stack or platform.
  • Legacy software reconstruction and rebuilds, using pinned upstream source inventories as references.
  • Automated regression and fidelity testing for generated software, since assertions are derived from a running oracle rather than authored by hand.
  • Benchmarking and model selection for agentic coding and computer-use products across desktop, mobile, and web.

• Industry relevance: The work packages the execution substrate as part of the experimental contract — versioned task-isolated workers pinning graphical runtimes, automation interfaces, and build toolchains — addressing the practical problem that long-horizon agent rollouts must preserve interactive state reliably for hours and scale across concurrent runs. The auxiliary finding that a persistent programmable runtime cut wall-clock time from 4.12 to 3.04 hours and per-task cost from $90.50 to $41.58 speaks directly to the economics of running such agents in production.

Future Directions

• Close the interaction and computation gap: Agents reproduce static interface structure more reliably than interactions and computed outputs; improving behavioral rather than structural fidelity is the clearest target.
• Raise full-fidelity success: With the best model passing all programmatic tests on only 2.8% of tasks, the distribution of near-misses and the specific assertions that break them are an open area for both training and analysis.
• Improve agent-side verification: The paper notes final-loop closure remains low across all models and that later checkpoints verify their own rendered outputs more often, but reports that these descriptive comparisons do not establish causation — a controlled study is a logical next step.
• Tighten isolation and scale the task pool: Android does not yet attest a packet-level egress filter, and Web cannot enforce a source-blind boundary because the client implementation is what the server sends; extending guarantees and continuing to draw tasks from a continually refreshed pool of open-source applications are both stated directions.

Target Audience

Researchers and engineers working on computer-use agents, agentic software engineering, and agent evaluation infrastructure; teams building or choosing benchmarks for GUI and coding agents; and practitioners interested in generating scalable, automatically verified training data from executable references rather than human-written specifications.

Authors’ abstract

Computer-use agents (CUAs) have advanced along two separate lines: graphical interaction and software development through code and the command line. Real digital work requires both, interleaved rather than stacked end to end. We study hybrid CUAs that autonomously decide when to explore an interface, implement software, and run and visually verify their artifacts. We introduce RecreationWorld, a five-platform framework built around recreation: given a running reference, an agent must discover its behavior and build a faithful implementation with no prescribed workflow. RecreationWorld provides reproducible environments on Ubuntu, macOS, Windows, Android, and Web, plus a unified harness with native GUI control and coding tools. The running reference serves as an oracle for hidden behavioral tests, providing execution-grounded rewards. We scale trajectory generation with high-quality open-source applications. Models trained on these trajectories improve across five out-of-distribution coding and hybrid computer-use benchmarks and more frequently verify their rendered outputs, providing evidence of transfer beyond recreation. For held-out evaluation, we introduce RecreationBench, comprising 250 diverse tasks across domains and platforms. Reference-grounded programmatic and visual assertions cover action-conditioned outcomes at multiple interaction depths; each is validated on the reference and by human reviewers before the suite is frozen for automatic scoring. GPT-6 Astra leads at 58.1% overall, but passes all programmatic tests on just 2.8% of tasks. Agents reproduce static interface structure more reliably than interactions and computed outputs, while generated applications remain smaller and more monolithic than their references. We release the benchmark, environments, and test suites.