PlayWorld: Benchmarking World Models with Agent Players over Long-Horizon Objectives

AI summary

Overview

Research area: Computer Vision / generative video world models — specifically, benchmarking interactive video world models that respond to user actions.

Technical level: Intermediate. The paper assumes familiarity with video generation and world-model terminology (rollouts, action controllability, spatial consistency), but its central idea — evaluating models by whether they accomplish high-level goals rather than whether they follow a fixed control script — is explained in accessible terms.

Scope: PlayWorld is a benchmark of 171 human-annotated scenarios that uses a multi-modal "Agent Player" to pursue long-horizon objectives inside nine interactive world models and scores the resulting 10–60-second rollouts along four capability dimensions plus basic video-quality and controllability metrics.

What This Paper Is About

Interactive video world models generate environments that respond to user controls, but comparing them fairly is hard: the same key presses produce different amounts of movement in different models, so a fixed predefined action trajectory may complete a 360-degree turn in one model and only a partial turn in another. The authors argue that human players actually judge these systems by pursuing a high-level objective — such as "turn around 360 degrees and check whether the environment stays consistent" — and adjusting their actions on the fly until the goal is met or abandoned. PlayWorld reproduces that behavior by handing a multi-modal agent a shared objective and a reference action sequence, and letting it adapt execution online so models can be compared under the same intent.

Key Contributions

1. A long-horizon objective-driven evaluation paradigm. Rather than fixed trajectories, each of the 171 cases pairs an initial world with a scene-grounded objective and a basic action sequence, and a multi-modal Agent Player adapts execution online.

2. The Agent Player with five online decisions. The agent observes generated frames, prior actions, scene description, and the objective, then issues Keep, Stop, Extend, Correct, or End to adapt action count and duration to each model's action granularity, inside a 40-step interaction budget.

3. A VQA rubric verifier over four capability dimensions. Sample-specific Yes/No rubric questions from Gemini 3.1 Pro score geometry consistency, interaction fidelity, out-of-sight evolution, and insight evolution on a 1–5 scale, gated by dimension-specific validation (Trajectory Validity, Subject and Reachability) so that instruction-following failures are not mistaken for memory failures.

4. A large-scale empirical study plus human validation. Nine world models (five web-based closed/open systems, four local chunk-wise models) produce over 1,400 interactive videos scored by more than 820 task-conditioned VQA questions, with 600 valid human pairwise judgments used to check that the VQA metrics track human preference rankings.

Main Findings

• Current models are unreliable on long-horizon interactive objectives. Across the nine evaluated models, out-of-sight evolution and insight evolution receive the lowest scores. Genie 3 leads overall at 2.12 (geometry 2.74, interaction 2.40, insight 1.51, out-of-sight 1.81); HappyOyster is second at 1.92; Matrix-Game-3.0 is lowest at 1.14.

• LingBot-World2 leads insight evolution but trails on out-of-sight evolution. It scores 1.95 on insight evolution — the best of any model — yet only 1.16 on out-of-sight evolution, and 1.82 overall.

• Sustained state evolution across time and occlusion is the primary bottleneck. Visible structure and immediate responses are handled better than semantic state changes over long horizons. During continuous observation a process may stay static, reset, or evolve out of causal order; after leaving view, a target may reappear unchanged, at the wrong location, or as a different entity.

• Long-horizon revisitation exposes global spatial inconsistency. In orbit and revisit cases, salient landmarks are repeatedly regenerated at new viewpoints instead of staying anchored — the paper's example is a camera orbiting 360 degrees around the Taj Mahal and the model regenerating the monument at multiple viewpoints, producing inconsistent copies. Individual frames look plausible while the sequence violates spatial uniqueness.

• Interaction fidelity is limited beyond simple collisions. Several leading models can stop a third-person character at an obstacle boundary, while first-person movement may still pass straight through solid obstacles. Walking into water often fails to produce ripples, splashes, or apparent depth change. Third-person models do not consistently bind input actions to the controlled character.

• Trajectory control and world-model capability diverge. SANA-WM achieves a relatively high overall trajectory-validation pass rate (80.4%) yet only modest rubric scores (overall 1.48), meaning it reaches the intended region but still struggles with memory, spatial consistency, and physical plausibility. Conversely, Hunyuan-GameCraft reaches 89.6% translation pass rate but a 40.3 Basic Ability Score.

• Pass rates vary widely across models. Genie 3 has the highest overall validation pass rate at 87.1%, followed by SANA-WM at 80.4% and HappyOyster at 79.6%. HY-WorldPlay is lowest at 41.6%, driven by 14.6% geometry consistency and 30.2% out-of-sight evolution pass rates.

• Preset + Agent beats both alternatives. On the same 25 cases for Genie 3 and HappyOyster, combining the reference sequence with online adaptation gives the highest Trajectory Score (1.08 on Genie 3, 1.12 on HappyOyster) and Human Preference (65.6% and 67.4%), while modifying only 12.0% and 14.9% of actions. Preset Only scored 0.92 / 1.00 with 39.6% / 40.8% preference; Agent Only scored 0.88 / 0.68 with 29.2% / 24.4%.

• Agent model choice matters little. Claude Haiku, Claude Sonnet, and Gemini 3.1 Pro differ only mildly in trajectory quality. Claude Sonnet has the highest Trajectory Score (1.24) but the highest Decision Latency (6.21 s/call); Claude Haiku has the highest Human Preference (57.8%) and lowest latency (3.83 s/call, versus 4.36 for Gemini 3.1 Pro). The authors adopt Claude Haiku 4.5.

• Automatic video-quality metrics only test basic ability. Most models do well on Video Quality while Action Controllability stays imprecise. HappyOyster earns the highest Basic Ability Score (76.4%) but that does not imply the strongest long-horizon world-model capability. Depth Stability and Subject Consistency can stay high when a model produces little camera motion at all.

• VQA metrics align with human preference. From 600 valid pairwise human judgments, Spearman's rho correlations between VQA scores and human preference are consistently positive across the four dimensions and overall.

Methodology in Plain English

Each evaluation case supplies the same three things to every model: an initial image, a plain-language long-horizon objective, and a human-written basic action sequence expressed in W/A/S/D, arrow keys, and WAIT. Candidate images come from Pexels and Google Images, covering natural, urban, and fantasy scenes with humans, animals, and vehicles; Gemini 3.1 Pro writes an environment caption that annotators verify and revise, and annotators then write the objective, a visually verifiable completion condition, the action sequence, and sample-specific VQA rubrics.

At run time an agent model looks at the currently generated frame, the remaining action sequence, the scene description, and recent history, then decides whether to keep going, stop early, extend a hold, correct or skip the next action, or end the case. A separate agent interface translates those decisions into each world model's native controls — browser automation for web-based systems, model-specific adapters for local ones — captures the resulting frame, and feeds it back, looping until "End" or 40 steps. Rollouts last roughly 10 to 60 seconds.

Scoring happens in two layers. A VQA rubric verifier (Gemini 3.1 Pro) first applies a dimension-specific validation gate — whether the rollout followed the intended path and reached the required viewpoint for geometry consistency and out-of-sight evolution, and whether the controlled subject responds and reaches the interaction region for interaction fidelity — then asks weighted Yes/No questions, aggregated into a 1–5 case score. Rollouts that fail validation receive the minimum score of 1. Insight evolution involves stationary observation, so it skips trajectory validation.

Alongside this, seven video-quality metrics (Aesthetic Quality, Imaging Quality, Motion Smoothness, Temporal Flickering, Temporal Consistency, Depth Stability, and Subject Consistency) and two controllability pass rates (Translation, Rotation) are computed. Camera poses come from VGGT on uniformly sampled frames; translation passes when target-normalized translation error is below 0.3 and rotation passes when mean geodesic rotation error is below 45 degrees. A Basic Ability Score averages ranks across all nine metrics into a percentage.

For human validation, five independent raters with video-generation experience provided judgments; Trajectory Score is a Gemini 3.1 Pro rating on a 0–2 scale, and Human Preference gives full credit to wins and half credit to ties.

Why This Matters

Impact on research. PlayWorld reframes world-model evaluation from "did the model follow this control script" to "did the model accomplish this goal," which removes a genuine confound: differences in action granularity across models were being mistaken for differences in spatial or memory capability. Its negative results — global spatial inconsistency under revisitation, weak persistent state evolution, thin physical response — give the field concrete failure modes to target rather than an aggregate quality number.

Real-world applications:

• Game and simulation development, where designers need to know whether a generated environment can be revisited coherently rather than just look good frame by frame.
• Embodied AI and robotics training, where agents depend on world models whose hidden state evolves causally when objects leave the camera's view.
• Authoring tools and interactive media, where creators script high-level objectives and need models to interpret them robustly regardless of control granularity.
• Virtual production and visualization, where a camera orbit around a landmark must not spawn duplicate copies of it.

Industry relevance. The evaluation spans both closed-source web systems and locally executed open models, and the Agent Player is designed to work through browser automation on web interfaces — a practical necessity as more capable world models ship only behind a web UI. The finding that a shared action prior plus light online adaptation outperforms both pure replay and full online planning also suggests a cheap, deployable recipe for anyone building automated evaluation or QA pipelines.

Future Directions

• Improve persistent state evolution. The paper identifies out-of-sight and insight evolution as the weakest dimensions; the natural next question is what architecture or training signal would let a model maintain a target's identity and progress through an unobserved interval.

• Fix global spatial anchoring. The repeated-regeneration-of-a-landmark failure suggests models rely on local appearance continuity rather than a persistent global 3D representation, which is an open modeling problem the benchmark now makes measurable.

• Broaden agent-model and control-strategy study. The authors show agent model choice has limited effect on a constrained decision task and note latency depends on provider infrastructure; whether that insensitivity holds for harder objectives or full online planning is untested.

• Close the metric gaps. Since automatic video-quality metrics can stay high when camera motion is minimal, and since controllability thresholds do not guarantee reaching the objective-specific target state, better automated proxies for long-horizon completion remain to be developed.

• Extend coverage. The paper notes HY-World2 had to skip cases whose initial images could not support global scene modeling, and that Hunyuan-GameCraft cannot always be evaluated on insight evolution because it requires non-empty action input per chunk — protocol mismatches that future benchmark versions may need to handle.

Target Audience

Researchers and engineers building or evaluating interactive video world models, benchmark designers working on generative-model evaluation methodology, and practitioners in game development, embodied AI, simulation, and virtual production who need to know where current world models break down. Readers with a general computer-vision background will follow the argument; those implementing the benchmark will need the appendices, which cover closed-loop execution details, model-specific adapters, and metric definitions (the supplied text is truncated partway through the appendix's VQA verifier discussion, so some implementation details referenced there are not included here).

Authors’ abstract

Video world models simulate future states conditioned on current observations and user actions. Recent systems have demonstrated impressive video consistency and action controllability over long sequences. However, fairly comparing these interactive models remains challenging. In practice, a human player typically evaluates a world model by pursuing long-horizon objectives through interaction. For example, a user may turn around 360 degrees to see whether the environment remains consistent, or walk into the water and inspect whether realistic water ripples are generated. The action sequence required to achieve the same objective may vary substantially between models, making fixed action-conditioned evaluation unsuitable for cross-model comparison. To address this, we employ multi-modal Agent Players to interact with world models toward specified long-horizon objectives. Building on this paradigm, we introduce PlayWorld, a benchmark providing 171 scenarios, each with a specified objective. To evaluate performance thoroughly, we assess models along four core dimensions: geometry consistency, interaction fidelity, out-of-sight evolution, and insight evolution. In addition, we incorporate basic ability metrics for video quality and controllability. Experiments across nine state-of-the-art world models reveal that current models remain unreliable on long-horizon interactive objectives, particularly in maintaining spatial consistency and persistent state evolution. Code and data are available at https://github.com/kxding/PlayWorld.