World-in-World: World Models in a Closed-Loop World

AI summary

Overview

• Research area: Computer vision / generative world models, embodied AI, and decision-making benchmarks.
• Technical level: Intermediate (accessible conceptually, but assumes familiarity with video generation, reinforcement learning-style planning, and embodied agents).
• Scope: This paper introduces World-in-World, the first open benchmark that evaluates generative world models by how much they actually help embodied agents succeed at tasks inside a closed interaction loop, rather than by visual quality alone.

What This Paper Is About

Generative "world models" — systems that predict future video frames given an agent's observation and a proposed action — can now produce strikingly realistic simulations. But nobody had rigorously tested whether these simulated futures actually help an agent make better decisions. Existing benchmarks score visual realism or controllability in isolation (open-loop), leaving the question of embodied usefulness unanswered. World-in-World closes that gap by plugging many different world models into a single closed-loop planning framework across four embodied tasks and measuring task success as the primary metric.

Key Contributions

1. A closed-loop benchmark for world models: The first open platform that evaluates world models through embodied interaction across four tasks (Active Recognition, Active Embodied Question Answering, Image-Goal Navigation, and Robotic Manipulation), using task success rather than visual quality as the headline metric.
2. A unified online planning strategy and action API: A three-stage proposal–simulation–revision loop (a policy-guided beam search) combined with an API that translates agent actions into whatever control format each world model needs (text prompts, camera trajectories, or low-level action tokens), so heterogeneous models can be compared fairly.
3. A post-training recipe and the first data-scaling law for world models in embodied settings: Fine-tuning pretrained video generators on a modest amount of action–observation data from the target domain, with systematic study of how performance scales with training data and model size.
4. Three empirical surprises: Visual quality does not predict task success; scaling action-conditioned post-training beats upgrading the base video generator; and giving the world model more inference-time compute substantially improves closed-loop performance.

Main Findings

• World models consistently improve base policies: Adding a world model raised performance across all four tasks. In Active Recognition, Runway Gen4 reached 64.79% accuracy with 4.06 mean steps versus a VLM baseline at 50.27% and 6.24 steps. In ImageNav, post-trained Wan2.1 hit 45.14% success versus 35.42% for the VLM baseline.
• Visual quality is a poor predictor of task success: Models with strong aesthetic and image-quality scores did not reliably outperform weaker-looking ones. What correlated with success was controllability — how faithfully the generated video matched the intended action, measured as 1−LPIPS between predicted and ground-truth observations.
• Post-training beats upgrading the pretrained generator: Wan2.1† improved AR accuracy from 58.26% to 62.61% and ImageNav success from 38.19% to 45.14%; SVD† went from 57.71% to 60.98% on AR and 40.28% to 43.05% on ImageNav. Notably, the much larger Wan2.2 (A14B) needed 40K post-training instances just to roughly match post-trained Wan2.1, suggesting action-conditioned data matters more than raw model scale or web-video pretraining.
• A clear data-scaling trend: Going from 400 to 80K post-training instances raised Wan2.1† from 60.25% to 63.34% and SVD† from 56.80% to 60.98% on AR. Larger models (14B) absorbed supervision better and saturated later than small ones (1.5B).
• Inference-time scaling helps substantially: Increasing the average number of world-model rollouts per episode from 3 to 11 improved SVD†'s Active Recognition accuracy from 53.36% to 60.98% — more simulated futures yield better decisions.
• Manipulation is the hardest domain: Gains were far more modest on robotic manipulation (best: SVD† at 46.5% success versus a 44.5% VLM baseline), because contact-rich physics, friction, and articulated object motion are much harder to simulate than pure viewpoint changes.
• Panoramic context is not a free win: Post-training on panoramic versus front-view input gave inconsistent benefits, partly because converting panoramas to perspective views loses resolution and hurts downstream perception.

Methodology in Plain English

The researchers placed a variety of world models — image-based ones like PathDreamer and SE3DS, and video-based ones like SVD, LTX-Video, Hunyuan, Wan2.1/2.2, Cosmos-Predict2, NWM, and Runway Gen4 — inside a single decision-making loop. At every step, an agent's base policy (a vision-language model, a heuristic, or a 3D diffusion policy) proposes several candidate action plans. A translation layer converts those plans into whatever input the world model expects (a text description, a camera path, or low-level action tokens). The world model then "imagines" the future observation sequence for each candidate. A revision policy scores the imagined futures and picks the best plan, which is executed in the real simulator, and the loop repeats with the new observation.

To test whether off-the-shelf video generators could be adapted, they fine-tuned several of them on action–observation data collected from the same simulators used for evaluation — Habitat-Sim scenes from HM3D (disjoint from test scenes to prevent memorization) for the navigation and recognition tasks, and RLBench demonstrations for manipulation. They then varied the amount of post-training data, the model size, and the number of inference-time rollouts to characterize scaling behavior. Evaluation covered 551 Active Recognition episodes (29 Matterport3D scenes), 144 ImageNav episodes (87 HM3D scenes), 184 A-EQA questions (54 scenes), and four RLBench manipulation tasks with 50 episodes each.

Why This Matters

• Impact on research: The paper reframes how the field should judge world models — not by how pretty their outputs look, but by whether they improve an agent's decisions. It provides infrastructure (unified action API, planning loop, benchmark) that makes cross-model comparison possible for the first time, and identifies controllability rather than fidelity as the key bottleneck.
• Real-world applications:
  • Robotics: Using learned simulators to preview and select manipulation or grasping actions before committing, reducing real-world trial and error.
  • Autonomous navigation: Letting delivery robots, drones, or warehouse vehicles mentally rehearse routes and viewpoints to gather more informative observations.
  • Assistive and embodied question answering: Enabling agents in smart homes or AR/VR to reason about occluded or distant objects by imagining alternate viewpoints.
  • Simulation and game content: Determining which generative models are worth deploying as interactive environments rather than passive video generators.
• Industry relevance: Companies building embodied AI, robotics foundation models, and generative video systems now have a concrete signal that scaling action-conditioned post-training data yields more value than scaling the base generative model — a direct guide for where to invest compute and annotation effort.

Future Directions

• Improving generalization to unseen environments: Off-the-shelf video generators often fall back on web-video priors or ignore action controls in novel embodied settings; the authors suggest unified action representations and curriculum or domain-specific data collection as remedies.
• Long-horizon planning and memory: Current world models simulate only short-term changes and lack mechanisms for accumulating spatiotemporal history. Spatial and episode-level memory are proposed as ways to maintain scene context over extended horizons.
• Physics-aware prediction: Modeling contact, friction, compliance, and the state changes of articulated or deformable objects remains largely unsolved; promising routes include physics-guided motion generation and inferring physical properties as conditioning signals.
• Stronger proposal and revision policies: Since overall performance is bounded by the base policy that generates candidates and the scorer that ranks them, better policies (and more human-aligned reward models) could raise the ceiling of the whole framework.

Target Audience

Researchers and engineers working on world models, video generation, embodied AI, and robot learning will get the most value, along with benchmark designers and practitioners deciding whether to invest in generative simulators for decision-making. Reinforcement learning and planning researchers interested in model-based methods will also find the proposal–simulation–revision formulation and the inference-time scaling results directly relevant. The paper is written at an intermediate level: no deep math background is required to follow the main arguments, but some familiarity with embodied tasks and generative models helps.

Authors’ abstract

Generative world models (WMs) can now simulate worlds with striking visual realism, which naturally raises the question of whether they can endow embodied agents with predictive perception for decision making. Progress on this question has been limited by fragmented evaluation: most existing benchmarks adopt open-loop protocols that emphasize visual quality in isolation, leaving the core issue of embodied utility unresolved, i.e., do WMs actually help agents succeed at embodied tasks? To address this gap, we introduce World-in-World, the first open platform that benchmarks WMs in a closed-loop world that mirrors real agent-environment interactions. World-in-World provides a unified online planning strategy and a standardized action API, enabling heterogeneous WMs for decision making. We curate four closed-loop environments that rigorously evaluate diverse WMs, prioritize task success as the primary metric, and move beyond the common focus on visual quality; we also present the first data scaling law for world models in embodied settings. Our study uncovers three surprises: (1) visual quality alone does not guarantee task success, controllability matters more; (2) scaling post-training with action-observation data is more effective than upgrading the pretrained video generators; and (3) allocating more inference-time compute allows WMs to substantially improve closed-loop performance.