WorldCast: Exploring Intelligent Visual Generation in World State Prediction

AI summary

Overview

Research area: Computer vision and multimodal generative AI, specifically the evaluation of unified image-and-video generation systems on a task the authors call world state prediction.

Technical level: Advanced. The paper assumes familiarity with diffusion and unified multimodal architectures, LLM-as-a-judge evaluation protocols, and interleaved versus grid generation formats.

Scope: The paper proposes a four-level hierarchy of visual generative capabilities, introduces a 320-case, eight-category benchmark (WorldCast) with 1,352 requested visual states, and evaluates 12 image and video generation models on multi-step prediction of how a real-world scene evolves.

What This Paper Is About

Generative models can now produce striking single images and videos, but it is unclear whether they can do something harder: take a messy bundle of context (an instruction, a photo of the starting scene, a goal picture, a diagram, a written rule) and predict the successive visual states a real-world process passes through. The authors argue that existing benchmarks mostly test static appearance, leaving this higher-level "world state prediction" ability largely unmeasured. WorldCast is their attempt to measure it, by scoring models on ordered sequences of keyframes that must be causally and physically coherent, not just pretty.

Key Contributions

1. A four-level capability hierarchy (L1–L4). The authors organize visual generative ability from basic text-to-image (L1), through editing, controllable generation and video (L2), up to L3 and L4 tasks that require tightly entangled understanding, reasoning and generation. They state that higher levels remain relatively underexplored despite model progress.

2. A formalization of the "world state prediction" task. Given an instruction p and one or more context images, a system G must output an ordered sequence of K visual states. Input images may play different roles: initial state, goal, object or scene reference, physical property, visual rule, or intermediate demonstration.

3. The WorldCast benchmark. 320 manually designed cases spanning 1,352 requested visual states across eight categories — three at L3 (Observation-Conditioned Transitions, Physical Simulation, Adaptive Task Completion) and five at L4 (Visual Cues, Physical Properties, Visual Goal Realization, Logic and Rules, In-Context Procedural Learning).

4. A unified evaluation protocol plus a diagnostic study. Image and video models are scored under one state-sequence interface using an Agentic Video Adapter (GPT-6 Astra via Codex) to extract keyframes from videos, and an LMM-as-a-Judge (GPT-5.5) with human-annotated, judge-only hints. The paper then analyzes interleaved generation, language-model enhancement, and agentic coding tools.

Main Findings

• Frontier systems show early L4 ability. GPT-Image-2.5 scores 72.8 overall, with 73.8 on L3 and 71.6 on L4, and scores above 60 on all five L4 tasks. The authors describe this as an early-stage capability for visual state prediction on real-world tasks.

• Large gaps between models, widening at L4. Nano Banana Pro reaches 42.8 overall; the best open-source model, SenseNova-U1, reaches 20.1 overall. SenseNova-U1 falls from 24.4 on L3 to 15.0 on L4, while GPT-Image-2.5 drops only 2.2 points between levels. The weakest entrant, Qwen-Image (a generation-only diffusion model), scores 5.6 overall, which the authors read as evidence that understanding and reasoning are required.

• Video models do not automatically win. Seedance 2.0 (47.2), Gemini Omni (41.6) and Wan 3.0 (39.9) are all outperformed by both GPT-Image systems (72.8 and 66.2). The authors suggest interleaved image models may benefit from video-like temporal supervision plus linguistic reasoning.

• Interleaved generation helps but is not the whole story. Switching GPT-Image-2.5 from interleaved to grid output drops its overall score from 72.8 to 50.1 (−22.7), with Logic and Rules collapsing from 74.3 to 23.0. Nano Banana Pro barely changes (42.8 to 40.6), and GPT-Image-2.5 with grid output still beats interleaved Nano Banana Pro.

• Context accumulation is a real burden. Over 195 four-step cases, SenseNova-U1 falls from 42.6 to 27.2 first-to-last step and Nano Banana Pro from 67.7 to 48.7; giving Nano Banana Pro a GPT-6 Astra enhancer reduces that drop to 82.6 → 76.4. GPT-Image-2.5 remains relatively stable.

• Language-model pairing matters. Adding GPT-6 as an enhancer for Nano Banana Pro improves results across L3–L4 but still trails the GPT-Image family. Swapping GPT-5.5 for GPT-6 as the enhancer for GPT-Image-2 raises the overall score only marginally, with a decline in Physical Simulation.

• Agentic loops give the biggest single gain. Using Codex-based harnesses with the same generation budget, an inspection loop lifts GPT-Image-2.5 from 72.8 to 80.2 (+7.4), and a coding-first inspection loop lifts it to 82.4 (+9.6). Gains concentrate on L4 Visual Goal Realization (61.3 → 81.3) and Logic and Rules (74.3 → 85.7), and on L3 Physical Simulation (69.6 → 88.6). Code-based simulation can be slow, with a single run sometimes exceeding one hour.

• Weak categories cluster around reasoning. Video models struggle most with L4 Logic and Rules; both GPT-Image models, Wan 3.0 and Gemini Omni score lowest within L3 on Adaptive Task Completion; Visual Goal Realization is the hardest L4 category for GPT-Image-2.5 (61.3) and Nano Banana Pro (28.5).

• Automatic evaluation is validated. Blind pairwise comparisons by six human experts against judge scores give Pearson r ≥ 0.97 across four dimensions, with winner agreement ≥ 86%. Adapter-selected versus human-selected keyframes on 100 videos correlate at Pearson 0.9546.

Methodology in Plain English

The authors first build a taxonomy: what can a generative model do, from making one picture up to simulating a changing world? They then design test cases that sit at the top two rungs. Each case gives a model a written instruction plus one or more images that can mean different things — "here is the start", "here is the goal", "here is a material property", "here is a rule from a game", "here is a manual showing a procedure". The model must output an ordered set of pictures showing how the world gets from start to finish. Cases were drawn from egocentric and game datasets, science and educational videos, YouTube assembly videos, illustrated manuals, and hand-designed scenarios, with some scenes synthesized or enhanced by image tools.

Because video models and image models produce output differently, the authors force everything onto one yardstick: an ordered list of keyframes. A video model's clip is converted into keyframes by an agent (GPT-6 Astra driving Codex) that picks frames covering the required states; image models that produce grids are cropped into individual states; models that can interleave text and images simply produce those. Scoring is done by an LLM judge (GPT-5.5) on three dimensions — Task Completion (itself split into Progress Alignment over the sequence and Outcome Completion of the final state), Causal Plausibility, and Visual Consistency — each rated on a 0–3 scale and combined into a 100-point score. To keep the judge honest, human annotators wrote case-specific hints about the objects, milestones and constraints that matter, and those hints are shown to the judge but never to the model being tested.

Finally, the authors run diagnostic experiments: they change the output format (interleaved vs. grid), swap in stronger language models as prompt enhancers, and wrap models in agentic harnesses that inspect and revise their own outputs, optionally using code to simulate intermediate states.

Why This Matters

Impact on research. The paper reframes generative model evaluation away from static image quality toward sequences of causally linked states, and supplies a hierarchy (L1–L4) that gives other researchers a shared vocabulary for what "intelligent generation" means. Its three diagnostic threads — interleaving, language enhancement, and agentic tool use — are presented as concrete research directions rather than just leaderboard numbers. The reported gap between the best proprietary system (72.8) and the best open-source system (20.1) is a useful signal about where open unified models currently stand.

Potential real-world applications (grounded in the task types the benchmark uses):

• Robotic and embodied planning: predicting intermediate states of household activities and manipulation tasks, as in the paper's egocentric and assembly examples.
• Instructional and procedural assistance: turning illustrated manuals into visual step-by-step guidance, including transferring a procedure to a visually different situation.
• Scientific and educational visualization: rendering how a physical experiment or phenomenon unfolds from an initial setup.
• Game and simulation content: predicting state transitions in digital games, puzzles and rule-governed environments.

Industry relevance. The results suggest that capability at this level depends on the whole stack, not just the renderer: the paired language model, the output format, and the surrounding agent harness all move scores substantially. For companies building unified multimodal products, that implies investment in interleaved generation, strong language-side reasoning, and agentic inspection loops — and it warns that grid-style output alone severely limits performance on reasoning-heavy tasks.

Future Directions

1. Models that accept heterogeneous artifacts as guidance. The towel-folding trajectory shows a coding agent producing a simulation that the generator could not reliably use. The authors argue future systems must dynamically interpret agent-produced artifacts rather than treat them as pixel-level anchors.

2. Distilling code-based simulation into direct generation. Because a single code-simulation run can take over an hour, the authors propose distilling the code-simulate-to-visual-render process into unified models during training, so complex queries can be answered with faster inference.

3. Robustness to accumulating context. Step-wise accuracy degrades as the sequence grows for several models, and the fix (a stronger enhancer) only partially closes the gap. Maintaining reliable generation over longer horizons remains open.

4. Closing the open-source and reasoning gaps. The steep drop from L3 to L4 for open-source models, and the persistent weakness on Visual Goal Realization and Logic and Rules, point to a need for models that plan and reason while generating — the "dynamic planning and reasoning, deeply unified" capability the abstract calls for.

Target Audience

Researchers and engineers working on unified multimodal models, image and video generation, or world models, who want a benchmark for multi-step, context-grounded generation. It is also useful for evaluation specialists interested in LMM-as-a-judge protocols with human hints, and for product teams deciding whether to invest in interleaved generation or agentic harnesses. Readers without a background in generative modeling will find the capability hierarchy and qualitative examples accessible, but the experimental sections assume familiarity with the model landscape.

Authors’ abstract

Humans interpret multimodal observations of the world and forecast how it will evolve. Recent multimodal generative systems have demonstrated similar capabilities: they can integrate multimodal context and synthesize visual outputs that depict changes in the world. To characterize this emerging trajectory, we organize model capabilities into a four-level hierarchy and discover that higher-level tasks remain relatively underexplored despite model progress. To bridge this gap, we explore model performance on multi-step world state prediction, a high-level task that requires models to forecast successive future world states from multimodal observations and diverse references. Such a task requires models to possess entangled understanding and generation capabilities, as well as extensive world knowledge to translate human action sequences, physical transformations, and abstract visual reasoning into coherent visual outputs. We introduce WorldCast, a benchmark comprising 320 manually designed cases across eight categories, and evaluate 12 representative models. The conducted experiments yield three key insights: (a) advances in frontier models are rapidly pushing the boundaries of intelligent world modeling, with the GPT-Image-2.5 System scoring 72.8 and demonstrating early-stage visual state prediction capabilities on real-world tasks; (b) WorldCast reveals substantial gaps between models: even Nano Banana Pro reaches only 42.8 and the best open-source model scores 20.1, with deficits in handling complex context, entangled references, and reasoning-heavy tasks; (c) strong generative models effectively draw on the capabilities of their underlying language models. These findings highlight the need for multimodal generative models that are capable of dynamic planning and reasoning while also being deeply unified to faithfully simulate real-world states.