State World Models: Execute the Rules, Generate the Pixels

AI summary

Overview

Research area: Computer vision, specifically interactive video world models and action-conditioned video generation.

Technical level: Intermediate to Advanced. The framing is intuitive, but the paper assumes familiarity with image-to-video diffusion backends, reference-conditioned generation (text, image, video references), vision-language-model judging, and state-machine style rule programs.

Scope: The paper proposes Stateful Harness, a runtime that executes explicit rules over a persistent semantic and spatial world state and then asks a frozen video model to render the resulting event, together with WorldState-Bench, a 50-pair counterfactual benchmark for testing whether hidden state changes produce the rule-mandated visible outcome.

What This Paper Is About

Current video world models can generate video from a single image conditioned on actions, but most of this work is about camera movement and exploration. Navigation control does not guarantee that an action obeys persistent rules when the outcome depends on state the user cannot see in the current frame — for example, whether a chest opens should depend on whether the player holds a key, even when the starting image and action are identical. The authors' goal is to make world-state evolution authoritative and use a frozen video model purely as a renderer, so that hidden state, not the generator's whim, decides what appears on screen.

Key Contributions

1. A separated architecture (Stateful Harness). World evolution is split from video rendering: a rule program resolves each user action against an explicit semantic ledger and an editable 3D "clay model" scene, the resolved event is compiled into a text instruction plus a preview clip, and a frozen video model renders it. A vision-language model then observes the generated clip and proposes state updates that the runtime admits only if compatible with the fixed rules.

2. WorldState-Bench and the Change-to-Comply protocol. A benchmark of 50 counterfactual twin pairs drawn from 49 hidden variables and 21 scenario themes, where each pair shares an initial frame, action script, and rule program but differs in one hidden value that requires a different visible ending. It introduces pair-level metrics (PairFollow, PairStrict) and TwinContrast, alongside clip-level EndFollow, CntAbsent, and Strict.

3. A controlled measurement across 11 frozen backends. Four image-to-video generators (Wan2.2 A14B, LongCat-Video 13.6B, HunyuanVideo 1.5, MiniMax-H3) and seven interactive world models (LingBot-World v2, Echo-WM Flash, FantasyWorld, AlayaWorld, InSpatio-World, Astra, HY-WorldPlay) are evaluated without training or fine-tuning, under both Rule-only and Rule+end instruction probes.

4. Evidence that the harness closes the gap with frozen weights. With the same MiniMax-H3 weights, the full harness reaches 92% PairFollow and 90% PairStrict, versus 32% PairFollow under Rule-only and 64% PairFollow (but only 4% PairStrict) under Rule+end, and it sustains multi-turn episodes of 15, 29, and 11 turns across three genres without retraining.

Main Findings

• Unwrapped backends rarely satisfy both sides of a counterfactual pair. Under Rule-only prompting, the best PairFollow among the 11 evaluated backends is 32%, achieved by both Wan2.2 A14B (65% EndFollow, 38% TwinContrast, 54% Strict, 63% CntAbsent, 26% PairStrict) and MiniMax-H3 (66% EndFollow, 32% TwinContrast, 29% Strict, 30% CntAbsent, 4% PairStrict).

• Naming the required ending helps, but does not establish joint compliance. Under the Rule+end probe, MiniMax-H3 moves from 32% to 64% PairFollow, yet PairStrict reaches only 4% and CntAbsent 30%. The highest unwrapped PairStrict is 26%, for Wan2.2 A14B.

• Absence of forbidden content is not success. Astra, HY-WorldPlay, and InSpatio-World all fall below 20% EndFollow (11%, 9%, and 19%) despite CntAbsent scores of 96%, 94%, and 92%, meaning they frequently produce no required event at all rather than a wrong one. The paper explicitly notes these two failure modes must be separated.

• The harness with the same frozen model reaches 92% PairFollow and 90% PairStrict. The full Stateful Harness on MiniMax-H3 also reports 96% EndFollow, 95% Strict, 98% CntAbsent, and 96% TwinContrast, an improvement of 30 to 86 points over the raw baseline of its track on every metric in Table 1.

• Human raters agree. In a blind pairwise comparison against four widely used unwrapped baselines, Stateful Harness is preferred in 98% of comparisons, with no comparison won by a baseline. Reported win shares are 96% versus MiniMax-H3 and Wan2.2, and 100% versus AlayaWorld and Echo-WM.

• Gains decompose into two separable effects. A component-chain decomposition attributes the improvement to outcome specification (+32 points) and to executed state with compiled references (+28 points), with both increments significant under paired exact McNemar tests over the 50 pairs.

• State persists over long episodes. Three real harness episodes run 15, 29, and 11 turns on the same frozen weights, spanning a stamina-gated town exploration (stamina 28/100 at act 1 falling to 0/100 at act 4, episode ending at t=81 s), a dating narrative (affection rising 47/100 to 54/100 to 75/100, ending in "lovers" at t=158 s), and a turn-based duel (arbiter HP 160 to 0 across turns 4–11).

• Reference channels matter individually. First-frame anchoring raises the full split from 74.2 to 76.1 and setting adherence from 71.1 to 89.6; adding the control video reaches 79.0 on navigation with interaction rising from 69.0 to 77.7, while consistency dips from 83.8 to 75.3 and recovers to 77.8. The same control video under the raw text protocol yields 67.1 on the full split, and clay appearance leaks into outputs when the instruction does not identify the preview as a layout reference.

• Results transfer to an external benchmark. On the WBench averages of the twenty models from Ying et al. (2026) plus two harness entries, video quality is 78.0 versus a text-driven class average of 77.4, physical plausibility 68.8 versus 66.9, setting adherence rises from 71.1 under the official protocol to 87.2, and the interaction score of 77.7 is reported as the highest in the comparison.

Methodology in Plain English

The authors treat a video model the way an LLM "harness" treats a language model: the model supplies pixels, but a surrounding program supplies the logic.

Building the world. Given a world description and an initial image, a coding agent writes a small typed game definition containing the state schema, available actions, rules, events, goals, and visual semantics. A feed-forward geometry model lifts the image into labeled cuboids on a ground plane, producing an untextured, editable "clay model" of the scene. A ledger of typed fields records resources, entity attributes, progress, attitudes, and terminal status. For a benchmark twin pair, the two instances get different hidden values but the same first frame and layout.

Each turn. The user picks an action. The rules — never the video model — decide the outcome: guards are evaluated against the current state and action, producing a provisional state, a rule event, and a spatial edit script (add, move, or remove a box, or change the camera path). An optional neural component may propose dialogue or narration, but schema validation, clamping, bounded retries, and fallback keep its output inside the set of events the rules allow, so it can change wording but not the mandated transition.

Turning the event into pixels. The event is compiled into a VisualScript instruction with slots for actor, action, target, reaction, dialogue, effects, end state, and shots, using a fixed compiler so identical inputs give identical text. A procedural camera program renders the edited clay model into a preview clip that carries layout and camera guidance. A frozen video model receives the text, the preview clip, and a visual anchor (the initial image, then the last frame of the previous clip), and generates the next clip.

Closing the loop. A vision-language model watches the generated clip and proposes observation-based updates to semantic or spatial fields. The runtime accepts a proposal only if it is well formed and consistent with the fixed rules; a visible detail that contradicts the settled outcome is logged as a rendering inconsistency and cannot override the rule result. The updated state governs the next action, while the model weights stay frozen throughout.

Evaluation. Each benchmark case is rendered under two instruction conditions — Rule-only (original case text plus the selected branch and its consequence) and Rule+end (a sentence explicitly stating the required terminal state) — and judged by Qwen3.8-27B-VL answering fixed-choice questions on frames sampled at 3 fps plus the final frame, without access to the hidden value or rule text. Clips with no discernible event or unjudgeable content are marked Unclear and receive no credit. Case construction uses FLUX.2 to generate shared first frames at 1280×720, audited over eight rounds across six dimensions to confirm the hidden value leaves no visible cue.

Why This Matters

Impact on research. The paper reframes a question the field has mostly answered with more data and bigger generators: whether a video world model is rule-governed is not the same as whether it is visually plausible or navigable. By holding the video model fixed and moving all reasoning into an explicit runtime, it provides a clean experimental control — any remaining failure is a rendering or observation problem, not a reasoning one — and it supplies a benchmark that isolates hidden-state sensitivity rather than visual quality, prompt adherence, or camera consistency.

Real-world applications:

• Interactive entertainment and generated games: persistent inventory, quest progress, and combat state that survive across turns, as in the 15-turn town episode and the 11-turn duel.
• Training and simulation: procedural scenarios where the same trainee action must have different consequences depending on hidden conditions such as remaining resources or prior steps taken.
• Authoring tools for non-programmers: new genres are added by writing a new executable game definition — the paper reports three genres instantiated training-free on the same runtime and weights.
• Controllable content pipelines: using a cheap editable 3D proxy plus an instruction to steer a frozen commercial video model, without fine-tuning it.

Industry relevance. The approach requires no engine integration and no weight updates; it works through the stock text, image, and video reference interfaces of existing backends, with one generation request per turn. That makes it a plausible wrapper layer for studios and platform teams who already license video generation APIs and want persistent, auditable interaction semantics on top.

Future Directions

• Harder trigger structures. The paper reports that zone-triggered cases (12 clips) are the most demanding because compliance additionally requires the render to make a region's state visible, which is exactly the motivation given for the spatial channel. Extending spatial reasoning to more complex layout-dependent rules is the natural next step.

• Scaling beyond 50 pairs. WorldState-Bench covers 49 hidden variables across 21 themes, with only 2 form-selection cases and 6 character-response cases against 42 mechanism-selection cases. Broader coverage of hidden-variable families and longer causal chains would test whether the 92% PairFollow figure holds.

• Judging robustness and rendering consistency. Evaluation depends on a single VLM judge with fixed-choice questions, and the channel ablation shows consistency dropping from 83.8 to 75.3 when the control video is added. Improving render fidelity under strong spatial conditioning, and stress-testing judge agreement, are open problems the results point to.

• Observation quality as the bottleneck. The loop's ability to carry state across turns rests on a vision-language model correctly reading generated clips and the runtime rejecting incompatible proposals. How much long-horizon drift accumulates over episodes well beyond 29 turns is not established by the reported episodes.

Target Audience

Researchers and engineers working on interactive video generation, world models, and game-like generative environments; practitioners building agent or "harness" layers around frozen generative models; and benchmark designers interested in counterfactual evaluation protocols. It is also useful for readers who want a compact, quantitative argument for why explicit state and executable rules remain necessary even as video generators improve.

Authors’ abstract

Existing world models can generate action-conditioned video from a single image. Many focus on camera movement, allowing users to explore the generated scene. However, navigation alone does not ensure that an action follows persistent rules when its outcome depends on state that is invisible in the current frame. We present Stateful Harness, a framework that separates world evolution from video rendering to support stateful and rule-governed interaction. It maintains an evolving world state under specified rules, renders each resulting event with a frozen video model, and uses the generated clip to update the world state for the next turn. At each turn, the harness resolves the action selected by the user and compiles the resulting event into an instruction, while a preview clip from an editable 3D scene provides spatial and camera guidance. To evaluate rule compliance, we build WorldState-Bench, a benchmark of 50 counterfactual twin pairs that share an initial frame and action but differ in hidden state and required outcome. Under rule-only prompting, the best of the 11 evaluated video backends produces the required endings for both cases in 32% of pairs. For MiniMax-H3, specifying the required endings raises this rate from 32% to 64%, while the full harness reaches 92% with the same frozen model. These results support separating rule execution from video rendering for world interactions with hidden state.