Programmable World Model

AI summary

Overview

Research area: Computer vision and generative world modeling, specifically interactive video world models that simulate playable environments.

Technical level: Advanced. The paper assumes familiarity with diffusion-based video generation, conditioning mechanisms (ControlNet-style adapters), 3D bounding boxes, and camera geometry. The high-level idea is accessible, but the implementation details are specialized.

Scope: The paper proposes a framework that separates explicit, programmatically controlled world-state evolution from neural video rendering, and introduces a benchmark for measuring whether generated videos stay faithful to that maintained state.

What This Paper Is About

Existing video world models can generate realistic, interactive footage, but they do not maintain a reliable, persistent record of what is happening in the world. They cannot easily be instructed to follow explicit rules (for example, "the door opens only when this switch is hit"), and they handle off-screen entities, inventory, health, and task progress only implicitly, if at all. This paper's goal is to make world behavior programmable and verifiable by splitting the problem into two parts: a small engine that deterministically tracks world state and the rules governing it, and a generative video model that merely renders what that state implies.

Key Contributions

1. A decoupled architecture for programmable worlds. The Programmable World Model separates executable world-state evolution from visual observation generation. A coding agent converts natural-language instructions into programs defining entity states, attributes, relations, and transition rules, which a lightweight engine executes to maintain a persistent canonical world state.

2. State-augmented 3D oriented bounding boxes as an intermediate representation. Each entity is represented by a compact 3D OBB (position, extent, orientation) augmented with persistent identity, semantic category, dynamic state, and appearance info. A deterministic state compiler projects these boxes along the target camera trajectory and rasterizes them into pixel-aligned identity, semantic, and motion-direction control maps.

3. An automatic data curation pipeline. Because structured annotations for camera geometry, instance identity, semantics, and object trajectories are scarce in unlabeled video, the authors build a pipeline combining camera estimation (ViPE), VLM-based semantic category discovery (Qwen3-VL), video instance segmentation and tracking (SAM3), and 3D OBB estimation (WildDet3D).

4. CombatStateBench. A controlled benchmark that evaluates whether generated videos are consistent with engine-maintained world states, focusing on visible alive-character counts and the visual realization of death states under varied camera and entity motion.

Main Findings

• Strong state-faithfulness numbers. On CombatStateBench, the method achieves 94% Count Accuracy and 98% State Accuracy, substantially outperforming existing interactive video world models.

• Explicit state beats learned state prediction. Related systems such as StatePlay predict game-state variables with a model, and MASS advances an authoritative typed state using a learned Logic Engine. Both remain vulnerable to transition-prediction errors that accumulate over long horizons. Maintaining state through explicit, executable rules makes it directly programmable, editable, and verifiable.

• Representation choice is a trade-off between control and cost. Text descriptions are cheap to author but impose no geometric constraints; 2D boxes and masks are viewpoint-dependent; full 3D scenes, articulated models, and G-buffers give finer control but demand richer supervision and force the system to resolve more degrees of freedom at inference.

• Training and inference follow opposite directions. Training recovers structural representations from already-realized dynamics; inference must generate those structural trajectories from high-level state transitions. This asymmetry means representations that are easy to extract during training may be hard to produce at inference. The 3D OBB abstraction was chosen to sit in the middle of this spectrum.

• Object motion must be disentangled from camera motion. The direction map is computed from an entity's world-space velocity, rotated into the target camera frame, and quantized into seven states (forward, backward, left, right, static, up, down). A static entity stays "static" even when camera movement shifts its projected position, resolving an ambiguity that image-space controls cannot.

• Long-horizon coherence comes from two memory systems. Chunk-autoregressive generation combines a bounded multi-scale temporal latent history (recent, mid-range, long-range, plus a persistent anchor of the initial observation) with a geometry-aligned spatial memory that lifts completed frames into world space and reprojects them into new target views, following AlayaWorld.

• Identity and semantics are handled separately. Persistent entities are assigned fixed slots from a bank of learnable identity embeddings (randomized across samples, fixed within a sequence), while semantic labels are embedded via a pretrained text encoder. This lets same-category entities share semantics while remaining individually trackable, which matters for entities absent from the initial observation.

• Renderer training is efficient. The pretrained camera-controlled backbone (LingBot-World-v1) and its camera-conditioning pathway stay frozen; only the newly attached Structured Spatial ControlNet and cross-chunk conditioning parameters are optimized.

Methodology in Plain English

The system has three stages that run in a loop.

First, programming. Given one reference image and a written description of the world, an off-the-shelf 3D detector recovers the visible objects and their layout. A vision-language coding agent then writes a program that names those objects, assigns them identities, attributes (health, faction, ownership), relationships, and rules for how actions change things. A small engine executes that program. Crucially, the engine holds facts the camera cannot see: inventory, task progress, whether an off-screen character is alive.

Second, compilation. The engine's world state is not directly renderable, so a deterministic compiler projects every entity's 3D box through the target camera and rasterizes three aligned maps: which instance is where, what category it is, and which way it is moving. It uses depth-aware rasterization so overlapping boxes resolve correctly.

Third, rendering. These maps, plus the camera trajectory and prior video frames, condition a pretrained video generation model. The model is not asked to invent positions or identities; it fills in appearance, texture, articulation, and secondary motion that the box representation leaves unspecified. Generation proceeds in chunks, with each finished chunk feeding both a temporal latent history and a world-space spatial memory so that the scene stays coherent when the camera returns to previously seen areas or occludes objects.

To train the renderer, the authors built a pipeline that takes raw video and automatically recovers everything the conditioning maps need: camera intrinsics and poses, metric depth, object categories, instance tracks, and per-frame 3D boxes with world-space trajectories.

Why This Matters

Impact on research. The paper reframes an important question in world modeling: not "how do we generate more realistic video" but "where should explicit structure end and generative completion begin." Its analysis of the training-inference asymmetry for structural representations is a useful conceptual contribution independent of the specific system, and separating state from rendering gives the field a cleaner target for evaluation.

Real-world applications:

• Rapid game prototyping. Designers describe mechanics in natural language and get a playable, visually rendered environment without building assets, animation rigs, or physics code.

• Simulation and training environments. Robotics, autonomous driving, and defense simulations need deterministic, inspectable state (positions, counts, outcomes) alongside photorealistic imagery. Explicit state makes results auditable in a way a neural predictor cannot be.

• Multiplayer or multi-agent interactive media. Because the canonical state is shared and view-independent, multiple observers or players can interact with the same persistent world while each receives their own rendered viewpoint.

• Content authoring with programmatic control. Filmmakers or level designers could script entity behavior and camera paths while letting a generative model handle the visual realization.

Industry relevance. The approach points toward a practical division of labor for interactive entertainment and simulation: cheap, verifiable CPU-side logic on one side, expensive generative rendering on the other. It also suggests a route to scaling training data automatically from unlabeled video, which matters for any company that cannot afford hand-annotated 3D supervision. The reliance on a frozen pretrained backbone plus a lightweight control adapter is a cost profile that fits production constraints.

Future Directions

• Reducing dependence on the box abstraction. 3D OBBs cannot express articulated poses, cloth, or fine deformation. Determining how much additional structure can be added before inference-time state construction becomes as hard as the animation problem itself is an open question.

• Validating the programming interface beyond combat. CombatStateBench covers one scenario family with countable entities and binary alive/dead states. Whether agent-written rules remain correct and verifiable for richer mechanics, long branching objectives, or many interacting entities is untested.

• Scaling the data pipeline to open-domain video. The curation pipeline depends on multiple pretrained components (ViPE, SAM3, WildDet3D, Qwen3-VL). Error propagation and failure cases across diverse unlabeled footage are not characterized.

• Extending to multiplayer and multi-view settings. The paper motivates shared state for multiplayer worlds but demonstrates a single-observer interaction loop. Coordinating multiple simultaneous agents, their actions, and consistent rendering across divergent cameras remains future work.

• Longer horizon and state-error accumulation. While the method avoids learned transition errors, it is not immune to mistakes in the agent-written programs themselves, nor to renderer drift over very long rollouts. Measuring and bounding both over extended interaction is an open problem.

Target Audience

Researchers and graduate students working on video generation, world models, and interactive generative environments, particularly those interested in controllability, persistent state, and structured conditioning interfaces. It is also relevant to graphics and game engine practitioners evaluating where learned rendering can replace or augment conventional pipelines, and to engineers building simulation or prototyping tools that require both photorealism and verifiable, scriptable behavior. Readers without background in diffusion-based video generation will find the conceptual framing accessible but the method sections demanding.

Authors’ abstract

Recent video world models generate increasingly realistic and interactive visual experiences, yet lack reliable mechanisms for maintaining persistent world state and enforcing programmable rules over extended interactions. We introduce Programmable World Model, a framework that decouples world-state evolution from visual observation generation. An agent translates natural-language instructions into executable programs that specify entity states and state-transition rules, enabling direct control over individual entities and their interactions. A lightweight engine executes these programs to update and maintain an explicit, persistent global world state, including off-screen entities and non-visual attributes. To connect world state with visual generation, we introduce state-augmented 3D oriented bounding boxes (OBBs) as an intermediate representation. This representation, together with the target camera trajectory, is deterministically compiled into pixel-aligned spatiotemporal conditioning signals for a pretrained video model serving as the generative renderer. This design allows users to create playable games with predefined mechanics, direct control over individual entities, and persistent world state throughout gameplay. We further introduce CombatStateBench, a benchmark for evaluating programmable world models. On CombatStateBench, our method achieves 94% Count Accuracy and 98% State Accuracy, substantially outperforming existing interactive video world models while supporting coherent long-horizon generation. These results demonstrate the effectiveness of separating explicit state evolution from generative rendering for building persistent, programmable worlds.