Generative World Simulation with DAO and JING

AI summary

Overview

• Research area: Computer vision and generative world models — specifically interactive, egocentric audiovisual generation coupled to an explicit, persistent simulation state.
• Technical level: Advanced. The work combines large-scale audiovisual diffusion/flow-matching backbones, multi-stage distillation, autoregressive streaming generation, memory retrieval, and a hand-designed world engine.
• Scope (one sentence): The paper introduces a Generative World Simulation framework that pairs DAO, a computable world engine that maintains shared world state, with JING, an Egocentric Interactive Experience Model built by adapting MiniMax-H3, and evaluates JING on the WBench benchmark.

Links given in the paper: blog at https://xgenlabs.ai/research/generative-world-simulation and model at https://huggingface.co/XGENlabs/XGEN-JING.

What This Paper Is About

Interactive world models can already respond to user inputs such as camera movement or text instructions, but they generally do not hold an explicit, queryable record of the world that persists when an observer leaves, changes viewpoint, or returns. The paper's goal is to connect two halves of a system: a world engine (DAO) that stores characters, objects, spaces, their relationships, world time, and active events, and an experience model (JING) that generates what a specific observer sees and hears next from that state. The result is a continuous loop in which actions change the world, world changes condition the next generated experience, and those experiences shape the next round of human or agent choices.

Key Contributions

1. JING: controllable first-person audiovisual experiences. JING combines character, object, and scene references with observation history and current actions to generate continuous audiovisual content. Users steer exploration with keyboard and mouse inputs and express manipulation and communication intents through text instructions, across diverse settings and visual styles.
2. DAO: computable and persistent shared world state. DAO explicitly represents characters, objects, spaces, and their relations, maintains world time and active events, processes actions according to interaction rules, carries changes forward, retains unchanged information, and preserves state as observers change viewpoints, leave, or return.
3. Continuous closed-loop interaction in a shared world. DAO and JING connect action consequences, local observations, and subsequent choices in an ongoing cycle, so that different choices from the same starting context lead to different developments, each building on its history.
4. A complete training and data recipe plus benchmark evaluation. The paper describes a multimodal data pipeline (acquisition, filtering, temporal segmentation, captioning, camera and reference conditioning), a five-stage JING training procedure, and JING's placement on the WBench Full and Navigation splits.

Main Findings

• WBench Full split (289 cases, pool of 15 entries): JING (Bidirectional) ranks 1 with an average of 81.0, and JING (4-step AR) ranks 2 with 79.9. The per-metric breakdown for JING (Bidirectional) is 80.9 video quality, 88.5 setting adherence, 76.7 interaction adherence, 85.7 consistency, and 73.3 physics; for JING (4-step AR) it is 79.8, 88.9, 79.5, 81.4, and 69.9. For comparison, Kling 3.0 scores 79.4 average and Wan 2.7 scores 78.4.
• WBench Navigation split (158 cases, pool of 39 entries): Alaya-EVOKE-Turbo ranks 1 with 82.0, followed by JING (Bidirectional) at 81.9 in rank 2 and JING (4-step AR) at 81.0 in rank 6. Also reported: JoyAI-Echo-1.5 (Bidirectional) 81.6, JoyAI-Echo-1.5 (4-step AR) 81.0, Zing-0.5 81.0, HiDream-O1-World 80.9, Kling 3.0 79.0 (rank 10), Wan 2.7 78.1 (rank 13), and Genie 3 73.9 (rank 29).
• Distillation largely preserves quality. The paper states that the training and distillation pipeline transfers the bidirectional model to a four-step causal generator while largely preserving overall performance, based on the 81.0 versus 79.9 Full average and 81.9 versus 81.0 Navigation average.
• Flexible reference conditioning. Generated sequences retain recognizable scene features (the paper names a clock tower and a bakery interior) while incorporating supplied character and object appearances, demonstrated for scene-only, scene-plus-character, and scene-plus-character-plus-object configurations.
• Capabilities shown qualitatively. JING generates egocentric exploration across environments and observer scales (including beetle scale), action-conditioned object manipulation (a hammer striking heated metal, a young dragon igniting a woodpile, hands releasing a sky lantern), character communication with generated speech in Chinese, English, and Japanese, action-conditioned branching from a shared fountain opening, reference-conditioned branching from a shared bakery opening (characters Clara, Tom, and Eli), and embodied-robot observations (mug placement, ball retrieval for a dog, plant watering).
• Embodied results are illustrative only. The paper explicitly states that the household-robot scenarios illustrate a potential observation interface for embodied agents and do not measure physical robot task success.
• DAO supports autonomous agents. In a six-event chain driven by agent intentions, a blocked lane is cleared, a misreading triggers an argument, a partial witness observes it, another character intervenes, evidence is checked, and social repair follows — all arising from agents pursuing their own intentions rather than explicit user triggers.
• One event, different experiences. DAO derives observer-specific information from a common world state so that Clara, Eli, and Nori each receive a first-person view of the same interaction at the same simulation time.
• World state persists through absence. In a departure-and-revisit comparison, generation without DAO shows a previously ignited napkin still burning with no evidence of an intervening response, while generation with DAO reflects the attendant extinguishing the fire during the observer's absence.
• Not reported in the provided content: numerical runtime-efficiency results, quantitative DAO ablation numbers, and quantitative metrics for the qualitative capability demonstrations. The abstract states that generation quality, action responsiveness, state persistence, long-term consistency, and runtime efficiency are examined, but only the WBench table and qualitative figures appear in the supplied text.

Methodology in Plain English

Data. The team assembled training material aimed at egocentric experience: first-person videos from publicly accessible platforms, game and interactive-video footage collected by the team and external partners, videos rendered in controllable 3D environments, and model-generated videos of interaction and dialogue. These are supplemented with subsets of the open-source SpatialVID, Sekai, and DL3DV datasets.

Processing. Automated media checks screen resolution, frame rate, decoding reliability, sharpness, exposure, compression artifacts, and frozen frames. Videos that pass are semantically assessed by a vision-language model examining sampled frames, short high-frame-rate windows, and audio for first-person perspective, motion continuity, shot transitions, visual overlays, and unrelated background music. Qwen3.6 is then used to find segment boundaries, adjusted so that scene revisits keep departure and return together and interactions and dialogue keep complete exchanges together. Captions are generated with a first-person perspective, and camera conditions are produced by ViPE, which estimates intrinsics and six-degree-of-freedom poses that are smoothed and filtered. Reference images for characters, objects, or scenes are cropped from frames or generated from frames and prompts. Everything is packaged into sharded LMDB datasets holding video and audio latents, text embeddings, and optional camera or reference-image conditions.

Model. JING adapts MiniMax-H3 to generate audiovisual latent chunks autoregressively from reference images, a text instruction, a camera trajectory, and the previously generated history. Reference images are encoded both by Qwen3-VL (together with text) for semantic conditioning and by the video VAE for visual reference tokens. Camera poses are converted into a six-dimensional relative motion representation, turned into control features by a dedicated encoder, and injected into video tokens in each Transformer block through feature-wise linear modulation. The latent temporal partition is rearranged from (5, 5, ..., 2) to (2, 5, 5, ...) so the initial 2+5 latent frames form an initialization chunk followed by five-latent-frame chunks, aligning generation boundaries with the VAE's overlapping decoding windows for incremental decoding. Attention context for each chunk combines reference tokens, the fixed first chunk, a recent-history window, and retrieved memories, with causal access across chunks. A memory archive stores pooled query and key descriptors per video latent frame and per group of stereo audio latents; the query descriptor from the last completed chunk scores earlier units, and selected units are replays with dedicated position slots and memory-role embeddings to build memory keys and values.

Training. Five stages: (1) train a bidirectional teacher with the camera-control module under bidirectional attention; (2) train a causal autoregressive model using teacher forcing augmented with resampling forcing, where historical segments are noised, reconstructed, and reused as context; (3) add the audiovisual memory and vary recent-context length, memory selection, and capacity so the model learns to use different historical combinations; (4) initialize a few-step student with Velocity Moment Matching (VMM), which matches conditional velocity moments at intermediate noisy states using a frozen teacher and a trainable auxiliary network; (5) refine the student with Distribution Matching Distillation (DMD), using a frozen teacher and a score model trained on student-generated samples.

World engine. DAO represents world state at step t as (G_t, B_t, E_t, ω_t): an entity graph of places, characters, and objects with poses, appearance, semantic attributes, and spatial/attachment relations; actor and interaction state covering intentions, motion, activities, and attributes such as emotions, needs, and object use; an active event set; and a world clock. Inputs pass through action admission, which resolves targets and checks interaction rules and constraints such as reachability and target availability, and can reject violating inputs. Admitted inputs drive an ordered update that applies queued edits, advances motion, resolves spatial constraints and dynamic obstacles, then advances activities and events, carrying unchanged state forward. Local readout selects relevant entities by sensing range, field of view, and occlusion to produce observer-specific conditions for JING. Snapshots and event history allow the world to be restored after a session ends or a process restarts.

Why This Matters

The paper's distinctive move is separating shared, persistent world state from per-observer experience generation. Most interactive video world models produce convincing next observations but have no explicit record of what happened; DAO gives actions consequences that survive viewpoint changes, departures, and returns, and lets several observers see different local views of the same evolving world. The staged distillation result also matters practically: it shows that a bidirectional camera-controllable teacher can be compressed to a four-step causal generator with only a small drop in benchmark average (81.0 to 79.9 on Full), which is the regime interactive applications need.

Real-world applications the paper's capabilities suggest:

• Interactive entertainment and games: explorable generated environments where player actions, object manipulation, and character conversations persist and shape later encounters.
• Embodied agent development: first-person generated observation interfaces from a robot's viewpoint, as illustrated by mug placement, ball retrieval, and plant watering, with the explicit caveat that physical task success is not measured.
• Multi-agent simulation and social scenarios: world engines where non-player characters act autonomously from their own memories and local observations, producing chains like the blocked-lane, argument, witness, and reconciliation sequence.
• Training and prototyping environments: controllable, revisitable generated scenes with consistent character, object, and scene references, useful for content creation and scenario rehearsal.

Industry relevance: the work sits at the intersection of video generation, interactive entertainment, and robotics simulation. It provides an architectural pattern (explicit state engine plus a conditioned generator) that others can adopt without needing the full system, and the released model weights and blog entry indicate an intention for external use.

Future Directions

• Quantifying what is currently shown qualitatively. The paper reports WBench numbers for JING but describes DAO evaluation as qualitative demonstrations and ablation experiments; publishing those ablations and runtime-efficiency figures would let others compare engine designs directly.
• Extending persistence and history depth. DAO uses snapshots and event history, and JING bounds historical context through a recent window plus a limited memory budget; how far the world and its generated experiences can scale in time and entity count remains open.
• Closing the loop with real embodied agents. The robot scenarios are explicitly illustrative and do not measure task success, so connecting DAO and JING to physical systems, or measuring generated-observation usefulness for policies, is a natural next step.
• Multi-observer consistency at scale. The paper demonstrates one event seen from Clara, Eli, and Nori; whether many observers can be served from the same shared state with consistent, coherent generated views over long sessions is not established.

Target Audience

This paper is most useful to researchers and engineers already comfortable with diffusion and autoregressive video generation, multimodal conditioning, and inference-time distillation, who are working on interactive world models, generative game environments, or egocentric audiovisual synthesis. It is also relevant to simulation and robotics researchers interested in using explicit world state to make generated observations persist across sessions and observers, and to technical readers tracking how benchmark performance on WBench Full and Navigation is reported and compared rather than absolute new capability claims.

Authors’ abstract

We present a Generative World Simulation framework that couples persistent shared world state with continuous first-person audiovisual generation. The framework integrates DAO, a computable world engine, and JING, an Egocentric Interactive Experience Model, to support sustained interaction for humans and agents. DAO maintains the states of characters, objects, and spaces and their relationships, advancing the world according to interaction rules and world time while preserving the consequences of past actions. From this evolving state, DAO derives observer-specific conditions for JING, which combines these conditions with character, object, and scene references, observation history, and current actions to generate each observer's audiovisual experience. Users can explore environments, manipulate objects, and communicate with characters through viewpoint control and text instructions, with their actions shaping both the shared state and subsequent experiences. By connecting world-state evolution with experience generation in a continuous interaction loop, the framework offers an approach to building interactive worlds that retain their history, evolve through action, and persist beyond any individual observer.