Reading, Writing, Sharing, and Interacting with the World through Video

AI summary

Overview

• Research area: Computer vision, specifically video foundation models (video understanding and video generation) and video world models, surveyed through a single "world modeling" lens.
• Technical level: Advanced. The paper is a formal survey with definitions, notation tables, and a taxonomy; it assumes familiarity with video encoders, Video-LLMs, diffusion/flow-based generation, and model-based reinforcement learning.
• Scope (one sentence): A survey that reframes video understanding, video generation, unified video models, and interactive video world models as four operations — reading, writing, sharing, and interacting — on the relation between an observed video and the latent world that produced it.

What This Paper Is About

Video foundation models are getting very good at describing and synthesizing video, while video world models try to capture the actual dynamics of the world a video shows. The paper argues these are two sides of one question: what relationship does a model learn between an observed video and the evolving world behind it? To answer it, the authors treat an observed video as a partial, lossy, and temporally ordered projection of a latent temporal world, and organize the literature around four operations on that relationship: reading (inferring world properties from video), writing (synthesizing video from world conditions), sharing (unifying the two in one framework), and interacting (predicting and updating world states under actions).

Key Contributions

1. A world-modeling lens. The four-operation framework (reading, writing, sharing, interacting) places video understanding, video generation, unified video models, and video world models into one account, rather than grouping models by input/output modality.
2. A two-axis classification of interactive world models. Within the "interacting" paradigm, models are separated by prediction space (observation-space versus latent-state prediction) and transition interface (passive prediction versus action-conditioned rollout and closed-loop interaction) — separating what a model predicts from how its predictions are conditioned, updated, and used.
3. Comprehensive scope with distilled insights. Across tasks, representations, models, systems, datasets, and benchmarks, the survey compares how the four operations represent state, learn from video, accept control, preserve memory, and are evaluated, and it distills recurring design patterns and failure modes into explicit insights and open directions.
4. Open resources. A public reading list, taxonomy, code, data, and other resources are released on a project page and an accompanying GitHub repository ("awesome-video-world-model").

Main Findings

• Temporal coherence is not world consistency. The paper distinguishes two properties: a trace is temporally coherent if neighboring frames agree locally (a discrepancy under a tolerance δ), and world-consistent if its observations are compatible with at least one admissible world trajectory satisfying assumed dynamics, identity persistence, spatial relations, and physical/causal constraints. Neither implies the other: an object that changes identity after leaving the frame, or a room whose layout changes between visits, can be locally smooth yet unexplainable; a hard cut violates local smoothness yet can remain world-consistent.
• Recorded video is world-consistent by construction; generated video is not. Because a real trajectory produced a recording, the coherence/consistency distinction only bites for traces a model writes or predicts — so world-modeling claims must be tested for consistency, not merely for smoothness.
• An "evidence ladder" for world-modeling claims. Each of the four operations is scored at four increasing levels: (1) plausible output, (2) faithful to evidence or conditions, (3) consistent world state, and (4) decision utility. Stronger claims about an internal world model require higher rungs — e.g., for writing: perceptual quality → adherence to text/motion/reference conditions → world consistency (identity, layout, physics) → generated data or plans that improve downstream models.
• Seven descriptive labels for video scenarios. Regular short video, sounding video, egocentric video, streaming and online video, long-form video, interactive video, and domain-specific video — each imposing different demands (e.g., causal state updates and bounded rolling memory for streaming; evidence retrieval and long-range memory for long-form; action fidelity, persistent state, closed-loop latency, and decision utility for interactive). These are described as complementary characteristics, not mutually exclusive categories.
• Three objects, carefully separated. The latent temporal world W (a stochastic process with dynamics conditioned on an optional action/intervention), the observed video V (a trace produced by a sensing process with camera configuration), and a model's represented state s, which in general is not equal to the true world state and may be incomplete or inaccurate.
• Convergence of previously separate lines. Video-LLMs, large video generators, unified understanding-and-generation models, and action-conditioned simulators are converging on the same challenge of connecting observations to dynamics; the survey notes this spans work from video encoders and generators to unified models and action-conditioned simulators.
• A shift in evaluation emphasis. Across tasks, representations, models, datasets, and benchmarks, the survey traces a move away from clip-level recognition accuracy and perceptual fidelity toward persistent state, controllable evolution, physical and causal validity, and evaluation grounded in interaction and downstream decisions.
• A gap relative to prior surveys. Existing surveys cover general world models, interactive video world models, 3D/4D world modeling, multimodal LLMs, long-form video generation, and audio-visual foundation models, but do not jointly account for how video models connect observations to evolving world states across understanding, generation, unification, and interaction.

Methodology in Plain English

This is a survey, so the "method" is conceptual organization plus literature synthesis rather than experiments.

• Formal scaffolding first. The authors define the latent world, the observed video (a partial, lossy, temporally ordered trace), the model's represented state, and the four operations as mappings: reading maps video to world properties; writing samples a trace from a condition (text, image/video prefix, audio, camera motion, trajectories, layouts); sharing couples a reading map and a writing map through at least one shared component (parameters, tokens/representations, an interface, or a training objective); interacting rolls out a transition model conditioned on actions and conditions, with a decoder producing visible output when needed.
• Consistency defined as an evaluation target, not a metric. Because the sensing process and the set of admissible world trajectories are unobserved, world consistency is framed as something benchmarks must probe — via revisits, occlusion, and interventions — rather than something computed directly.
• Organization by operation, not by modality. After shared background (video scenarios, Video-LLM foundations such as encoder–connector–LLM pipelines, and generative modeling foundations such as VAE latents and diffusion/flow denoising), the survey walks the four operations in sequence, then distills cross-cutting challenges and closes with a conclusion.
• Claims checked against evidence type. Throughout, the survey separates what a model's design supports, what its training data supervises, and what its evaluation actually demonstrates, explicitly avoiding the inference that strong performance on individual video tasks implies consistent or generalizable world modeling.
• Scope rules. Coverage is restricted to publicly available research contributing architectural, methodological, or evaluation evidence, with foundational older studies retained where needed.

Note: the supplied excerpt ends within Section 4.3 (generative modeling foundations), so the detailed contents of Sections 5–8 and 9 are represented here only through the roadmap and summary statements the paper makes about them.

Why This Matters

• For research: It supplies shared vocabulary and a common yardstick — the coherence/consistency split and the four-level evidence ladder — for comparing models that currently get evaluated with incompatible metrics (recognition accuracy for readers, perceptual fidelity for writers, downstream reward for interactors). It also reframes "does this model have a world model?" as an evidence question rather than a branding question.
• Real-world applications (drawn from the application areas the paper names):
  • Autonomous driving — driving world models such as GAIA-1/GAIA-2, Vista, DriveDreamer, DrivingWorld, Cosmos and Cosmos-Drive-Dreams, with nuScenes and DriveLM as representative resources.
  • Robotics and embodied agents — egocentric and robotic learning, policy training and evaluation via simulated futures (Ego4D, EPIC-KITCHENS, Ego-Exo4D, and simulation-oriented systems such as UniSim, iVideoGPT and Dreamer 4).
  • Healthcare — surgical video as a domain-specific setting (Cholec80), where errors carry real cost.
  • Education and instructional media — instructional video resources such as HowTo100M and COIN, plus communication and entertainment use of generated video.
• Industry relevance: The surveyed systems are largely industrial-scale (Sora and Sora 2, Kling, Veo 3, Movie Gen, HunyuanVideo, Wan, Seedance 2.0, Cosmos, Genie 3, Matrix-Game/2.0). The push toward real-time responsiveness, long-horizon memory, and world-consistent generation maps directly onto product needs for interactive media, simulation, and data engines for training embodied policies.

Future Directions

1. Long-horizon consistency and memory. How to keep identity, layout, and state persistent across long rollouts and revisits, given mechanisms like rolling memory, self-forcing, and world-memory modules surveyed in the interacting stream.
2. Physical and causal grounding. Moving from visually plausible motion to dynamics that respect physical and causal constraints, and developing benchmarks that probe them rather than reward appearance.
3. Measurable synergy between understanding and generation. The survey calls for transfer measured under matched data, capacity, and compute — i.e., showing that sharing actually improves reading, writing, or both, rather than merely co-locating them.
4. Standardized evaluation of world properties. Turning world consistency from a definition into practice: benchmarks built on revisits, occlusion, and interventions, and evaluation tied to downstream decisions, planning, and policy ranking rather than frame quality alone.

Target Audience

Researchers and advanced graduate students in computer vision, generative modeling, and embodied AI who work on video understanding, video generation, unified multimodal models, or learned simulators; benchmark designers and evaluation researchers who need principled criteria beyond perceptual quality; and technical leads in industry building video generation, interactive media, simulation, or robot/driving data engines. Beginners will find the framing intuitive but will need background in Video-LLMs and diffusion models to follow the formalism.

Authors’ abstract

Video foundation models have advanced rapidly in understanding and generating visual sequences, while video world models seek to capture the underlying dynamics of the world they depict. Despite their different objectives, both concern the relationship between video observations and an evolving world. This survey adopts world modeling as a unifying lens for examining these developments. Viewing an observed video as a partial, lossy, and temporally ordered projection of a latent temporal world, we organize the literature around four complementary operations: reading infers world properties from observed video; writing synthesizes video from specified or learned world conditions; sharing integrates understanding and generation through common modeling mechanisms; and interacting predicts and updates world states under actions and evolving observations. This perspective distinguishes temporal coherence in observed or generated sequences from consistency in the underlying world state. Across tasks, representations, models, datasets, and benchmarks, we trace a shift in emphasis from clip-level recognition and perceptual fidelity toward persistent state, controllable evolution, physical and causal validity, and evaluation grounded in interaction and downstream decisions, and we distill the open challenges that follow.