World in World: Explore the World with World Models

AI summary

Overview

Research area: Computer vision, specifically controllable video generation and autoregressive video world models.

Technical level: Advanced. The paper assumes familiarity with denoising diffusion/flow-based video generation, causal autoregressive rollouts, transformer self-attention, rotary position embeddings, and classifier-free guidance.

Scope: The paper introduces World in World (WiW), a training-free inference-time interface that converts heterogeneous control inputs into camera- and time-labelled "clean visual states" which a frozen causal video world model reads through its existing self-attention layers.

What This Paper Is About

Interactive video world models can generate long rollouts that continue from what they have already produced, but steering them precisely—for example, re-observing a recorded video from a new camera path—usually demands task-specific modules or additional training. WiW's goal is to let one frozen pretrained model accept many kinds of visual control (source observations, projected geometry, rendered subjects, and previously generated frames) without changing a single weight. The central claim is that control of a world model is better framed as a problem of constructing, localising, and regulating visual evidence than as a problem of adapting the model.

Key Contributions

1. World in World (WiW), a training-free visual-evidence interface. All pretrained parameters stay frozen; no control-specific adapters or fine-tuning are introduced. Control evidence is injected only at inference time through the backbone's native self-attention.

2. Complementary evidence construction for event-synchronised exploration. Four evidence channels are defined and unified into one representation: source-video observations (appearance), depth-based target-view projections (spatial layout), rendered subject geometry (completion of newly exposed surfaces), and retrieved generated history (long-horizon consistency). Each carries token-level spatial support weights and its own activation schedule across denoising steps.

3. Two attention-level mechanisms for evidence use. Correspondence-guided attention routing (CGAR) uses persistent point identities plus camera geometry to route queries toward geometrically matching source-video tokens when appearance matching is ambiguous. Evidence-wise attention CFG (EWA) independently regulates each auxiliary channel's contribution at the attention-response level, adding no extra network function evaluations.

4. Demonstration of one frozen backbone across many tasks. Camera-controlled rerendering, long-horizon revisiting, human-motion transfer, bullet-time generation, video stabilization, video editing, and K/V sharing between two generation instances all run on the same frozen model.

Main Findings

• Strongest overall quantitative result. On DAVIS and OpenVid-1M, WiW reaches the highest average across seven VBench dimensions (Overall 85.192) and the lowest camera trajectory errors among compared methods (ReCamMaster, TrajectoryCrafter, WorldForge, InSpatio-World, UniWorld-View, CameraAnything).

• Best camera-following accuracy. TransError of 0.0686 and RotError of 2.83 degrees—both the lowest reported, with the rotation error substantially below the next best (CameraAnything at 3.19 degrees).

• Best or near-best perceptual quality. Highest PSNR (23.15) and SSIM (0.787), with LPIPS (0.1217) effectively tied with UniWorld-View (0.1217); the one VBench dimension where a baseline clearly leads is Temporal Flickering.

• Target-view warping is the single most critical component. Removing it degrades every metric, raising rotation error roughly 3.4x and translation error roughly 10.9x above the full method. Removing source-camera Plücker conditioning also increases camera errors, supporting the retention of both target-view layout and source-view camera information.

• CGAR and EWA both matter. Ablating either degrades subject consistency, background consistency, and camera accuracy, indicating that localising evidence and controlling its strength are distinct, non-redundant problems.

• Evidence channels address different failure modes. Rendered body geometry specifically guides completion of subject surfaces never observed in the source video, while historical retrieval restores appearance and layout when the camera returns to a region whose states have already been evicted from the rolling cache.

• Guidance is free in compute terms. EWA reuses attention responses from the same denoising forward pass, so multi-channel guidance costs no additional network function evaluations.

• One interface, many applications. Bullet-time rendering, stabilization, editing, human motion transfer, and K/V sharing between two generation cases are all expressed as different evidence configurations over the same frozen backbone.

Methodology in Plain English

The starting observation is that causal video world models already have a standard way of reading visual information: their self-attention layers consume the first frame of a video and recently finalised outputs as "clean" latent states, tagged with camera pose and time. WiW exploits this existing pathway instead of building new ones. Any control signal is converted into the same format—an image-like state plus camera parameters, a timestamp, a per-token validity weight, and a schedule saying during which denoising steps that channel should be active.

Four kinds of evidence are built. Source frames supply appearance. Those frames are depth-estimated and reprojected into the requested camera to give an explicit spatial layout, with visibility masks deciding which reprojected pixels are trustworthy. When the new camera reveals parts of a moving subject that the source video never showed, an avatar reconstructed from the source footage and driven by body-model parameters is rendered from the target view to propose plausible shape and appearance there. Finally, because the model's rolling cache only holds recent frames, previously generated states are archived and retrieved when the camera revisits an old region—selected by how much of the current view they cover and how similar the viewing direction is, with a cap on how many are used.

Two regulation mechanisms decide how this evidence is read. The routing mechanism tracks points with persistent identities across the source video and uses depth plus camera geometry to link current queries to the source tokens they actually correspond to, biasing attention toward those matches. The guidance mechanism compares the attention response produced with evidence against the response produced without it, subtracts the component that simply duplicates the model's existing output, and amplifies only the complementary direction—scaled by an independent strength per evidence channel and bounded by the native response magnitude so it cannot overwhelm generation.

The whole system is instantiated on the publicly released causal-fast checkpoint of LingBot-World 2.0 with all weights frozen, and evaluated on DAVIS and OpenVid-1M using VBench dimensions, camera trajectory errors estimated from the generated videos, and standard image fidelity metrics.

Why This Matters

Research impact. The paper reframes controllable world-model generation from "train a new conditioning pathway per control type" to "construct and orchestrate visual evidence for a fixed model." If that framing generalises, it suggests a path toward a single pretrained world model that absorbs new control modalities—text, actions, audio, physics constraints—without retraining, which is a meaningful shift in how the community might think about controllability.

Real-world applications:

• Interactive content creation and virtual production: directors could re-shoot a captured performance from a different camera angle while preserving the original timing and appearance of the event.
• Game and simulation asset generation: navigating a recorded scene along arbitrary camera trajectories, or sharing generated latent state between simulation instances.
• Video post-production: stabilization and editing performed as inference-time control rather than dedicated pipelines, plus bullet-time effects from ordinary footage.
• Embodied-agent simulation: re-observing a recorded dynamic scene from an agent's viewpoint, which is the core operation a learned simulator must support.

Industry relevance. Video generation labs, VFX and virtual production studios, game engine developers, and robotics simulation teams all have an interest in extracting new viewpoints from existing footage without retraining large models. Training-free control is attractive commercially because it avoids the compute and data cost of fine-tuning, and because it lets a single deployed model serve many control tasks.

Future Directions

• Generality beyond one backbone. WiW is demonstrated only on LingBot-World 2.0's causal-fast checkpoint. Whether the evidence interface transfers to other causal video models—and whether attention-layer semantics are stable enough across architectures—remains open.

• Reducing dependence on external geometry priors. The pipeline relies on off-the-shelf components (DepthCrafter for depth, LHM++ and SMPL-X for human geometry) whose errors propagate into generation, and whose human-specific design limits which subjects can be handled. Extending rendered-geometry evidence to arbitrary articulated or non-human subjects is a natural next step.

• Scaling long-horizon memory and retrieval. The history bank grows throughout a rollout and is controlled by a bounded top-K retrieval that ranks by coverage and viewing-direction compatibility. More principled compression, indexing, or learned retrieval policies for very long explorations are an obvious extension.

• Evaluation and robustness. Camera errors are estimated from generated video rather than measured directly, and there is no systematic study of what happens when evidence channels contradict each other or when one channel is systematically wrong. Characterising and arbitrating evidence conflicts is an important open problem.

Target Audience

Researchers and graduate students working on video generation, world models, 4D scene reconstruction, and controllable generative models; engineers building interactive video tooling, virtual production pipelines, or embodied-agent simulators; and anyone interested in the broader question of how to extend a frozen pretrained generative model's capabilities at inference time without retraining.

Authors’ abstract

Autoregressive video world models enable interactive, long-horizon exploration, but flexible control remains challenging. Exploring a source video from new viewpoints requires the generated rollout to remain synchronised with the recorded event, place observed content in the requested view, plausibly complete newly exposed regions, and recover previously generated appearance on revisits. Existing methods typically address these requirements through task-specific modules or additional training. We present World in World, a training-free inference-time interface that converts heterogeneous control evidence into camera- and time-labelled clean visual states, which are read through the native self attention of a frozen causal video model. The evidence comprises source-video observations, target-view scene projections, geometry renderings that guide completion of newly exposed subject regions, and retrieved generated states beyond the rolling cache. Each evidence source carries token-level support and its own availability schedule. A correspondence router combines persistent point identities with geometry to establish token correspondences, guiding supported queries towards matching source-video tokens. Evidence-wise attention CFG (EWA) then independently regulates each auxiliary channel's additional contribution using attention responses from the same denoising forward pass. The shared interface supports camera-controlled rerendering, long-horizon revisiting, and human-motion transfer with the same frozen backbone. We evaluate World in World on camera-controlled video rerendering under diverse viewpoint changes, assessing perceptual quality, temporal consistency, and camera-following accuracy.