World-in-Context: Interactive Video World Modeling with In-Context Conditioning

AI summary

Overview

Research area: Computer Vision — generative video world models, specifically interactive video generation conditioned on user control signals (view mode, character action, camera motion).

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching video backbones, latent-space conditioning, self-attention, rotary positional encodings, and Plücker ray representations.

Scope (1 sentence): The paper proposes World-in-Context, a framework that turns view mode, discrete actions, and camera trajectories into tokens inside a shared self-attention space (In-Context Conditioning, or ICC), plus a control-aware temporal positional encoding (IT-RoPE), and evaluates it on the WBench benchmark with a 5B-parameter model.

What This Paper Is About

Interactive video world models let a user steer a generated video by supplying controls, and the model predicts how the visual world responds. Current pipelines handle these controls poorly: the viewing mode (free camera, first-person, third-person) is usually buried in a text caption, and character action is often conflated with camera motion, with each control injected through a separate architectural branch so their interdependencies are only captured indirectly. The goal is a single unified interface where view, action, and camera signals are all conditioned in the same attention space, so the model captures how they interact and can switch viewpoints mid-video.

Key Contributions

1. In-Context Conditioning (ICC) for interactive world modeling. View mode, action, and camera signals are projected into condition tokens and prepended to the visual latents in the same self-attention sequence, replacing separate AdaLN modulation, cross-attention, or additive-injection pathways.

2. Explicit view-mode modeling with emergent switching. Discrete view tokens for Free, Fpp, and Tpp are added to the context instead of being described in text. The authors report that this disentangles regime-specific motion and that in-video view switching emerges from training on single-view clips alone, becoming reliable with a small set of switching clips.

3. Joint action-camera decoupling and analysis. Discrete actions and continuous camera trajectories are modeled as coexisting but separate conditioning signals, and controlled ablations show they play complementary roles: action drives local command fidelity, camera anchors global trajectory stability.

4. Interaction-Transport RoPE (IT-RoPE). A temporal positional encoding that parameterizes attention with control-induced transport operators accumulated along the execution sequence, so the model sees the intervening control progression between distant frames rather than only endpoints.

Main Findings

• Navigation parity with a much larger model: World-in-Context-5B reaches a WBench Navigation average of 81.5, essentially matching 22B EchoWM at 81.6, and slightly above the 5B Zing-0.5 at 81.0.
• Large jump in perspective switching: It attains a 93.8% success rate on the WBench Fpp↔Tpp set, versus 18.8% for the best 5B baseline. Alaya-EVOKE (14B) and DreamX-World (5B) both score 18.8; EchoWM scores 32.0.
• Switching emerges without switching supervision: The base model, trained only on single-view clips, reaches 37.5% on Fpp↔Tpp versus 6.25% for a text-based view-conditioning baseline. Fine-tuning on 500 switching clips raises this to 93.8%.
• Interaction score improves: Unified in-context conditioning lifts the Interaction score from 84.2 (Zing-0.5) to 86.0, with Table 1 also reporting Quality 80.9, Setting 81.0, Consistency 87.2, and Physical 72.3.
• ICC's advantage grows with richer conditioning: With camera-only control, ICC ties other strategies (76.3 vs 76.3–77.0). Adding joint action-camera inputs yields +0.6 Navigation and +1.1 Interaction; adding view tokens yields +1.0 and +1.4, reaching 81.2 and 86.8.
• IT-RoPE beats standard 3D-RoPE and endpoint-only encoding: Full IT-RoPE improves Navigation from 80.5 to 81.2 and Interaction from 86.2 to 86.8; the endpoint-only variant reaches 80.5 / 86.6, showing that accumulating intervening control matters.
• Action and camera are complementary: Action-only conditioning gives +6.5 translation and +3.9 rotation accuracy over camera-only but introduces visual artifacts; camera-only gives +2.1 geometric and +1.9 photometric consistency but insufficient displacement. Joint training achieves the best rotation accuracy (90.6) and geometric consistency (90.8) while keeping translation fidelity at 89.3.
• Favorable data efficiency: With only 50K+250 clips, ICC reaches a 75.0% switching rate, already above additive injection trained on twice the data (71.9% at 100K+500). At 100K+500, ICC beats the largest additive baseline (125K+750) on both Navigation (81.5 vs 80.3) and Switch (93.8% vs 78.1%).
• View tokens change motion statistics: Across 100 scenes, text-based conditioning produces overlapping oscillation profiles for Fpp and Free, while explicit view tokens separate pronounced periodic Fpp oscillations from smoother Free trajectories.
• Causal streaming is nearly as good: WiC-Causal, trained autoregressively with historical latents in the context, scores 81.2 Navigation average and 90.6% switching on WBench — small degradation from the bidirectional 81.5 / 93.8%.
• Extends to richer game controls: WiC-Game, trained on a compact subset of a single Counter-Strike 2 map (Dust II) from EWM data, executes added fire/duck/walk tokens in-domain, on unseen maps (Mirage, Nuke), and on synthetic-start-image real-world-style scenes.

Methodology in Plain English

The starting point is an existing 5B video generation backbone (Wan2.2-TI2V-5B) trained with a flow-matching objective in latent space. Instead of building a separate control channel for every new signal, the authors convert each control into a sequence of tokens with the same hidden dimension as the video latents and simply prepend those tokens to the latent sequence. Because everything lives in one self-attention sequence, the model can directly learn how a view mode, an action, and a camera pose relate to each other at any frame.

Each control type gets its own encoding. View mode is a learned categorical embedding per frame, restricted to Free, Fpp, or Tpp. Actions are split into sub-slots (move, turn, translation delta, rotation delta), each encoded by a small slot-specific network plus a zero-initialized residual branch for continuous magnitudes, with time, position, and slot embeddings added. Camera poses become tokenized Plücker ray fields plus a global relative-pose embedding and a local rigid-body increment embedding. Text is reserved strictly for describing world state, not for specifying view mode.

Because token-level conditioning says what the controls are at each frame but not how control evolved between distant frames, the authors add IT-RoPE. Each control input induces a small transformation matrix, and multiplying these along the sequence gives an accumulated "transport" operator between any two frames — one path for action, one for camera. These operators add a correction term to the temporal attention score, gated per view regime and token type, masked to contiguous valid segments, and centered by subtracting the identity so the correction vanishes for static intervals. Learnable scales are initialized at zero so the backbone's behavior is preserved at the start of training.

Training uses 100K single-regime navigation clips assembled from public video datasets (DL3DV, RealEstate, Sekai, SpatialViD, EgoExo, ABot-World-Explorer), filtered for shot continuity (TransNetV2), motion (UniMatch optical flow and VMAF), and aesthetics (Dover), plus 500 Fpp↔Tpp switching sequences from EgoExo and internal data. View labels come from Qwen3-VL-32B classification cross-checked against reconstructed trajectory statistics; camera trajectories from ViPE; discrete actions are largely inverted from those camera trajectories, with Fpp head-bobbing oscillations filtered out first. Evaluation is on WBench's Navigation split and its Perspective Switching Fpp↔Tpp suite, with controlled re-implementations of competing conditioning strategies trained on identical data.

Why This Matters

The paper argues that the main bottleneck in interactive world models is not raw generation quality but how controls are wired into the network. Reframing multi-condition interaction as context modeling gives one interface that scales with the number of control signals, rather than a growing pile of side branches — a claim the authors support with widening margins as conditioning gets richer (camera → action+camera → +view).

Real-world applications:

• Game engines and playable generative prototypes, where keyboard/gamepad actions, camera, and view mode must be independently controllable (demonstrated with Counter-Strike 2 fire/duck/walk tokens).
• Film and animation previsualization, where directors steer camera paths and character motion separately and cut between first- and third-person shots inside one generated take.
• Robotics and embodied simulation, where first-person embodiment versus free-camera inspection of a reconstructed scene are distinct operating modes.
• Interactive virtual tours and real-estate or e-commerce scene exploration, where smooth free roaming and embodied walkthroughs share a single model.

Industry relevance: The headline result is efficiency — a 5B model matching a 22B one on navigation and vastly exceeding every 5B baseline on view switching — which matters for serving interactive video at scale. The data-efficiency results (matching a baseline trained on 2.5× the switching data) and the near-lossless causal variant further suggest the approach is deployable for streaming, low-latency interaction rather than offline batch generation.

Future Directions

• Scaling data and model size. The authors explicitly position the work as an algorithmic study using a modest 100K navigation clips and 500 switching sequences, and note that overall capacity depends on data quality, model size, and training scale — leaving open how far the ICC advantage compounds at larger scale.
• Beyond the three core control modalities. Only view, action, and camera are studied in depth; the game-control extension (fire, duck, walk) hints at arbitrary new token types, but how hundreds of compositional controls behave in the shared attention space is untested.
• Long-horizon streaming and drift. Causal rollout is demonstrated with a sink token plus a sliding window of recent frames and shown to reproduce previously seen landmarks on loop closure; how far this holds over much longer horizons is not established.
• Action annotation without inverse estimation. Character actions are largely recovered from camera trajectories via ViPE rather than real game telemetry, which the paper acknowledges as a substitute for synchronized engine data; the effect of that approximation on fidelity is an open question.
• Evaluation breadth. Results center on WBench plus qualitative rollouts; comparisons against methods with undisclosed training-data scales are handled via controlled re-implementations rather than disclosed-budget comparisons.

Target Audience

Researchers and engineers working on generative video, world models, embodied AI, and interactive content creation — particularly those building controllable video diffusion/flow-matching systems or working on conditioning architectures. It is most useful to readers who already understand transformer attention, latent video diffusion, and camera parameterization; beginners can extract the conceptual message (unified in-context tokens beat separate control branches) but will need background reading for the IT-RoPE and transport-operator sections.

Authors’ abstract

Interactive video world models require coordinated control over viewing perspectives, character actions, and camera motion. Existing methods typically leave view modes implicit in text captions, conflate action with camera motion, and inject controls through fragmented architectural pathways that capture interdependencies only indirectly. In this work, we present World-in-Context, an interactive world modeling framework powered by In-Context Conditioning (ICC). Rather than relying on specialized conditioning pathways, World-in-Context unifies view, action, and camera controls into explicit tokens contextualized directly within a shared self-attention space. Our formulation reveals that (i) explicit view tokens disentangle conflicting dynamics and unlock emergent view switching from single-view data, while (ii) action and camera inputs play complementary roles in governing local responsiveness and global stability. To preserve temporal continuity across execution intervals, we further introduce Interaction-Transport RoPE (IT-RoPE), parameterizing temporal attention with accumulated control transitions. On WBench, World-in-Context-5B achieves navigation performance on par with 22B EchoWM ($81.5$ vs. $81.6$) and attains a $93.8\%$ perspective-switching success rate, substantially outperforming existing 5B baselines ($18.8\%$). Controlled analyses further demonstrate compounding advantages of World-in-Context under richer conditioning mixtures, favorable data efficiency, and natural applicability to causal rollouts.