Olaf-World: Orienting Latent Actions for Video World Modeling

AI summary

Overview

Research area: Computer Vision / Machine Learning — specifically, action-controllable video world models and unsupervised latent action learning.

Technical level: Advanced. The paper assumes familiarity with variational autoencoders (β-VAE), diffusion transformers, flow-matching objectives, and self-supervised video encoders like V-JEPA2.

Scope: This paper proposes a sequence-level alignment objective (SeqΔ-REPA) that makes latent actions learned from unlabeled video transferable across visual contexts, and uses it to pretrain an action-controllable video world model (Olaf-World).

What This Paper Is About

Video world models can predict future frames given an action, but training them typically requires frame-aligned action labels, which are expensive and domain-specific. Latent action learning tries to solve this by inferring actions directly from unlabeled video, but the resulting action spaces are inconsistent: the same semantic action (e.g., "move forward") maps to different latent directions in different scenes, breaking transfer. This paper fixes that inconsistency by anchoring latent actions to the observable effects of control (how video features change over time), measured from a frozen self-supervised video encoder.

Key Contributions

1. Characterizes cross-context non-identifiability in latent action learning. The authors formalize (with a proposition in Appendix A) why step-wise reconstruction objectives cannot identify a global action coordinate system — any context-dependent reparameterization of the latent codes yields identical prediction losses, so the same semantic action can map to arbitrary directions per clip.

2. Proposes SeqΔ-REPA, a sequence-level control-to-effect alignment objective. Instead of aligning static features (as in prior representation-alignment work), it aligns the integrated latent action over a clip to the net temporal feature difference computed from a frozen video encoder (e.g., V-JEPA2).

3. Introduces Olaf-World, a two-stage pretraining pipeline: (1) train a latent action model with SeqΔ-REPA on passive video, then (2) freeze it and use its latents as a unified control interface to pretrain an image-to-video diffusion transformer world model.

4. Demonstrates cross-context transfer and data-efficient adaptation. Latents are more linearly decodable, action prototypes match across domains, zero-shot action transfer is more faithful, and adaptation to new control interfaces (e.g., Unreal Engine action labels) works with as little as 1 minute of labeled video.

Main Findings

• Stronger latent structure: In cross-domain linear probing (Macro-F1), Olaf-World reaches 0.8138 (1st→1st) and 0.6250 (1st→3rd), versus AdaWorld's 0.6004 and 0.4820. The gains are largest on the harder third-person view subset (0.8256 vs 0.4827), showing the latents are both more separable and more context-invariant.

• More consistent action semantics: Cross-context prototype similarity matrices become diagonal-dominant, meaning each action in one domain is most similar to its counterpart in another domain. AdaWorld's similarity matrix is uniformly high, indicating actions are not uniquely identified under context shift. Residual confusion remains for yaw "look" actions, which the authors attribute to genuine physical differences between egocentric and third-person camera rigs.

• Better zero-shot action transfer: When a latent action sequence extracted from a reference clip is applied to a new target frame, AdaWorld exhibits temporal wash-out, agent drop-out, and motion drift, while Olaf-World preserves target appearance and follows the intended motion.

• Best controllability after adaptation: Across adaptation budgets of 0, 1, and 50 labeled videos, Olaf-World achieves the lowest translational and rotational relative pose error (RPE) on both first- and third-person domains, with comparable VBench visual quality. At the 50-video budget in 3rd-P, RPE-Rot drops to 0.2082 versus AdaWorld's 0.3353.

• Better out-of-distribution generalization: After adapting to one action space and testing on 50 diverse unseen initial frames, Olaf-World retains the lowest RPE, indicating the latent control does not overfit to adaptation visuals.

• Ablations confirm both design choices matter: Removing Δ (aligning static features instead of temporal effects) drops cross-domain Macro-F1 substantially (e.g., 1st→3rd falls from 0.6250 to 0.5287). Removing ℓ₂ normalization also hurts. Video encoders (V-JEPA2, VideoMAEv2) dramatically outperform an image-only encoder (DINOv3) and the no-reference baseline, showing temporal modeling in the reference encoder is essential.

• Adaptation scales gracefully: More labeled videos and higher LoRA ranks steadily improve action accuracy while visual quality stays roughly constant, and full fine-tuning serves as an upper-capacity reference.

Methodology in Plain English

The authors start from a standard recipe: an inverse-dynamics encoder looks at two consecutive frames and guesses a latent "action," and a forward decoder tries to reconstruct the next frame from the current frame plus that latent action. This trains fine per clip but gives no incentive for the latent space to be shared across clips — so a "move forward" latent in one scene has no relation to a "move forward" latent in another.

Their fix uses a simple observation: while actions are never labeled, their effects are visible. If you run a frozen self-supervised video encoder (like V-JEPA2) over a short clip and average the frame-to-frame feature differences, you get a compact "effect direction" that captures what changed — motion, perspective shift, object movement — without being tied to static appearance. Because it is a temporal difference averaged over time, it cancels out scene-specific details.

They then take the average of the latent actions inferred over the same clip, project it through a small MLP into the encoder's feature space, and use cosine similarity to pull that projection toward the encoder's effect direction. This becomes an extra loss term added to the standard β-VAE objective. The shared reference gives all clips a common target for what an action means, effectively pinning down the previously unconstrained latent basis.

For the world model stage, they freeze the trained latent action model and run it over large unlabeled video to produce action sequences. Those sequences condition a pretrained image-to-video diffusion transformer via AdaLN-Zero modulation, with actions grouped to match the backbone's temporal compression factor of 4. To adapt to a real environment with labeled actions, they learn a small embedding table mapping discrete environment actions to latents, initialize it with class prototypes computed by running the frozen LAM on labeled segments, and finetune the adapter plus a rank-16 LoRA on the backbone.

Why This Matters

Impact on research: The paper reframes latent action learning as a representation alignment problem rather than a purely generative one, and provides a formal argument for why locally-consistent objectives cannot produce globally-consistent action spaces. This gives the field a concrete diagnostic (cross-domain linear probing) and a general recipe (anchor to observable effects) that could apply to any latent-variable dynamics model, not just world models.

Real-world applications:

• Interactive game and simulation engines: Enabling control interfaces to be learned from gameplay footage without manually logging keyboard/mouse inputs.
• Robotics and cross-embodiment transfer: Effect-aligned latents could act as embodiment-agnostic "skills," with a small per-robot adapter mapping physical joint commands to latent skills (the authors flag this as future work).
• Autonomous driving simulation: Pretraining controllable driving world models from passively collected dashcam or fleet video, then adapting to a specific vehicle's control schema with minimal labeled data.
• Content creation and VFX: Providing a transferable motion interface so a director can lift a camera or character motion from one clip and replay it in a completely different visual scene.

Industry relevance: The data-efficiency result is the most commercially relevant claim — adapting a pretrained world model with roughly 1 minute (1 video) or 2 hours (50 videos) of labeled interaction is far cheaper than collecting frame-aligned labels at scale, which is the current bottleneck for controllable video generation products and embodied AI pipelines.

Future Directions

• Cross-embodiment skill transfer in robotics: The authors explicitly propose using effect-aligned latents as transferable skills bridging human and robot action spaces, with embodiment-specific adapters. Testing this on real hardware is an open question.

• Relaxing the yaw-action limitation: Rotational camera actions ("look left/right") remain the hardest case for cross-context alignment because the same control induces physically different motion under egocentric versus third-person rigs. Whether a better reference encoder or a pose-aware effect signal closes this gap is unresolved.

• Scaling the backbone and adaptation capacity: The authors show controllability improves with LoRA rank and full fine-tuning, but they use rank 16 in main experiments for efficiency. The upper bound of controllability with larger adaptation budgets is not established.

• Reference encoder choice and pretraining objective: V-JEPA2 and VideoMAEv2 both work but V-JEPA2 shows better in-domain probing while VideoMAEv2 shows better cross-domain scores. A principled understanding of which temporal encoder properties matter most for effect-direction targets remains open.

Target Audience

This paper is most valuable for researchers and practitioners working on video generation, world models, and embodied AI, particularly those interested in controllable video synthesis or robot learning from unlabeled video. It will also interest representation learning researchers working on identifiability and disentanglement, since the core argument is a latent-coordinate symmetry result. Engineers building interactive simulation or content-creation tools will find the adaptation protocol (prototype-initialized adapter plus LoRA) directly actionable. Readers without background in variational inference, diffusion transformers, or self-supervised video representation learning will find the technical sections challenging, though the Introduction and Figure 2 convey the core intuition accessibly.

Authors’ abstract

Scaling action-controllable world models is limited by the scarcity of action labels. While latent action learning promises to extract control interfaces from unlabeled video, learned latents often fail to transfer across contexts: they entangle scene-specific cues and lack a shared coordinate system. This occurs because standard objectives operate only within each clip, providing no mechanism to align action semantics across contexts. Our key insight is that although actions are unobserved, their semantic effects are observable and can serve as a shared reference. We introduce Seq$Δ$-REPA, a sequence-level control-effect alignment objective that anchors integrated latent action to temporal feature differences from a frozen, self-supervised video encoder. Building on this, we present Olaf-World, a pipeline that pretrains action-conditioned video world models from large-scale passive video. Extensive experiments demonstrate that our method learns a more structured latent action space, leading to stronger zero-shot action transfer and more data-efficient adaptation to new control interfaces than state-of-the-art baselines.