Skip to content
AI.info

Research

One Prior, Two Dynamics: Repurposing Video Diffusion Transformers for Self-Supervised Latent Action Learning

O2LaF shows a pretrained video diffusion transformer can supply both inverse and forward dynamics for latent action learning, with code, making robot learning from unlabeled video more practical.

arXiv
2611.26617
Published
2026-11-19
Authors
Yuxin Jiang, Wenqiang Sun, Jiaqi Li, Yuchao Gu, Lvmin Zhang, Ivor Tsang, Mike Zheng Shou

AI summary

Overview

  • Research area: Computer vision and self-supervised learning for embodied AI — specifically latent action learning built on top of pretrained video Diffusion Transformers (video DiTs).
  • Technical level: Intermediate. The paper assumes familiarity with diffusion/flow-matching, LoRA adaptation, and the standard inverse-dynamics/forward-dynamics formulation of latent action models, but its core idea is stated in one sentence: use the same pretrained video model for both halves of the pipeline.
  • Scope: O²LaF ("One Prior, Two Dynamics — a self-supervised Latent Action learning Framework") repurposes a single pretrained video DiT into an inverse dynamics model and a forward dynamics model, trains them end-to-end through a compact latent-action bottleneck, and evaluates the resulting actions on camera navigation and robotic manipulation probing, plus zero-shot cross-context and cross-embodiment transfer.

What This Paper Is About

Videos show how the world changes but not which actions caused the change, so latent action learning tries to recover a compact action code from pairs of frames. Prior work typically learns these dynamics largely from scratch, or plugs a pretrained video model into only one half of the pipeline (either the forward model or, less often, the inverse model), leaving a gap on the other side. The authors ask whether one pretrained video DiT can supply the prior for both inverse and forward dynamics, and whether the resulting latent actions are transferable across visual contexts and embodiments.

Key Contributions

  1. O²LaF framework ("one prior, two dynamics"). Both the inverse dynamics model (IDM) and forward dynamics model (FDM) are initialized from the same pretrained video DiT and adapted independently with LoRA, then optimized end-to-end through a 32-D Gaussian latent-action bottleneck.
  2. A feed-forward IDM design. The video DiT is converted from an iterative generator into a single-pass transition encoder using learnable action queries, an asymmetric attention mask (queries read only the two endpoint states and cannot talk to each other or to video tokens), and a variational action head.
  3. A causal, action-conditioned FDM design. The generator keeps its generative role but uses frame-causal attention with joint prediction of all future states, plus high-noise timestep sampling — both intended to remove shortcuts so the model must actually use the inferred latent action.
  4. A systematic study of how the two dynamics should be factorized, covering adaptation method (frozen head vs. LoRA), FDM scale, temporal factorization (causal joint vs. bidirectional joint vs. autoregressive), role-specific noise levels across a 5×5 sweep, and alternative IDM backbones including V-JEPA2, VideoChat3, and an ST-Transformer trained from scratch.

Main Findings

  • State-of-the-art few-shot action probing. Freezing the learned encoder and fitting linear probes, O²LaF outperforms prior latent-action representations on MIND (discrete navigation: four simultaneous ternary controls, joint exact-match accuracy) and on DreamDojo's In-lab Eval (continuous manipulation, EEF R²).
  • Human-to-robot representation transfer without robot adaptation. O²LaF reaches an EEF R² of 0.277 on GR-1 versus 0.149 for DreamDojo, despite training its encoder exclusively on egocentric human demonstrations from EgoDex.
  • Better zero-shot camera-navigation transfer. Across 24 reference trajectories (12 first-person, 12 third-person) transferred to 50 and 20 novel contexts (840 cross-context generations total), O²LaF has the lowest ViPE camera-motion errors and the highest VBench imaging quality in both perspectives — e.g., first-person translation NRMSE 1.289 and rotation RPE 0.786, versus 2.539 / 2.055 for AdaWorld and 1.627 / 1.300 for Olaf-World; third-person 0.511 / 0.330 versus 1.342 / 1.217 and 0.985 / 0.701.
  • Latent-action classifier-free guidance helps. With action CFG = 3, results improve over the no-CFG variant (first-person imaging quality 0.698 vs 0.672; third-person rotation RPE 0.330 vs 0.364), which the authors note contrasts with prior findings that CFG is less effective for predefined action controls.
  • Cross-embodiment cycle consistency beats DreamDojo. Over 85 video pairs spanning 20 manipulation tasks, O²LaF wins on all self-reconstruction and cross-cycle metrics (H→R→H and R→H→R) — for example human self-reconstruction PSNR 26.72 / SSIM 0.903 / LPIPS 0.148 versus 23.38 / 0.871 / 0.223.
  • Adapting the backbone matters more than adding a head. A frozen video DiT with a trainable action head scores 43.88 (first-person, k=1) and 22.15 (third-person, k=1), while LoRA-128 reaches 72.81 and 52.08; rank-16 LoRA is close to rank-128.
  • A stronger FDM improves the learned actions. With a fixed 1.3B IDM, scaling the FDM from 1.3B to 14B raises first-person accuracy from 67.14 to 72.72 at k=1 and from 80.76 to 85.98 at k=50.
  • Causal joint prediction beats alternatives. Autoregressive prediction (h=1) collapses to 55.45 / 70.13 (first-person, k=1 / k=50), and bidirectional joint prediction trails causal joint at k=50 (83.71 vs 87.95 first-person; 92.36 vs 94.37 third-person).
  • The two dynamics want different noise. In a 5×5 sweep over (t_IDM, t_FDM), moderate IDM noise often beats clean input, low FDM noise gives poor representations (target shortcuts), and maximal FDM noise is not always optimal. Both navigation perspectives favor (0.3, 0.98), while manipulation shows different optima across arms.
  • Probing accuracy does not predict transfer. V-JEPA2 as the IDM backbone gives the best probing numbers (88.62 / 96.14 on MIND) but leaks source appearance in cross-context generation, confirmed by color probing (color R² of 0.514 / 0.816 / 0.742 for L/a/b), whereas Wan2.2 stays near or below zero (−0.297 / 0.081 / 0.223) and better preserves the target scene.
  • Action learning and generation are in tension. High-noise-only FDM training helps third-person probing (60.75 / 94.86 at k=1 / k=50) but degrades generation. A two-stage deployment that freezes the IDM and trains a bidirectional FDM recovers generation quality, and initializing that FDM from the jointly trained model improves further (first-person ViPE 1.260 / 0.725 versus 1.289 / 0.786 from the pretrained initialization).
  • Qualitative motivation. Feeding clean latents to a frozen video DiT with the text prompt "What is the action in the video?" produces cross-attention that follows active hands and manipulated regions as the interaction changes.

Methodology in Plain English

The starting point is a pretrained text-to-video DiT whose VAE turns a video into K+1 spatiotemporal latent states, trained with rectified flow (interpolating clean latents with Gaussian noise, t=0 clean, t=1 pure noise, target velocity ε − x).

Inverse side. Instead of running iterative denoising, the IDM does a single forward pass on a partially noised video. Learnable "action queries" are inserted, one per transition, each tagged with a temporal position encoding. An asymmetric attention mask enforces three rules: video tokens attend causally across frames, each query reads only frames x_i and x_i+1, and queries neither read each other nor get read by video tokens. A linear Gaussian head maps each query's final representation to a mean and variance for a 32-D latent action, sampled with the reparameterization trick during training and taken as the posterior mean at inference. The main configuration uses a clean-input IDM (π_IDM = δ₀).

Forward side. The FDM receives the clean first frame plus the temporally aligned action sequence ([0, z₀, …, z_{K−1}]), injected into every Transformer block through AdaLN. All future states are noised at a shared timestep and all their velocities are predicted jointly in one pass. Bidirectional temporal attention is replaced with frame-causal attention so tokens at x_i cannot see x_{>i}; this blocks the "just copy the previous clean frame" shortcut that autoregressive prediction would allow. Timestep sampling deliberately emphasizes high noise (π_FDM = 0.7 π_base + 0.3 π_high, with π_high supported on [0.9, 1)), because at t=1 the flow-matching loss is exactly reconstruction of the future latent from the clean first frame and the action — forcing the FDM to depend on z.

Training. The two models are trained end-to-end: a flow-matching velocity loss on the FDM plus a KL regularizer toward N(0, I) on the action posterior, with β = 1e−4. Gradients flow through the sampled latent actions back into the IDM. Backbones are Wan2.2-TI2V-5B with rank-128 LoRA; the VAE and text encoder stay frozen; new action-readout and action-conditioning modules are trained from scratch.

Evaluation. Navigation models train on MiraData and are probed on MIND with class-balanced logistic regression across 1–50 labeled videos. Manipulation models train only on EgoDex human videos and are probed on GR-1 robot demonstrations with Ridge regression across 1–32 episodes per task. For transfer, actions are extracted from a source video and used to condition generation from a different target first frame with no adaptation. For a fair world-model comparison, a separate bidirectional FDM is trained for each frozen encoder under identical protocols.

Why This Matters

The paper reframes where pretrained video knowledge should enter latent action learning: rather than bolting a video model onto one side of the pipeline, it shows the shared temporal-reasoning demand of inverse and forward dynamics can be served by the same prior. It also issues a methodological warning that is easy to overlook — a latent action that probes well may still encode appearance that sabotages downstream generation, so probing alone is an incomplete measure of transferability.

Real-world applications:

  • Robot learning from action-free human video, where human demonstrations supply the control signal for robot manipulation without robot-domain action labels.
  • Controllable video world models, where inferred latent actions steer generation from a new first frame in a new scene.
  • Virtual cinematography and camera control, transferring a reference trajectory's camera motion to new footage (first- and third-person).
  • Cross-embodiment policy interfaces, using latent actions as a shared control space between human and robot data.

Industry relevance spans robotics (cheap pretraining data from internet video), content creation and VFX (motion and camera transfer), and simulation/world-model tooling, since the same checkpoint serves both as an action encoder and as an action-conditioned generator.

Future Directions

  • Reconciling action learning with generation quality inside a single model. End-to-end training with causal, high-noise supervision yields strong actions but weak generation; the authors fall back on a two-stage deployment. Closing this gap without a second stage is left open.
  • Better evidence beyond linear probing. The V-JEPA2 result shows probing can reward appearance-entangled representations. The paper argues downstream generative evaluation is essential, but a principled combined metric is not established.
  • Domain- and embodiment-aware noise schedules. The 5×5 sweep shows navigation prefers (0.3, 0.98) while manipulation has different optima per arm, and the main experiments use one domain-agnostic configuration — adaptive or learned schedules are an obvious next step.
  • Scaling the inverse side. The FDM scale study holds the IDM

Authors’ abstract

Videos reveal how the world changes, but rarely the actions that cause those changes. Latent action learning recovers such supervision from visual transitions, yet often learns the underlying dynamics largely from scratch. We ask whether a pretrained video generator can instead provide the prior for both inverse and forward dynamics. We introduce O$^{2}$LaF, built on a simple principle: one prior, two dynamics. We initialize both models from the same pretrained video DiT and…

The full abstract on arXiv
Read the original paper