DreamX: Video Diffusion Model is a Universal Latent Action Learner

AI summary

arXiv: 2611.05377v1 [cs.CV] · 3 Nov 2026 · KAIST & NVIDIA · Project page: cvlab-kaist.github.io/DreamX

Overview

Research area: Computer vision and robot learning — specifically latent action models (LAMs), video diffusion models, cross-embodiment transfer, and vision-language-action policy learning.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching video models, transformer attention masking, LoRA adaptation, and variational bottlenecks.

Scope (1 sentence): The paper shows that a pretrained video diffusion transformer (Cosmos-Predict2.5 2B) can be repurposed to read out compact, embodiment-agnostic latent actions from its own spatiotemporal features, and that these actions transfer across humans and robots for video generation, action execution, and policy learning.

What This Paper Is About

Robot policies need large amounts of action-labeled demonstration data, which is expensive to collect, while human egocentric videos are abundant but carry no robot action labels and involve a different body and control space. Existing latent action models try to infer actions from visual transitions, but their representations tend to entangle the actual motion with incidental visual factors like viewpoint, scene appearance, and embodiment, so they transfer poorly. DreamX asks whether a pretrained video diffusion model — which already encodes how scenes evolve and already knows how to synthesize plausible futures — can serve directly as a universal latent action learner, producing a shared action interface usable by both humans and robots.

Key Contributions

1. A latent action learner built from a video diffusion backbone. DreamX appends learnable query tokens to the video token sequence and lets them read the backbone's spatiotemporal features through its existing attention, compressing each latent video frame from 1,200 frame tokens down to 4 latent-action tokens (a 300× reduction).

2. Generative supervision that separates motion from appearance. The same pretrained backbone acts as a decoder that reconstructs future frames conditioned only on the initial frame and the latent actions (text conditioning is removed), so the scene appearance is supplied by the initial frame and the latent actions are pushed to encode motion rather than visual detail.

3. Demonstrated cross-embodiment transfer without paired data. The frozen encoder/decoder pair supports any-to-any video transfer and human-to-robot action transfer — decoding a human video's latent action into executable robot commands — with no paired source–target training data.

4. Compact latent actions as policy targets. A variational bottleneck compresses each 2,048-dimensional latent-action token into a 32-dimensional code, which an observation-conditioned policy predicts instead of generating future video, cutting inference cost substantially.

Main Findings

• State-of-the-art on LARYBench. DreamX achieves the best result among latent action models on every benchmark split: average action-regression MSE of 0.17 versus 0.38 for the strongest baseline (DiLA), and average action-classification accuracy of 69.67% versus 62.25% for DiLA. Per-dataset regression MSE is 0.15 (CALVIN), 0.05 (VLABench), 0.26 (RoboCOIN), and 0.22 (AgiBotWorld-Beta); per-split classification accuracy is 68.03 (Atomic robot), 72.72 (Composite human), and 68.26 (Composite robot).

• Even the compressed codes beat all baselines. The 32-channel variant, DreamX + FAE-32, reaches 0.22 regression MSE and 67.15% classification accuracy, still above every prior method.

• Zero-shot human-to-robot video transfer on H&R. On the HumanAndRobot benchmark, with no paired human–robot training data, DreamX records FVD 57.44, LPIPS 0.179, PSNR 26.52, and SSIM 0.917, against AdaWorld (170.19 / 0.344 / 22.95 / 0.846) and VideoWorld2 (241.92 / 0.299 / 22.53 / 0.881). Baselines show visual degradation or leak a human hand into the output.

• Human motion decodes into robot motion. Executed open-loop with no visual feedback, latent actions from human videos reproduce the stroke structure of the letter "N" in a writing task, the forward-then-left direction change and ordering in single-arm pushing, and coordinated arm-specific motion in bimanual pushing. These are qualitative demonstrations; no success-rate metric for action transfer is reported in the provided content.

• Real-world drawer-closing policy. DreamX reaches an average subgoal success rate of 96.7% (90.0 upper, 100.0 lower, 100.0 both) versus 50.0% for Mimic-Video, at 1,474 ms inference latency versus 28,226 ms — a 19.1× speedup. Pushing-completion rates are 80%, 100%, and 100% across the three configurations.

• Action labels are not essential. Adding auxiliary action supervision changes LARYBench regression MSE only from 0.24 to 0.23 and classification accuracy from 65.3% to 64.8% (a slight decrease), suggesting the motion and action semantics come largely from video alone.

• Latent actions control motion, not appearance. Perturbing latent actions disrupts object manipulation and zeroing them largely suppresses motion, while the surrounding scene stays largely intact; query-to-video attention concentrates on hands and grippers during human and robot demonstrations.

Methodology in Plain English

Start with a pretrained video diffusion transformer, Cosmos-Predict2.5 2B, whose weights stay frozen. It normally turns a short clip into a sequence of latent frames, each patchified into many spatial tokens.

To read out an action, the researchers attach a small group of four learnable "query" tokens to each latent frame (up to 32 queries for a clip) and run everything through the model's existing spatiotemporal attention. Because query-to-video attention is frame-causal, a query only sees frames up to its own position, so it summarizes what changed up to that point. The contextualized queries are the latent action.

To make those queries mean motion rather than appearance, the same backbone is reused as a decoder. It must regenerate the future frames using only the first frame (which carries scene appearance and, via the pretrained generative prior, visual detail) plus the latent actions, with text conditioning removed. The only way to get the future right is to put the motion into the queries. When robot action labels happen to exist, the decoder also predicts action chunks from the robot's initial proprioceptive state as an auxiliary task; the video and action streams read the queries but not each other, under a masking scheme that keeps them separate.

Training uses standard flow matching on both video and action terms, with the action weight set to zero for clips that lack labels. The encoder and decoder are adapted with separate rank-16 LoRA modules for 78,000 steps on 480×640 video, using 2.19k hours across six datasets — 469 hours of human video from EgoDex and EgoVerse, plus robot data from Bridge, DROID, AgiBot, and MolmoAct2-BimanualYAM.

For transfer, the encoder is frozen: extract a latent action from a source video, then hand it to the frozen decoder together with either a target initial frame (producing a video) or a target robot's initial state (producing executable actions). No paired source–target data is used.

For control, a variational bottleneck (one self-attention block, six-layer transformer decoder, 50,000 steps, KL weight β = 0.01) squeezes each 2,048-dimensional token to 32 dimensions while keeping the token layout. The policy predicts these codes from the current frame and instruction, and a separate DiT action head predicts low-level actions from the generator's features, observation, instruction, and state — trained for 100,000 steps with rank-128 LoRA on two NVIDIA H200 GPUs. Human videos supervise only the latent codes; robot demonstrations add action supervision. Hardware is a bimanual YAM robot with 14-dimensional absolute commands (six joint positions plus one gripper command per arm) observed by a single top-view Intel RealSense D405.

Why This Matters

Impact on research. The paper argues against the prevailing recipe of training a small action encoder from scratch on single frame pairs. Instead it shows that a video generative model's internal features already constitute a strong action representation, and that a generative decoder is a better supervisory signal than pixel reconstruction alone because it can absorb appearance and leave motion in the bottleneck. This reframes video pretraining as an action-labeling mechanism, not just a video synthesizer, and the LARYBench gap (0.38 → 0.17 regression MSE) is large enough to suggest the frame-pair paradigm is a real limitation.

Real-world applications:

• Learning robot skills from internet human video, where no robot action labels exist — the human-to-robot decoding path requires no paired data.
• Cheap teleoperation and demonstration retargeting: capture a motion once, replay it on a different robot or in a different scene via the shared latent action.
• Lower-latency robot policies: predicting 32-dimensional codes instead of future video cuts per-call latency from 28,226 ms to 1,474 ms, which is the difference between a demo and something that can run in a control loop.
• Synthetic cross-embodiment video generation for data augmentation, visual simulation, and evaluating whether a policy has learned motion versus memorized appearance.

Industry relevance. Robot foundation model efforts (VLA models, world action models) are bottlenecked on demonstration data cost and on the fact that control spaces differ across hardware. A model-agnostic latent action interface that plugs into an off-the-shelf video backbone is a plausible way to pool heterogeneous human and robot video into one training signal, and the compact-code policy path addresses the inference-cost objection that has kept video-diffusion policies out of real-time deployment.

Future Directions

• Quantitative action-transfer evaluation. The human-to-robot action transfer is shown qualitatively (writing, single-arm pushing, bimanual pushing) on a 14-dimensional bimanual YAM; there is no reported success rate for the transfer itself, so measuring it across more tasks and embodiments is the obvious next step.
• Scaling and generality of the policy results. The real-world policy study covers a single drawer-closing task with 45 training trajectories and 10 trials per configuration; whether the 96.7% success rate and the latency advantage hold across longer-horizon, more varied manipulation is untested here.
• How much the video backbone matters. DreamX is instantiated only with Cosmos-Predict2.5 2B; the paper's central claim is that video diffusion priors are the source of the universality, so comparing across backbones of different sizes and pretraining corpora would test that claim directly.
• Closing the loop. Action transfer is executed open-loop without visual feedback, and the policy generates latent codes rather than future frames. Combining latent-action prediction with closed-loop visual feedback, and testing whether the codes support planning or world-model-style rollouts, is a natural extension.

Target Audience

Researchers and engineers working on robot learning, vision-language-action models, and video generative models — particularly those interested in cross-embodiment transfer and in extracting structure from pretrained video backbones rather than training task-specific encoders. It is also relevant to practitioners who need policies that run at practical latency but are unwilling to give up the representational benefits of video pretraining. Readers without background in diffusion models and transformer attention will find the method section dense, though the core idea (queries read motion, decoder supplies appearance) is accessible.

Authors’ abstract

Learning a universal action representation from video requires modeling dynamics that are consistent across different executions of the same action yet invariant to changes in embodiment and appearance. Existing latent action models infer actions from visual transitions, but their representations often entangle the underlying action with irrelevant visual variations, limiting transfer across embodiments. We introduce DreamX, which repurposes a pretrained video diffusion model as a universal latent action learner. Our approach extracts compact latent actions from the video model's rich spatiotemporal features, compressing each latent video frame by $\boldsymbol{300\times}$ ($\boldsymbol{1{,}200 \rightarrow 4}$ tokens). In addition, by leveraging the video model's generative prior, these latent actions capture action-relevant dynamics disentangled from visual appearance. The resulting latent space serves as a shared action interface across embodiments (human and robot), environments, and action parameterizations. On LARY-Bench, DreamX achieves state-of-the-art performance among latent action models. Moreover, actions encoded in one setting remain effective in another: the latent space supports any-to-any video transfer and zero-shot action transfer across diverse embodiments and environments. Furthermore, this shared interface also enables policy learning in a compact latent action space, reducing the computational cost of video-diffusion-based policies.