Native Action-Prior Learning from Videos for World Action Models

AI summary

Overview

Research area: Robot learning and world action models (WAMs) — specifically, how to pretrain robot action policies from observation-only video without action labels.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching generative models, Mixture-of-Transformers architectures, attention masking, and imitation-learning benchmarks such as LIBERO and RoboTwin.

Scope: The paper proposes NAVA-WAM, a two-stage world action model that pretrains its action-generating expert directly from observation-only videos via future-video flow matching, then post-trains on action-labeled demonstrations for robot control, and evaluates it on simulation benchmarks and a physical Franka FR3.

What This Paper Is About

World action models that couple future visual prediction with action generation are limited by their reliance on action-annotated robot trajectories, which are costly to collect and hard to standardize across embodiments. Existing ways of using cheap observation-only videos either learn vision representations that must later be adapted for control, or infer latent actions that must later be grounded to robot commands — both inserting an indirect intermediate stage. NAVA-WAM instead asks whether observation-only videos can directly optimize the action policy itself, with no intermediate interface.

Key Contributions

1. Native action-prior learning. A paradigm that directly pretrains the action policy (the Action-DiT) from observation-only videos, rather than pretraining a vision representation or a latent-action space and transferring it later. The paper frames this as a direct path from scalable video to the action policy, avoiding representation bottlenecks.

2. Transition-structured joint attention for action-free pretraining. The Video-DiT is frozen and the Action-DiT is optimized purely through future-video flow-matching loss. Attention is organized around visual transitions — the clean preceding state, a noised future state, and the corresponding Action-DiT tokens — so the video objective propagates supervision into the Action-DiT through the joint-attention pathway. The paper argues this realizes an implicit inverse–forward dynamics factorization without separate inverse and forward models.

3. Asymmetric attention for post-training and efficient inference. After action labels become available, video–action flow matching jointly trains both streams, but the visual stream is decoupled from the evolving action sample. This lets the visual keys and values be cached after the first denoising step so that iterative denoising runs only through the Action-DiT, giving action-only inference without generating future videos.

4. Empirical validation across simulation, out-of-distribution settings, ablations, and a real robot. Evaluation covers LIBERO/LIBERO-Plus and RoboTwin 2.0 in in-domain and out-of-domain regimes, action-label efficiency, probe-based action relevance, inference cost, and deployment on a Franka FR3.

Main Findings

• LIBERO / LIBERO-Plus: NAVA-WAM reaches 99.0% on LIBERO (in-domain) and 83.5% on LIBERO-Plus (out-of-domain). For comparison, in the same table Image-WAM scores 98.4% / 83.1% and Being-H0.7 scores 99.2% / 82.1%, so NAVA-WAM has the highest LIBERO-Plus score in the table but a slightly lower LIBERO score than Being-H0.7.

• RoboTwin 2.0: NAVA-WAM reaches 88.5% on Clean (in-domain) and 73.6% on Random (out-of-domain). The paper states this is an improvement of 3.5 points on Clean and 17.9 points on Random over the strongest baseline, with a much smaller Clean-to-Random drop.

• Real-robot deployment: On a physical Franka FR3, post-trained on DROID without task-specific fine-tuning, NAVA-WAM succeeds in 14/15 trials (93.3%), versus 10/15 (66.7%) for DreamZero and 8/15 (53.3%) for π0.5. On task T1, which requires placing a cube on the left side of a bowl, NAVA-WAM succeeds in 4/5 trials while both baselines fail in all five.

• Action-label efficiency: On RoboTwin 2.0 with 10 / 25 / 50 demonstrations per task, NAVA-WAM scores 66.5 / 72.0 / 73.6, versus 60.5 / 67.6 / 69.7 for no pre-training, 58.2 / 60.6 / 68.1 for representation-based pre-training, and 61.6 / 65.0 / 71.2 for latent-action-based pre-training. At 25 demonstrations, NAVA-WAM (72.0%) surpasses all baselines trained with 50, which the paper presents as evidence of both better pre-training and better label efficiency.

• Action relevance of learned representations: Ridge-regression R² to ground-truth actions at 10% / 50% / 100% probe-data budgets: NAVA-WAM 0.247 / 0.320 / 0.329; representation-based 0.080 / 0.157 / 0.153; latent-action (CoMo) 0.088 / 0.192 / 0.203; latent-action (DynaMo) 0.052 / 0.143 / 0.154.

• Inference efficiency: NAVA-WAM requires 4.30 TFLOPs per action chunk with 73.6% Random-domain success, compared with Image-WAM at 4.09 TFLOPs and 37.6%, Fast-WAM at 3.56 TFLOPs and 6.3%, and HALO at 67.69 TFLOPs and 26.4%.

• Qualitative attention and motion transfer: Attention visualizations show the pretrained Action-DiT attending to interaction-relevant regions such as the gripper and manipulated objects, while the CoMo latent-action baseline shows more diffuse attention over static regions. In a motion-transfer analysis, Action-DiT activations extracted from a source transition reproduce the source motion when transferred to different target observations across simulation, real-world, and sim-to-real examples.

• Data scaling: Random-domain success improves consistently with the size of the observation-only pre-training corpus at both tested action-label budgets. The paper does not report the absolute size of the pre-training corpus.

• Theoretical grounding: Proposition A.1 shows the reduction in Bayes-optimal future-state prediction error from conditioning on the transition representation is non-negative. Proposition A.2 shows that exposing the Action-DiT only to a Gaussian-corrupted future state bounds the achievable prediction error away from zero for any non-zero corruption level, preventing a degenerate shortcut in which the representation simply copies the clean prediction target.

Methodology in Plain English

NAVA-WAM builds on a Mixture-of-Transformers design with two experts that share attention but keep separate parameters: a Video-DiT (initialized from Wan2.2-5B) that models visual dynamics, and a 1B-parameter Action-DiT with hidden dimension 1,024 that generates continuous robot actions.

Stage one — pretraining from video only. A frozen spatiotemporal VAE encodes video clips into latent visual states, and adjacent pairs define transitions. For each transition, the model builds a segment containing the clean previous state, a noised version of the future state, and the Action-DiT tokens for that transition. An attention mask isolates these segments. Inside a segment, the Action-DiT queries can look at both the previous and the noised future visual states, which lets it build an internal representation of the transition — effectively doing inverse dynamics. Meanwhile, the noised future-state queries can also look at the Action-DiT tokens, so predicting the future video depends on what the Action-DiT has represented — effectively doing forward dynamics. The Video-DiT is frozen, so the only way to reduce the future-video flow-matching loss is to update the Action-DiT.

The key design detail is that the Action-DiT never sees the clean future state, only a noised version. That blocks the trivial shortcut of passing the answer straight through.

Stage two — post-training on robot demonstrations. With action labels available, both streams are trained jointly using flow matching: a video loss and an action loss, weighted by λv and λa (both set to 1 in the experiments). Here the attention becomes asymmetric — Video-DiT queries attend only to visual tokens, while Action-DiT queries attend to both visual and action tokens. This keeps the visual stream independent of the action sample being denoised.

Inference. Because the visual stream does not depend on the action sample, the Video-DiT keys and values can be computed once and cached. The first forward pass evaluates both streams at the first denoising step; all subsequent denoising steps run only through the Action-DiT, reusing the cache. No future video is ever generated, which is what keeps the TFLOP cost low while retaining the benefit of video-based priors.

Training data. Pretraining uses observation-only videos from Open X-Embodiment, AgiBotWorld, and EgoDex. Downstream, LIBERO training uses 50 demonstrations per task; RoboTwin 2.0 uses only 50 Clean-domain demonstrations per task with no Random-domain demonstrations.

Why This Matters

Impact on research. The paper reframes a common assumption in robot learning: that video data must be distilled into a representation or a latent action space before it can help a policy. NAVA-WAM argues the video objective can be applied directly to the action-generating module, and backs this with ablations showing it outperforms both representation-based and latent-action-based pre-training at every tested label budget. The theoretical propositions give a concrete reason the design avoids the shortcut problem that latent-action methods must engineer around. If the results generalize, this reduces the architectural complexity the field currently pays for using unlabeled video.

Real-world applications:

• Tabletop and household manipulation, such as the three Franka FR3 tasks evaluated here, where a policy must follow spatial relations like placing an object to the left of another.
• Warehouse and logistics picking, where environments and object arrangements shift constantly and out-of-distribution robustness matters more than in-domain accuracy.
• Manufacturing and assembly, where demonstrations are expensive to collect and embodiments differ, making action-label efficiency valuable.
• Assistive and service robotics, where a policy that generalizes from passively observed video could be adapted to new tasks with few teleoperated demonstrations.

Industry relevance. The central cost problem in robot learning is action-labeled data collection. A method that extracts useful action priors from observation-only video — including human video from sources like EgoDex — and that then reaches competitive performance with 25 demonstrations per task instead of 50 offers a direct path to lowering that cost. The action-only inference design (4.30 TFLOPs per chunk, comparable to Fast-WAM's 3.56 and Image-WAM's 4.09, and far below HALO's 67.69) matters for real-time closed-loop control on deployed hardware.

Future Directions

• How far does observation-only scaling go? The paper shows Random-domain success improving with corpus size, but does not report absolute corpus sizes or an apparent saturation point. Whether the trend continues, and at what cost, is open.

• Generalization beyond the evaluated embodiments. Real-robot evaluation covers a single Franka FR3 platform and three tabletop tasks, and LIBERO/RoboTwin are simulated. Whether the learned action prior transfers across substantially different morphologies and action spaces is untested here.

• Characterizing what the Action-DiT actually learns. The probe R² and attention visualizations suggest transition-relevant information, and the motion-transfer experiment suggests it is transferable, but there is no systematic account of which physical quantities the representation encodes or where it fails.

• Interaction with semantic priors and longer horizons. The paper studies future-video supervision over visual state transitions; how native action-prior learning would combine with large-scale vision–language–action pretraining, or with much longer action chunks and horizons, is not addressed.

Target Audience

• Robotics and embodied-AI researchers working on policy learning, world models, and video pretraining for control.
• Practitioners building manipulation systems who need to stretch limited teleoperation budgets or adapt policies across environments.
• Machine learning researchers interested in flow matching, Mixture-of-Transformers designs, and attention-mask engineering for cross-modal supervision.
• Readers tracking the debate between vision-representation, latent-action, and direct-policy approaches to using unlabeled video — this paper stakes out and empirically defends a fourth position, and its baseline table against DreamZero, π0.5, Image-WAM, Fast-WAM, HALO, JEPA-VLA, and latent-action methods makes that comparison concrete.

Authors’ abstract

World action models integrate future visual dynamics with robot action prediction, but their scalability remains limited by the need for action-annotated robot trajectories. Observation-only videos contain rich evidence about interaction dynamics, but existing approaches typically use them either to pretrain visual representations that must later be adapted for control, or to infer latent actions that are subsequently grounded to robot commands. We present NAVA-WAM, which introduces native action-prior learning by directly pretraining the action policy from observation-only videos, avoiding indirect representation-to-control transfer or a separate latent-action model. Our training consists of two stages. First, we pretrain on observation-only videos, where future-video flow-matching supervision over visual transitions is propagated through transition-structured joint attention to optimize the Action-DiT and learn action-relevant priors. Second, we use action-labeled demonstrations to post-train the Action-DiT for robot control through joint video--action flow matching, while asymmetric attention decouples the visual branch from iterative action denoising and enables efficient action-only inference. Extensive experiments show that NAVA-WAM consistently outperforms prior approaches under both in-distribution and out-of-distribution settings, while demonstrating strong action-label efficiency and effective real-robot generalization. These results establish native action-prior learning as an effective approach to directly pretrain action policies from observation-only videos, providing a scalable path beyond action-labeled robot data.