mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs

AI summary

Overview

Research area: robot learning, specifically generalist manipulation policies, generative video models, and Vision-Language-Action (VLA) architectures.

Technical level: Advanced. The paper assumes familiarity with flow matching, diffusion transformers, inverse dynamics models, and VLA baselines such as OpenVLA and the π₀.₅ series.

Scope: The paper introduces mimic-video, a Video-Action Model (VAM) that freezes a pretrained internet-scale video backbone, extracts its intermediate latent representations, and conditions a flow matching-based action decoder on them to produce low-level robot actions.

What This Paper Is About

Current Vision-Language-Action models inherit their priors from vision-language backbones pretrained on static images and text, so they carry no innate knowledge of physics, object motion, or temporal procedure. That knowledge must instead be learned from scarce, expensive expert robot teleoperation data, creating a data-efficiency bottleneck. The goal of this paper is to replace static image-text conditioning with a pretrained generative video model whose latent representations already encode visual dynamics, leaving the action decoder with only the simpler job of inverse dynamics.

Key Contributions

1. A new class of method: Video-Action Models (VAMs). mimic-video couples a pretrained language-conditioned video diffusion backbone (Cosmos-Predict2) with a lightweight flow matching action decoder, where the decoder acts as an Inverse Dynamics Model (IDM) conditioned on intermediate video model hidden states.

2. A partial denoising inference strategy. Instead of fully generating future video frames, the method integrates the video flow field only to an intermediate flow time τ_v, extracts activations after the k-th video model layer, and conditions the action decoder on those features. The video backbone remains frozen during action decoder training.

3. An "oracle" case study isolating prediction from execution. Training an action decoder on latents from ground-truth future video frames yields near-perfect success rates whether or not the backbone was finetuned on the target distribution, which the authors use to argue that control effectively reduces to visual prediction and that policy performance scales with video model quality.

4. State-of-the-art results across simulated and real-world embodiments with reported 10x better sample efficiency and 2x faster convergence compared to traditional VLA architectures.

Main Findings

• SIMPLER-Bridge average success rate: mimic-video (scratch) reaches 46.9% and 56.3% with per-task τ_v tuning, versus 35.4% for the π₀.₅-style VLA (scratch) baseline, 43.8% for ThinkAct, 45.0% for FLOWER, 16.0% for Octo (finetuned), and 14.6% for OpenVLA (finetuned). Per-task scores for mimic-video (scratch) were 37.5 (Put Carrot on Plate), 37.5 (Put Spoon on Towel), 12.5 (Stack Blocks), and 100.0 (Eggplant); with per-task τ_v tuning these become 54.2, 41.7, 29.2, and 100.0.

• LIBERO average success rate: mimic-video (scratch) achieves 93.9%, compared with 85.9% for the π₀.₅-style VLA (scratch), 88.6% for DiT Policy (finetuned), 84.1% for OpenVLA (finetuned), 83.1% for Octo (finetuned), 79.7% for Diffusion Policy (scratch), and 96.9% for OpenVLA-OFT (finetuned). Its per-suite scores were 94.2 (Spatial), 96.8 (Object), and 90.6 (Goal).

• Real-world bimanual dexterous manipulation: mimic-video scores 72.0 on Packing and 93.0 on Package handover, versus 42.6 and 74.1 for a DiT-Block Policy with wrist cameras and 11.0 and 30.0 for a DiT-Block Policy using workspace view only. mimic-video was conditioned on only the single workspace camera view.

• Sample efficiency: the action decoder reaches the maximum success rate of the VLM-conditioned decoder using only 10% of the training data. Reducing to one episode per task, a 98% reduction in action data, still yields a 77% average success rate, which the authors place as competitive with their Diffusion Policy baseline.

• Convergence speed: the mimic-video action decoder converges faster and to a higher asymptotic success rate than the π₀.₅-style VLA decoder, even though the VLA baseline saw task-specific action data during FAST-pretraining.

• High-fidelity video is not required: peak autonomous policy success on SIMPLER occurred at the highest flow time τ_v = 1, meaning pure noise conditioning and a single forward pass of the video backbone. A separate sweep on held-out BridgeDataV2 validation data found the lowest action reconstruction MSE at an intermediate flow time of approximately τ_v ≈ 0.4, which matches the near-perfect performance observed in the oracle case study.

• Video quality gates policy quality: in the case study, conditioning on predicted video latents from a standard off-the-shelf model versus one finetuned on robotics data showed a pronounced scaling behavior, while oracle latents from ground-truth future frames gave near-perfect success rates in both cases.

• Task-specific data used in the real-world tasks: the video backbone was finetuned on a broader 200-hour corpus, while the action decoders were trained on 1h 33m (512 episodes) for sorting and 2h 14m (480 episodes) for stowing. The paper does not report additional dataset sizes beyond these and the LIBERO/BridgeDataV2 descriptions.

Methodology in Plain English

The system has two parts, each trained with flow matching, which learns to transport Gaussian noise into data along a continuous time parameter τ running from 0 (clean data) to 1 (noise).

The first part is the video model: an open-source 2B latent Diffusion Transformer, Cosmos-Predict2, which operates on video frames encoded by a pretrained 3D tokenizer. Its input concatenates clean latent patches from a context prefix of 5 frames with noisy latent patches for the future frames to be produced. Each transformer layer alternates self-attention over the full video sequence, cross-attention to language instructions encoded by T5, and a two-layer MLP.

The second part is the action decoder, a smaller Diffusion Transformer. It encodes the robot's proprioceptive state and a sequence of future actions through two separate MLPs, concatenates them along the sequence dimension, and uses learned absolute positional encodings. Each decoder layer cross-attends to the video model's intermediate representations, self-attends over the action sequence, and passes through a two-layer MLP, with residual paths and AdaLN modulation driven by a low-rank bilinear-affine encoding of both the video flow time τ_v and action flow time τ_a. During training, the soft token for proprioceptive state is randomly replaced with a learned mask token to avoid overfitting.

Training happens in two phases on disjoint parameters. First, the video backbone is finetuned with Low-Rank Adapters (LoRA) on robotics video datasets so it captures the robot domain while retaining pretrained temporal reasoning. Second, the action decoder is trained from scratch on video representations from the now-frozen backbone. Independent flow times τ_v and τ_a are sampled per iteration, with τ_v drawn from a logit-normal distribution matching video pretraining and τ_a distributed proportionally to the square root of τ_a − 0.001, following prior work.

At inference, the video flow field is integrated from noise to an intermediate τ_v, the first k layers of the video model process that partially denoised latent, and the extracted activations condition the action decoder, which then fully denoises a chunk of robot actions. When τ_v = 1, the integration step becomes redundant and one forward pass of the video backbone suffices, enabling real-time control. The authors report τ_v = 1 as a good default balancing performance and speed, with the optimal value being task-dependent.

Why This Matters

The paper argues that the dominant VLA recipe assigns the wrong burden: static image-text pretraining provides semantics but no physics, forcing robot data to teach dynamics from scratch. If the authors' claim holds, the expensive part of policy learning shifts from scarce teleoperation data to abundant internet video, changing how robot foundation models should be pretrained.

Potential real-world applications:

• Dexterous bimanual manipulation in contact-rich, heavily occluded settings, demonstrated on package sorting and tape stowing with 16-DoF humanoid hands on Panda arms.
• Warehouse and logistics picking, handover, and packing, where the paper's Package Sorting task (pick, handover, place) directly maps to operational workflows.
• Low-data industrial deployment, since policies can be trained from roughly an hour or two of task-specific demonstrations per task.
• Household and service robotics requiring long-horizon tasks under novel objects and instructions, the regime LIBERO-Goal and LIBERO-Object test.

Industry relevance: the result that only a single workspace camera suffices, while a comparable diffusion policy needed wrist cameras to reach 42.6 and 74.1 on the real-world tasks, has direct implications for hardware cost and sensor integration. The 10x sample-efficiency claim and 2x convergence claim bear on the cost of data collection and retraining cycles. The 2B-parameter open-source video backbone also means the approach is buildable without proprietary video foundation models.

Future Directions

• Multi-view video backbones. The authors state that relying on a single-view video backbone restricts policies to a fixed, single workspace view and suggest natively multi-view video architectures would improve spatial reasoning and occlusion robustness.

• A unified large-scale cross-embodiment VAM. Applying the VAM recipe to train one unified cross-embodiment model is described as a necessary step to unlock the full generalization of video foundation models. The paper states this has not yet been done.

• Further work on error sources in video latents. The paper reports that the intermediate video model representations from which the decoder reconstructs actions behave differently from the video latents themselves, and defers a detailed discussion of these mechanisms to Appendix E.

• Addendum to the limitations list. The paper's limitations section is truncated in the provided content after the second point ("Finally, ou"), so any third limitation is not reported here.

Target Audience

Robotics and machine learning researchers working on generalist manipulation policies, imitation learning, and generative models for control. It is most useful to readers already familiar with VLAs, diffusion and flow matching policies, and benchmark suites such as SIMPLER and LIBERO, and to engineers evaluating whether video-pretrained backbones can replace vision-language backbones in production robot policies with limited teleoperation data.

Authors’ abstract

Prevailing Vision-Language-Action Models (VLAs) for robotic manipulation are built upon vision-language backbones pretrained on large-scale, but disconnected static web data. As a result, despite improved semantic generalization, the policy must implicitly infer complex physical dynamics and temporal dependencies solely from robot trajectories. This reliance creates an unsustainable data burden, necessitating continuous, large-scale expert data collection to compensate for the lack of innate physical understanding. We contend that while vision-language pretraining effectively captures semantic priors, it remains blind to physical causality. A more effective paradigm leverages video to jointly capture semantics and visual dynamics during pretraining, thereby isolating the remaining task of low-level control. To this end, we introduce mimic-video, a novel Video-Action Model (VAM) that pairs a pretrained Internet-scale video model with a flow matching-based action decoder conditioned on its latent representations. The decoder serves as an Inverse Dynamics Model (IDM), generating low-level robot actions from the latent representation of video-space action plans. Our extensive evaluation shows that our approach achieves state-of-the-art performance on simulated and real-world robotic manipulation tasks, improving sample efficiency by 10x and convergence speed by 2x compared to traditional VLA architectures.