ViPRA: Video Prediction for Robot Actions

AI summary

Overview

• Research area: Robotics, robot learning from video, vision-language-action (VLA) models, latent action representations, continuous control.
• Technical level: Advanced. The paper assumes familiarity with VQ-VAE tokenization, latent action models, flow matching, imitation learning, and video-language model pretraining.
• Scope (one sentence): The paper presents ViPRA, a pretraining-and-finetuning framework that converts a video-language prediction model into a continuous robot control policy by learning motion-centric latent actions from actionless human and robot videos, then mapping them to robot actions with a flow matching decoder using only 100 to 200 teleoperated demonstrations.

What This Paper Is About

Robot demonstrations are expensive to collect and are tied to a specific robot body, while videos of humans and teleoperated robots are abundant but usually have no action labels. The paper asks whether a video prediction model can be turned into a robot policy, and answers by training a model to predict both the what (future visual observations) and the how (motion-centric latent actions) of a scene, then decoding those latents into smooth continuous robot actions with very little labeled data.

Key Contributions

1. A scalable method for extracting fine-grained motion-centric latent actions from unlabeled human and robot videos, supervised by perceptual losses and an optical flow consistency loss.
2. A joint pretraining framework for robot control that predicts future visual states and latent action sequences inside a single unified video-language model.
3. A data-efficient pretraining-finetuning pipeline that combines flow matching with action chunking to produce smooth, high-frequency continuous control, running at up to 22 Hz.
4. Empirical gains of 16% on the SIMPLER benchmark and 13% on real world tasks over the strongest prior continuous control baselines.

Main Findings

• SIMPLER success rates: ViPRA-AR reaches 69.8% average success versus LAPA at 53.1% and OpenVLA at 38.6% in the discrete setting. ViPRA-FM reaches 62.5% average success versus Scratch-FM at 41.7%, π0 at 27.1%, UniVLA at 42.7%, UniPI at 1.7%, and VPT (list of discrete baselines) at 51.0%. The paper states ViPRA-FM outperforms Scratch-FM by 20.8%, π0 by 35.4%, and UniVLA by 19.8%.
• SIMPLER grasp rates: ViPRA-AR averages 76.1%, the highest across discrete methods, versus LAPA 73.9%, OpenVLA 68.8%, VPT 66.7%, and Scratch-AR 65.6%. ViPRA-FM averages 71.9%, higher than UniVLA 50.0%, π0 36.5%, Scratch-FM 60.4%, and UniPI 23.1%.
• Real world manipulation: ViPRA-FM achieves the highest average success rate at 54.1%, versus π0 at 40.1% and Scratch-FM at 23.8%, using substantially fewer labeled demonstrations. Discrete policies were excluded from real-world evaluation because bin-based predictions produced unstable spikes and occasionally triggered safety stops on the Franka arm.
• Future-state prediction matters: Removing state prediction (ViPRA–SP2) drops average SIMPLER success from 69.8% to 59.4% (AR) and from 62.5% to 53.2% (FM). A state-only variant (ViPRA–LA) reaches 60.7%, and adding a 1-step latent variant with state prediction (ViPRA–AC) reaches 59.2% (AR) and 44.8% (FM). Adding state prediction at finetuning time (ViPRA+SP3) degrades performance to 53.1% (AR) and 31.3% (FM) due to compounding errors.
• Action chunking matters: The full model using chunking plus future state prediction reaches the best results of 69.8% (AR) and 62.5% (FM).
• Optical flow consistency helps: Removing the flow loss raises perplexity from 5.01 to 5.63, entropy from 1.59 to 1.74, and action MSE from 0.84 to 0.92, indicating the latent space encodes motion rather than background appearance.
• Human and robot video co-training helps: On LIBERO-10, co-training achieves 0.79 success and 0.84 action MSE; robot-only achieves 0.72 and 0.91; human-only achieves 0.69 and 0.99. Human-only pretraining still yields positive transfer.
• Latent actions transfer across videos: A cross-video rollout test shows that injecting latents encoding upward motion from one video into the first frame of a downward-moving video causes the reconstruction to move upward, demonstrating transferable, dynamics-aware semantics.
• Codebook usage is structured: Token-position and code-index histograms show that both the codebook entry and its position in the latent action sequence encode information about motion direction and dynamics.
• Smoothness versus discrete policies: ViPRA-FM produces smooth continuous trajectories while LAPA exhibits local discontinuities and random spikes, often around contact events; such discontinuities triggered the robot's emergency brake in real world deployment.
• Inference speed: With KV caching, ViPRA's flow matching decoder runs at 1.95 Hz per chunk, supporting effective control rates up to 22 Hz at chunk size 14. The paper states this rate is matched only by one other 7B-parameter model (Kim et al., 2025).

Methodology in Plain English

The researchers break the problem into three stages.

First, latent action learning. They take actionless video clips of humans and robots and train a model to compress each transition between frames into a short sequence of discrete tokens, called latent actions. Because these tokens are learned from the video itself, no action labels are needed. Each latent action uses a small shared codebook of size 8, with multiple discrete tokens per timestep. To make sure the tokens describe real motion rather than appearance, they train a decoder to reconstruct future frames using a pixel-level L1 loss, a perceptual (LPIPS) loss, and an optical flow consistency loss computed with RAFT, which is switched on only after a warm-up period to avoid instability from poor early reconstructions.

Second, multimodal pretraining. They build on the instruction-tuned LWM-Chat-1M video-language model. Two extra modules are added: an embedding head that maps latent action tokens into the model's token space, and a small MLP decoder that autoregressively predicts the next latent token from the transformer's hidden state. During pretraining, the model receives the two most recent frames and a task description, and is trained with cross-entropy to predict both a future frame that is H steps ahead and the latent action sequence leading to it.

Third, continuous adaptation. For downstream control, they add an action encoder and a flow matching decoder. The decoder learns a flow field that transforms noise into a chunk of continuous robot actions, trained with a flow matching loss and Beta(1.5, 1) noise interpolation. At inference, the flow field is integrated over 10 uniform steps with Δs = 0.1.

The finetuning data requirements are small: 100 to 200 teleoperated demonstrations. In simulation, they collect 100 diverse multi-task trajectories in SIMPLER using a pretrained VLA model. In the real world, they collect 180 trajectories per task via GELLO teleoperation across 5 cup colors and 10 object types, and cap evaluation at an effective 3.5 Hz closed-loop control rate.

Why This Matters

Impact on research. The paper shows that a video prediction model can be repurposed as a robot policy without ever needing action labels during pretraining, and it argues against the assumption that latent action pretraining must be treated as pure autoregressive policy learning. By jointly modeling what changes and how, and by using a continuous flow matching decoder rather than discretized action bins, it produces control that is smoother and safer on real hardware.

Real-world applications:

• General-purpose manipulation robots that need to handle unfamiliar objects, colors, textures, and shapes after training on only a few hundred demonstrations.
• Bimanual coordination tasks, since the paper reports transfer from a single-arm and human video pretraining corpus to coordinated dual-arm Franka Panda setups.
• Low-cost robot deployment pipelines, where collecting thousands of hours of labeled action data is infeasible but large video corpora are already available.
• Safe physical manipulation near people, where the paper's emphasis on smooth continuous trajectories and avoidance of abrupt motor torque jumps matters for hardware safety.

Industry relevance. The paper's main practical claim is data efficiency: it reports competitive or better performance than π0 and OpenVLA, which used OpenX action-labeled robot demos containing 970k trajectories, while ViPRA's decoder uses only 100 to 200 teleoperated demonstrations. The support for control rates up to 22 Hz is relevant for deployment on physical hardware, and the paper has released models and code at https://vipra-project.github.io.

Future Directions

• More dexterous embodiments. Extending beyond WidowX and Franka to dual-arm humanoids or multi-fingered hands, which the authors say demands finer latent actions that can represent precise contact dynamics and inter-arm coordination.
• Additional sensing modalities. Incorporating tactile or force feedback, wrist cameras, proprioception, or depth to strengthen perception-control alignment, especially in dynamic or unstructured environments.
• Broader and richer video data. Expanding to ego-centric datasets with human-hand interactions, tool use, or short navigation episodes to improve embodiment invariance and temporal grounding.
• Scaling behavior and data efficiency. The paper leaves open how passive video pretraining scales and where diminishing returns emerge, and how to best balance unlabeled video against smaller quantities of demonstrations.
• Predictive modeling and planning. The latent action decoder can be interpreted as a predictive world model, which the authors suggest could support reinforcement-learning-based alignment and test-time planning with VLM-based reward models.

Target Audience

This paper is most useful for robotics researchers and machine learning engineers working on vision-language-action models, imitation learning, and latent action representations, particularly those focused on reducing labeled demonstration requirements. It also benefits practitioners building general-purpose manipulation policies for industrial or service robots, and researchers studying how video foundation models can transfer physical dynamics knowledge to control. Readers without background in latent variable models, flow matching, or robot learning will find the paper challenging, since the method section assumes familiarity with those techniques.

Authors’ abstract

Can we turn a video prediction model into a robot policy? Videos, including those of humans or teleoperated robots, capture rich physical interactions. However, most of them lack labeled actions, which limits their use in robot learning. We present Video Prediction for Robot Actions (ViPRA), a simple pretraining-finetuning framework that learns continuous robot control from these actionless videos. Instead of directly predicting actions, we train a video-language model to predict both future visual observations and motion-centric latent actions, which serve as intermediate representations of scene dynamics. We train these latent actions using perceptual losses and optical flow consistency to ensure they reflect physically grounded behavior. For downstream control, we introduce a chunked flow matching decoder that maps latent actions to robot-specific continuous action sequences, using only 100 to 200 teleoperated demonstrations. This approach avoids expensive action annotation, supports generalization across embodiments, and enables smooth, high-frequency continuous control upto 22 Hz via chunked action decoding. Unlike prior latent action works that treat pretraining as autoregressive policy learning, ViPRA explicitly models both what changes and how. Our method outperforms strong baselines, with a 16% gain on the SIMPLER benchmark and a 13% improvement across real world manipulation tasks. We have released models and code at https://vipra-project.github.io