RynnVLA-Latent: Scaling Robotic Manipulation Foundation Models with 3D-Grounded Latent Actions

AI summary

Overview

• Research area: Robotics — vision-language-action (VLA) models, robot manipulation foundation models, and latent-action pretraining from unlabeled video.
• Technical level: Advanced. The paper assumes familiarity with VLA architectures, flow matching, Mixture-of-Transformers designs, and self-supervised representation learning.
• Scope: A framework (RynnVLA-Latent plus its latent-action labeler RynnLAM) that converts 30.1K hours of heterogeneous human, robot, and simulation video into 745.52M 3D-grounded latent-action descriptors used to pretrain a 4B-parameter manipulation policy, evaluated on LIBERO, LIBERO-Plus, VLABench, RoboTwin 2.0, and six real-world tasks.

What This Paper Is About

Scaling robot policies is bottlenecked by two things: action-labeled robot demonstrations are expensive, and different robots use incompatible control spaces (different dimensions, frames, and meanings), so human and web videos cannot simply be pooled with robot data under one action-prediction objective. The authors build a shared "latent action" interface — a compact descriptor of what changed between two video frames, grounded in 3D motion and separated from camera ego-motion — so that any video can supervise policy pretraining without native robot commands. The policy learns to predict these descriptors from images and language, then a short embodiment-specific post-training step retargets the same network to real robot actions.

Key Contributions

1. 3D-grounded latent-action supervision (RynnLAM). A factorized encoder that splits a frame transition into 8 motion tokens (512-D), a global action latent (64-D), and a camera latent (32-D), trained with residual future-feature prediction, camera-compensated 3D-flow supervision, contrastive feature warping, camera-pose decoding, and a gradient-reversal objective — no native robot actions or proprioception required.
2. Scalable VLA pretraining from heterogeneous video (RynnVLA-Latent). A Mixture-of-Transformers policy (RynnBrain VLM backbone plus a narrow action expert) pretrained with flow matching to predict 6-step latent-action chunks from multi-view images and language, using 30.1K hours from 25 sources, 16.63M episodes, 3.057B frames, and 745.52M latent-action descriptors.
3. A two-stage decoupling of motion learning from control alignment. Pretraining uses latent actions and no proprioception; post-training reuses both the pretrained backbone and the action-expert trunk, adds proprioception, and only swaps a thin output projection to the target robot's action dimension — no separate latent-to-action decoder.
4. Representation- and policy-level validation. Frozen-representation probes on LARY (relative-action MSE 0.38 for the full descriptor), four simulation benchmarks, six real-world tasks on dexterous-hand and mobile-gripper platforms, plus representation ablations and data-scaling results.

Main Findings

• LIBERO: 98.8% success. The 4B policy reaches 98.8% on LIBERO and 84.7% on LIBERO-Plus without benchmark-specific adaptation.
• VLABench: 51.9% average across five tracks, and RoboTwin 2.0 bimanual: the abstract reports 88.6% average success, while the introduction reports 88.3% average success under the "Native" protocol — the two figures in the paper differ, and per-task breakdowns are not included in the provided content.
• Real-world deployment: the highest observed success rates among compared methods on six real-world manipulation tasks spanning a stationary dexterous-hand platform (Marvin-WUJI: dual 7-DOF Tianji arms plus dual 20-DOF Wuji hands, 54 total degrees of freedom) and a mobile gripper platform (Astribot-S1: wheeled base, dual 7-DOF arms, parallel grippers). Exact per-task success percentages are not reported in the provided content.
• Latent actions predict control, not just appearance. On the LARY low-level control regression track, the full 608-D descriptor achieves average MSE 0.27 (absolute) and 0.38 (relative), versus 0.93 relative for LAPA, 0.77 for Moto, and 0.55 for DreamDojo. General encoders are far weaker: V-JEPA 2 at 0.25, DINOv3 at 0.19, Wan2.2 at 0.30, FLUX.2-dev at 0.35 absolute average.
• Every component matters. Ablations on relative-action MSE: full RynnLAM 0.38; without camera decoupling 0.47; without gradient reversal 0.49; without 3D flow 0.50; frozen encoder 0.52; without motion-token residual forcing 0.57. Absolute-action ablations follow the same ordering (0.27 full vs. 0.32–0.37 for variants).
• Raw features are strong on the absolute track but not comparable to the full descriptor. RynnLAM raw features score 0.14 absolute average (best in that column), but the paper states the full 608-D descriptor gives better control predictability than any individual component in its representation probes (Appendix E), and only the full descriptor is used as the pretraining target.
• The descriptor is deliberately not camera-invariant. Camera-pose decoupling acts only on the global action latent; the full descriptor retains the camera latent, and the authors explicitly do not claim exact viewpoint invariance or physical identifiability.
• Data pipeline details matter for scale. Latent labels come from frame pairs sampled every 4 frames across 250 parallel shards, producing a policy training index of 121.06M samples (about 946K optimizer steps per epoch at batch size 128). Human video and real-robot video each contribute roughly 40% of duration, simulation 20%.

Methodology in Plain English

Step 1 — Collect and annotate video. The team assembled 25 video sources: real robot demonstrations (e.g., RoVid-X, AGIBOT-Beta, DROID, OpenX-Embodiment), simulation (RoboCasa, BEHAVIOR-1K, InternData-A1, RoboTwin 2.0), and human video (HowTo100M, EgoVid, EgoVerse, EgoDex, SthSthV2, EPIC_KITCHENS). Every clip goes through the same automatic pipeline: Qwen3-VL writes captions, DPFlow computes dense optical flow, and Depth Anything 3 estimates monocular depth. No action labels are used.

Step 2 — Learn what "changed" between two frames. RynnLAM takes a pair of frames and compresses the transition into three parts: local motion tokens, a global action latent, and a camera latent. It is trained to (a) predict the residual change in visual features rather than the whole future image, with the error normalized by the size of that change; (b) predict dynamic 3D scene flow after compensating for estimated camera motion, so static regions should show near-zero flow; (c) make flow-warped source features closer to the target features than to the un-warped source features, weighted by motion magnitude; and (d) decode relative 6-DoF camera pose from the camera latent while a gradient-reversal trick makes the same decoder fail on the action latent — pushing ego-motion into one slot and manipulation motion into another. Training runs 36K steps with the encoder frozen, then 20K steps unfrozen.

Step 3 — Label the whole corpus offline. Only the encoder is needed at this stage, so it stamps every frame pair with a 608-dimensional descriptor, giving 745.52M latent actions.

Step 4 — Pretrain the policy. RynnVLA-Latent pairs a pretrained vision-language backbone with a narrower action expert that attends to the VLM's tokens at every layer. It is trained with flow matching to generate 6-step chunks of latent actions from up to 6 camera-view slots, the instruction, and a noisy interpolation of the target. Missing views are simply masked out, so one model handles many camera setups. No robot actions or proprioception are used.

Step 5 — Retarget to a real robot. Post-training keeps the backbone and the action-expert trunk, adds proprioceptive state, and replaces only the final projection so the same flow-matching objective now outputs native robot commands (with a 6-D rotation representation and masking for unused dimensions). Inference integrates the learned velocity field from noise with Euler steps, computes the language-image prefix once, and executes the whole action chunk before replanning.

Why This Matters

Research impact. The paper argues that the limiting factor for VLA scaling is the supervision interface, not model size, and offers a concrete recipe: geometric grounding plus explicit camera-motion separation produces latent actions that are measurably more predictive of low-level control than transition-reconstruction latents. The LARY probe numbers (0.38 relative MSE vs. 0.93 for LAPA, 0.77 for Moto, 0.55 for DreamDojo) give the community a comparable yardstick, and the two-stage pretrain-then-retarget design means unlabeled video scale and robot-specific alignment can be pursued independently.

Real-world applications.

• Dexterous manipulation: the Marvin-WUJI setup (dual 7-DOF arms, dual 20-DOF hands, 54 DoF) targets fine multi-finger tasks that are hard to teleoperate at scale.
• Mobile manipulation: the Astribot-S1 (wheeled base, dual 7-DOF arms, 5 kg payload, ±0.1 mm repeatability) covers fetch-and-place style tasks where the base moves.
• Cross-embodiment fleets: one pretrained policy can be retargeted to different arms and grippers rather than training each robot from scratch.
• Learning from internet and egocentric video: human cooking and assembly footage (e.g., HowTo100M, EPIC_KITCHENS, EgoDex) becomes usable policy supervision without any robot in the loop.

Industry relevance. The release spans a project page, GitHub repository, Hugging Face collection, and ModelScope collection, and the recipe leans on existing open datasets and off-the-shelf perception models (Depth Anything 3, DPFlow, Qwen3-VL). The economic claim is direct: expensive action-labeled demonstrations are needed only for the final embodiment alignment, while the bulk of pretraining runs on video that already exists.

Future Directions

• Enforcing viewpoint invariance. The authors state that camera decoupling encourages separation but does not guarantee exact viewpoint invariance, and that the canonical camera-slot interface alone does not guarantee invariance to an unseen viewpoint — a natural next target.
• Closing the RoboTwin reporting gap. The abstract (88.6%) and introduction (88.3%) report different RoboTwin 2.0 averages; a full per-task breakdown under the Native protocol would clarify where bimanual performance still lags.
• Pushing past 51.9% on VLABench. Average success across five VLABench tracks is the weakest simulation number reported, suggesting semantic grounding and long-horizon reasoning remain open relative to the near-saturated 98.8% on LIBERO.
• Scaling the latent-action corpus further. The paper reports improvements with increasing latent-action pretraining data, which invites questions about where that curve saturates, how source-weighting (p_d ∝ W_d^0.7) should be tuned, and whether the 608-D descriptor is the right size.
• Latent actions beyond policy learning. The authors note that the same descriptors condition cross-embodiment video generation in a companion model (RynnWorld-Latent), pointing to shared latents as a general interface for world modeling as well as control.

Target Audience

Robotics and embodied-AI researchers building VLA or world-action models; engineers working on cross-embodiment policy transfer and robot foundation models; and practitioners who want to pretrain manipulation policies on existing video corpora instead of collecting new teleoperation data. Readers need comfort with transformer architectures, flow matching, and self-supervised objectives — the paper is not an introductory read, but its central idea (learn a shared, geometrically grounded description of motion from any video) is accessible to anyone who has trained a policy on demonstrations.

Authors’ abstract

Scaling vision-language-action (VLA) models is constrained by scarce action-labeled demonstrations and incompatible control spaces across robot embodiments. We present RynnVLA-Latent, a framework that turns heterogeneous interaction videos into scalable supervision for robot policy pretraining. At its core, RynnLAM learns continuous, 3D-grounded latent actions from visual transitions without requiring native robot controls. By combining residual future-feature prediction, camera-compensated 3D-flow supervision, and camera-motion decoupling, RynnLAM encourages representations that capture control-relevant motion rather than appearance changes alone. We apply RynnLAM to 30.1K hours of human, robot, and simulation videos, producing 745.52M latent-action descriptors. RynnVLA-Latent then learns to predict latent-action chunks from visual observations and language through flow matching, before adapting its pretrained backbone and action expert to embodiment-specific robot actions. The resulting 4B policy achieves 98.8% success on LIBERO, 84.7% on LIBERO-Plus without benchmark-specific adaptation, 51.9% average success across five VLABench tracks, and 88.6% average success on the RoboTwin 2.0 bimanual benchmark. It also achieves the highest observed success rates among the compared methods on six real-world manipulation tasks across dexterous-hand and mobile-gripper platforms. Together with representation ablations and data-scaling results, these findings demonstrate the potential of latent-action supervision for scaling robot policy learning beyond native action-labeled data.