WorldBridge: Bridging Video and Action for Task Generalization in World Action Models

AI summary

Overview

Research area: Robot learning — specifically World Action Models (WAMs), a class of embodied models that jointly predict future video and robot actions, closely related to vision-language-action (VLA) models.

Technical level: Advanced. The paper assumes familiarity with flow matching, Mixture-of-Transformers architectures, video VAEs, and imitation-learning benchmarks.

Scope: The paper studies how video supervision and action supervision each contribute to generalizing to new tasks, then introduces WorldBridge, a two-expert WAM trained on a large heterogeneous robot-plus-egocentric-hand corpus, evaluated across six simulated manipulation benchmarks and two real robot platforms.

What This Paper Is About

Most robot generalization research tests robustness to nuisance changes — new object layouts, camera angles, lighting — while the robot performs tasks it already saw in training. This paper instead targets task generalization: executing behaviors that lie entirely outside the original training task set. The authors ask a specific, measurable question: if you give a World Action Model more video of a new task, versus more video plus a little action labeling, how much does each help — and they then build a model around the answer.

Key Contributions

1. A controlled study separating video generalization from action generalization. Holding in-distribution (ID) video-action data fixed on RoboTwin 2.0, the authors vary only the availability of out-of-distribution (OOD) videos and OOD action labels, producing a three-point progression of OOD success rates (25.07% → 55.90% → 70.70%).

2. WorldBridge, a World Action Model with separate video and action experts. The model uses a Mixture-of-Transformers architecture with a joint flow-matching objective, stage-specific structured attention masks that route information from memory to current observation to predicted future video to actions, two-scale visual memory, and an "action forcing" training scheme with per-group noise levels inside an action chunk.

3. A unified data processing pipeline spanning robots and egocentric human video. Robot trajectories and human hand demonstrations are standardized into a shared 46-dimensional action representation (14D arm joints, 14D end-effector poses, 2D grippers, 12D dexterous hands, 4D waist) with dimension-wise validity masks, plus kinematic, temporal, and signal-anomaly filtering; egocentric hands are re-expressed in the current camera frame and mapped to wrist poses, gripper openness, and finger targets.

4. Broad empirical evaluation. State-of-the-art-among-compared-methods results on RoboTwin 2.0 (96.23% full data, 80.9% clean-to-random), RoboCasa GR1 TableTop (82.8%), DOMINO Level-1 (57.6%), and an 83.0% result on RMBench matching the best in the comparison, plus real-world deployment on AgiBot G2 and ARX AC-1.

Main Findings

• Video alone buys a lot of transfer. Training only on in-distribution video-action pairs gives a 25.07% success rate on out-of-distribution RoboTwin 2.0 tasks.

• Unlabeled video of the new tasks nearly doubles it. Adding OOD task videos while keeping action supervision restricted to the ID set raises OOD success to 55.90%, which the authors attribute to broader video coverage exploiting inverse-dynamics capabilities the model already has.

• One labeled demo per new task adds a large further jump. Keeping the video set fixed and adding just one action-labeled demonstration per OOD task raises success to 70.70%. The authors read this as sparse paired examples sharpening the video-to-action bridge for visual transitions the video data already covers.

• Strong benchmark numbers. 96.23% on RoboTwin 2.0 full data, 80.9% in the RoboTwin 2.0 clean-to-random (C2R) setting, 82.8% on RoboCasa GR1 TableTop, 57.6% on DOMINO Level-1, and 83.0% on RMBench (matching the best result in the comparison). Results on SimplerEnv-WidowX and RoboDojo are listed as evaluated benchmarks but their numbers are not reported in the available text.

• Real-world deployment works across two platforms. The same jointly pretrained checkpoint is fine-tuned separately for AgiBot G2 and ARX AC-1 and evaluated on long-horizon execution, memory-dependent tasks, spatial localization, and bimanual coordination. Quantitative real-world success rates are not reported in the available text.

• A large curated corpus is actually usable. After curation the corpus contains 1,275,715 robot episodes (9,729.58 hours) and 2,105,846 ego episodes (12,591.63 hours) — 3,381,561 episodes and 22,321.21 hours total, retained from 3,888,958 episodes and 28,104.87 hours (79.42% hour retention). Retention varies sharply by source, e.g. 16.91% for RoboMIND and 98.19% for GenRobot.

• Inference cost can be cut without retraining. Because the attention mask is directional (no feedback from actions to video), the video branch can be denoised for only n synchronized steps and the remaining N − n action steps reuse the cached per-layer video key-value features; the paper reports a typical N = 10 and omits future-video RGB decoding at deployment.

Methodology in Plain English

The core experiment. The authors take RoboTwin 2.0 and fix a task partition into "in-distribution" and "out-of-distribution" before adding any extra data. They then train three variants: (a) ID video-action pairs only, (b) ID pairs plus OOD video without OOD actions, (c) the same videos plus a single action-labeled demo per OOD task. Comparing success on the OOD tasks isolates what video coverage contributes versus what action grounding contributes.

The model. Two transformer "experts" — one for video, one for actions — share parameters only through joint self-attention. Observations are encoded with a video VAE and instructions with a T5 text encoder. Both future video latents and action chunks are generated by flow matching: clean targets are linearly corrupted toward Gaussian noise, and the network predicts the velocity that would undo the corruption, integrating back to clean outputs at inference. The video and action flow timesteps are sampled independently, so the two experts routinely operate at different noise levels.

The bridge. The attention mask is what makes this a video-to-action bridge rather than two models glued together. During pretraining, video tokens see only video tokens (so video prediction stays action-independent) while action tokens see everything. During post-training, the mask is tightened: memory and current-frame tokens interact with each other, future-video tokens see current and future video, and action tokens see future-video and action tokens. Information is therefore forced to flow memory → present → predicted future → action. For strongly memory-dependent tasks, the looser pretraining mask is used so actions can read memory directly.

Memory. Two four-frame histories are kept: short-term frames at offsets [−8, −6, −4, −2] and long-term frames at [−128, −96, −64, −32] on the action time grid. The long-term stride equals the 32-step action horizon, so long-term memory spans roughly the progression of the whole task. Because the causal video VAE compresses time by a factor of four, the current observation is repeated four times and a zero padding frame is prepended; after encoding, the padding latent is dropped, leaving three clean conditioning latents.

Action forcing. Normally one noise level is shared across a whole action chunk, which lets each action token be solved mostly from its own noisy input. Action forcing instead samples an independent flow timestep per group of g actions, so a chunk contains a mix of nearly-clean and heavily-corrupted actions. The intent is to make each token lean on the visual context rather than on itself. At inference this is switched off — all action tokens are denoised jointly at a shared timestep.

Data. Rather than assuming every dataset speaks the same language, the pipeline audits each source's fields, maps them into the fixed 46-dimensional layout with separate state and action masks, converts rotations to unit quaternions in (x, y, z, w) order, calibrates grippers to 0 = closed and 1 = open, and then filters: trimming stationary episode boundaries (motion detected at a 10⁻⁴ threshold), dropping trajectories under 48 frames, checking forward-kinematics poses against recorded end-effector poses, detecting sudden changes with median and Savitzky–Golay residuals, checking state-action directional agreement over offsets within ±15 frames, and rejecting out-of-range values relative to per-repository 1st/99th percentiles. For egocentric video, future hand keypoints (21 per hand in MANO order) are transformed into the current camera frame so a moving head-mounted camera does not look like hand motion; wrist pose, gripper openness (thumb tip to a virtual index/middle fingertip, normalized by palm width), and six finger targets per hand are extracted, and windows with more than 20% rejected action timesteps are discarded.

Training. Robot and ego data are sampled with probabilities 0.7 and 0.3, weighted within each category by usable physical duration. Chunks contain H = 32 action targets and eight future video frames (one video frame per four action timesteps), robot actions at 15 Hz and ego actions at 30 Hz. Multi-camera sources are tiled into a single T-shaped image (main view on top, two wrist views below) and resized to 320 × 384. Pretraining runs 60,000 optimizer steps on 64 GPUs with 12 samples per GPU and two gradient-accumulation steps, an effective batch size of 1,536, AdamW (β₁ = 0.9, β₂ = 0.95, weight decay 0.01, gradient clipping at 0.5), a base learning rate of 10⁻⁴, and linear warmup from 10⁻¹⁰ to 9.9 × 10⁻⁵ over the first 2,000 steps. Video and action flow-matching losses are equally weighted.

Deployment. In real-world experiments, a server computes the next action chunk while a client keeps executing the current one, with at most one outstanding request; stale actions are discarded on arrival, and the chunk is resampled onto a 100 Hz servo grid handled by a C++ controller. Action denoising is compiled with torch.compile and replayed through CUDA Graphs to cut launch overhead.

Why This Matters

The paper reframes robot generalization as two separable problems — can the model imagine how a new task unfolds, and can it convert that imagination into motor commands — and shows empirically that these two abilities respond very differently to different kinds of data. The 25.07% → 55.90% → 70.70% progression is a concrete, quotable argument that unlabeled video is a first-class training signal for manipulation, and that a single demonstration per new task can be worth far more than its raw data volume suggests. That has direct consequences for how robot data budgets should be spent.

Real-world applications:

• Household and service robots that must handle tasks introduced after deployment, where collecting hundreds of demonstrations per new task is impractical but video of the task is abundant.
• Factory and warehouse manipulation, where new part geometries and assemblies appear regularly and a handful of teleoperated demos plus existing process video could be enough to bring a policy up to speed.
• Learning from human egocentric video (wearable cameras, AR/VR capture) as a scalable supervision source, since the pipeline converts hand keypoints into the same 46-dimensional action space used for robots.
• Cross-embodiment deployment, where one pretrained checkpoint is fine-tuned for different arms and hands — here AgiBot G2 and ARX AC-1 — instead of training a bespoke policy per robot.

Industry relevance: The work sits squarely on the current WAM/VLA research frontier alongside cited efforts such as DreamZero, Zero-WAM, and native video-action pretraining. The engineering contributions — a 22,321.21-hour curated corpus, a shared action representation with validity masks, and an inference stack that reuses video key-value features plus CUDA Graph replay — are as much about making these models trainable and deployable at scale as about the architecture itself.

Future Directions

• Quantifying the minimum action labeling needed. The paper shows one demo per OOD task jumps success from 55.90% to 70.70%, but the shape of that curve — whether two demos help as much as the first, and which tasks need none — is the natural follow-up.

• Diagnosing the bridge directly. The framing separates video generalization from action generalization, but the paper measures only end-task success. A direct probe of inverse-dynamics quality on OOD visual transitions would test the stated explanation (that gains come from leveraging existing IDM capability).

• Extending the shared action space. The 46-dimensional layout covers arm joints, end-effector poses, grippers, dexterous hands, and only four waist/torso slots. Locomotion, mobile bases, and full-body humanoids are not represented, which limits how far the cross-embodiment claim can stretch.

• Pushing inference cost down further. The decoupling trick, torch.compile, and CUDA Graphs help, but deployment still involves multi-step denoising with a typical N = 10 and frozen early video features that only approximate joint denoising; tighter approximations or distillation are open.

Target Audience

Robotics and embodied-AI researchers already comfortable with diffusion/flow-matching policies and transformer architectures; engineers building generalist manipulation stacks who need to decide how to allocate a data budget between video and teleoperated demonstrations; and readers tracking the WAM/VLA literature who want a clean empirical decomposition of what video pretraining actually contributes to task transfer. Readers new to the area will find the framing accessible but the method sections dense.

Authors’ abstract

Achieving the broad task generalization of embodied models in physical environments remains a central challenge in robot learning. We study this challenge in World Action Models (WAMs) through two complementary capabilities: video generalization, which predicts the visual evolution of new tasks, and action generalization, which translates these predictions into executable behavior through a video-to-action bridge. By varying video and action supervision for tasks outside an initial training set, we examine their respective contributions to task transfer. On RoboTwin 2.0, using only in-distribution video-action supervision yields a 25.07% success rate on out-of-distribution tasks. Introducing videos of these tasks without their action labels raises success to 55.90%, while adding just one action-labeled demonstration per task further improves it to 70.70%. These findings suggest that broader video coverage enables task transfer by leveraging existing inverse-dynamics capabilities, while even sparse action supervision further strengthens those capabilities. Guided by this perspective, we introduce WorldBridge, a WAM with separate video and action experts connected through structured attention and trained with joint flow matching. WorldBridge incorporates visual memory at two temporal scales and action forcing, alongside a unified data processing pipeline that standardizes and filters heterogeneous robot trajectories and egocentric hand-derived supervision. Across multiple manipulation benchmarks, WorldBridge achieves state-of-the-art performance among the compared methods, including 96.23% on RoboTwin 2.0 full data setting and 80.9% on clean to random setting, 82.8% on RoboCasa GR1 TableTop, and 57.6% on DOMINO Level-1. Our study highlights the complementary roles of video prediction and action grounding in developing robots that generalize across tasks.