Learning Transferable World-Action Models from Task-Paired Human Videos

AI summary

Overview

Research area: Robot learning / world-action models — specifically, how to pre-train video-based robot policies on human video instead of costly robot data.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching video models, action-chunked imitation learning, and cross-embodiment transfer.

Scope: The paper proposes WATCH, a cross-video pre-training objective in which a world-action model predicts one human demonstration's actions and future frames while conditioned on a different demonstration of the same task, and measures what this buys before and after robot post-training.

What This Paper Is About

World-action models learn to predict both actions and their visual consequences by watching video, but standard pre-training predicts the next frames within a single demonstration. Because the context and target share the same scene, camera, and demonstrator, the model can lean on appearance rather than learning the interaction that actually transfers across scenes and embodiments. The authors ask whether forcing the model to predict one demonstration from another — using only task-paired human videos, with no matched human–robot data — produces representations that transfer better to robot control.

Key Contributions

1. WATCH pre-training objective. A cross-video formulation where the model is conditioned on a reference human demonstration (its video and actions) and must predict a different demonstration's action chunk and future frames from that demonstration's current observation and state. The reference is built from task-matched human-to-human pairs, so embodiment and environment variation is supplied without any paired robot data.

2. Pairing pipeline over human video. Query–reference pairs are constructed from different episodes of EgoVerse: human actions are derived from body configurations and expressed in a shared human–robot Cartesian task space, candidates are gated by a semantic task taxonomy, and then ranked by dynamic-time-warping distance between motion trajectories. This yields 10.99M query–reference pairs (10.0K hours of paired 3-second windows), with 10.52 references per query on average, including 6.35 cross-source matches; 90.7% of queries have at least one cross-source reference. Selected matches show higher trajectory-shape cosine similarity by 0.36 than same-task random cross-source pairs.

3. Analysis of what the objective learns before robot post-training. Sensitivity tests, motion retrieval, held-out-embodiment regression, and attention visualizations show the shift from source-specific appearance toward task-relevant motion and hand–object interaction.

4. Downstream evaluation across backbones and protocols. Gains are shown with two backbones (Cosmos Policy on Cosmos-Predict2.5, and Cosmos3 Edge-2B), two post-training protocols (in-context and context-free), and three benchmarks (MimicDroid, TableTop-Sim, RoboCasa Kitchen-H50).

Main Findings

• In-context robot success rises substantially on MimicDroid. With Cosmos Policy, average success goes from 40.9% to 68.3% (L1 75.0% → 91.7%, L2 29.4% → 72.5%, L3 18.3% → 40.8%). With Cosmos3, average goes from 59.8% to 77.7% (L3 30.8% → 55.0%). The largest gains are on L2 and L3, the levels that shift appearance/objects and hold out the layout.

• The prior survives without a reference at test time. Under context-free post-training, MimicDroid improves from 49.3% to 69.0% with Cosmos Policy and from 49.7% to 63.4% with Cosmos3; TableTop-Sim (bimanual ALOHA) improves 59.6% → 65.4% and 65.8% → 72.0%; RoboCasa Kitchen-H50 (single-arm Franka, 50 human demonstrations per task) improves 56.0% → 63.6% with Cosmos Policy (a 7.6-point gain) and 55.0% → 55.8% with Cosmos3, which already saw robot data.

• Conditioning is more sensitive to task than to source. Across 137 held-out episodes, action MSE rises from 0.0322 to 0.0335 when the reference comes from a different collection source, and to 0.0367 when the reference is a different task from the same source; only the task change is significant after Holm correction.

• Cross-embodiment motion alignment improves. On 359 held-out EgoVerse windows and 396 RoboMIND windows from three robots, WATCH retrieves matching motions from different human sources and different robots at earlier ranks than both the original video model and the within-video baseline. A regressor trained on two embodiments and tested on the third predicts nine future-motion quantities at Spearman correlation 0.18 (human) and 0.24 (robot), against 0.06 and 0.01 for the original video model.

• Better prediction on unseen robots. On RoboMIND, which no model saw during pre-training, WATCH lowers action MSE from 0.046 to 0.039 and raises acc@0.1 from 55.6% to 68.5% relative to the within-video baseline; FVD improves to 154.9 and FID to 8.33, versus 290.7 / 12.48 for the original video model and 1485.1 / 53.53 for within-video pre-training.

• Human pairs approximate robot-pair representations. WATCH trained on 1K hours of human pairs reaches mean match@1 of 0.76 on human windows and 0.60 on robot windows against a counterpart trained on 1K hours of RoboMIND robot pairs with the same objective, versus 0.21 and 0.19 for the original video model.

• Joint action–video supervision matters. Comparing action-only, video-only, and joint supervision under the same cross-video pairing, only joint training concentrates both action-slot and video-slot attention on the acting hand and manipulated object.

• The objective beats within-video pre-training and co-training. On in-context MimicDroid averages, none 40.9%, within-video 51.2%, co-training 60.0%, same-source top-1 58.6%, WATCH 68.3%. With references replayed at 2× speed (faster than the robot can follow), within-video pre-training loses its whole gain (30.3% versus 30.7% with no pre-training), while WATCH retains most of its own (54.1%). WATCH also leads on all three context-free benchmarks.

• It scales with paired human video. With 25% of the cross-video data, WATCH matches within-video pre-training on the full dataset on the unseen in-context levels; it surpasses that baseline at 50% on L3 and 75% on L2, and exceeds it on context-free L2 at around 50%.

Methodology in Plain English

Start with a world-action model: a video model that has been extended so that, alongside predicting future frames, it also predicts the actions that produced them. Normally you train it by showing it a video, giving it the current frame, and asking for the next frames and the actions in between. The catch is that the model can cheat — the future looks a lot like the present, so it can copy appearance instead of learning the physical interaction.

The authors change the setup. They give the model two clips of the same task recorded by different people in different rooms with different cameras. One clip is the "query": the model sees its first frame and state. The other is the "reference": the model sees that person's whole clip and their actions. The model must then produce the query's actions and its future frames. Since the reference's background, lighting, and body are useless for painting the query's future, the only thing worth transferring is the interaction — what the hand does to the object, and in what order.

To build these pairs at scale, they take human egocentric video from EgoVerse, convert body motion into a common end-effector-style action space shared with robots (a 57-dimensional vector covering head, left/right end-effector, and left/right hand), filter candidate pairs by task label, and then rank by how similar the motion trajectories are. Each query gets up to 8 matches from its own source and up to 7 from other sources, and one is sampled at each training step. The training loss is the same joint flow-matching objective as the base recipe (action term weighted at λ = 48), applied only to the generated slots.

For robot deployment, they try two things: in-context post-training, where the robot prediction is still conditioned on a human reference, and context-free post-training, where the reference slots are simply removed and the model behaves like an ordinary behavior-cloning policy.

Why This Matters

Impact on research. The paper isolates a specific flaw in the dominant video pre-training recipe — within-clip next-frame prediction lets appearance shortcut the learning signal — and offers a cheap fix that needs no robot hardware. It also gives evidence that human–human pairs induce representations close to robot–robot pairs, which reframes human video as a pre-training substrate rather than only as auxiliary co-training data.

Real-world applications.

• Lowering the cost of building generalist manipulation policies for humanoids and arms, since paired human video is far cheaper to collect than teleoperated robot trajectories.
• Faster adaptation to new factory or warehouse layouts, where the largest gains in this paper occur (L2 and L3: changed appearance/objects and held-out layouts).
• Learning from demonstration in homes and kitchens, where robot data collection is impractical but egocentric human video is abundant.
• Reference-driven task specification, where a human shows a task once and the robot attempts it — the in-context setting evaluated here.

Industry relevance. Robot foundation model teams currently spend heavily on teleoperation. A pre-training stage that consumes only human video, plugs into existing video backbones (Cosmos-Predict2.5, Cosmos3), and works under both in-context and standard behavior-cloning post-training is directly applicable to production pipelines.

Future Directions

• Extending beyond Cartesian action spaces. The method depends on a shared Cartesian end-effector space for pairing, action priors, and warm-starting the robot action head. Joint-space policies are not demonstrated; the authors note this would need a Cartesian intermediate target or a learned joint-to-end-effector map.
• Pairing from less structured video. Pair construction currently requires task labels and hand tracking. Label-free gating and pose estimation would be needed to use unlabeled or hand-pose-free video.
• Combining human and robot cross-video data. The authors show cross-video prediction also applies to robot pairs, and leave open whether robot data helps more entering through the same cross-video objective than alongside it as in co-training.
• Further scaling. The scaling curve has not saturated — every level is still rising at the full corpus — so more paired human video from more collection sources is the first lever, along with extending pairing beyond a single human dataset.

Target Audience

Researchers and engineers working on robot foundation models, video world models, and imitation learning — particularly those building pre-training pipelines and deciding how much robot teleoperation data is actually necessary. Readers without background in flow-matching video models or action-chunked policies will find the empirical results accessible but the methodological sections dense.

Authors’ abstract

World-action models inherit broad visual and physical priors from video pre-training. However, standard within-video next-frame prediction allows them to rely on appearance shared between the context and future, rather than learning interactions that transfer across scenes and embodiments. We introduce WATCH, a cross-video pre-training that learns from task-paired human demonstrations collected across diverse environments. Conditioned on the video and actions of one demonstration, the model jointly predicts another demonstration's actions and future frames from its current observation. Predicting one demonstration from another encourages the model to learn task-relevant interactions that transfer across scenes and demonstrators. Our pre-training shows that these capabilities can be learned from readily scalable human-human pairs alone, without requiring matched human-robot demonstrations. Analyses before robot post-training show improved motion alignment across embodiments and better action and video prediction on unseen robots, with representations closer to those learned from robot pairs. After robot post-training, WATCH improves MimicDroid in-context success from 40.9% to 68.3% with Cosmos Policy and from 59.8% to 77.7% with Cosmos3, with the largest gains on the unseen levels. Gains persist under context-free post-training across robot embodiments. WATCH outperforms within-video pre-training and co-training baselines, supporting cross-video prediction as a scalable approach to learning transferable world-action representations.