SWAM: Scaling World Action Models with Cross-Modal Alignment

AI summary

Overview

Research area: Robot learning / computer vision — specifically world action models (WAMs), which jointly predict future video and robot actions conditioned on language, built on top of pretrained video generation models.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching transformers (DiT), mixture-of-transformers (MoT) architectures, vision-language-action (VLA) models, and manipulation benchmarks such as RoboTwin 2.0.

Scope (one sentence): An empirical study of how to combine ~47,000 hours of general video, egocentric video, UMI human demonstrations, and robot demonstrations — and how to design language and action supervision — so that language–video priors from video pretraining are preserved and extended into generalizable robot control.

What This Paper Is About

World action models start from a pretrained video model that already understands how language describes scenes and how scenes evolve over time, then add an action-prediction branch so the model can output robot commands. The paper argues that existing work treats this language–video alignment as a given, even though introducing action learning can damage it. The authors build a large multisource data pool and systematically test which kinds of language annotation, which kinds of action supervision, and which data mixtures and architectures best preserve language–video alignment while extending it to control.

Key Contributions

1. A curated multisource pretraining pool of approximately 47,000 hours (47,016.7 hours in the reported total), spanning 10,909.6 hours of general video (Panda-70M, OpenVid), 10,986.3 hours of egocentric video (EgoSuite, Ego4D, EgoVerse), 15,284.9 hours of in-house UMI demonstrations (411,754 trajectories), and 9,836.0 hours of robot demonstrations (768,356 trajectories retained from ABC, AgiBot, InternData-A1, Galaxea, RoboCOIN, RoboMind). The data, training code, and model weights are released.

2. A three-level empirical study of cross-modal alignment: language–video alignment (caption detail, scene/lighting information, temporal granularity, caption–video correspondence, instruction diversity), language–video–action alignment (how much UMI action-label coverage matters), and joint training (data-source mixing, sampling ratios, architecture and conditioning design).

3. A concrete WAM design: an MoT architecture with the video branch initialized from Wan2.2-TI2V-5B and a randomly initialized action branch, connected by shared masked attention that passes information unidirectionally from video to action, with text cross-attention present only in the video branch so that action prediction reuses pretrained language–video correspondences.

4. Strong generalization results under a Clean-only post-training recipe: 88.10% success on RoboTwin Easy (Clean-to-Clean) and 61.56% on Hard (Clean-to-Random), plus 40.00% success across three real-world tasks versus 23.33% for the strongest baseline.

Main Findings

• Richer captions improve control generalization. At subtask granularity, Hard success rose from 47.20% (Brief) to 49.60% (Medium) to 53.52% (Detailed); adding environment information reached 55.04%, the best Easy (83.92%) and Hard result in that group.

• Subtask-level beats episode-level language. Under Detailed descriptions, switching from Episode to Subtask descriptions raised Hard success from 51.60% to 53.52%.

• Caption–video correspondence matters. Randomly shuffling Detailed Subtask descriptions against their video segments within each trajectory dropped Hard success from 53.52% to 52.00%.

• Rephrasing and rewriting shared robot instructions helps. Across nine robot-data subsets containing 145,430 trajectories but only 592 task-level instruction entries, Original instructions scored 84.64% Easy / 50.80% Hard; Rephrased (100 paraphrases per instruction) reached 86.08% / 54.76%; Detailed + Env. per-trajectory descriptions reached 86.48% / 56.16%.

• UMI action labels scale usefully. Keeping all UMI video but varying only the fraction of trajectories carrying action labels, Hard success rose from 46.72% (0%) through 47.76% (25%), 49.92% (50%), 50.16% (75%) to 51.28% (100%) — a 4.56 percentage-point gain.

• Robot-data pretraining is far better than none. Direct post-training from Wan reached 34.56% Hard success, while pretraining on robot data reached 56.16%.

• General video pretraining adds further gains under a fixed budget. A UMI/Ego setup at 20k steps scored 82.64% Easy / 53.84% Hard; adding general videos and training to 45k steps kept Easy at 82.64% and raised Hard to 56.48%.

• RoboTwin 2.0 results. Post-trained only on Clean demonstrations, the final recipe reaches 88.10% (Easy, Clean-to-Clean) and 61.56% (Hard, Clean-to-Random) across the 50 tasks.

• Real-world generalization. On a bimanual Flexiv Rizon 4 platform across cup stacking, towel folding, and gift-in-box, the model reaches 40.00% SR under the Hard setting versus 23.33% for OpenWAM, the strongest baseline.

Methodology in Plain English

The authors start from a pretrained video generator and add a parallel "action branch" that predicts robot commands. The two branches keep separate residual streams but share attention, and the sharing is one-directional: the action branch can look at video information, but video learning never depends on the action branch. This design lets the huge amount of video-only data keep training the video side without requiring action labels. Language enters the video branch through cross-attention; the action branch gets language only indirectly, through its attention to the video branch, so it inherits whatever language–video understanding the video model already has.

Both branches are trained with flow matching — video latents and actions are noised and the model learns to predict the velocity that removes the noise — with the two losses weighted equally. All samples contribute to the video objective; only samples with valid action annotations contribute to the action objective, so general videos, egocentric videos, UMI demonstrations, and robot demonstrations can all be mixed into one pretraining run.

Data curation is a substantial part of the work. Videos are standardized into LeRobot 2.1 format, checked for encoding failures and low-quality content (using VLM-based review to drop animations, static scenes, TV programs, motion blur, and uninformative clips), long videos are segmented into subtasks, and captions are generated at three levels of detail (Brief, Medium, Detailed) plus scene and lighting annotations. Because the model modifies Wan's design, the video branch is first adapted on general video to limit disruption of pretrained priors. UMI and robot demonstrations additionally supply 16-dimensional bimanual end-effector state–action records (per hand: 3D position, 4D quaternion in wxyz order, 1D gripper), with quality checks and masks for invalid action labels.

Evaluation is deliberately harsh: models are post-trained on RoboTwin 2.0 Clean demonstrations only, then tested both under clean conditions and under randomized backgrounds, clutter, lighting, table height, and camera placement. Real-world testing likewise separates Easy (familiar objects moved) from Hard (new objects, new backgrounds, or both).

Why This Matters

The paper reframes WAM data work as a supervision-design problem rather than a data-volume problem. Its central practical finding is that detailed, accurately matched, subtask-level language descriptions measurably improve robot control generalization — a cheap intervention compared with collecting more robot teleoperation data. Its second finding is that handheld UMI human demonstrations, which can be collected at scale outside a lab, provide action supervision that transfers to robot control despite the visual and embodiment gap.

Real-world applications:

• Household manipulation: the demonstrated tasks (cup stacking, towel folding, gift-in-box, snack bagging) map directly onto domestic robotics.

• Scalable data collection pipelines: handheld gripper-plus-smartphone UMI capture makes it feasible to crowdsource manipulation experience rather than relying on costly robot teleoperation.

• Warehouse and logistics picking/placing, where background texture, lighting, and object appearance vary constantly — exactly the Hard-condition settings tested.

• Robustness testing standards: the Clean-to-Random protocol gives a repeatable way to measure whether a policy generalizes to unseen scenes rather than memorizing a fixed setup.

Industry relevance: the release of data, code, and weights, plus a recipe that trains a single model across 50 tasks and transfers to real hardware, lowers the barrier for teams building manipulation policies on video-pretrained backbones. The finding that detailed captioning of existing robot datasets (as few as 592 instructions across 145,430 trajectories) yields several points of gain is directly actionable for anyone sitting on a large, poorly annotated dataset.

Future Directions

• Scaling beyond the current budget. The abstract states that scaling pretraining under the selected configuration brings further gains, but the full scaling results and the architecture-comparison details are beyond the truncated portion of the paper; a natural next step is extending scale studies to much larger data and model sizes.

• Better use of video-only data. The unidirectional attention design already lets unlabeled video train the video branch without touching the action branch; how far this can be pushed, and whether the action branch can benefit more from large-scale action-free video, remains open.

• Narrowing the human-to-robot gap. UMI action labels help even though UMI data "differ visually from robot execution"; explicit view and embodiment alignment could plausibly increase that transfer.

• Beyond success rate on three tasks. Real-world evaluation covers three tasks and 30 trials per task per setting; larger, more varied task suites and finer metrics (the paper also reports partial success rate, PSR) would test whether the recipe holds broadly.

Target Audience

Researchers and engineers working on robot manipulation, VLA and world-action models, and video-generation backbones for embodied AI. It is most useful for those designing pretraining data mixtures and language-annotation pipelines, and for practitioners deciding whether human demonstration data (UMI) or additional robot teleoperation is the better investment. Readers without a background in diffusion/flow-matching transformers and manipulation benchmarks will find the experimental methodology accessible but the architectural details demanding.

Authors’ abstract

World action models jointly predict video and actions conditioned on language. Existing studies focus primarily on architectures and training mechanisms, typically treating language-video alignment acquired through video pretraining as an established foundation. However, introducing action learning may impair this alignment. Principled guidance on data and supervision design for maintaining language-video alignment during action learning remains limited. To investigate this,…