OpenWAM: An Open Framework for Composable World-Action Models

AI summary

Overview

Research area: Robotics and robot learning, specifically world-action models (WAMs) that couple future video prediction with robot control, plus inverse and forward dynamics learning.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching generative models, transformer architectures, mixture-of-experts routing, vision-language-action (VLA) policies, and standard robot-learning benchmarks such as LIBERO.

Scope: The paper introduces an open framework, model family, and evaluation protocol built on a shared causal robot-video backbone, in which video-action interaction is configurable, and separately trained inverse and forward dynamics components can be composed with adapted video predictors.

What This Paper Is About

Existing world-action models vary their video backbone, interaction structure, supervision, and inference procedure all at once, so it is hard to tell which design choice causes which result. OpenWAM fixes a common causal robot-video foundation and a shared training and evaluation pipeline, then varies only the way future video and action streams interact, allowing joint, video-then-action, action-then-video, and decoupled generation to be compared directly. The second half of the paper asks a different question: when only the video predictor is adapted to a new task, do independently trained inverse and forward dynamics components remain useful, and does supervision from counterfactual (non-demonstrated) transitions help them transfer?

Key Contributions

1. A controlled world-action model framework. OpenWAM combines a causal video backbone pretrained on robot video without action labels with configurable video-action models, all under shared training and evaluation protocols.

2. Four video-action interaction programs in one architecture. A shared Mixture-of-Transformers (MoT) backbone supports Joint, video-then-action (VTA), action-then-video (ATV), and Decoupled generation, differing only in generation order and future cross-modal attention, while sharing the backbone, tokenization, and objective family.

3. Local-context dynamics interfaces with counterfactual supervision. Inverse (IDM) and forward (FDM) dynamics models are trained independently on demonstrations plus alternative action continuations branched from the same simulator states, and are evaluated as replaceable components composed with separately trained predictors.

4. Controlled empirical analysis. The paper measures the effects of robot-video pretraining, interaction design, transition coverage, and conditioning scope on both closed-loop policy performance and component reuse, in simulation and on a real bimanual robot.

Main Findings

• Robot-video pretraining plus causal adaptation gives a large LIBERO-Long gain. Holding the VTA architecture, task supervision, and downstream training fixed, the original Wan2.2-5B initialization reaches 68.4% success on LIBERO-Long, while the robot-video-pretrained causal backbone reaches 97.8% (+29.4 points). The paper states this measures the combined effect of pretraining and causal adaptation, not either factor individually.

• All four interaction programs perform strongly on LIBERO. Averaged over three random seeds: VTA 99.4 (Object), 98.4 (Goal), 98.6 (Spatial), 97.8 (Long), mean 98.6; Decoupled mean 98.0; Joint mean 97.6; ATV mean 96.8. Reported baselines include LingBot-VA (mean 98.5), Fast-WAM (97.6), Motus (97.7), OpenVLA-OFT (97.1), pi-0.5 (96.9), pi-0 (94.1), GR00T-N1 (93.9), and OpenVLA (76.5).

• Real-world bimanual control also succeeds. On three tasks (Toast Bread, Rubik's Cube, Sort Cups), VTA reaches 92.0, 90.0, 94.4 (mean 92.1) and Joint reaches 90.0, 94.0, 91.7 (mean 91.9).

• Counterfactual supervision makes a frozen local-context inverse model transferable. Across four held-out LIBERO-90 targets, a frozen local-context IDM trained on mixed supervision reaches 84.0% mean success (74, 86, 88, 88 per task) and counterfactual-only reaches 84.5% (80, 92, 90, 76). By contrast, a demonstration-only local-context IDM reaches 21.5% mean, a demonstration-only full-context IDM 44.5% (88, 90, 0, 0), and a mixed-supervision full-context IDM 47.0% (54, 92, 4, 38). Under matched mixed supervision, local conditioning improves target success by 37.0 percentage points.

• Transfer is uneven without counterfactual data. The demonstration-only full-context IDM performs well on LIBERO-90 Tasks 64 and 74 but obtains 0% success on Tasks 21 and 45, whereas counterfactual-only and mixed local-context IDMs remain effective across all four targets.

• Source-task performance is largely retained. On LIBERO-Long with the LIBERO-Long video predictor: full-context IDM demonstration-only (native VTA) 97.8%, full-context IDM mixed 92.2%, local-context IDM counterfactual-only 90.6%, local-context IDM mixed 94.4%, and local-context IDM demonstration-only 25.8%. Reference points are a task-tuned VTA finetuned policy at 93.0% mean on the transfer targets and the original unadapted VTA policy at 10.5% mean.

• Counterfactual supervision improves action-conditioned forward dynamics. Evaluated on 2,560 counterfactual futures from 160 contexts across ten LIBERO tasks with 16 action branches per context: RGB MSE drops from 14.35 (demonstration-only) to 9.40 (counterfactual-only, a 34.5% reduction) and 9.62 (mixed, 33.0%). Outcome identification at K=2 rises from 68.3% to 93.6% (counterfactual-only) and 91.8% (mixed); at K=16 it rises from 21.1% to 71.3% (counterfactual-only) and 67.7% (mixed).

• Interaction order matters less than the shared foundation. The four programs share the same initial checkpoint and training recipe, and their LIBERO means span 96.8 to 98.6, indicating multiple effective configurations rather than one specialized policy.

Methodology in Plain English

The team starts from a general-purpose video model, Wan2.2-5B, and continues pretraining it on over 10,000 hours of heterogeneous robot and human-interaction video without any action labels. Because the original model is not temporally causal, they introduce chunk-causal prediction during this stage: a future chunk of video may attend to the current observation and earlier chunks and previously generated chunks, but not to later chunks, while tokens within a chunk are denoised jointly.

They then attach an action branch using a Mixture-of-Transformers architecture. Video and action trajectories become two temporally aligned token streams processed by modality-specific transformer branches, which meet in cross-modal attention. The paper treats temporal causality and cross-modal interaction as separate design dimensions. By changing only the generation order and which future tokens can see which others, the same backbone yields four programs: Joint (video and action denoised together with bidirectional future attention), VTA (video first, then actions conditioned on the completed visual trajectory), ATV (actions first, then video conditioned on the completed action trajectory), and Decoupled (no future cross-modal attention). All use the same latent flow-matching objective, with teacher forcing during training for sequential programs and conditioning on generated trajectories at inference.

For the dynamics study, the researchers define local-context components that see only the current observation, proprioceptive state, and a supplied future trajectory, with no language, no past actions, and no observations before the current timestep. The inverse model maps a supplied visual trajectory to actions; the forward model maps a supplied action sequence to future observations. To broaden supervision beyond successful demonstrations, they restore selected simulator states and execute alternative future action sequences from those same states, producing counterfactual transitions. Training can then mix these with task demonstrations using a weight eta; the experiments use a 60% counterfactual / 40% demonstration mix (eta = 0.6). Simulator state is used only to control data collection and is not given to the model. At inference, a task-conditioned video predictor proposes a future trajectory and a frozen inverse model converts it into actions, or an action generator proposes actions and a frozen forward model predicts their visual consequences.

Evaluation uses closed-loop task success in four LIBERO suites and on a real bimanual platform (two Franka Research 3 arms with parallel-jaw grippers, three RGB views), plus RGB prediction error and outcome identification for forward dynamics, where a prediction must be closest in RGB MSE to its matching realized future among K same-state outcomes (K=2 and K=16).

Why This Matters

OpenWAM's main contribution is methodological rather than a single benchmark number: it isolates the variables that other world-action models change together, and it shows that an action-grounding component trained once can be reused after the visual predictor is adapted to a new task, provided that training included transitions beyond successful demonstrations. The finding that counterfactual transitions make a local-context inverse model transferable while a comparable full-context model does not (84.0% versus 47.0% mean under matched mixed supervision) is a concrete, testable result about conditioning scope and data coverage.

Real-world applications:

• Industrial and warehouse manipulation, where a visual predictor can be retargeted to new objects or layouts while a frozen action component continues to produce executable commands.
• Contact-rich assembly and deformable-object handling, the settings the paper's real-world tasks (toaster insertion, Rubik's Cube manipulation, cup sorting) approximate.
• Data-efficient robot deployment, where cheaper non-successful or exploratory rollouts substitute for additional expert demonstrations when training dynamics models.
• Candidate-action evaluation, using a forward dynamics model to predict the visual consequence of proposed actions before executing them.

Industry relevance: The framework, pretrained checkpoint, code, and project page are released openly (code at github.com/OpenWAM/OpenWAM, project page at openwam.stanford.edu), and the paper positions itself as a common testbed for comparing WAM interaction designs. That matters for teams choosing among joint versus sequential video-action generation, and for those wanting to swap action components without retraining the whole system.

Future Directions

• Real-robot composability. The paper states that composability is established primarily in simulation and that demonstrating comparable reuse on physical robots will require more data-efficient physical dynamics supervision.
• Beyond task-adapted video predictors. The transfer experiments adapt the video predictor, so the work does not establish zero-shot transfer of the complete video-action model.
• A single multi-objective model. Appendix E shows that policy, IDM, and FDM objectives can be trained jointly in one checkpoint, but the authors have not shown that one multi-objective model matches the strongest specialized OpenWAM variants across all modes.
• Long-horizon forward dynamics and deployment. The forward dynamics study focuses on local, short-horizon prediction, leaving long-horizon planning and physical deployment as natural extensions. The paper also notes that the same local-transition interface could use failed or exploratory physical-robot transitions when such data are collected.

Target Audience

Robotics and machine learning researchers working on world models, video-based prediction, vision-language-action policies, and imitation learning; engineers building configurable robot manipulation stacks who need to compare interaction designs under controlled conditions; and readers interested in how training data coverage beyond successful demonstrations affects the reuse of learned dynamics components. The paper is also relevant to those studying transfer and task adaptation in robot policies, since its central transfer experiment keeps action components frozen while adapting only the visual predictor.

Authors’ abstract

World-action models (WAMs) couple future prediction with robot control, yet existing systems often vary the video backbone, interaction structure, supervision, and inference procedure simultaneously, making their design choices difficult to compare. We introduce OPENWAM, an open world-action modeling framework built around a common causal robot-video foundation and configurable video-action interaction. Starting from Wan2.2-5B, we perform causal robot-video pretraining on over 10,000 hours of video, then integrate an action expert through a shared Mixture-of-Transformers architecture that supports joint, video-then-action, action-then-video, and decoupled generation. OPENWAM achieves high success rates on four LIBERO suites and real-world bimanual tasks; robot-video training with causal adaptation improves VTA success on LIBERO-Long from 68.4% to 97.8%. The same configurable architecture naturally extends to inverse and forward dynamics, allowing us to study how counterfactual transitions improve independently trained dynamics models beyond demonstrations alone. When only the video predictor is adapted to a new task, a frozen local-context inverse dynamics model trained on counterfactual data and demonstrations achieves 84.0% mean success across four held-out LIBERO-90 tasks, compared with 47.0% for a full-context inverse model and 21.5% for a local-context model trained only on demonstrations. For forward dynamics, counterfactual supervision reduces RGB prediction error by 34.5% and raises outcome identification from 21.1% to 71.3% among 16 same-state outcomes. OPENWAM provides a common testbed for comparing WAM interaction designs and for studying dynamics learning from video data beyond successful demonstrations.