Vision-Language-Action or World-Action? Just Action Models

AI summary

Overview

• Research area: Robot learning, specifically the design of control policies built on pretrained visual models (vision-language-action models, world-action models, and representation-based supervision).
• Technical level: Intermediate (the core idea is simple to state, but the paper assumes familiarity with flow-matching action experts, VAE latents, and joint-embedding/predictive representation learning; the appendices are Advanced).
• Scope: A single paper that argues the VLA-versus-WAM debate conflates three separable design choices and proposes one shared training recipe — an action loss plus a "foresight" loss on frozen future representations — that is evaluated on LIBERO, LIBERO-Plus, and RoboTwin 2.0.

What This Paper Is About

Two research camps disagree about what a robot control policy should inherit from pretrained models: vision-language-action (VLA) models adapt vision-language representations with an action loss, while world-action models (WAMs) start from image or video generators and additionally learn to predict future observations. The paper argues that this comparison bundles three independent choices — how the backbone is initialized, what objective is applied to recorded future observations, and which modules are kept at inference — so a gap between a VLA and a WAM cannot be attributed to any one of them. The goal is to show that learning from recorded futures does not require a visual generative objective at all, and that the same "foresight" recipe works across generative, self-supervised, and vision-language backbones.

Key Contributions

1. A unifying recipe (JAM). The authors propose Just Action Model, a pretrained visual backbone plus an action expert trained with an action loss and a foresight loss, where recorded future observations enter only as frozen prediction targets and the supervision branch is discarded after training.
2. Fixed-architecture controlled comparisons. Replacing the generative losses of FastWAM and ImageWAM with foresight, while holding architecture, data, optimizer, training budget, and deployment path fixed, preserves or improves success — and future-offset targets beat current-offset targets.
3. A cost argument. With cached foresight targets, JAM reduces estimated online training FLOPs per sample per update by 73.0% relative to FastWAM and 36.8% relative to ImageWAM.
4. A cross-family instantiation. The same objective is applied to generative (Wan, Ovis-U1/Yak, FLUX.2), self-supervised (DINOv2-B/L), and vision-language (Qwen3.5-2B) backbones, with the strongest generative and vision-language instantiations reaching comparable control performance.

Main Findings

• Foresight can replace generation. For FastWAM, swapping video denoising for foresight on pretrained Wan targets moves LIBERO from 97.00% to 97.45% and LIBERO-Plus from 55.31% to 55.58%. For ImageWAM, the swap moves LIBERO from 92.90% to 96.75% and LIBERO-Plus from 65.27% to 79.95%.
• Where you read out matters more than the objective swap. For JAM (Wan), moving from offset [0] (current frame) to future-only offsets [1,2] raises LIBERO-Plus from 44.62% to 55.58%, and adding the current position back ([0,1,2]) lowers it to 54.18%. For JAM (Ovis-U1), LIBERO-Plus rises from 69.65% at [0] to 79.95% at [1].
• The best future target is architecture-dependent. JAM (Wan) performs best with the frozen VAE tokenizer alone (64.54% on LIBERO-Plus), and even a randomly initialized Wan anchor reaches 58.63%. JAM (Ovis-U1) performs best with the pretrained Yak anchor (79.95%), versus 76.43% for the VAE tokenizer and 72.51% for random Yak.
• Pretrained backbone weights matter; the anchor's pretraining matters less. With a pretrained Wan backbone, pretrained and random Wan anchors reach 97.45% and 97.65% on LIBERO; with a random backbone they reach 71.15% and 84.40%.
• The recipe transfers across backbone families. Under the same action-plus-foresight objective, JAM (Ovis-U1) reaches 96.75% / 79.95% (LIBERO / LIBERO-Plus), JAM (Qwen3.5-2B) 96.55% / 81.14%, JAM (DINOv2-B) 95.50% / 68.19%, and JAM (DINOv2-L) 96.30% / 70.39% — all above ImageWAM (Ovis-U1) at 92.90% / 65.27%.
• Adding foresight helps a VLA. For Qwen3.5-2B, adding foresight raises LIBERO-Plus success from 77.03% to 81.14% with architecture and training settings fixed.
• Headline benchmark movement versus FastWAM. Success rates go from 97.0% / 55.3% / 91.2% to 98.2% / 64.5% / 91.4% (LIBERO / LIBERO-Plus / RoboTwin 2.0).
• RoboTwin 2.0 remains close. JAM (FLUX.2) has the highest listed score at 93.1%; JAM (Wan) 91.4% versus FastWAM 91.2%; LingBot-VA, which uses embodied pretraining, leads JAM (Wan) by 0.8 points. JAM (Qwen3.5-2B) trails JAM (FLUX.2) by 3.30 percentage points.
• LIBERO is competitive but not dominant. JAM (Wan) reaches 98.2% versus FastWAM's 97.0%, while π0.5 and LingBot-VA, both with embodied pretraining, stay 0.3 points ahead of JAM (Qwen3.5-2B) and JAM (Wan).
• LIBERO-Plus shows the largest gains. JAM has the best average in all three groups; it exceeds FastWAM and ImageWAM (Ovis-U1) by 9.2 and 14.7 points and trails them only in the background and robot-state categories, respectively.
• Training cost falls. Estimated FLOPs per sample per update drop from 15.52 TFLOPs (FastWAM) to 4.19 (JAM (Wan)), and from 5.81 TFLOPs (ImageWAM (Ovis-U1)) to 3.67 (JAM (Ovis-U1)). Measured GPU time on one GH200 falls by 67.3% and 37.8% in the same two comparisons.
• Probing suggests future information becomes more accessible. Two MLP readouts reconstruct the current frame and the frame 16 steps ahead from frozen policy features; JAM trained with offset [1] recovers the future change more clearly in the shown example and achieves the highest mean future-frame PSNR among compared policies.
• Attention shifts with the objective and backbone. Averaged over 100 reference frames, DINOv2-L gives its largest attention share to image keys, JAM (Ovis-U1) to text keys, and Qwen3.5-2B to action keys.

Methodology in Plain English

The authors start by taking apart the VLA-versus-WAM comparison into three knobs: what pretrained model seeds the visual backbone, what loss is applied to the recorded future frames, and what actually runs at test time. Then they build a recipe that keeps the first and third knobs free while simplifying the second.

In JAM, the policy is a pretrained visual backbone plus an action expert trained with flow matching, exactly as in the baselines. On top of the action loss, a small prediction head takes features of the current observation from the middle of the backbone and tries to match features that a frozen model — the "future anchor" — extracts from recorded future frames. The match uses a cosine-distance loss with gradients stopped on the target side. Recorded futures never become inputs to the action path, no image or video is ever generated during training, and the prediction head plus anchor branch are thrown away after training. Because the targets are fixed, they can be computed once and cached, which is where the training-cost savings come from.

The evaluation is deliberately staged. First, they take FastWAM and ImageWAM and change only the objective — swapping the generative loss for the foresight loss while keeping architecture, data, optimizer, training budget, and deployment path identical — and sweep which temporal offset is supervised and which frozen model supplies targets (pretrained backbone copy, randomly initialized copy, or the VAE tokenizer alone). Second, they hold the recipe fixed and plug in backbones from three families: a video generator (Wan), an image generator (Ovis-U1/Yak, FLUX.2), a self-supervised image encoder (DINOv2-B/L), and a vision-language model (Qwen3.5-2B). Everything is trained and tested in simulation on LIBERO, LIBERO-Plus (a fixed 1,506-task perturbed subset), and RoboTwin 2.0. Supervision is applied at roughly 60% of backbone depth (block 18 of 30 for Wan, Yak layer 11 of 18, layer 13 of 24 for Qwen3.5), with the foresight weight λ_F set to 0.6 for Wan and 0.3 for the others.

Why This Matters

Impact on research. The paper reframes a live architectural debate as a set of separable variables and shows that the expensive, generator-dependent part of WAM training — the visual generative loss and its decoder/denoising machinery — is not what produces the gains. That gives the field a cheap baseline that any backbone can adopt, and it means comparisons between VLA and WAM families need to control for target offsets and anchor choice before attributing differences to pretraining. It also reports negative-ish evidence honestly: a randomly initialized future anchor can still work, while a randomly initialized policy backbone is clearly worse, which sharpens the question of what pretraining actually contributes.

Potential real-world applications (the paper demonstrates these only in simulation):

• Robotic manipulation policies for tabletop and drawer/bowl-style tasks, where the LIBERO and LIBERO-Plus suites are proxies.
• Bimanual manipulation, as covered by the 50 RoboTwin 2.0 tasks under Clean and Randomized settings.
• Deployment settings where training compute or serving latency is the binding constraint, since the foresight branch is removed at inference and cached targets cut online training cost.
• Reusing existing pretrained assets — video generators, image editors, self-supervised encoders, or vision-language models — when a team already has one and does not want to stand up a video-generation training pipeline.

Industry relevance. The 73.0% reduction in estimated online training FLOPs per sample per update (and 67.3% measured GPU-time reduction on one GH200 versus FastWAM) is a direct cost argument for teams training policies on limited hardware. The recipe also avoids a video decoder or denoising loop during both fine-tuning and deployment, which simplifies the serving stack.

Future Directions

• Isolating pretraining effects. The authors explicitly note that architecture and action-interface differences prevent the cross-backbone comparison from isolating the effect of pretraining alone; a controlled study of backbone pretraining is left open.
• Explaining the architecture-dependent target preference. The paper offers a hypothesis — that Wan's strided window latents already trace scene change whereas Yak encodes only the 16-step endpoint — but does not test it.
• Choosing target offsets in principle. Future-offset targets beat current-offset targets consistently, yet the current position can still carry future context through bidirectional attention; how to select readout offsets and layers systematically is unresolved.
• From simulation to physical robots. All results are in simulation, and the ethics statement flags unsafe deployment of simulation-trained policies without physical validation as a relevant risk.

Target Audience

Researchers and engineers working on robot manipulation policies and pretrained-model adaptation will get the most from this paper, particularly those deciding whether to invest in video-generation-style world models or in lighter representation-level supervision. It is also useful for practitioners who already own a vision-language or self-supervised backbone and want a training objective that does not require a generator. Readers new to the area will need background in flow-matching action heads and joint-embedding prediction to follow the objective and ablations.

Authors’ abstract

The vision-language-action (VLA) and world-action model (WAM) communities debate whether control policies should inherit pretrained knowledge from vision-language models or from visual generators. The debate, however, bundles three separable choices: backbone initialization, the objective applied to future observations, and whether the machinery for that future loss is kept at inference. A gap between a VLA and a WAM therefore cannot simply be attributed to any one of them. We show that decoupling these choices yields a common framework for both families, which we term Just Action Model (JAM): a pretrained visual backbone with an action expert, trained with an action loss and a foresight loss. The foresight loss trains current-observation features to predict frozen encodings of the recorded future through a prediction head discarded after training, so no visual generation is involved in either training or inference, and the backbone can be chosen freely. We first compare objectives within fixed policy architectures, then evaluate the foresight recipe across backbone families. The resulting JAM policies outperform their matched FastWAM and ImageWAM counterparts on LIBERO and LIBERO-Plus, and achieve similar or slightly higher success rates on RoboTwin. Compared with FastWAM, JAM improves the success rates from 97.0%/55.3%/91.2% to 98.2%/64.5%/91.4%, respectively. With cached foresight targets, JAM reduces estimated online training FLOPs per sample per update by 73% relative to FastWAM. Our results show that future supervision improves control without visual generation or additional inference machinery, and strong foresight performance is attainable with either generator-initialized or vision-language backbones. Project Page: https://y0uroy.github.io/JAM