LaLa-World: Scaling Latent World Models with Natural Language Actions

AI summary

Overview

Research area: Computer Vision / robot learning — specifically latent world models (JEPA-style) for planning and policy learning.

Technical level: Advanced. The paper assumes familiarity with joint-embedding predictive architectures, model predictive control, diffusion-based action heads, and vision-language-action models.

Scope (one sentence): The paper proposes LaLa-World, a latent world model that uses natural language as a shared action interface so that a single future predictor can be trained jointly on heterogeneous control trajectories and ordinary image/video data, and shows this helps both MPC planning and downstream policy fine-tuning.

What This Paper Is About

Latent world models predict how visual representations will change in response to actions, which makes them cheap to roll out for planning — but their action-conditioned predictors are almost always trained on one simulation or robotics dataset at a time, because numerical action vectors have domain-specific dimensions and physical meanings (the paper's example: [2, 4] can mean horizontal/vertical displacement in one domain and [4] a force in another). Image and video data, which is abundant, usually has no numerical action labels at all, so it cannot be used to train these predictors. The paper's goal is to remove this bottleneck by expressing both numerical actions and visual changes in natural language, so one predictor can learn across many domains and data sources at once.

Key Contributions

1. A language interface for actions and visual change. LaLa-World converts numerical actions from seven simulation and robotics datasets into natural-language templates (e.g., "move 2 left and 4 up and", "push with force 2 downward"), and pairs image/video transitions with caption or editing-instruction text, so control data and video data share one conditioning format.

2. A predictor that is itself a language model. Rather than bolting on a separate text encoder, the future predictor is built on a pretrained language model (Qwen3.5-0.8B by default), which reads the language condition as text and predicts future latent representations in the same network, reading outputs from learned query tokens.

3. A joint-trained LWM that beats per-domain LWMs in MPC planning. A single predictor trained across all domains achieves the highest average planning score in both benchmark groups, exceeding existing LWMs that were trained separately per domain or task.

4. A matched latent-vs-generative comparison and a policy variant. Against an RAEv2-based generative world model baseline matched on vision encoder, language conditions, and training data, LaLa-World reaches comparable or better planning performance at lower training and planning cost; the pretrained model is then fine-tuned into LaLa-Policy, which jointly predicts action chunks and future latent representations.

Main Findings

• Best average MPC planning among LWMs on simulation tasks. LaLa-World reaches 72.1% average success across Wall, PointMaze, Push-T, MetaWorld-R, and MetaWorld-RW, versus JEPA-WMs at 64.6% (a 7.5 percentage-point gap) and DINO-WM at 62.3%. Per-task, it scores 95.8 on Wall, 56.2 on PointMaze, 81.2 on Push-T, 75.0 on MetaWorld-R, and 52.1 on MetaWorld-RW.

• Largest single-task gain is on Push-T. Success rises from 68.8% (JEPA-WMs) to 81.2%.

• Highest zero-shot scores on the held-out RoboCasa tasks. 40.6% on Reach and 46.9% on Place; RoboCasa is excluded from training.

• Best average on the second benchmark group. LaLa-World averages 50.5% across Rope (66.7), Granular (58.3), RoboCasa-R (40.6), RoboCasa-P (46.9), and DROID (40.2), versus JEPA-WMs at 44.8% and DINO-WM at 36.8%. V-JEPA 2-AC scores 42.9% but is not evaluated on all five (no Rope or DROID entries).

• Image and video data help both accuracy and robustness. Adding image/video transitions raises the average score from 59.3 (control trajectories alone) to 61.6 (all data). When action templates are paraphrased at evaluation (e.g., "move the gripper" → "shift the robot hand"), paraphrasing drops the average by 6.8 points with control-only training, while the all-data model remains robust.

• Planning works from qualitative language. With magnitudes replaced by "slightly", no modifier, or "far", the predictor still produces successful planning examples in Wall and RoboCasa even when exact magnitudes are unspecified.

• Faster training than the generative baseline. Under a 12-hour budget on eight H100 GPUs, LaLa-World reaches the 12-hour performance of five-step RAEv2 in two hours, and later reaches 59.4% planning performance versus RAEv2's best of 48.0%.

• Faster planning. Full-depth LaLa-World plans 2.9× faster than full-depth RAEv2 with one denoising step and 13.9× faster with five denoising steps. The abstract summarizes this matched comparison as comparable planning performance at 6× less training cost and 13.9× faster planning.

• Policy fine-tuning is competitive and much faster at inference. LaLa-Policy averages 97.7% on LIBERO (above Cosmos Policy's 97.3%, π0.5's 96.9%, and CogVLA's 97.4%) and 66.4% on RoboCasa with 300 demos (above Video Policy's 66.0%) and 57.3% with 50 demos (above Cosmos Policy's 56.0%). Average inference time per trajectory is 0.86 s vs 4.45 s on LIBERO (5.2×) and 2.12 s vs 11.71 s on RoboCasa (5.5×).

• World-model pretraining is a useful initialization. Initializing the predictor from LaLa-World beats both random initialization and Qwen3.5 initialization on LIBERO and RoboCasa, extending Cosmos Policy's finding about video-model priors to latent world models.

• No pixel reconstruction objective is required. The model is trained purely on squared error between predicted and target future representations; a 12-layer transformer decoder (trained separately with L1 and LPIPS losses, following RAEv2's decoder recipe) is used only for visualization, so decoded images may show visual inaccuracies.

Methodology in Plain English

The setup has two parts: making the data speak one language, and building a model that understands it.

Making the data uniform. Every numerical action in the control datasets is rewritten through fixed templates into a short sentence that preserves direction and magnitude. Separately, ordinary video clips from OpenVid (with captions) and image-editing pairs from OmniEdit and ShareGPT (with instructions) are turned into transition examples where the caption or edit instruction is the "action" describing the visual change. All of these become the same kind of training example: past observations, a language condition, and the next observation.

The model. A frozen vision encoder (DINOv2-S/14 or DINOv3-L/16) turns each frame into a latent vector. Those latents are projected through a two-layer MLP, given 3D positional embeddings indexed by time and 2D spatial location, and fed into a language model together with the tokenized language condition. Learned query tokens read out the predicted next latent, which is trained to match the encoder's representation of the actual next frame. Proprioceptive state is deliberately excluded, both as input and target, because it is defined differently in each domain.

Planning. Given a start observation and a goal observation, model predictive control with the Cross-Entropy Method samples candidate action sequences, verbalizes them with the training templates, rolls each one forward through the predictor, and scores it by the squared distance between the final predicted latent and the encoded goal. The best sequence is executed (in simulation, the corresponding numerical actions are recovered and run), and the loop repeats where re-planning is needed.

Policy. For LaLa-Policy, learned action queries are added alongside the existing future-prediction queries. Their outputs become conditioning features for a diffusion-transformer action generator (the Isaac-GR00T action-head architecture), which denoises an action chunk from Gaussian noise. The frozen encoder stays fixed while the pretrained predictor and new policy modules are trained jointly on LIBERO and RoboCasa demonstrations.

Why This Matters

Impact on research. The paper argues that visually convincing generation is not a prerequisite for a useful world model: matching future latents can support planning and policy learning while being cheaper to train and dramatically cheaper to query. It also provides what the authors describe as the first matched comparison of latent and generative world models for planning with the same encoder, language conditions, and training data — filling a gap where earlier speed comparisons differed in action interface, encoder, predictor size, and training data. And it shows a way to absorb unlabeled video into world-model pretraining, which was previously closed off to action-conditioned latent models.

Real-world applications (plausible extensions of the demonstrated capabilities):

• Robotic manipulation in homes and kitchens, where the paper already evaluates zero-shot on RoboCasa Reach and Place tasks.
• Warehouse and logistics picking, where fast per-trajectory inference (0.86 s on LIBERO, 2.12 s on RoboCasa) matters for throughput.
• Natural-language robot instruction, since the model accepts both precise commands with numbers and vague ones like "move slightly left".
• Simulation-style interactive environments for games and driving, which the paper's related work identifies as domains where language-conditioned world models already act as interactive simulators.

Industry relevance. The efficiency angle is the practical one: planning 13.9× faster than a diffusion-based alternative and running policies at up to 5.5× the speed of Cosmos Policy reduces the compute needed for both deployment and experimentation, and the model's dependence on an 0.8B-parameter predictor and a small language-model backbone keeps it in a modest size class.

Future Directions

• Scaling data and model size. The authors list scaling training data and model size, extending prediction horizons, and evaluating policies on real robots as future work.
• Testing the latent-vs-generative comparison further. The current matched study covers one generative baseline (RAEv2) and one planning protocol; it is an open question how the conclusions hold under other generative world models or longer horizons. The authors note that evaluating RAEv2 with 50 denoising steps would require approximately 25 days on 8 H100 GPUs, which bounds how thorough such comparisons can be.
• Reducing reliance on fixed action templates. The converted control data uses fixed templates, which the paper itself identifies as exposing the predictor to a narrow range of language — motivating the video/image data, but leaving room for richer action phrasing.
• Improving how predicted futures are read out. The decoder is trained separately and is not part of world-model training, so decoded predictions can contain visual inaccuracies; better readout could make the imagined futures easier to inspect and trust.

Target Audience

Researchers and engineers working on world models, model-based reinforcement learning, robot manipulation, and vision-language-action policies — particularly those interested in JEPA-style latent prediction, in efficient alternatives to video-generation-based world models, and in cross-domain pretraining. Readers without background in latent predictive architectures or MPC will find the method sections dense, though the core idea (put actions into language so one model can learn from everything) is easy to grasp.

Authors’ abstract

This paper shows that JEPA-style latent world models (LWMs) can learn across diverse domains and data sources by adopting the language-conditioned interface of video generation models. We establish this approach as an effective and efficient basis for planning with imagination and downstream policy fine-tuning. To this end, we introduce LaLa-World, a Language-conditioned Latent World model that learns from simulation and robot trajectories together with broader image and video data. Two designs make this work. First, we describe both numerical actions and visual changes in language, enabling joint training of a single future predictor across heterogeneous datasets. Second, we build the predictor on a language model, allowing it to interpret language conditions while predicting future latent states. The resulting predictor achieves higher average MPC planning performance than existing LWMs trained separately for each domain or task. Adding image and video data further improves average planning performance and robustness to action phrasing. In a matched comparison with an RAEv2-based video generation model baseline, LaLa-World reaches comparable planning performance with $6\times$ less training cost and $13.9\times$ faster planning. Beyond planning, the pretrained LaLa-World serves as an initialization for LaLa-Policy, which jointly predicts actions and future visual representations. LaLa-Policy performs competitively with Cosmos Policy on LIBERO and RoboCasa, with up to $5.5\times$ faster inference.