Astronex-World 1.0: Real-Time Interactive World Model Foundation

AI summary

Overview

• Research area: Computer vision — generative video world models, controllable video diffusion, and real-time interactive simulation.
• Technical level: Advanced (assumes familiarity with diffusion/flow-matching models, Transformer attention, and autoregressive generation).
• Scope: A technical report describing Astronex-World 1.0, a 5-billion-parameter open video world-model foundation built on the Wan2.2-TI2V-5B prior that supports camera-, action-, and event-conditioned video generation in both a bidirectional and a causal, real-time-streaming form.

What This Paper Is About

World models aim to predict how a scene will evolve given observations, camera motion, and actions — the same capability an autonomous vehicle or robot needs to reason about what happens next. Pretrained video diffusion models are not naturally suited to this: they attend to the whole clip at once, cannot be told how the camera moves or what actions are taken in an executable way, and degrade when generated frames are fed back into their own context. This paper describes a foundation model that bridges those gaps by converting a 5B video diffusion prior into a controllable, causally-decodable, streamable world model, and reports benchmarks showing it competes with models two to four times larger.

Key Contributions

1. A shared 5B controllable video backbone released in two forms — a bidirectional model for high-quality offline generation and supervision, and a causal model with block-causal attention and cross-block KV caching for persistent, few-step interactive generation. Both support text-to-video and image-to-video.
2. A unified control interface — PRoPE injects full camera intrinsics and extrinsics into all 30 Transformer layers; a 64-dimensional continuous action stream plus an embodiment ID modulates every visual layer; a timed text-event interface inserts changes mid-rollout; and a post-trainable action-sequence output head is reserved for embodied AI and driving.
3. A five-stage training recipe (bidirectional control adaptation → block-causal teacher forcing → online UniPC trajectory distillation → mixed-domain causal SFT → asymmetric DMD/DMD2 with motion preservation) that runs entirely on two NVIDIA L20 48 GB GPUs — a notably modest budget.
4. Efficiency-focused evaluation showing a 5B model post-trained on two L20s lands in the WBench Full score range of 13.6B–22B competitors, while streaming in real time on a single GPU.

Main Findings

• Real-time streaming on one GPU: The causal model generates 832×480 video at 24 fps using eight-step UniPC, an eight-latent-frame block, a 20-frame local history window, and four persistent sink frames — streaming on a single NVIDIA L20 48 GB.
• Benchmark scores: 73.5 on WBench Navi (navigation-only, 158 cases) and 70.0 on WBench Full (289 cases, including event editing, subject action, and perspective switching). Interaction drops sharply on Full (47.6 vs. 63.4) because the current adapter exposes only one timed text-switch point.
• Favorable scale comparison: On WBench Full, the 5B model surpasses the 13.6B LongCat-Video and 14B Helios, sits within one point of the 22B LTX-2.3, and beats YUME 1.5 — a model post-trained from the same 5B Wan2.2 prior on A100 GPUs.
• Events stay local: Because blocks before the switch are already cached and never recomputed, an inserted event changes frames after it by a mean of 9.5/255 in RGB while leaving pre-switch frames nearly untouched (1.8/255). Events are therefore a feature of the causal form only.
• Long-horizon stability: Block-causal attention, a sliding local KV window, persistent initial sink frames, and camera-frustum-overlap retrieval (enabled only for revisit patterns like loops and zigzags) mitigate the motion decay, geometric drift, and color shift typical of autoregressive rollout.
• Motion preservation in distillation: A dedicated motion term comparing mean adjacent-frame latent change between generated and real clips prevents the DMD objective from collapsing to a low-motion conditional mean.
• Camera recovery matters: Adding the action branch weakened camera responsiveness, requiring a dedicated 200-update recovery step (producing the "restore-2200" checkpoint) before the model was usable for both controls.

Methodology in Plain English

The authors did not train a video model from scratch. Instead, they started from Wan2.2-TI2V-5B, an existing open text-and-image-to-video model, and taught it new behaviors in sequence.

First they attached two new control paths using low-rank adaptation and newly initialized modules: a camera path that encodes where the viewpoint is and how it moves (via PRoPE, which turns camera matrices into a positional transformation inside attention), and an action path that turns a 64-dimensional action vector plus a robot/vehicle identity into per-layer modulation of the network. They trained this on camera-control data and real robot trajectories, then repaired the camera response afterward.

Second, they converted the model from looking at the whole video at once to only looking backward. The video is chopped into blocks of latent frames; within a block, frames can see each other, but across blocks, attention only reaches previously generated history. Finished blocks get cached so they are never recomputed. This conversion was done by training a teacher-forced version on clean history.

Third, because the causal model was slow to sample, they distilled a dense 25-step sampler trajectory into a 12-step student using online distillation, then fine-tuned on a mix of camera, first-person game, driving, and robot manipulation data to restore subject motion. Finally, they applied distribution matching distillation (DMD/DMD2) — using the frozen bidirectional model as the "real" teacher and an online-updated fake-score network — to make few-step outputs match the teacher distribution, adding a motion-preservation term. The causal model is evaluated; the bidirectional checkpoint is both a released product and the training teacher.

Why This Matters

Impact on research: The paper demonstrates that a mid-sized open video prior can be converted into a controllable, streamable world model with a modest two-GPU training budget, and that it can match models several times larger. This lowers the compute barrier for world-model research and provides an open, checkpointed foundation — including reserved action input/output interfaces — that others can post-train on driving or robotics data rather than building from scratch.

Real-world applications:

• Autonomous driving: Action- and camera-conditioned rollouts let developers simulate ego motion, traffic events, and road dynamics for planning and scenario testing.
• Robotics and embodied AI: The reserved 64-D action output head supports inverse-dynamics or action-sequence decoding on real robot trajectories (the paper trains on DROID and NVIDIA PhysicalAI data).
• Interactive content and games: Timed event prompts combined with persistent KV caching allow a single image-to-video start to be extended into a long interactive session rather than restarting each turn.
• Simulation and synthetic data generation: Controllable camera trajectories and event insertion enable generating labeled video for training perception systems, including rare or unsafe scenarios.

Industry relevance: The efficiency claim is the headline for practitioners — real-time interactive generation on a single 48 GB GPU means deployment costs are plausible for product teams, not just research labs. Open weights, a GitHub repository, and a Hugging Face release make the foundation directly usable and modifiable, positioning it as infrastructure for embodied AI and autonomous driving pipelines.

Future Directions

• Closing the interaction gap: WBench Full interaction scores (47.6) trail navigation-only scores (63.4). The adapter currently supports only one timed text-switch point and merges multiple events; supporting multiple sequential timed events and a richer action vocabulary is a clear next step.
• Improving few-step quality: Four-step generation is supported but less sharp and stable than eight-step, so the distillation budget and DMD node schedule remain open to improvement.
• Post-training the reserved action interface: The action-sequence output head, event branch, and embodiment conditioning are designed for downstream domain adaptation on driving and robot data, but this report does not demonstrate those post-training results.
• Scaling and generalization: Whether the five-stage recipe holds at larger backbone sizes, and how it performs across more diverse domains than the four training datasets, remains untested.

Target Audience

Researchers and engineers working on video generation, world models, embodied AI, and autonomous driving simulation. It is most valuable to readers who already understand diffusion/flow matching and Transformer attention, and who want either a reproducible training recipe for controllable video generation or a deployable open checkpoint for post-training on their own robotics or driving data. Readers seeking an introductory explanation of world models will find the paper dense, as it is written as a technical report rather than a tutorial.

Authors’ abstract

We present Astronex-World 1.0, an open controllable video world-model foundation. Given a text prompt (text-to-video) or an initial observation (image-to-video), the model predicts future visual states under frame-aligned camera trajectories, continuous actions, and an embodiment identifier, and accepts text events inserted at a specified position of a rollout. The family provides a bidirectional model for full-context generation and a causal model with block-causal attention and cross-block KV caching for persistent generation, both built on the Wan2.2-TI2V-5B prior. PRoPE injects camera intrinsics and extrinsics, while a 64-dimensional action stream modulates every Transformer layer. A five-stage training path develops bidirectional camera and action control, converts the backbone to block-causal generation, distills a few-step student, restores mixed-domain dynamics, and applies asymmetric DMD/DMD2 distribution matching. The causal model generates 832x480 video at 24 fps. All five training stages run on two NVIDIA L20 48 GB GPUs, and the causal model streams in real time on one. It scores 73.5 on WBench Navi and 70.0 on WBench Full. On Full, this 5B model is above the 13.6B LongCat-Video and the 14B Helios, within one point of the 22B LTX-2.3, and above YUME 1.5, which is post-trained from the same 5B prior on NVIDIA A100 GPUs. The reserved action input and output interfaces allow post-training for embodied intelligence and autonomous driving.