Orbis 2: A Hierarchical World Model for Driving

AI summary

Overview

Research area: Computer Vision — generative world models for autonomous driving (video prediction, representation learning, action-conditioned simulation).

Technical level: Advanced. The paper assumes familiarity with flow matching, diffusion forcing, teacher forcing, DINOv2 features, VQGAN tokenizers, DiT/ST-Transformer backbones, and linear-probing evaluation protocols.

Scope: This paper presents Orbis 2, a two-level hierarchical driving world model that separates long-horizon abstract forecasting from short-horizon high-fidelity frame generation, and introduces a diffusion-forcing pretraining plus teacher-forcing fine-tuning recipe.

What This Paper Is About

Most driving world models predict the future at a single level of abstraction, generating frames one at a time while optimizing for visual fidelity rather than the spatial and semantic understanding needed for downstream tasks. The authors build a model that splits this job in two: an abstract predictor that forecasts coarse scene structure far into the future, and a detail predictor that renders realistic short-horizon frames conditioned on that forecast. They also investigate how the training objective shapes the quality of the model's internal representations, and show that pretraining with diffusion forcing followed by teacher forcing yields both richer representations and more stable long rollouts.

Key Contributions

1. A hierarchical driving world model. A two-level architecture in which a high-level predictor operates in an abstract latent space at coarse temporal resolution, and a low-level predictor generates high-fidelity frames in a pixel-aligned latent space conditioned on the high-level prediction.
2. A two-stage training paradigm for predictive representation learning. Pretraining with a diffusion forcing objective (all frames independently corrupted and denoised) followed by short teacher-forcing fine-tuning (next-frame prediction from clean context). The paper reports this improves representation quality over next-frame prediction alone and is independent of the hierarchical architecture.
3. A compressed DINO latent for the abstract level. Rather than predicting directly in full 768-dimensional DINOv2 feature space, the authors learn a 16-dimensional abstract latent aligned to frozen DINOv2 embeddings with a weak reconstruction regularizer (λ_rec = 0.1), which they report is necessary for stable long-horizon generation with flow matching.
4. State-of-the-art results across complementary axes. Long-horizon perceptual fidelity (FVD), long-horizon stability (a newly introduced FVD-slope metric over chunked FVD), representation quality (linear probing on Cityscapes segmentation and KITTI depth), and steering responsiveness on counterfactual trajectories.

Main Findings

• Best long-horizon stability, competitive absolute FVD. On 6-second rollouts, Orbis 2 reports FVD of 98.97 on nuPlan, 98.01 on Waymo, and 187.53 on nuPlan-turns. Cosmos-v2.5 (Predict-2B, post-trained, Video2World) achieves lower FVD on the nuPlan benchmarks (67.74 and 126.26) but is 2× larger and trained on over 3× more driving data from an undisclosed mix; Orbis 2 outperforms it on Waymo (98.01 vs 116.12).
• Best FVD-slope. Orbis 2 achieves FVD-slope of 46.45 on nuPlan-turns over four 4-second chunks, versus 93.89 for Epona, 99.67 for Orbis, 189.34 for Cosmos-v2.5, 258.00 for GEM, and 434.67 for Vista.
• Strongest internal representations. Linear probing gives Orbis 2 a segmentation mIoU of 58.29 on Cityscapes and depth RMSE of 4.157 on KITTI, best among the compared driving world models (Vista 39.76 / 4.884, GEM 45.81 / 4.373, Orbis 52.39 / 4.321, Epona 35.81 / 5.720, Cosmos-v2.5 43.20 / 6.384). Frozen DINOv2-Base still scores higher on probing (60.01 mIoU, 3.890 RMSE).
• Representation quality correlates with rollout stability. The authors report that models with stronger semantic and geometric probing results also achieve more stable long-horizon prediction, suggesting representation quality may matter more than perceptual fidelity alone for world modeling.
• Hierarchy design matters, and the best configuration is specific. In the design study, the chosen setup — abstract top predictor at 2 fps over detailed bottom predictor at 10 fps — achieves 49.39 mIoU / 4.498 RMSE for the top branch and FVD 209.15 with FVD-slope 47.87. Alternatives fail in characteristic ways: a flat detail-level model drifts (FVD 170.02, FVD-slope 101.63), a flat abstract model is stable but detail-poor (FVD 490.04, FVD-slope 48.38), a detail-latent top predictor diverges out-of-distribution (FVD-slope 81.40), and an abstract top predictor at 10 fps loses motion (FVD 551.38, FVD-slope 137.07).
• Diffusion forcing improves representations; teacher forcing stabilizes rollouts. Ablation on the abstract branch: teacher forcing only gives 55.01 mIoU / 4.355 RMSE; diffusion forcing only gives 58.03 / 4.274; the combined DF + TF recipe gives 58.29 / 4.157, exceeding the abstract tokenizer (57.19 / 4.574).
• Compression is required for stable abstract prediction. Predicting in the full 768-dimensional DINO space degrades progressively over the horizon, whereas the 16-dimensional compressed latent supports stable long-horizon generation even with small models.
• Best steering responsiveness on counterfactual trajectories. With average displacement error computed using VGGT as an external inverse dynamics model on 100 six-second nuPlan-turns samples (500 generations), Orbis 2 achieves 1.20 m on original trajectories and 2.18 / 0.53 / 1.65 / 0.73 m on speed ×0.5, speed ×1.5, yaw rate ×0.5, and yaw rate ×1.5 counterfactuals. Epona is slightly better on speed ×0.5 (2.08 vs 2.18).
• Fast inference and moderate compute. The 1.07B-parameter model runs at 3.64 FPS with 19 GB VRAM — at least 5× faster than all baselines — and was trained in under 6k H100-GPU-hours (6 × 10²¹ FLOPs at 25% MFU).
• Better on frequent driving classes, worse on rare ones. Class-wise probing shows gains over DINOv2-Base on road (+1.02), sidewalk (+4.42), sky (+4.89), vegetation (+2.73), and car (+2.13) mIoU, with losses on train (−20.57), bus (−8.30), motorcycle (−7.38), and rider (−5.84).
• Hierarchy beats a single mixed-abstraction branch. A 512M single-branch model with token factorization reaches 47.16 mIoU, 4.680 RMSE, FVD 184.69, FVD-slope 96.55; the hierarchical 257M + 279M configuration reaches 49.39, 4.498, 209.15, and FVD-slope 47.87.

Methodology in Plain English

The system learns two separate ways of describing a driving frame. The first is a fine-grained latent produced by a VQGAN-style tokenizer encoder (86M-parameter ViT encoder, 91M-parameter CNN decoder) that keeps pixel-level detail. The second is a compact 16-dimensional abstract latent produced by an encoder trained to match frozen DINOv2 features through a projection head, with a weak pixel-reconstruction loss (weight 0.1) to keep visual information without hurting semantic alignment.

A high-level predictor then works only in the abstract space, at 2 Hz with a 6-frame (3-second) window — 5 frames of input, 1 frame predicted — conditioned on the current ego action. Its output is a subgoal. A low-level predictor works in the detailed space at 10 Hz with a 10-frame (1-second) window — 5 frames of context, next 5 frames generated — conditioned on that subgoal. At inference the two roll out autoregressively, with sliding windows of 1 frame for the high-level predictor and 5 frames for the low-level one. Both predictors are Spatio-Temporal Transformers (ST-Transformer, extended from the DiT block): the backbone is 512M parameters, the detail predictor adds 43M via adaLN spatial conditioning, totaling 555M, for 1067M parameters overall. An EMA of the weights (decay 0.9999) is used at inference.

Training has several stages. The tokenizers are trained first. The high-level predictor is pretrained with a diffusion forcing loss — every frame in the sequence gets its own independent noise level and the model predicts velocities for all frames jointly — then fine-tuned with a teacher forcing loss that predicts only the last frame from clean context. The low-level predictor is trained with teacher forcing conditioned on ground-truth abstract latents during training and predicted ones at inference. The predictors are first pretrained unconditionally, then fine-tuned for steering conditioning by adding an embedded steering vector to the timestep embedding and feeding it through adaptive layer normalization, applied only to the top-level branch.

Data: both levels are trained for a single epoch over 5,890 hours of video (OpenDV 1124h, BDD100k 659h, PhysicalAI-AV 1633h, NATIX 2374h); the tokenizers use 2.6 million frames curated from BDD100K, OpenDV, Honda HAD, Honda HDD, ONCE, nuScenes, and nuPlan. Steering conditioning uses a 500h subset of the NVIDIA PhysicalAI AV dataset with IMU-derived planar trajectories. Evaluation uses linear probing for segmentation (Cityscapes mIoU) and depth (KITTI RMSE), FVD on Waymo, nuPlan and nuPlan-turns (800 nuPlan sequences, 400 for Waymo and nuPlan-turns), chunked FVD over consecutive 4-second windows, the new FVD-slope scalar, and ADE on trajectories estimated by VGGT.

Why This Matters

The paper argues that a world model's internal understanding of scene structure and geometry is as important as how realistic its generated frames look, and it provides a concrete architectural and training recipe for improving both. It also introduces FVD-slope as a way to measure whether video quality decays over long rollouts — a failure mode that short-horizon FVD can hide. The finding that diffusion forcing pretraining improves representations, combined with the observation that stronger representations track with more stable rollouts, gives the field a reusable training insight rather than only an architecture.

Real-world applications that this type of model could support, as implied by the paper's framing of world models for planning, simulation, and policy learning:

• Closed-loop simulation for testing and validating autonomous driving policies in counterfactual scenarios that did not occur in the logged data.
• Data augmentation and synthetic scenario generation for training perception and planning systems, including rare or under-represented situations.
• Action-conditioned rollout for planning, where a model predicts how the scene evolves under candidate steering commands before committing to one.
• Cheap, long-horizon evaluation: the model's fast inference (3.64 FPS, 19 GB VRAM) makes large-scale rollout-based testing more practical than with the compared baselines.

Industry relevance: driving world models are central to autonomous-vehicle simulation and validation pipelines. The paper's explicit efficiency comparison — 1.07B parameters trained in under 6k H100-GPU-hours, at least 5× faster inference than all baselines — is aimed at making this class of model affordable to train and deploy. The hierarchical design also separates the cheap abstract forecast from the expensive frame generator, which is relevant for real-time or resource-constrained deployment.

Future Directions

• Understanding why diffusion forcing helps. The authors state that the reason for the representation improvement is not fully studied; they hypothesize that independent per-frame noise levels act as implicit data augmentation and that different noise patterns make the model learn related tasks such as denoising, interpolation, and forecasting, but they do not isolate these factors. They leave a detailed analysis of noise schedules and corruption patterns to future work.
• Tightening the link between abstraction levels. The hierarchy uses separate abstract and fine-grained latent spaces, each optimized with a different objective. The authors note this separation clarifies each level's role but limits interaction between levels, and call improving the connection between the two spaces an important direction.
• Extending beyond a single camera. The study uses a 2,374-hour subset of front-camera data from the NATIX collection, and the authors state they plan to extend the approach to multi-camera data.
• Improving rare-class representation. The predictor outperforms DINOv2-Base on frequent driving classes but lags by 5 to 20 mIoU points on rarer classes such as bus, train, rider, and motorcycle — an open gap in the learned abstract latent.

Target Audience

Researchers and engineers working on video generation, driving world models, and embodied AI, especially those interested in hierarchical architectures, flow-matching objectives, and training-objective design for representation quality. It is also relevant to practitioners building autonomous-driving simulation and validation pipelines who care about long-horizon rollout stability, action controllability, and inference cost. Readers need a working background in diffusion/flow-matching models, transformer backbones, and representation-probing evaluation to follow the method and results tables closely.

Authors’ abstract

Current world models operate at a single level of abstraction, with most prioritizing perceptual fidelity while lacking the spatial reasoning and semantic understanding required for real-world downstream tasks. We present a hierarchical driving world model that factorizes future prediction across two levels operating at distinct temporal and abstraction scales: a high-level predictor that forecasts coarse scene structure over extended temporal horizons, and a low-level generator that produces detailed predictions conditioned on the high-level output. This decomposition yields high perceptual fidelity while also capturing strong spatial and semantic representations. We further show that pretraining with a diffusion forcing objective yields substantially richer internal representations than the standard teacher forcing objective, while teacher forcing -- predicting only the next frame from clean context -- produces more stable autoregressive rollouts. We therefore introduce a generic two-stage training paradigm that pretrains the model with diffusion forcing and fine-tunes with teacher forcing, combining the representational benefits of the former with the rollout stability of the latter. Our approach achieves state-of-the-art results across the standard suite of driving world model evaluations on established benchmarks, including long-horizon generation fidelity, steering responsiveness evaluated on counterfactual scenarios, and internal representation quality. Project page with code, demo, checkpoints and qualitative results: https://lmb-freiburg.github.io/orbis2.github.io/