OmniNWM: Omniscient Driving Navigation World Models

AI summary

Overview

Research area: Computer vision for autonomous driving — generative world models that simulate driving scenes, respond to control inputs, and supply reward signals for planning.

Technical level: Advanced. The paper assumes familiarity with diffusion transformers, latent variable models, Plücker ray coordinates, 3D semantic occupancy, and closed-loop evaluation for motion planning.

Scope in one sentence: OmniNWM is a single probabilistic driving world model that jointly generates panoramic RGB, semantic, and metric-depth video (and lifts them to 3D semantic occupancy), controls generation with normalized panoramic Plücker ray-maps, and derives dense intrinsic rewards from the generated occupancy to close the loop with a vision-language-action planner.

What This Paper Is About

Existing driving world models are fragmented: they usually generate only RGB video, they drift over long rollouts, and their camera control breaks when the sensor setup changes because motion is entangled with each rig's extrinsics. They also lack a built-in way to score planner behavior, typically borrowing external black-box reward models that suffer distribution shift.

OmniNWM's goal is to unify the three dimensions the authors identify as core — state, action, and reward — in one framework: a joint multimodal generator for state, a geometry-invariant control encoding for action, and occupancy-grounded dense rewards for reward, all running as a closed-loop simulation cycle.

Key Contributions

1. A unified State-Action-Reward framework. A Panoramic Diffusion Transformer (PDiT) jointly optimizes panoramic RGB, colorized semantics, and metric depth in a shared latent manifold, then lifts the pixel-aligned outputs into 3D semantic occupancy via a lightweight geometric mapping. The paper argues this avoids the inter-modal misalignment of modular "independent generation" designs and that the resulting occupancy — and therefore the reward — stays consistent with the visual observation.

2. Canonical geometric action encoding with normalized panoramic Plücker ray-maps. A parameter-free encoder maps trajectories into ray-maps, which are then rigidly transformed into a canonical reference frame (for example, the initial front view) so that motion dynamics are decoupled from sensor intrinsics and rig geometry. This is claimed to enable precise, zero-shot trajectory control across different datasets and camera configurations.

3. Structured panoramic forcing. During training, structured noise is injected along both the temporal (frame-wise drift) and spatial (inter-view inconsistency) dimensions, expanding the valid data manifold into a tubular neighborhood so the learned transition behaves contractively. The paper frames this as stochastic manifold thickening that mitigates exposure bias and autoregressive error accumulation.

4. A closed-loop simulation cycle with intrinsic occupancy-grounded dense rewards. Rewards are computed directly from the generated 3D occupancy volume using collision, drivable-area, and velocity terms, and are used to evaluate a Qwen-VL-based agent (OmniNWM-VLA) whose planned trajectory is re-encoded into ray-maps to condition the next generation step.

Main Findings

• Video generation quality: OmniNWM reports 5.45 FID and 23.63 FVD at 224×400 resolution with 17-frame clips on the nuScenes validation set, without heavy volumetric conditions such as voxel grids or point clouds. By comparison, UniScene is listed at 6.45 FID / 71.94 FVD and DiST-4D at 7.40 FID / 25.55 FVD.

• Camera control accuracy: Without canonical normalization, the model shows rotation error 1.71 rad and translation error 9.75 m. With normalization, errors fall to 0.16 rad and 1.18 m — reductions of 90.6% and 87.9%. The paper reports 1.18 m TransErr versus 7.56 m for UniScene on the 150-scene nuScenes validation set.

• Occupancy prediction: OmniNWM reaches 33.3 IoU and 19.8 mIoU on nuScenes-Occupancy, described as SOTA, outperforming generative baselines such as OccGen (23.4 IoU, 14.5 mIoU) and OccScene (12.2 mIoU), as well as camera-based and LiDAR-based discriminative methods.

• Panoramic depth: Abs. Rel. 0.23, δ<1.25 of 0.81, δ<1.25² of 0.93 — better than the generative baseline DiST-4D (0.39 / 0.58 / 0.81) and the discriminative baselines SurroundDepth (0.28 / 0.66 / 0.84) and M2Depth (0.26 / 0.73 / 0.87).

• Planning evaluation: OmniNWM-VLA achieves 87.3% Scenario Pass Rate across 150 nuScenes validation scenes at 12 Hz, surpassing Impromptu-VLA and Qwen-2.5-VL evaluated with identical textual prompts.

• Long-horizon stability: With no structured forcing (pure autoregression), FVD rises from 26.79 at 17 frames to 386.72 at 201 frames. Standard scheduled sampling reaches 178.65 FVD at 201 frames. Structured panoramic forcing keeps FVD at 25.22 at 201 frames, starting from 23.63 at 17 frames.

• Zero-shot generalization: Without fine-tuning, OmniNWM reports 9.51 FID, 79.24 FVD, 0.28 rad RotErr, and 1.65 m TransErr on 17-frame nuPlan clips, while baselines such as Vista, MagicDrive, UniScene, Epona, and DiST-4D show high trajectory errors (above 7 m TransErr). Qualitative results also cover 200-frame nuPlan and in-house generation and 3-view and 6-view camera rigs.

• Reward validation: Rewards derived from generated occupancy correlate with rewards derived from ground-truth occupancy and LiDAR+Box data at average Pearson correlations of r_GT = 0.96 and r_LiDAR = 0.94, addressing circularity concerns in closed-loop evaluation.

• Occupancy input ablation: Joint semantic and depth generation yields gains of 3.0 and 2.7 mIoU respectively over the ablated configurations, reaching the reported 33.3 IoU / 19.8 mIoU.

• Reactive interaction: The paper reports emergent interactive behavior — for example, a truck yielding when the ego vehicle cuts in — which it attributes to sampling from a learned equilibrium distribution rather than scripted heuristics.

Methodology in Plain English

The authors build one model rather than several. A pretrained 3D VAE compresses RGB, palette-colorized semantics, and metric depth into latents that are concatenated channel-wise into a single joint latent; a diffusion transformer denoises that joint latent so all three modalities stay pixel-aligned. Semantics are discretized back with nearest-neighbor matching to a fixed palette after decoding. A separate lightweight lifting module — an EfficientNet-B7 U-Net plus 3D convolutions and an outer product — turns the generated RGB, depth, and semantics into a 3D occupancy volume.

For control, the authors avoid feeding raw camera poses or waypoints. Instead they build Plücker coordinates (a 6-dimensional per-pixel ray representation) from every camera in the rig and then rigidly transform all rays into one canonical reference frame, so the representation describes motion relative to the rig's starting view rather than the rig's specific calibration. This is a parameter-free encoding injected as geometric constraints.

For training stability, they corrupt context latents with two independent noise terms — one varying across frames and one across views — instead of unstructured Gaussian noise, so the model learns to recover from the two dominant failure modes. This supports both frame-level autoregression (for precise planning) and clip-level autoregression (for efficient long generation).

For reward, the model queries the generated occupancy volume and computes a scalar from three dense penalties: collision (scaled by speed), driving off the drivable surface, and deviation from a target velocity, with weights 0.5, 0.3, and 0.2. These rewards score planner trajectories, and the chosen trajectory is re-encoded as ray-maps to drive the next generation step, forming the loop. The planner itself, OmniNWM-VLA, is built on Qwen-VL with a "Tri-Modal Mamba-based Interpreter" that fuses RGB, depth, and semantics before tokenization, and outputs dense (x, y, θ) trajectories at 12 Hz.

Training uses nuScenes and nuScenes-Occupancy on 48 NVIDIA A800 GPUs with batch size 48, AdamW, learning rate 1×10⁻⁴, and weight decay 0.01, in three stages: single-view (17 frames, 224×400) for 10k iterations, joint 6-view panoramic generation for 3k iterations, and variable-length (17 or 33 frames) and resolution up to 448×800 fine-tuning for 3k iterations.

Why This Matters

The paper argues that a world model is only useful for autonomy if it can predict states, accept actions, and score them — and it positions OmniNWM as the first to do all three in one probabilistic framework with intrinsic rather than external rewards. If the reported zero-shot cross-rig and cross-dataset generalization holds, it changes how such models can be deployed: one model per fleet rather than one model per sensor calibration.

Real-world applications:

• Simulation-based testing of planners: Generated panoramic videos with occupancy let planners be evaluated on rare or dangerous scenarios without on-road testing.
• Synthetic training data: Pixel-aligned RGB, semantics, and depth from one generator can augment perception training, especially for underrepresented camera rigs.
• Cross-platform deployment: A canonical action encoding could allow a model trained on one vehicle's camera layout to run on another with a different rig.
• Closed-loop policy assessment: Occupancy-grounded dense rewards give a quantitative signal for comparing driving policies in a generated environment.

Industry relevance is direct: autonomous driving companies spend heavily on simulation and data collection, and a model that generates aligned multimodal sensor data with controllable trajectories and built-in safety scoring addresses both costs. The paper's comparison table explicitly positions earlier systems such as Drive-WM as relying on external image-based reward models, while OmniNWM derives rewards intrinsically from generated 3D semantic occupancy.

Authors’ abstract

Autonomous driving world models are expected to work effectively across three core dimensions: state, action, and reward. However, existing methods are typically restricted to fragmented modality modeling, short-horizon drift, and imprecise action control, while lacking intrinsic mechanisms for policy evaluation. In this paper, we introduce OmniNWM, an Omniscient panoramic Navigation World Model that addresses all three dimensions within a consistent probabilistic framework. For State, OmniNWM generates panoramic videos of RGB, semantics, metric depth, and 3D occupancy, ensuring pixel-level alignment across modalities with joint distribution modeling. To mitigate autoregressive exposure bias, we propose a structured panoramic forcing strategy to stabilize long-horizon generation via stochastic manifold thickening. For Action, we introduce canonical geometric action encoding with normalized panoramic Plücker ray-maps. This representation decouples motion dynamics from sensor intrinsics, enabling precise, zero-shot trajectory control across heterogeneous datasets and camera configurations. For Reward, we derive intrinsic occupancy-grounded dense rewards directly from generated 3D volumes, establishing a reliable closed-loop simulation cycle for evaluating diverse planning agents. Extensive experiments demonstrate that OmniNWM achieves SOTA performance in generation fidelity and control precision, with remarkable zero-shot robustness to novel scenes on NuPlan and in-house datasets with distinct camera rigs. Project page is available at https://arlo0o.github.io/OmniNWM/.