DriveLaW:Unifying Planning and Video Generation in a Latent Driving World

AI summary

Overview

Research area: autonomous driving world models, specifically the unification of video generation and motion planning in a shared latent space.

Technical level: Advanced. The paper assumes familiarity with diffusion models, rectified flow, VAEs, DiT (Diffusion Transformer) architectures, and closed-loop planning benchmarks.

Scope: DriveLaW is a "chained" world model in which a video generator's internal latents are injected directly into a diffusion planner, and the paper reports state-of-the-art single-view video generation on nuScenes and state-of-the-art closed-loop planning on NAVSIM.

What This Paper Is About

Recent driving world models can either synthesize future video or plan trajectories, but most treat these as two separate output streams inside one architecture, so the visual imagination never actually becomes the state the planner uses. DriveLaW addresses this by making the planner operate on the latent representation produced by the video generator itself, so that what the model imagines and what the model drives are grounded in the same internal features. The goal is a single system whose generated futures and planned trajectories are inherently consistent, without needing reinforcement learning or learned trajectory scorers at test time.

Key Contributions

1. A unified latent world model that shifts from parallel to chained generation and planning. The authors argue that latents learned from large-scale video generation carry better semantic coherence and spatial structure than traditional Bird's Eye View (BEV) features or Vision-Language Model (VLM) features, and they inject these generative priors into the planner.

2. A specialized two-part architecture: DriveLaW-Video, a spatiotemporal video generator with a novel noise reinjection mechanism aimed at structural inconsistency and blurring in high-speed scenarios, and DriveLaW-Act, a diffusion planner conditioned directly on video latents.

3. A three-stage progressive training strategy that resolves the optimization tension between high-fidelity video synthesis and reliable trajectory generation.

4. Empirical validation showing state-of-the-art FID and FVD for single-view video generation on nuScenes and a new record in closed-loop PDMS on the NAVSIM benchmark, achieved without post-training or auxiliary scorers.

Main Findings

• Video generation leads on nuScenes validation: DriveLaW reaches 4.6 FID and 81.3 FVD, compared with Epona at 7.5 FID / 82.8 FVD, Vista at 6.9 / 89.4, DriveWorld at 7.4 / 90.9, DrivingGPT at 12.8 / 142.6, DriveDreamer at 52.6 / 452.0, and DriveGAN at 73.4 / 502.3. The abstract states this surpasses the previous best-performing work by 33.3% in FID and 1.8% in FVD.

• New planning record on NAVSIM Navtest: DriveLaW attains 89.1 PDMS, with NC 99.0, DAC 97.1, TTC 96.7, Comfort 100, and EP 81.3. This improves over the parallel generation–planning baseline Epona (86.2 PDMS) by 2.9 PDMS, over DriveVLA-W0 (87.2) by 1.9, and over PWM (88.1) by 1.0. It also exceeds traditional end-to-end planners such as DiffusionDrive (88.1), ReCogDrive-IL (86.5), WoTE (88.3), UniAD (83.4), TransFuser (84.0), PARA-Drive (84.0), VADv2 (80.9), Ego Status MLP (65.6), and Constant Velocity (20.6).

• No post-training or post-processing required: The reported PDMS is achieved without reinforcement learning or learned scorers.

• Open-loop nuScenes planning also improves: DriveLaW records average L2 error of 1.15 m and average collision rate of 0.24%, versus Epona at 1.25 m and 0.36%. Per horizon, DriveLaW is 0.44 / 1.10 / 1.91 m at 1 s / 2 s / 3 s and 0.15 / 0.10 / 0.48% collision, against Epona's 0.61 / 1.17 / 1.98 m and 0.01 / 0.22 / 0.85%. The 1-second collision rate is therefore higher for DriveLaW (0.15% versus 0.01%), while 2-second, 3-second and averaged collision rates are lower.

• Video pretraining scales into planning gains: PDMS rises from 85.9 with no driving-domain pretraining (scratch), to 87.0 at 76k samples, 87.8 at 3.8M, and 89.1 at 7.6M — a +3.2 PDMS improvement for the fully pretrained generator.

• Video latents beat other representations: Under the same diffusion planner, video latents give 89.1 PDMS versus 86.5 for VLM hidden states and 84.1 for BEV features, an improvement of 5.0 over BEV and 2.6 over VLM.

• Early denoising steps are the best conditioning signal: Conditioning the Action DiT on the video latent at denoising step t=1 yields 89.1 PDMS, t=5 yields 86.9, and t=10 collapses to 23.2 (with NC 81.7, DAC 63.4, TTC 67.6, Comfort 0, EP 15.4).

• The three-stage curriculum matters: Removing the first stage gives FID 5.0 but FVD 109.3; removing the second stage gives FID 5.0 and FVD 93.2; the full strategy gives 4.6 FID and 81.3 FVD.

Methodology in Plain English

The system is built from two chained pieces. The first, DriveLaW-Video, is a 2B-parameter video DiT initialized from LTX-Video pretrained weights. It compresses video into a causal latent space with a spatiotemporal VAE at 32×32×8 resolution and 128 channels, a 1:192 compression ratio (1:8192 pixel-to-token), which is more compact than the typical 1:48 or 1:96 used elsewhere. The VAE encoder uses 3D causal convolutions so each timestep only sees the past and present. Instead of finishing all denoising in latent space, the final rectified-flow step at t=t1 is decoded and refined in pixel space, which restores high-frequency detail such as highlights, dynamic shadows, and fine road textures without a separate super-resolution module. The transformer is adapted from PixArt-α, uses self-attention for spatiotemporal modeling and cross-attention for conditions like navigation commands, applies RMSNorm to queries and keys, and uses Rotary Positional Embedding with normalized fractional coordinates to reduce spatial drift over long horizons. Ego kinematics are converted into structured natural-language motion prompts rather than numeric encodings.

The noise reinjection mechanism targets degradation from large motion: at each denoising step the model predicts a clean latent, decodes it to a temporary image, converts to grayscale, and applies a discrete Laplacian kernel to get a high-frequency response map. An adaptive threshold (a tuned multiple of the response standard deviation) defines a mask, which is downsampled to latent resolution by nearest-neighbor interpolation. Noise is injected only inside that mask before the next denoising step, forcing the model to regenerate detail rather than smooth it away while leaving smooth regions like sky untouched.

The second piece, DriveLaW-Act, is a 133M-parameter diffusion planner. It encodes noised actions, ego status and high-level commands, and conditions on latents cached from the Video DiT's first denoising step; those video features serve as keys in the planner's cross-attention. Training uses a flow-matching objective.

Training proceeds in three stages: long clips at low resolution (740×352×121) to learn motion and temporal coherence; shorter clips at high resolution (1280×704×25) to sharpen spatial detail; then trajectory fine-tuning where the planner is conditioned on the frozen video latents. The first two stages run 30k iterations at batch size 4, learning rate 1e-5, weight decay 5e-2. The trajectory stage uses batch size 192 for 44k steps, learning rate 3e-5, weight decay 1e-5, feeding the past four camera frames and supervising 2 Hz trajectory points over the next 4 seconds. Inference uses 30 sampling steps for video and 5 for trajectories.

Data: training uses 8 Hz camera video from nuScenes (1,000 scenes from Boston and Singapore; 850 for training and validation, 150 for testing) and NuPlan (1,200 hours of human driving data from four cities), with planning evaluated at 2 Hz on NAVSIM, which builds on OpenScene and splits into Navtrain (103k samples) and Navtest (12k samples). Evaluation uses FVD and FID for video, and PDMS — combining no-at-fault collision, drivable-area compliance, ego progress, time-to-collision and comfort — for planning.

Why This Matters

Impact on research: The paper challenges the common design in which a world model generates video and a separate head or module plans, showing that the generator's own mid-denoising features are a stronger planning representation than BEV or VLM features. It also provides evidence of a scaling relationship between video pretraining data volume and closed-loop driving performance, and it demonstrates that strong planning scores can be obtained without reinforcement learning or learned trajectory scorers.

Real-world applications:

• Training and validating autonomous driving policies in simulation, where a world model generates rare or dangerous scenarios.
• Generating synthetic driving video to supplement scarce long-tail data for perception and planning teams.
• On-vehicle or offline trajectory planning that must stay consistent with what the perception stack predicts will happen next.
• Closed-loop benchmark evaluation of AD stacks where trajectories are scored on safety and comfort rather than only displacement error.

Industry relevance: The work is a collaboration between Huazhong University of Science and Technology and Xiaomi EV, and the code is released at a Xiaomi Research repository, indicating direct relevance to a production autonomous driving effort. Its focus on removing the need for post-training or scorer modules addresses practical deployment concerns about latency and complexity in the planning stack.

Future Directions

• The ablations show a sharp collapse when conditioning on late denoising steps (t=10 yields 23.2 PDMS); understanding and stabilizing the choice of latent timestep across scenarios is an open question.
• The paper reports scaling of planning quality with video pretraining sample count up to 7.6M; whether the trend continues at larger corpora is not established here.
• DriveLaW is evaluated on nuScenes and NAVSIM; generalization to other cities, sensor configurations, and closed-loop simulators used in industry is not reported.
• DriveLaW-Video is evaluated as single-view generation; extension to multi-view or multi-modal (LiDAR, occupancy) generation, which other world models target, is not explored.
• The three-stage curriculum requires separately staged resolution and frame-count settings (740×352×121 then 1280×704×25); simplifying this schedule is not addressed.

Target Audience

Researchers and engineers working on autonomous driving world models, end-to-end planning, and video diffusion, particularly those interested in representation coupling between generative models and control. It is also relevant to readers tracking latent-space conditioning strategies, progressive training curricula for video diffusion, and closed-loop planning benchmarks such as NAVSIM. Readers without background in diffusion models and driving benchmarks will find the method sections difficult, as no beginner-level primer is provided.

Authors’ abstract

World models have become crucial for autonomous driving, as they learn how scenarios evolve over time to address the long-tail challenges of the real world. However, current approaches relegate world models to limited roles: they operate within ostensibly unified architectures that still keep world prediction and motion planning as decoupled processes. To bridge this gap, we propose DriveLaW, a novel paradigm that unifies video generation and motion planning. By directly injecting the latent representation from its video generator into the planner, DriveLaW ensures inherent consistency between high-fidelity future generation and reliable trajectory planning. Specifically, DriveLaW consists of two core components: DriveLaW-Video, our powerful world model that generates high-fidelity forecasting with expressive latent representations, and DriveLaW-Act, a diffusion planner that generates consistent and reliable trajectories from the latent of DriveLaW-Video, with both components optimized by a three-stage progressive training strategy. The power of our unified paradigm is demonstrated by new state-of-the-art results across both tasks. DriveLaW not only advances video prediction significantly, surpassing best-performing work by 33.3% in FID and 1.8% in FVD, but also achieves a new record on the NAVSIM planning benchmark.