LaGen: Towards Autoregressive LiDAR Scene Generation

AI summary

Overview

Research area: Computer Vision, specifically 3D/4D LiDAR scene generation for autonomous driving world models.

Technical level: Advanced. The paper assumes familiarity with latent diffusion models, range-view LiDAR representations, autoregressive sequence modeling, and standard autonomous driving datasets and benchmarks.

Scope: LaGen is an autoregressive latent diffusion framework that generates long-horizon, frame-by-frame, interactive LiDAR scenes from a single initial frame, using bounding boxes and ego states as control conditions.

What This Paper Is About

Existing LiDAR generation methods mostly produce isolated single frames or non-interactive 4D sequences, while LiDAR prediction methods require many frames of history and emit all future frames deterministically along a fixed trajectory. Neither paradigm supports the long-horizon, decision-reactive simulation that closed-loop autonomous driving needs, where only an initial frame is available and the ego vehicle's actions change from step to step. LaGen addresses this by generating each new LiDAR frame autoregressively from the previous one, conditioned on bounding boxes and ego-vehicle state, so that user edits or new decisions at any step propagate naturally into future frames.

Key Contributions

1. First autoregressive LiDAR scene generation framework. LaGen generates high-fidelity long-horizon 4D LiDAR scenes frame by frame from a single-frame input, with interactivity that prior 4D generators and multi-frame predictors lack.

2. A multimodal conditional feature encoder. The framework encodes the previous frame's LiDAR (via range image and VAE latent), object-level 3D bounding box projections (via cross-attention with separate previous/current key and value features), and ego states (speed, acceleration, steering, transformations) into the denoising process.

3. Scene Decoupling Estimation (SDE) module. Decouples the previous frame into foreground objects and background, repositions object point clouds using bounding box center displacement between frames, and re-projects them as object-level estimates. This gives the model finer-grained object detail than bounding box masks alone and improves interactive editing.

4. Noise Modulation (NM) module. Injects Gaussian noise at varying levels into the conditional latent features of the previous frame and the SDE estimates, reducing over-reliance on prior frames to counter the train-inference distribution mismatch that causes error accumulation in autoregressive rollouts.

5. Extensive evaluation on nuScenes and KITTI-360 covering generation fidelity, temporal consistency, downstream 3D detection, object-level quality, long-horizon prediction, interactive editing, and ablation studies.

Main Findings

• State-of-the-art generation fidelity on nuScenes. LaGen reaches MMD 0.11 and JSD 2.77 (both lower is better), compared to 1.52 / 5.43 for the strongest baseline LiDARCrafter and 1.84 / 4.50 for DriveLiDAR4D — a roughly tenfold reduction in MMD.

• Strong cross-dataset generalization on KITTI-360. With 64-beam scans, LaGen records MMD 1.41, JSD 5.75, FRD 1109.83, and FPD 73.50, outperforming LiDARGen, LiDM, and OpenDWM across all four metrics.

• Superior temporal consistency on long sequences. On the CTC metric, LaGen scores 0.60, 0.97, 1.46, and 2.22 at frames 1–4, versus 0.89–2.38 for baselines; it is the only method reported at frame 10, where it achieves 3.84. TTCE is comparable to baselines (2.71, 3.62), indicating fidelity is not sacrificed for consistency.

• Dramatic gains in long-horizon prediction. With bounding boxes, LaGen reports Chamfer Distance 0.50 at 0.5s and 2.34 at 9.5s, and L1 error 0.32 at 9.5s. 4D-Occ with 6 input frames degrades to CD 21.08 and L1 6.49; ViDAR with 2 frames reaches CD 6.37 and L1 6.06. The box-free variant also beats all baselines, showing the model learns scene dynamics beyond box priors.

• Downstream 3D detection improves. Training on LaGen-generated data yields mAP 0.52 and NDS 0.49, above UniScene (0.51 / 0.47), OpenDWM (0.45 / 0.47), and LiDARCrafter (0.41 / 0.41).

• Better object-level quality. For all object categories, LaGen achieves CD 0.855 and EMD 0.822, improving on UniScene (0.939 / 1.245). Car and pedestrian metrics show the same trend.

• Both modules contribute measurably. Removing SDE worsens CTC from 0.60 to 0.71 and CD at 8.0s from 2.16 to 2.23. Removing NM worsens CD at 9.0s from 2.25 to 3.16, with gains growing over longer horizons — exactly the error-accumulation behavior the module targets.

• Interactive editing works. Moving a vehicle's bounding box causes LaGen to correctly restore occlusion effects at the new position; deleting a vehicle causes the model to complete previously occluded road surface.

• Inference cost is tunable. Performance saturates around 50 denoising steps (0.43 s/frame), and 10 steps (0.21 s/frame) gives near-real-time generation with minor accuracy loss.

Methodology in Plain English

LaGen represents each LiDAR sweep as a range image — a 2D grid where each pixel stores depth and intensity — using spherical projection that maps 3D points onto azimuth/elevation coordinates. This converts unstructured point clouds into a format that standard 2D image diffusion models can handle.

The generator is a Latent Diffusion Model: a VAE compresses range images into a compact latent space, and a UNet learns to denoise latents there. At sampling time, the UNet iteratively removes noise from a random latent to produce a new frame's range image, which is then unprojected back to 3D points.

Three conditioning signals steer the denoising. The previous frame's LiDAR latent is aligned to the current sensor viewpoint using the relative pose transform, then modulated with a FiLM layer. Bounding boxes are converted into per-category binary masks in range-view space, encoded, and injected through cross-attention — with the current frame providing values and the previous frame providing keys, which lets the model reason about how objects moved. Ego state vectors are embedded as an additional token at each UNet level.

Two modules strengthen consistency. SDE splits the previous point cloud into object points and background points. Object points are transformed to the new sensor frame and then shifted by the displacement between matched bounding box centers in consecutive frames, producing a per-object estimate of where each object's points should be now. Background points are rotated by the sensor's rotation component. These estimates are encoded and injected as extra conditions, giving the model object-level structure rather than just coarse box locations. NM adds Gaussian noise at a randomly sampled level to the latents of the previous frame and SDE estimates during training, so the model learns to tolerate imperfect conditions — which is exactly the situation at inference, where those conditions are generated rather than ground truth.

Inference is a loop: generate one frame, use it as input for the next, and regenerate SDE estimates along the way. Because each step accepts whatever bounding boxes and ego states the simulator supplies, edits at any frame immediately affect subsequent generation.

Why This Matters

Impact on research. LaGen reframes LiDAR generation as an autoregressive, interactive world-modeling problem rather than a one-shot synthesis or fixed-trajectory prediction problem. It shows a single-frame-input diffusion model can remain temporally consistent over ten or more frames, which substantially lowers the input requirements that have constrained prior prediction work. The SDE and NM modules are general techniques for handling object-level structure and train-inference mismatch that could transfer to other autoregressive generative settings.

Real-world applications:

• Data augmentation for perception training. Generating diverse, high-fidelity LiDAR clips to train 3D detectors in conditions that are rare in collected datasets, which the downstream detection experiment supports.
• Closed-loop simulation. Running a planner inside LaGen, feeding its decisions back as bounding box and ego state conditions, so the simulated world reacts to the planner's actions rather than following a fixed replay trajectory.
• Safety-critical scenario evaluation. Constructing rare or dangerous situations — cut-ins, sudden pedestrian crossings, occluded obstacles — by editing bounding boxes mid-sequence and observing realistic consequences.
• What-if and counterfactual testing. Moving or removing objects at any frame and checking whether a planner's behavior changes appropriately, supported by the demonstrated vehicle-move and vehicle-remove editing results.

Industry relevance. Autonomous driving companies rely on data acquisition and simulation, both of which are expensive and both of which this work directly addresses. The tunable inference cost (0.21 s/frame at 10 denoising steps) makes near-real-time simulation plausible. The published code lowers the barrier to adoption. Multi-return and non-spinning LiDAR limitations mean direct deployment on some sensor configurations requires further work.

Future Directions

1. Multi-return LiDAR support. The current range-view representation assigns one depth per pixel, which cannot capture sensors that return multiple echoes per ray. A natural extension is adding a separate depth channel per return.

2. Non-spinning LiDAR representations. LaGen targets spinning mechanical sensors; solid-state and other non-spinning configurations require new scene representations, which the authors flag as unexplored.

3. Robust cross-frame object association. The SDE module matches bounding boxes by nearest center within the same semantic category, which degrades in crowded scenes with many same-class objects. Using tracklets from the simulator to establish correspondence is the suggested remedy.

4. Broader world-model integration. Extending LaGen toward full closed-loop simulation and unified world modeling for autonomous driving, potentially combining it with video generation or natural language control, which the current bounding-box-only conditioning does not cover.

Target Audience

Autonomous driving researchers working on simulation, data generation, and end-to-end planning; generative modeling researchers interested in diffusion models applied to 3D and sequential sensor data; LiDAR perception engineers evaluating synthetic data for training; and graduate students entering the intersection of world models and autonomous driving who already have background in diffusion models and LiDAR data representations.

Authors’ abstract

Generative world models for autonomous driving (AD) are of great value in applications such as data augmentation, closed-loop simulation, and safety-critical scenario evaluation. Unlike the widely studied image modality, in this work we explore generative world models for LiDAR data. Existing generation methods for LiDAR predominantly focus on single frame generation or lack the capacity for interactive simulation, while existing prediction approaches require multiple frames of historical input and can only deterministically predict multiple frames at once. Both paradigms fail to support long-horizon interactive generation. To this end, we introduce \textbf{LaGen}, which, to the best of our knowledge is the first autoregressive framework capable of generating long-horizon LiDAR scenes in a frame-by-frame, interactive manner. LaGen is able to take a single-frame input as a starting point and effectively utilize bounding box information as conditions to generate high-fidelity 4D scene. In addition, we introduce a scene decoupling estimation module to enhance the model's interactive generation capability for object-level content, as well as a noise modulation module to mitigate error accumulation during long-horizon generation. We extensively evaluate LaGen's performance in controlled data generation and long-horizon scene generation on the nuScenes dataset. The experimental results demonstrate that LaGen achieves state-of-the-art performance, especially on later frames. The code is publicly available at: https://github.com/szzhou88/LaGen.