AutoScape: Geometry-Consistent Long-Horizon Scene Generation

AI summary

Overview

Research area: Computer vision / generative AI — specifically long-horizon 3D-consistent driving scene and video generation using diffusion models.

Technical level: Advanced. The paper assumes familiarity with latent diffusion models, classifier guidance, VAEs, point-cloud rendering, and standard video generation benchmarks (FID, FVD).

Scope: AutoScape is a two-stage framework that generates sparse geometry-consistent RGB-D keyframes with a diffusion model and then interpolates dense frames between them, producing driving videos that stay 3D-consistent for over 20 seconds (250 frames).

What This Paper Is About

Generating driving videos with diffusion models works well for short clips, but quality and geometric consistency degrade over long horizons — the paper states that for a 15–20 second split, the prior state-of-the-art (Vista) has FID 157.9 and FVD 1614.5. The authors argue that the key bottleneck is degrading geometric consistency, not just appearance. AutoScape's goal is to keep appearance and geometry aligned across hundreds of frames by generating sparse, highly consistent RGB-D keyframes as global anchors and then filling in dense frames via video interpolation.

Key Contributions

1. AutoScape, a hierarchical framework that jointly generates the appearance and geometry of long-range driving scenes by separating the problem into keyframe generation and interpolation.
2. A new RGB-D diffusion model with geometry-aware conditioning (conditioned on rendered point clouds from previously generated keyframes) and warp-consistent guidance, intended to enforce long-range 3D consistency alongside high visual fidelity.
3. A two-stage training pipeline — large-scale RGB-D pre-training (13 million curated images from nuScenes, Argoverse2, and SA1B) followed by rendering-conditioned fine-tuning on nuScenes — plus a data-filtering step that removes the 20% most inconsistent samples.
4. Reported state-of-the-art results on long-horizon driving scene generation, including comparisons against Vista and two variants of WonderJourney.

Main Findings

• Overall quality gains over the prior state of the art: Against Vista, AutoScape reduces FID from 68.3 to 35.1 and FVD from 629.8 to 359.0, reported as improvements of 48.6% and 43.0%.
• The margin grows with horizon length: In the 15–20 second split, scores improve from [FID, FVD] = [157.9, 1614.5] for Vista to [56.8, 657.4] for AutoScape, reported as improvements of 64% and 59.3%.
• Consistent wins across every time split: AutoScape reports the best quantitative performance in all of the 0–5s, 5–10s, 10–15s, 15–20s, and overall splits, whereas Vista degrades sharply beyond 5 seconds (FID rising from 37.2 at 0–5s to 157.9 at 15–20s).
• Generalization without fine-tuning: Evaluated on Argoverse2 with no fine-tuning, AutoScape reports [FID, FVD] = [49.2, 317.9] versus Vista's [80.4, 614.2].
• User study preference: From 22 valid responses, AutoScape was preferred in 88.39% of cases for best 3D consistency over a long sequence, against Vista and both WonderJourney variants.
• Ablations (all measured on the overall split):
  • – RGB-D pre-training: FID/FVD degrade from 35.1/359.0 to 47.6/650.0.
  • – Data filtering (training on the unfiltered noisy data): FID/FVD degrade to 43.5/463.0, a reported gap of 8.4 and 104.
  • – Warp consistent guidance: FID/FVD degrade to 38.5/380.2.
  • – Depth generation (replacing the RGB-D model with an RGB model plus a monocular metric depth predictor): FID/FVD degrade to 39.2/511.4; the paper attributes the large FVD drop to worse 3D consistency.
• Qualitative behavior: AutoScape is reported to maintain 3D consistency through significant view changes and to generate night-time scenes and out-of-distribution corner cases such as rainy weather, uncommon vehicle types (e.g., ambulances), and rural landscapes.
• Controllability: Generation can be steered with text prompts, HD map layers (lane boundary, lane divider, pedestrian crossings), and object bounding boxes via a ControlNet branch.

Methodology in Plain English

The framework runs two stages.

Stage 1 — sparse keyframe generation. The vehicle trajectory is first broken into sparse keyframe viewpoints. Viewpoint selection is automatic: starting from one trajectory endpoint, the next keyframe is the first viewpoint whose distance from the previous keyframe exceeds 10 m or whose view-angle difference exceeds 20°. Each existing RGB-D keyframe is back-projected into 3D point clouds using depth and camera parameters, then re-rendered onto the next keyframe's camera plane, producing a coarse, hole-ridden image plus a visibility mask. A latent diffusion model (built on Stable Diffusion's inpainting U-Net, extended with five extra input channels) then generates the new RGB-D keyframe conditioned on those rendered points, the mask, HD maps, object boxes, and text. The VAE is modified to encode and decode RGB plus depth jointly as a 4-channel input at 16-bit precision, with depth normalized to 0–1 over a 300 m maximum. The process runs auto-regressively backwards along the trajectory, and each new keyframe is back-projected and added to the growing point cloud.

Stage 2 — dense interpolation. Between each pair of consecutive RGB-D keyframes, an off-the-shelf point-cloud-conditioned video diffusion model (ViewCrafter) is used without fine-tuning to fill in the intermediate frames from the rendered point clouds.

To keep long sequences from drifting, the paper adds warp-consistent guidance: a masked mean-squared-error between the predicted RGB-D frame and the rendered keyframe points, computed only over pixels visible in both. The gradient of this loss is added to the diffusion score during sampling — classifier-guidance style — steering generation toward geometric agreement with prior keyframes. The 5% of pixels with the largest loss are masked out for robustness. The same loss is used before training to filter out the 20% most inconsistent samples from the rendering-conditioned training set (350k samples generated → 280k used).

Training used 32 A100 GPUs for 50k iterations (batch size 1024, lr 1e-4) in pre-training, and 8 A6000 GPUs for 20k iterations (batch size 512, lr 1e-4) in rendering-conditioned training. Inference uses DPM-Solver with 50 steps and classifier-free guidance strength 7.5.

Why This Matters

Research impact: The paper reframes long-horizon scene generation as a geometry problem rather than a purely temporal-module problem, and shows that explicit RGB-D modeling plus test-time guidance can beat a strong video diffusion baseline by a wide margin at long horizons. It also offers a recipe for scaling RGB-D generative training using pseudo-labeled depth, which is relevant to any work limited by scarce depth data.

Real-world applications:

• Autonomous driving simulation and verification — generating photorealistic, controllable driving scenes for testing and validating perception and planning systems.
• Rare/corner-case data augmentation — the paper demonstrates out-of-distribution scenarios such as rainy weather, ambulances, night scenes, and rural landscapes, which are exactly the cases that are hard to collect in reality.
• Generative closed-loop simulators — the paper situates its work in the lineage of systems like DriveArena, where generated scenes feed an interactive simulator.
• Robotics and mixed reality — the introduction explicitly names these as applications where synthesizing realistic visual data is important.

Industry relevance: The controllable conditioning on HD maps and 3D bounding boxes, plus a stated 20-second/250-frame horizon, targets practical simulation needs. Training details (GPU counts, iteration counts, batch sizes) are reported, which aids reproducibility and cost estimation.

Future Directions

• Extending the horizon further. The paper notes that unbounded scene generation is being explored by concurrent work such as InfiniCube, and positions its own 20-second horizon as a step rather than an endpoint.
• Systematic comparison of interpolation models. The authors state that point-cloud-conditioned video diffusion models from concurrent methods are potentially applicable to the interpolation stage and explicitly leave systematic comparisons to future work.
• Reducing dependence on pseudo-labeled depth. Pre-training depth comes from the monocular estimator Metric3D, and the paper attributes inconsistencies partly to noisy depth where consecutive keyframes do not align perfectly — improving depth supervision is a natural next target.
• Better handling of dynamic objects and occlusions — named as sources of inconsistent rendered keyframe points that require the data-filtering workaround, suggesting room for a more principled solution.

Target Audience

Researchers and engineers working on generative video, driving-scene simulation, and 3D-consistent scene synthesis, particularly those already familiar with latent diffusion models and video diffusion. It is also useful for practitioners in autonomous driving simulation and synthetic data generation who want to understand how to trade sparse-but-consistent anchors against dense-but-drifting frame synthesis. Readers without diffusion-model background will find the framework diagram and stage decomposition accessible, but the training and guidance details assume prior knowledge.

Authors’ abstract

This paper proposes AutoScape, a long-horizon driving scene generation framework. At its core is a novel RGB-D diffusion model that iteratively generates sparse, geometrically consistent keyframes, serving as reliable anchors for the scene's appearance and geometry. To maintain long-range geometric consistency, the model 1) jointly handles image and depth in a shared latent space, 2) explicitly conditions on the existing scene geometry (i.e., rendered point clouds) from previously generated keyframes, and 3) steers the sampling process with a warp-consistent guidance. Given high-quality RGB-D keyframes, a video diffusion model then interpolates between them to produce dense and coherent video frames. AutoScape generates realistic and geometrically consistent driving videos of over 20 seconds, improving the long-horizon FID and FVD scores over the prior state-of-the-art by 48.6\% and 43.0\%, respectively.