GeoV2V: Geometry-Grounded Video Diffusion Model for Driving Scene Generation

AI summary

Overview

Research area: Computer vision — novel view synthesis (NVS) and generative video modeling for autonomous driving simulation.

Technical level: Advanced. The paper assumes familiarity with diffusion models, transformer backbones (DiT), latent video conditioning, 3D Gaussian Splatting, NeRF, and LiDAR/depth reprojection.

Scope: The paper proposes GeoV2V, a geometry-grounded video-to-video diffusion framework that re-synthesizes driving videos from laterally shifted ("lane-shift") camera trajectories, together with Para4D, a CARLA-generated synthetic dataset of synchronized multi-lane driving videos used as training supervision.

What This Paper Is About

Simulating what an autonomous vehicle would see if it shifted into an adjacent lane requires generating video from viewpoints that were never recorded. Reconstruction-based methods (3DGS, NeRF) stay geometrically faithful but degrade badly under such large viewpoint extrapolation, while "reconstruct-then-restore" pipelines produce pretty frames that drift geometrically and temporally. The authors' goal is a single end-to-end video diffusion model that takes an existing driving video plus geometric priors and outputs a temporally coherent, geometrically consistent video along a new parallel trajectory — without bounding-box annotations.

Key Contributions

1. Para4D dataset. A synthetic dataset built with the CARLA simulator containing synchronized videos captured from multiple parallel lanes within the same dynamic scene, with parallel camera trajectories offset by up to 4 meters, plus LiDAR and accurate camera parameters. It supplies explicit supervision for lane-shift view synthesis, which real-world datasets lack.
2. GeoV2V framework. A video-to-video diffusion model built on the Wan 2.1 backbone that conditions on both the full source video and multi-modal geometric priors (LiDAR-colored points and depth-completed reprojected points) to generate novel-trajectory driving videos without annotations such as 3D bounding boxes.
3. Hierarchical geometry injection. A design in which dense depth-reprojection priors condition the first 15 layers of the denoising network (to fix coarse scene structure) and sparse LiDAR reprojection priors condition the last 15 layers (to correct fine geometric detail), each through a lightweight ControlNet of 61M parameters alongside a 1.3B DiT backbone.
4. Empirical comparison of paradigms. A systematic comparison of reconstruction-based, reconstruct-then-restore, and video-to-video paradigms for lane-shifting NVS on Waymo, nuScenes, and Para4D, plus evidence that synthetic-only training transfers zero-shot to real driving data.

Main Findings

• Interpolation on recorded trajectories (Waymo): GeoV2V reaches 25.46 PSNR, 0.716 SSIM, and 0.132 LPIPS on Waymo Static, and 23.42 PSNR, 0.692 SSIM, 0.149 LPIPS on Waymo Dynamic. Its LPIPS is the best in both settings; on Waymo Dynamic its PSNR is below EmerNeRF (23.91) and StreetGS (23.54).
• Lane shifting on real data (Table 2, FID/FVD at 1 m / 2 m / 4 m): GeoV2V obtains FID 17.53 / 23.59 / 32.58 and FVD 159.00 / 232.06 / 328.69. It has the lowest FID at 1 m and the lowest FVD at every offset; at 2 m and 4 m, StreetCrafter reports lower FID (22.56 and 28.53).
• Lane shifting on Para4D (Table 3): At 1 m / 2 m / 4 m offsets GeoV2V scores PSNR 25.18 / 23.88 / 21.31, SSIM 0.738 / 0.697 / 0.608, LPIPS 0.040 / 0.051 / 0.080, and FID 17.53 / 25.25 / 41.17 — the best of all compared methods on both pixel-level and perceptual metrics, with LPIPS roughly 4–7× lower than the closest baseline.
• Ablation on video input: Removing the source video costs little in metrics (23.28 vs 23.88 PSNR at 2 m) but visibly distorts generated vehicles, because the video mainly supplies fine detail that geometry cannot resolve.
• Ablation on geometry priors: Removing the dense prior drops PSNR from 23.88 to 17.18 and LPIPS from 0.051 to 0.124; removing the sparse LiDAR prior gives 22.86 PSNR and 0.058 LPIPS; removing both gives 16.45 PSNR and 0.128 LPIPS.
• Ablation on Para4D supervision: Training with misaligned side views instead of synchronized multi-lane pairs drops 2 m performance to 20.92 PSNR, 0.593 SSIM, 0.102 LPIPS, versus 23.88 / 0.697 / 0.051 with Para4D.
• Implicit preservation of complex visual dynamics: The model keeps blinking turn signals, brake lights, lens droplets, and other imaging artifacts consistent across frames (Figs. 1, 2, 7), even though these factors are never modeled explicitly.
• Synthetic-to-real transfer: The model shows promising zero-shot transfer to real-world scenes when trained only on synthetic data, without additional fine-tuning.

Methodology in Plain English

The pipeline has three moving parts.

First, for each frame the system builds a 3D "cheat sheet" of the scene by combining two sources: colored LiDAR points (accurate but sparse) and dense point clouds obtained by running depth estimation and depth completion on the image and reprojecting pixels into 3D. These are rendered separately into the desired new viewpoint. Because reprojection leaves mesh-like holes, the authors fill them with a depth-aware morphological procedure: morphological closing identifies candidate holes, and each hole pixel is filled by distance-weighted averaging over nearby valid pixels whose depth is close to that of the nearest valid pixel. This avoids the classic failure of naive inpainting, which smears foreground colors into regions that were genuinely occluded.

Second, the diffusion model receives the original video, the two rendered geometric priors, and the camera pose of each target frame. The source video is concatenated with the noisy target latents along the frame dimension, so temporally aligned source and target frames sit at a fixed relative offset in the rotary positional embedding — no extra attention machinery needed. Camera extrinsics are encoded per frame and added into the corresponding frame latents. The dense prior drives the first 15 transformer layers and the sparse LiDAR prior the last 15, on the theory that early DiT layers set coarse camera viewpoint while later layers refine detail; LiDAR gets its own ConvNeXt encoder to cope with sparsity.

Third, training is deliberately restrained: the feed-forward layers of the Wan-DiT backbone are frozen and only the self-attention blocks and camera-embedding modules are fine-tuned, preserving large-scale pretrained features. Because LiDAR patterns differ across datasets, the LiDAR branch is trained separately after the rest of the network converges, in a two-stage scheme that limits overfitting to a particular sensor distribution.

Evaluation splits into two regimes. On Para4D, synchronized parallel-lane videos give true ground truth, so PSNR, SSIM, LPIPS, and FID are computed directly at 1 m, 2 m, and 4 m lateral offsets. On Waymo and nuScenes, no paired shifted trajectories exist, so lane-shift evaluation is "off-record" and uses FID and FVD; interpolation on recorded trajectories uses a {1-0-1-0-1} temporal sampling scheme where odd frames condition the model and even frames are held out for scoring.

Why This Matters

The paper argues that real driving video is produced by many interacting factors — geometry, illumination dynamics, optical media — and that methods which explicitly model only a chosen subset inevitably fail on the rest. Its contribution is to show that an end-to-end video diffusion model conditioned on geometry, trained with the right supervision, can absorb the unmodeled factors implicitly. It also identifies a data problem the field had not solved: no real dataset contains synchronized parallel-lane footage, so cross-lane supervision has to be synthesized.

Real-world applications:

• Closed-loop autonomous driving simulation: generating the visual consequences of candidate lane changes before committing to them.
• Corner-case generation for perception testing: producing lane-shifted views of rare events, unusual lighting, and sensor contamination such as lens droplets.
• Data augmentation for planning and perception models, expanding single-trajectory logs into multi-viewpoint training data.
• Validation of world models, by supplying a geometrically grounded baseline against which driving world models can be checked.

Industry relevance: the method needs no bounding boxes, which are labor-intensive and dataset-specific, and it transfers from synthetic to real data — both attractive properties for groups that must scale simulation across new cities and sensor rigs without re-annotating.

Future Directions

• Metrics for visual dynamics. The authors explicitly flag that PSNR and SSIM are largely insensitive to dynamic lighting and optical artifacts, so these are currently judged only qualitatively; designing metrics that capture them is posed as the key open problem, especially since lens contamination and changing traffic signals are critical corner cases.
• Closing the FID gap at large offsets. StreetCrafter reports lower FID than GeoV2V at 2 m and 4 m on real data, so improving frame-level realism under extreme lateral displacement remains open.
• Generalizing beyond parallel lanes and lateral shifts, given that Para4D only models offsets up to 4 meters on parallel trajectories.
• Reducing reliance on synthetic supervision, since the training prior comes from CARLA and the paper reports zero-shot transfer as "promising" rather than solved.

Target Audience

Researchers and engineers working on novel view synthesis, driving-scene generation, and world models for autonomous driving; practitioners building simulation and data-augmentation pipelines who need geometrically consistent multi-viewpoint video; and readers interested in how geometric priors (LiDAR, monocular depth) can be injected into large video diffusion transformers. The paper presumes comfort with diffusion architectures and 3D scene representations, so beginners will need background reading on NeRF, 3DGS, and DiT conditioning first.

Authors’ abstract

Novel view synthesis under trajectory changes is essential for autonomous driving simulation, yet existing methods struggle to generate consistent videos when extrapolating to unseen viewpoints such as lane shifts. Reconstruction-based approaches maintain geometric consistency but degrade in visual quality under large viewpoint changes, while reconstruct-then-restore methods often produce visually plausible frames yet suffer from geometric and temporal inconsistencies. We present GeoV2V, a geometry-grounded video-to-video diffusion framework that synthesizes temporally coherent driving videos along novel trajectories by integrating LiDAR and depth-derived geometric priors with full-video conditioning in the Wan 2.1 backbone. A key challenge in learning cross-lane transformations is the lack of synchronized multi-trajectory supervision in real-world datasets. To address this, we also introduce Para4D, a synthetic dataset capturing synchronized multi-lane driving videos within dynamic scenes, which provides explicit supervision for lane-shift generation and serves as a strong training prior for geometry-consistent view synthesis. Experiments on Waymo, nuScenes, and Para4D demonstrate state-of-the-art performance in visual quality, geometric consistency, and robustness to complex lighting dynamics under large trajectory shifts. Project page: https://xiyuche.github.io/GeoV2V/