GeoFlow: Efficient Driving Video Generation via Geometry-Aligned Priors

AI summary

Overview

Research area: Computer Vision — efficient generative video modeling for autonomous driving, specifically flow matching / diffusion-based driving video synthesis.

Technical level: Advanced. The paper assumes familiarity with flow matching, diffusion ODEs/SDEs, latent VAE representations, camera geometry (extrinsics, intrinsics, point clouds, splatting), and standard video generation metrics (FID, FVD).

Scope: The paper introduces GeoFlow, a framework that replaces the standard Gaussian noise initialization of driving video generation models with a geometry-aligned prior built from warped reference-frame depth, in order to reach high-quality multi-view driving videos in far fewer sampling steps.

What This Paper Is About

Driving video generators based on diffusion and flow matching produce state-of-the-art results but are slow, because they start from pure Gaussian noise and must reconstruct every scene element of every frame from scratch. The authors argue this is wasteful: since driving scenes are strongly governed by multi-view geometry and ego-motion, most future frame content is predictable from a reference frame and the given control signals. GeoFlow's goal is to start generation from a coarse, geometry-aligned prediction of the future instead of noise, thereby shortening and straightening the sampling trajectory so that high-fidelity videos can be produced in only a handful of steps.

Key Contributions

1. GeoFlow framework: An efficient driving video generation framework that initializes future-frame generation from reference-frame information rather than pure noise, fundamentally straightening and shortening the transport path between source and target distributions.
2. Geometry-Aligned Prior (GAP) distribution: A source distribution constructed from the inherent geometric consistency of driving videos, built by warping latent reference-frame features to future poses using metric depth and camera/ego-motion transformations.
3. Spatially-Adaptive Noise Injection: A continuous, parameter-free reliability mask that selectively injects Gaussian noise into unreliable regions (occlusions, dynamic objects, depth-ambiguous areas) so artifacts from depth errors and rendering are corrected without discarding valid geometry.
4. Demonstrated practicality: Extensive experiments showing significant few-step generation quality improvements at low adapting cost, including a plug-and-play generality analysis across base models with different architectures.

Main Findings

• Fewer steps, better quality: On NuScenes validation, GeoFlow reaches FID 6.8 and FVD 32.5 in 15 inference steps, compared with the OpenDWM baseline at 40 steps (FID 6.8, FVD 38.8). Other compared methods (MagicDrive-V2, DreamForge, Drive-WM, DriveDreamer-2, UniMLVG) are all listed with 30 to 50 steps or unreported step counts.
• 8 steps beats the baseline's 40 steps: GeoFlow at 8 steps achieves FVD 38.6, versus the baseline's FVD 38.8 at 40 steps, a 5× step reduction. GeoFlow's FVD is 49.2 at 5 steps, 35.0 at 10 steps, 32.5 at 15 steps, 32.6 at 20 steps, and 34.0 at 40 steps, while the baseline degrades from 123.5 at 5 steps to 38.8 at 40 steps.
• Generality across architectures: Applying GeoFlow to OpenDWM-tvae reduces 5-step FVD by 62.5% (204.8 to 76.7), to OpenDWM-vae by 59.6% (121.7 to 49.2), and to the stronger UniMLVG base model by 38.6% (72.6 to 44.6).
• Spatially-adaptive noise injection outperforms alternatives: Naïve warping without noise injection performs poorly and accumulates error at higher steps; global uniform noise helps but degrades higher-step performance; the full masked strategy performs best.
• Every mask component contributes: At 10 inference steps, no masks gives FID 8.2 / FVD 43.6; adding the occlusion mask gives 7.9 / 40.0; adding depth uncertainty gives 7.6 / 40.2; adding the dynamic-object mask gives 7.5 / 35.0. Removing the dynamic mask causes significant ghosting artifacts around moving objects.
• Robust to depth model swapping: Used zero-shot at inference in place of the training-time MapAnything-v1.0, MapAnything-v1.1 gives FID/FVD of 11.1/44.1 at 5 steps, 7.9/33.1 at 10 steps, and 7.8/35.6 at 40 steps; DepthAnything-3 gives 13.1/55.1, 8.0/38.2, and 7.2/37.3 at the same step counts.
• Fast training convergence: FVD drops significantly within the first thousands of iterations and saturates around 10,000 iterations; total adaptive training cost is less than 30 H100 GPU hours on 2 NVIDIA H100 GPUs at a learning rate of 5e-5.
• Inference speedup despite a shorter chunk: Even though GeoFlow uses an autoregressive chunk size of L=6 (1 reference frame plus 5 generated frames) versus the baseline's L=19, generating a 16-frame clip (8 steps × 4 passes) is 4.2× faster than the baseline (40 steps × 1 pass).
• Warping overhead is marginal: For a single 6-frame clip at 5 steps on one L20 GPU, geometry reconstruction takes 0.43 s (2.84%), feature rendering 0.49 s (3.24%), ODE solving 13.83 s (91.65%), and the total is 15.09 s.
• No architectural changes: Both baseline and GeoFlow share the exact same model architecture and control conditions, differing only in the source distribution.

Methodology in Plain English

The authors start from a reference frame and the known future ego-motion and camera parameters. First, they estimate metric depth of the reference frame, which also yields a depth uncertainty map. They encode the reference frame into a VAE latent and unproject those latent features into a 3D point cloud using the depth and camera parameters. Using the relative pose from ego-motion or trajectory planning, they transform the point cloud into the target future coordinate system and render it back to 2D with a Z-buffer-based feature splatting strategy, keeping the closest point per pixel to handle collisions. The result is a warped latent map — a geometrically aligned coarse guess of the future frame.

Because that warp is imperfect, they do not use it directly as the starting point. Flow matching needs a stochastic source for training stability, diversity, and error correction, and the warp contains distortions from depth inaccuracies and rendering. A simple global noise blend is a poor compromise because it also corrupts the well-aligned regions. So they build a per-pixel noise mask by taking the maximum of three cues: geometric invalidity from the Z-buffer (occluded or out-of-view areas), projected 3D bounding boxes for dynamic agents whose motion static warping cannot predict, and the normalized depth uncertainty map for ambiguous geometry such as sky or transparent surfaces. The final source distribution is a linear interpolation between the warped latents and standard Gaussian noise, weighted by this mask — low mask values preserve geometry, high values fall back to noise.

Training is then standard flow matching, but along the optimal transport path between the constructed prior and the ground-truth latent, with the analytical velocity defined as ground truth minus the prior. At inference, the sampler starts from the constructed prior instead of Gaussian noise and integrates the learned vector field. Frames are generated autoregressively in chunks of 6 (1 reference plus 5 new frames), and generation is evaluated on NuScenes at 256×448 resolution using FID per frame and FVD over 16-frame clips, following UniMLVG's protocol of 150 validation scenes and six-view 16-frame videos.

Why This Matters

The paper reframes acceleration for driving video generation as a problem of where you start, not just how you solve the ODE. That is orthogonal to existing acceleration strategies such as advanced samplers and distillation, and it requires no architectural changes, no distillation cost, and under 30 H100 GPU hours of adaptation. By showing a plug-and-play improvement across three different base configurations, it suggests that prior construction is a broadly applicable lever for few-step video generation.

Real-world applications:

• High-throughput data augmentation for autonomous driving, generating diverse driving data at lower production cost.
• Generative closed-loop simulation, where faster sampling increases the interaction frequency of simulated environments used to evaluate and train driving algorithms.
• Corner-case synthesis, including scarce scenarios such as extreme weather and hazardous traffic situations that are costly or unsafe to collect in the real world.
• Neural scene reconstruction and generative simulation pipelines that depend on producing photorealistic, multi-view, geometrically consistent driving video.

Industry relevance: autonomous driving companies that depend on large volumes of labeled, controllable driving video for training and simulation stand to benefit from lower inference latency (the paper reports a 4.2× speedup for a 16-frame clip) and reduced adaptation cost, making generative simulation more practical for deployment.

Future Directions

• Longer-horizon autogeneration: GeoFlow relies on geometric overlap between reference and target views and uses a short chunk of L=6 versus the baseline's L=19; extending valid reprojection to longer chunks is an open problem.
• High-step degradation: GeoFlow's FVD at 40 steps (34.0) is slightly worse than at 15 steps (32.5), suggesting room to improve behavior when more sampling steps are used.
• Dependence on depth quality: Performance varies with the depth estimator (MapAnything-v1.1 versus DepthAnything-3 versus the training-time MapAnything-v1.0) across step counts, so further robustness to inaccurate depth and rendering is a natural extension.
• Mask design: The reliability mask is hand-built from three fixed geometric and semantic cues with no learnable parameters; whether learned or additional cues could improve the prior is left open.
• Combination with other acceleration methods: The authors state their approach is orthogonal to samplers and distillation, implying untested combinations with those techniques.
• Evaluation scope: Results are reported on NuScenes with 150 validation scenes at 256×448 resolution; broader datasets, resolutions, and multi-reference-frame settings (as used by the UniMLVG † comparison with 3 reference frames) are not explored.

Target Audience

Researchers and engineers working on efficient generative video modeling, driving world models, and autonomous driving data synthesis; practitioners seeking low-cost ways to accelerate existing diffusion or flow matching video models without retraining from scratch; and readers with a background in diffusion/flow matching and multi-view geometry who want to understand how source-distribution design affects sampling efficiency.

Authors’ abstract

Generative models like Diffusion Models and Flow Matching have demonstrated remarkable capabilities in synthesizing high-fidelity driving videos, but are severely constrained by high inference latency due to the requirement of extensive sampling steps. We argue that this inefficiency stems from the prevailing reliance on a standard Gaussian source distribution, where consecutive frames are initialized as independent Gaussian noise. This paradigm disregards the rich spatiotemporal correlations inherent in driving videos, compelling the model to regenerate deterministic scene structures existing in previous frames from noise, which is both computationally redundant and prone to geometric inconsistency. To address this problem, we propose GeoFlow, a novel framework designed to achieve efficient driving video generation by harnessing explicit geometric priors. Instead of sampling from standard Gaussian noise, we leverage multi-view geometry and spatially-adaptive noise injection to construct a Geometry-Aligned Prior (GAP) distribution as starting point. This initialization bridges the gap between source distribution and data distribution, yielding a significantly straighter and shorter sampling trajectory. Extensive experiments demonstrate that GeoFlow can achieve remarkable efficiency of both training and inference: merely several hours of fine-tuning on baseline models can significantly boost few-step generation quality, while fully converged training drastically reduces number of inference steps required for state-of-the-art video generation.