LiFi: LiDAR Generation from Multi-View Images via Geometric and Semantic Collaborative Guidance

AI summary

Overview

Research area: Computer vision / cross-modal generative modeling for autonomous driving — specifically, synthesizing 3D LiDAR scenes from multi-view camera images.

Technical level: Intermediate to Advanced. The paper assumes familiarity with diffusion models, latent diffusion, range-image representations of LiDAR, and classifier-free guidance.

Scope (one sentence): This paper introduces LiFi, a latent diffusion framework that generates LiDAR scenes from multi-view camera images by jointly conditioning on a geometry branch (depth, uncertainty, and RGB aligned to the LiDAR scan pattern) and a cross-view semantic branch, combined through a dual-branch balanced classifier-free guidance strategy.

What This Paper Is About

Cameras are cheap and ubiquitous on vehicles, but LiDAR provides the precise 3D geometry that many perception systems rely on. Generating LiDAR point clouds directly from onboard multi-view camera images would let systems obtain 3D structure without LiDAR hardware, yet the problem is under-explored because RGB images (dense 2D appearance) and LiDAR (sparse, precise 3D measurements) are difficult to align.

Prior camera-to-LiDAR work such as LiDM focuses mainly on semantic cues and is typically limited to monocular input, without explicitly enforcing geometric alignment. LiFi is a dedicated framework that addresses both cross-modal geometric alignment and cross-view semantic aggregation for multi-view image input.

Key Contributions

1. A dedicated multi-view image-to-LiDAR framework. The authors state that LiFi is, to the best of their knowledge, the first dedicated framework capable of generating high-fidelity LiDAR scenes with geometric-visual alignment from multi-view images.

2. Three alignment-oriented components. A range-view-centered reverse sampling algorithm, an uncertainty-aware (uncertainty-modulated) geometric encoder, and a cross-view semantic encoder, intended to achieve precise cross-modal alignment at the geometric level plus effective global semantic guidance.

3. Dual-branch balanced classifier-free guidance. A CFG strategy that trains with full-conditional, single-branch-conditional, and unconditional settings, which the authors say enhances conditional guidance capability and collaborative learning between the two branches.

4. Extensive evaluation on nuScenes and KITTI-360. Including high-fidelity LiDAR generation, a downstream 3D object detection study, temporal consistency evaluation, and ablations.

Main Findings

• State-of-the-art LiDAR generation on nuScenes. LiFi reports MMD 2.78 (in 10⁻⁵), JSD 3.66 (in 10⁻²), FRD 906.71 and FPD 126.31, compared with LiDARGen (MMD 55.14, JSD 27.69), RangeLDM (72.01, 8.43, FRD 1172.06, FPD 441.16), LiDM (4.57, 10.61, FRD 2358.15, FPD 420.36), UniScene (107.18, 12.35, FRD 1636.60, FPD 230.89), OpenDWM (10.75, 7.69, FRD 2607.81, FPD 146.30) and LiDARCrafter (24.69, 7.37, FRD 1183.11, FPD 36.17). FRD and FPD are the metrics where LiDARCrafter beats LiFi on FPD (36.17 vs. 126.31).

• Strong results on KITTI-360. With MMD reported in 10⁻⁴, LiFi achieves MMD 1.99, JSD 8.77, FRD 1107.21 and FPD 21.40, versus LiDARGen (13.36, 13.67, 2415.23, 102.80), RangeLDM (4.31, 9.42, 1362.25, 68.22), LiDM (9.31, 19.00, 2894.65, 253.91) and OpenDWM (12.93, 14.63, 1966.36, 436.17).

• Better point-level fidelity and cross-modal consistency than camera-to-LiDAR baselines. On nuScenes, LiFi reports CD 0.83, EMD 50.08, DAS 1.02 and 0.52 s/frame inference, against LiDM (MMD 8.02, JSD 9.97, CD 4.57, EMD 99.71, DAS 2.17, 0.34 s/frame) and X-Drive (MMD 48.58, JSD 10.54, CD 4.73, EMD 179.62, DAS 1.52, 1.80 s/frame).

• Temporal consistency without explicit temporal modeling. Feeding consecutive camera frames produces LiDAR sequences with good temporal consistency. LiFi reports TTCE 2.67 and 3.61 at horizons 3 and 4, and CTC 0.80, 1.11, 1.52, 2.22 and 3.19 at horizons 1, 2, 3, 4 and 10, improving over UniScene, OpenDWM, OpenDWM-DiT and LiDARCrafter, which do not report a value at horizon 10.

• Value in a downstream detection task. With generated LiDAR, BEVFusion reaches mAP 0.526 and NDS 0.520, versus 0.529/0.533 with real LiDAR and 0.278/0.341 for camera-only BEVDet. The authors state this does not directly demonstrate that LiFi-generated scenes can independently support perception performance comparable to real LiDAR.

• Both branches and both conditions matter (ablation). Removing the geometry-aware branch degrades MMD/JSD to 5.20/4.27; removing the cross-view semantic branch gives 3.57/3.67; removing the depth condition gives 3.57/4.12; removing the RGB condition gives 2.80/3.92; removing uncertainty modulation gives 2.87/3.80; the full model gives 2.78/3.66.

• Dual-branch balanced CFG matters (ablation). Without classifier-free guidance, MMD/JSD is 6.32/4.85; without the dual-branch balanced design, 3.02/4.31. On guidance scale, w = 2.0 gives 5.71/3.76, w = 4.0 gives 3.45/3.81, and w = 8.0 gives 2.37/4.13 (better MMD but worse JSD); the reported full configuration uses w = 6.0 with 2.78/3.66.

• Training configuration. Distributed training on four NVIDIA H200 GPUs for 200 epochs, 176,000 optimization steps, total batch size 32, learning rate 1×10⁻⁴ with cosine annealing, DDIM sampler with T = 1024, 50 denoising steps at inference, approximately 20 hours of training, and six nuScenes camera images per generated scene.

Methodology in Plain English

LiFi represents LiDAR scenes as range images — a 2D array indexed by azimuth and elevation that follows the sensor's actual scanning pattern. This keeps the structured angular sampling and inherent sparsity of LiDAR data, and the transformation is reversible, so a generated 2D image converts deterministically back to a 3D point cloud. Generation happens with a latent diffusion model: a VAE compresses the range image into a latent space and a UNet performs the diffusion in that latent space (the VAE is initialized from pretrained RangeLDM weights).

The framework takes multi-view images, camera intrinsics and extrinsics, and extracts three visual priors with a pretrained vision foundation model (DepthAnything3 is used as the instantiation): per-pixel relative depth, an uncertainty map for that depth, and camera semantic tokens.

Geometry branch. Rather than projecting every depth-map pixel into a point cloud — which is expensive and does not align with the sparse LiDAR distribution — the authors invert the process. They take canonical 3D query points that follow LiDAR scan characteristics, project each into every camera view, keep only points that land inside image boundaries and fields of view, and resolve overlap or occlusion with a nearest-depth criterion. This "reverse sampling" yields a LiDAR-aligned relative depth map, uncertainty map, and RGB image. These, plus a 2D range-view coordinate encoding, are fed to a convolutional encoder. The uncertainty map is not just another input channel; it modulates features through FiLM scaling and shifting, so the encoder can amplify reliable geometry and suppress unreliable regions.

Semantic branch. Flattened semantic tokens from all views are augmented with three positional encodings — patch coordinates, view/camera identity, and camera extrinsics (the extrinsic matrix flattened to a 16-dimensional vector passed through a two-layer MLP). A multi-head self-attention cross-view interaction module then lets tokens from all cameras interact in one feature space, capturing long-range dependencies and cross-view consistency. The resulting features are injected into the UNet's Transformer blocks via cross-attention.

Guidance. Standard classifier-free guidance randomly drops all conditions to learn both conditional and unconditional distributions. LiFi extends this dropout beyond all-or-nothing: it also trains with only geometry or only semantics retained, so the model learns each branch's individual effect and their joint contribution. Inference uses the standard CFG formulation with a guidance scale.

Why This Matters

Impact on research. The paper targets an under-explored formulation — camera-to-LiDAR generation with explicit geometric alignment rather than semantic cues alone, and with multi-view rather than monocular input. It introduces a range-view-centered reverse sampling approach that samples camera images according to the LiDAR scan pattern instead of projecting dense depth maps, and it contributes a training-time guidance scheme for keeping two heterogeneous conditioning branches balanced. It also reports results on both nuScenes and KITTI-360 and includes temporal consistency and downstream detection evaluations, which broadens the standard evaluation of LiDAR generators.

Real-world applications:

• Cross-modal simulation, where camera-only driving logs are turned into LiDAR scenes for closed-loop testing.
• Data augmentation for 3D perception models, particularly in cost-sensitive sensor setups.
• Sensor completion, filling in or synthesizing LiDAR where hardware is absent or degraded.
• Controllable data synthesis for long-tail scenarios that are expensive to capture with real LiDAR.

Industry relevance. Cameras are low-cost and easy to deploy while LiDAR acquisition is expensive, so generating LiDAR-aligned scenes from existing camera fleets addresses a practical data bottleneck for autonomous driving. The downstream detection experiment, where a fusion model trained with generated LiDAR reaches mAP 0.526 and NDS 0.520 versus 0.529 and 0.533 with real LiDAR, is a concrete signal of industrial usefulness, though the authors caution it does not show the generated data substitutes for real LiDAR. Reported inference cost is 0.52 s/frame on a single NVIDIA H200 GPU, which is relevant for practical deployment.

Future Directions

• Reduce information loss from the range image representation. The authors note the representation is not entirely lossless, so some camera image details are omitted from the synthesized point clouds, and refining it could improve performance.
• Add explicit temporal modeling. The architecture is not specifically designed for temporal generation. Although consecutive frames already yield temporally consistent sequences, adding dedicated temporal consistency mechanisms could extend applicability.
• Extend to simulation, planning, and control. The conclusion names these as areas where LiFi could play an important role in complex environments and long-tail scenarios.
• Analyze performance across depth ranges and failure modes. The paper states that generation quality across different depth ranges and the associated failure modes are analyzed in the supplementary material, indicating this as an area for further study.

Target Audience

Researchers and practitioners working on cross-modal generative modeling, LiDAR scene synthesis, sensor simulation, and autonomous driving perception. It is most useful for readers already comfortable with diffusion models and point-cloud representations who want to understand how to align image-space geometry with LiDAR scan patterns, and for engineers looking to augment or simulate LiDAR data from existing camera fleets. Readers seeking an introductory treatment should note the reliance on latent diffusion, classifier-free guidance, and range-image parameterization.

Authors’ abstract

Generating LiDAR point clouds from multi-view camera images is a valuable task with applications in controllable data synthesis, cross-modal simulation and various perception tasks. However, most existing methods are not specifically designed for this task, and the quality of their generated LiDAR remains limited. Moreover, these methods typically focus only on semantic cues in images or generate LiDAR from a monocular camera image. In this work, we introduce LiFi (LiDAR From images), a dedicated framework for generating visually and geometrically aligned LiDAR scenes from multi-view camera images. LiFi guides the latent diffusion process through two complementary branches: geometry and semantics. The geometry branch constructs scene features from images through a reverse sampling algorithm and an uncertainty-aware geometric encoder combined with DepthAnything3, whereas the semantic branch provides high-level semantic representations via a cross-view interaction module. We further introduce a dual-branch balanced classifier-free guidance (CFG) strategy, which enhances the model's conditional generation capability while preserving the independent learning and collaborative controllability of the two branches. Extensive experiments demonstrate that LiFi outperforms state-of-the-art methods in generating high-fidelity LiDAR scenes. The code will be publicly available at: https://github.com/szzhou88/LiFi.