LiDAR-GS++:Improving LiDAR Gaussian Reconstruction via Diffusion Priors

AI summary

Overview

Research area: LiDAR-based novel view synthesis and reconstruction-based simulation for autonomous driving, combining Gaussian Splatting with diffusion priors. Technical level: Advanced. Scope: a single paper presenting LiDAR-GS++, a LiDAR Gaussian Splatting reconstruction method that uses a controllable LiDAR-to-LiDAR diffusion model to generate extra supervision for extrapolated viewpoints and a depth distortion-aware distillation strategy to integrate that generated data.

What This Paper Is About

Reconstruction-based LiDAR simulators built on Gaussian Splatting produce artifacts when the camera (or sensor) is moved to viewpoints outside the original driving trajectory (extrapolation), while doing well at viewpoints inside it (interpolation). This happens because a single traversal scan only captures the scene from one distribution of viewpoints, leaving parts of the scene under-fitted. The paper's goal is to make extrapolated LiDAR re-simulation geometrically consistent and high fidelity by generating extra, geometry-consistent LiDAR scans with a diffusion model and distilling them into the Gaussian representation.

Key Contributions

1. LiDAR-GS++ itself: a LiDAR reconstruction method that expansively reconstructs neural 2DGS fields via diffusion priors, reported as state-of-the-art against previous GS-based and NeRF-based re-simulation methods on both extrapolation and interpolation.
2. A controllable LiDAR generation model guided by coarsely extrapolated GS rendering, used to produce extrapolated LiDAR scans as extra supervision. The condition and the output are both LiDAR range views (single-modal, LiDAR-to-LiDAR), avoiding cross-modal prompts such as text, maps, or bounding boxes.
3. A Depth Distortion-Aware Distillation (DDAD) strategy for expansive reconstruction that identifies distorted, under-fitted regions using a depth discrepancy criterion and applies the loss only there, integrating generated scans while limiting hallucinations in already well-fitted regions.
4. A switch from 3DGS to a neural 2DGS field for the LiDAR scene representation, with Gaussian attributes predicted by four lightweight MLPs conditioned on ray direction and distance-of-flight.

Main Findings

• Extrapolation quality on Para-Lane: LiDAR-GS++ reports CD 0.102, F-score 0.923, PSNR 31.843, SSIM 0.871 for lane extrapolation @3.5m, compared with LiDAR-GS (CD 0.270, F-score 0.865, PSNR 30.742, SSIM 0.828), GS-LiDAR (0.305, 0.843, 29.279, 0.802), LiDAR-RT (0.482, 0.806, 30.430, 0.824) and LiDAR4D (1.518, 0.785, 29.464, 0.833).
• Interpolation quality on Para-Lane: LiDAR-GS++ reports CD 0.079, F-score 0.936, PSNR 32.232, SSIM 0.873, versus LiDAR-GS (0.090, 0.923, 32.103, 0.869), GS-LiDAR (0.086, 0.927, 29.654, 0.805), LiDAR-RT (0.159, 0.902, 30.979, 0.837) and LiDAR4D (0.112, 0.921, 31.631, 0.866).
• Extrapolation quality on Waymo (@3.5m lateral shift): LiDAR-GS++ reports FRID 11.669, FPVD 15.134, JSD 0.379, MMD 2.618, against LiDAR-GS (39.095, 34.018, 0.606, 13.301), GS-LiDAR (31.967, 78.84, 0.534, 8.263), LiDAR-RT (41.330, 57.551, 0.576, 11.738) and LiDAR4D (48.503, 52.651, 0.587, 9.726).
• Interpolation quality on Waymo: LiDAR-GS++ reports CD 0.052, F-score 0.972, PSNR 32.463, SSIM 0.975, versus LiDAR-GS (0.059, 0.968, 32.313, 0.973), GS-LiDAR (0.057, 0.971, 31.732, 0.952), LiDAR-RT (0.063, 0.962, 29.410, 0.917) and LiDAR4D (0.061, 0.964, 32.434, 0.970). MMD is stated to be at the 1e-5 scale.
• Efficiency: LiDAR-GS++ reports 26 minutes training and 16.2 fps inference, described as suboptimal — slower in training than LiDAR-GS (18 min) due to the extra expansive-reconstruction overhead, and second in inference only to LiDAR-RT (20.7 fps, attributed to OptiX hardware acceleration). GS-LiDAR reports 129 min and 10.8 fps; LiDAR4D reports 426 min and 1.7 fps.
• Ablation on components (Para-Lane, extrapolation): removing the neural 2DGS field ("w/o NGF") degrades to CD 0.417, F-score 0.825, PSNR 29.878, SSIM 0.809; removing diffusion priors ("w/o Diff") gives 0.264, 0.869, 30.777, 0.840; removing DDAD gives 0.163, 0.905, 30.701, 0.839; the full method gives 0.102, 0.923, 31.843, 0.871. On interpolation, "w/o DDAD" (0.085, 0.927, 32.013, 0.854) is worse than the full method (0.079, 0.936, 32.232, 0.873), which the authors attribute to generated data adversely affecting already well-fitted viewpoints.
• Diffusion prior transfers to another baseline: adding the diffusion prior and DDAD to GS-LiDAR yields CD 0.116, F-score 0.921, PSNR 29.530, SSIM 0.816 on lane extrapolation @3.5m, versus GS-LiDAR's reported 0.305, 0.843, 29.279, 0.802 in Table 1.
• Controllable LiDAR generation ablations: without position encoding, FRID 33.62, FPVD 15.17, JSD 0.51, MMD 13.76; without wavelet transforms, 29.31, 14.88, 0.50, 10.37; conditioned on bounding boxes and maps, 46.74, 109.51, 0.66, 17.72; the render-conditioned version (the paper's approach) reports 28.39, 14.46, 0.48, 6.20.
• Need for selective distillation: Chamfer Distance between generative results and reference real scans on Para-Lane shows residual detail discrepancies, which the authors use to motivate DDAD rather than fully blending generated and real scans.

Methodology in Plain English

The pipeline has three stages. First, a single-pass LiDAR driving clip is projected into a range view (intensity, depth, ray-drop channels) and reconstructed with a neural 2D Gaussian field. Each Gaussian carries a center, rotation, scale, intensity, ray-drop probability, opacity and a 32-dimensional feature token; four lightweight MLPs predict these attributes using the ray direction and distance-of-flight, so the representation can account for the direction- and distance-dependent attenuation of LiDAR returns. Rendering follows 2DGS ray-splat intersection with range-view rasterization and volume-rendering integration, trained with depth, intensity, ray-drop and scale-regularization losses.

Second, the authors create training data for a controllable LiDAR generation model. Because extrapolated ground truth is unavailable in single-pass clips, they simulate low-quality inputs by perturbing the neural 2D Gaussian field inputs with variance 0.2 and randomly dropping Gaussian primitives at a ratio of 0.1, producing about 27k training pairs from the Waymo Open and Para-Lane datasets (20 hours on 4 RTX3090 GPUs). The diffusion model is based on LiDM with a frozen VAE, plus Fourier position encoding at the input and wavelet transforms in the up/downsampling layers; it is trained once, not per scene, for 5e4 iterations on 8 A100 GPUs with batch size 16 per GPU, Adam and learning rate 5e-5, with DDIM sampling at 50 steps. At inference time the degraded renderings from manually shifted extrapolated viewpoints condition this model to produce plausible, geometry-consistent scans.

Third, these generated scans are added as extra supervision for expansive reconstruction (mixed 1:1 with real scans), but only in regions flagged as distorted. A distortion mask is computed by comparing the rendered depth against a median depth taken at the Gaussian intersection where transmittance is closest to 0.5; large discrepancies indicate under-fitted attributes. The reconstruction is trained for 7,000 iterations on an RTX3090 — 5,000 for single-traverse reconstruction and 2,000 for the mixed generative/real fine stage — starting from 500,000 randomly initialized GS anchors, using Scaffold-GS growth and split strategy, AbsGS accumulated gradients, densification from step 500 with split gradient threshold 0.002, Adam with per-parameter learning rates of {1,4,4,2,5} x 1e-3, and instance decomposition based on NSG for dynamic objects.

Evaluation uses the Para-Lane dataset (32x1800 resolution at 10 Hz, synchronized cross-lane LiDAR, 3 clips) and Waymo Open (64x2650 at 10 Hz, single lane, 4 clips). Interpolation samples every 10th viewpoint; Para-Lane extrapolation uses real scans from neighboring lanes; Waymo extrapolation artificially shifts viewpoints laterally by 3.5 m and is scored with generative metrics (FRID, FPVD, MMD, JSD) because extrapolated ground truth does not exist.

Why This Matters

Impact on research: this is presented as the first work to integrate diffusion priors into LiDAR Gaussian reconstruction, addressing extrapolation explicitly rather than only interpolation. It also argues that single-modal LiDAR-to-LiDAR conditioning (coarse renderings) is more suitable than cross-modal prompts such as text, maps, or bounding boxes for scene expansion, and shows the idea transfers to another baseline (GS-LiDAR).

Real-world applications:

• Closed-loop simulation and virtual regression testing for end-to-end autonomous driving algorithms, where scenes must be re-simulated from new viewpoints.
• Viewpoint-changing scenarios such as lane-changing for obstacle avoidance, which are exactly the extrapolated views where prior GS and NeRF simulators degrade.
• Reusing crowdsourced driving clips to reconstruct urban road scenes and test casually encountered corner cases.
• Supporting multi-modal (camera plus LiDAR) high-level driving agents, including Level-4 agents, which the authors say are held back by weaker LiDAR simulators relative to camera simulators.

Industry relevance: the work comes from CaiNiao Inc. (Alibaba Group), Zhejiang University and Sun Yat-sen University, and targets practical re-simulation on public urban roads with real-time inference capability, which matters for autonomous driving development pipelines that need scalable, physically realistic sensor simulation.

Future Directions

• Handling non-rigid motion of dynamic objects such as pedestrians, which the reconstruction process currently does not account for.
• Adding temporal consistency to the LiDAR generative model, which the authors state it currently lacks.
• Exploring more advanced generation models to improve fidelity and consistency of synthesized LiDAR.
• Progressive enlargement of the accessible LiDAR novel view synthesis range from limited initial real scans, which the authors describe as the opportunity created by combining reconstruction and diffusion.

Target Audience

Researchers and engineers working on LiDAR novel view synthesis, Gaussian Splatting, diffusion-based sensor generation, and autonomous driving simulation. It suits readers already familiar with 3D/2D Gaussian Splatting, NeRF-style rendering, and latent diffusion models, since the paper assumes that background and reports dense quantitative comparisons. Practitioners building closed-loop simulators or virtual regression-testing infrastructure for end-to-end driving stacks will find the extrapolation results and the reported training/inference costs most directly useful.

Authors’ abstract

Recent GS-based rendering has made significant progress for LiDAR, surpassing Neural Radiance Fields (NeRF) in both quality and speed. However, these methods exhibit artifacts in extrapolated novel view synthesis due to the incomplete reconstruction from single traversal scans. To address this limitation, we present LiDAR-GS++, a LiDAR Gaussian Splatting reconstruction method enhanced by diffusion priors for real-time and high-fidelity re-simulation on public urban roads. Specifically, we introduce a controllable LiDAR generation model conditioned on coarsely extrapolated rendering to produce extra geometry-consistent scans and employ an effective distillation mechanism for expansive reconstruction. By extending reconstruction to under-fitted regions, our approach ensures global geometric consistency for extrapolative novel views while preserving detailed scene surfaces captured by sensors. Experiments on multiple public datasets demonstrate that LiDAR-GS++ achieves state-of-the-art performance for both interpolated and extrapolated viewpoints, surpassing existing GS and NeRF-based methods.