SimULi: Real-Time LiDAR and Camera Simulation with Unscented Transforms

AI summary

Overview

Research area: Neural rendering for autonomous driving sensor simulation (3D Gaussian Splatting, LiDAR simulation, camera simulation).

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, NeRF, rasterization versus ray tracing, camera intrinsics and LiDAR sensor geometry.

Scope: The paper introduces SimULi, a real-time neural rendering pipeline that jointly simulates camera images (including non-pinhole models) and spinning LiDAR scans from real-world autonomous driving data, and benchmarks it against camera-only, LiDAR-only and joint camera-LiDAR baselines on the Waymo and PandaSet datasets.

What This Paper Is About

Testing self-driving vehicles requires simulators that reproduce what the vehicle's sensors would actually see, including scenarios that would be unsafe or impractical to collect on real roads. Existing neural rendering methods for this task are either too slow for large-scale use or restricted to idealized pinhole cameras, and methods that model cameras and LiDAR together tend to sacrifice the accuracy of one sensor to preserve the other because of unavoidable cross-sensor calibration inconsistencies. SimULi aims to reconstruct scenes from real sensor data and re-render both arbitrary camera models and LiDAR scans in real time without that quality tradeoff.

Key Contributions

1. LiDAR support built on 3DGUT. The authors extend 3D Gaussian Unscented Transform (3DGUT), which already supports non-linear camera models and time-dependent effects such as rolling shutter, to render rotating LiDAR sensors with irregular sampling patterns. This includes an automated, histogram-equalization-based tiling scheme that derives tiling parameters for any spinning LiDAR model without handcrafted heuristics, plus a ray-based culling strategy.

2. A factorized multi-sensor representation. Camera and LiDAR attributes are encoded into separate sets of 3D Gaussians, joined by a nearest-neighbor anchoring loss, instead of a single NeRF or a single Gaussian set. The authors state this mitigates the camera-versus-LiDAR accuracy tradeoff of prior methods and reduces mean camera and depth error by up to 40% compared to existing methods.

3. State-of-the-art fidelity without learned refinement networks. The pipeline surpasses prior LiDAR and camera state-of-the-art without relying on neural networks for refinement, unlike NeRF-based work (Tonderski et al., 2024; Yang et al., 2023b; Zheng et al., 2024) and 3DGS-based work (Zhou et al., 2025; Hess et al., 2025).

4. Real-time, general sensor support. The authors state SimULi is, to their knowledge, the first method to render in real time while supporting arbitrary camera and LiDAR models.

Main Findings

• Rendering speed. SimULi renders 10-20x faster than ray tracing approaches and 1.5-10x faster than prior rasterization-based work, while handling a wider range of camera models. Figure 1 states a 1.5-20x range overall.
• Waymo Interp. (static). SimULi reaches 30.15 PSNR, 0.881 SSIM, 0.241 LPIPS, 0.003 median L2 depth error, 0.007 mean relative L2, 0.064 intensity RMSE, 0.944 ray drop accuracy, 0.136 chamfer distance, 156.90 MP/s and 11.33 MR/s. The paper reports it outperforms all baselines by more than 2 dB PSNR; the strongest camera baseline listed is DeformableGS at 27.95 PSNR and the closest joint method is SplatAD at 27.82 PSNR. It beats LiDAR-only LiDAR-RT on every metric except ray drop accuracy (0.962 for LiDAR-RT, which uses a U-Net refinement network), and renders more than 10x faster.
• Waymo Dynamic. SimULi reaches 32.35 PSNR, 0.922 SSIM, 0.150 LPIPS, 0.002 median L2, 0.019 mean relative L2, 0.053 intensity RMSE, 0.932 ray drop accuracy, 0.148 chamfer distance, 179.45 MP/s and 10.56 MR/s, compared with SplatAD at 30.60 PSNR and 52.28 MP/s.
• PandaSet reconstruction. SimULi reaches 29.76 PSNR, 0.881 SSIM, 0.195 LPIPS, 0.002 median L2, 0.006 mean relative L2, 0.034 intensity RMSE, 0.997 ray drop accuracy, 0.206 chamfer distance, 137.74 MP/s and 11.10 MR/s. It improves over the second-best method (SplatAD, 28.58 PSNR) by more than 1 dB PSNR while rendering camera views 60% faster.
• PandaSet novel view synthesis. SimULi reaches 27.12 PSNR, 0.830 SSIM, 0.220 LPIPS, 0.006 median L2, 0.018 mean relative L2, 0.059 intensity RMSE, 0.970 ray drop accuracy, 0.331 chamfer distance, 136.33 MP/s and 11.02 MR/s, versus SplatAD at 26.73 PSNR.
• Comparison to the closest baseline. Relative to SplatAD, the authors report PSNR improvements of 0.4-1.7 dB without using CNNs for view dependence, cameras rendered 1.5-3x faster, and LiDAR accelerated by 10x.
• Ablation on factorization. Averaged across PandaSet, the full model reaches 27.12 PSNR, 0.331 chamfer distance, 136.33 MP/s and 11.02 MR/s. Encoding all sensors into one particle set with a direct LiDAR depth loss at lambda_d = 0, 0.001, 0.01 and 0.1 yields 26.61, 26.70, 26.39 and 25.78 PSNR respectively, with LiDAR rendering around 4.77-5.55 MR/s, i.e. roughly 2x slower than the factorized model. The authors also note anchoring outperforms camera-only reconstruction (lambda_d = 0).
• Ablation on camera components. Removing the bilateral grid drops PSNR to 25.99; removing the environment map drops it to 26.81, both below the full method's 27.12.
• Image quality improvements beyond 3DGUT. Although camera rendering builds on 3DGUT (27.23 PSNR on Waymo Interp.), the anchoring, appearance variation and environment map raise quality by 3 dB PSNR and reduce "floaters", with slightly faster rendering.
• Tiling parameters. A grid search over LiDAR tiling reports the best rendering speed at M=32 maximum points per tile and N_phi=16 elevation tiles (15.75 MR/s), while noting this choice does not affect quality.

Methodology in Plain English

The scene is modeled as a dynamic graph in the style of prior work, where each 3D Gaussian particle belongs either to a moving actor (with a 3D bounding box and a sequence of SE(3) poses adjusted by learnable offsets) or to the static background. At render time, object particles are transformed from their local coordinates to world coordinates using the pose for the current timestamp.

Rather than storing camera and LiDAR information in one particle set, the authors keep two: a camera set whose particles carry third-order spherical harmonics (48 coefficients) for view-dependent color, and a LiDAR set carrying 48 coefficients for view-dependent intensity and ray drop. Each set is trained with its own reconstruction losses, so cross-sensor calibration mismatch in the data cannot force one sensor to be sacrificed. To still let LiDAR geometry improve camera rendering, a nearest-neighbor anchoring loss pulls camera particles toward the surfaces learned by the LiDAR particles; because full nearest-neighbor search is expensive, each camera particle is assigned its K=50 nearest LiDAR neighbors, and these assignments are refreshed every 1000 iterations.

Rendering follows 3DGUT: each Gaussian is approximated by 7 sigma points projected through the sensor's projection function, which yields a 2D conic used for tiling and culling. Because sigma points are projected independently, time-dependent effects like rolling shutter are handled by folding sensor motion into the projection. For LiDAR, sigma points are converted to azimuth, elevation and range. Because LiDAR's measurement pattern is irregular (full 360 degrees in azimuth but roughly 20 degrees in elevation) and elevation samples are unevenly distributed, the authors compute a normalized cumulative distribution over elevation with a histogram of 400 bins and place elevation tile boundaries where the CDF crosses integer boundaries, then choose an azimuth tile count so beam counts per tile differ by at most 8 samples. A separate, denser tile grid is used for culling (1600 azimuth by 8 elevation) with a summed-area table so each particle's visibility is filtered in constant time (4 memory reads and 3 arithmetic operations). A 3D smoothing filter, adapted from AAA-Gaussians, handles LiDAR beam divergence around reflective surfaces.

Training minimizes a reconstruction loss (L1 photometric and SSIM for camera; distance, intensity and binary cross-entropy ray drop for LiDAR), the anchoring loss weighted at 0.01, and regularization terms including an entropy loss that encourages LiDAR particles toward binary opacity and sparsity, plus smoothness across learned per-frame affine color transforms and the environment map. Loss weights are 0.8 photometric, 0.2 SSIM, 0.01 distance, 0.1 intensity, 0.05 ray drop.

Why This Matters

The paper targets a practical bottleneck in autonomous vehicle development: simulators must reproduce raw sensor data, including fisheye cameras and spinning LiDAR, at speeds that make large-scale scenario testing feasible. It also argues that encoding each sensor separately is a cleaner answer to cross-sensor calibration error than weighting a shared representation.

• Autonomous vehicle testing: generating realistic, controllable sensor streams for rare or dangerous scenarios that cannot be collected safely in the real world.
• Perception model training: producing training data for models that consume raw fisheye and LiDAR input, avoiding the domain gap created when simulators only render ideal pinhole images.
• Sensor configuration studies: because the LiDAR tiling is derived automatically from any spinning sensor's beam pattern, the pipeline can be retargeted to different or hypothetical LiDAR layouts.
• Reconstruction and digital twins of driving scenes: building editable representations of captured environments for planning and validation.

Industry relevance: The work comes from NVIDIA, explicitly targets the multi-sensor suites used by autonomous vehicle companies, and reports throughput in millions of pixels and rays per second on NVIDIA A40 and A100 hardware, framing real-time performance as the enabling property for deployment.

Future Directions

• Extending to non-spinning LiDAR. The authors note that their experiments focus on spinning LiDARs but the tiling procedure can be extended to non-spinning sensors by also tiling along the azimuth dimension according to the beam pattern.
• Privacy mitigation. The ethics statement raises the risk that training data containing faces and license plates persists inside the model, suggesting semantic distillation into the representation, while noting that preprocessing the input data before training is a more effective mitigation.
• Practical use of the factorization. The authors state the factorized representation is of interest to many downstream applications, but leave that exploration open.
• Further sensor and effect coverage. The paper handwaves non-pinhole, rolling-shutter-like and beam-divergence effects but does not report an evaluation of the full space of sensor models, and does not report training times or particle counts, leaving cost of the recipe uncharacterized.

Target Audience

Researchers and engineers working on neural rendering, 3D Gaussian Splatting, and autonomous driving simulation, particularly those interested in multi-sensor reconstruction, LiDAR rendering, or real-time rasterization pipelines. Readers need prior familiarity with 3DGS, NeRF, and camera/LiDAR sensor models to follow the method section; the results tables are otherwise readable by anyone tracking benchmarks on Waymo and PandaSet.

Authors’ abstract

Rigorous testing of autonomous robots, such as self-driving vehicles, is essential to ensure their safety in real-world deployments. This requires building high-fidelity simulators to test scenarios beyond those that can be safely or exhaustively collected in the real-world. Existing neural rendering methods based on NeRF and 3DGS hold promise but suffer from low rendering speeds or can only render pinhole camera models, hindering their suitability to applications that commonly require high-distortion lenses and LiDAR data. Multi-sensor simulation poses additional challenges as existing methods handle cross-sensor inconsistencies by favoring the quality of one modality at the expense of others. To overcome these limitations, we propose SimULi, the first method capable of rendering arbitrary camera models and LiDAR data in real-time. Our method extends 3DGUT, which natively supports complex camera models, with LiDAR support, via an automated tiling strategy for arbitrary spinning LiDAR models and ray-based culling. To address cross-sensor inconsistencies, we design a factorized 3D Gaussian representation and anchoring strategy that reduces mean camera and depth error by up to 40% compared to existing methods. SimULi renders 10-20x faster than ray tracing approaches and 1.5-10x faster than prior rasterization-based work (and handles a wider range of camera models). When evaluated on two widely benchmarked autonomous driving datasets, SimULi matches or exceeds the fidelity of existing state-of-the-art methods across numerous camera and LiDAR metrics.