RadarGen: Automotive Radar Point Cloud Generation from Cameras

AI summary

Overview

Research area: Computer vision / multimodal generative simulation for autonomous driving — specifically, synthesizing automotive radar point clouds from multi-view camera imagery.

Technical level: Advanced. The paper assumes familiarity with latent diffusion models, transformer backbones, bird's-eye-view (BEV) projection, and radar signal attributes such as radar cross section (RCS) and Doppler velocity.

Scope (one sentence): RadarGen is a conditional latent diffusion framework that generates radar point clouds with location, RCS, and Doppler attributes from surround-camera images, evaluated on large-scale driving data against a feedforward baseline using a newly proposed metric suite.

What This Paper Is About

Most neural and generative simulators for autonomous driving produce only RGB imagery and video, with recent extensions to LiDAR. Radar — already ubiquitous in production vehicles for its low cost, light weight, and weather resilience — has received far less attention from the generative modeling community, so current simulators cannot reproduce radar's signal sparsity, radar cross section, and Doppler characteristics. RadarGen addresses this by learning a conditional distribution over radar point clouds given multi-view camera images at two consecutive timesteps, so that camera-only or edited scenes can be enriched with realistic radar without re-recording radar data.

Key Contributions

1. A probabilistic diffusion framework for radar generation. RadarGen is presented as the first probabilistic diffusion framework to generate realistic automotive radar point clouds — including location, RCS, and Doppler — from multi-view camera inputs, producing diverse measurements rather than a single deterministic prediction.

2. A latent diffusion methodology over BEV radar representations. The method trains on a BEV image representation of sparse radar attributes and conditions generation on BEV-aligned visual depth, semantic, and motion priors extracted from pretrained foundation models.

3. A comprehensive evaluation suite. Because no established benchmark existed for radar point cloud generation, the authors introduce metrics for geometric fidelity, radar attribute fidelity, and distribution similarity, computed for both the entire scene and foreground objects, and validate their design through ablations.

4. Demonstrated usability and controllability. Generated radar data can be interpreted by a detector trained on real data, and the framework supports applications such as scene editing and radar augmentation.

Main Findings

• Broad quantitative gains over the baseline on MAN TruckScenes. On the Entire Area, RadarGen reaches CD Loc. of 1.68 ± 0.39 versus 1.84 ± 0.48 for the baseline, IoU@1m of 0.31 ± 0.11 versus 0.23 ± 0.10, DA Precision of 0.26 ± 0.12 versus 0.14 ± 0.10, and DA F1 of 0.24 ± 0.12 versus 0.14 ± 0.09. Distribution similarity also improves, with MMD Loc. 0.056 ± 0.062 versus 0.368 ± 0.151, MMD RCS 0.09 ± 0.15 versus 0.36 ± 0.25, and MMD Doppler 0.31 ± 0.74 versus 0.65 ± 0.64.

• One metric favors the baseline. CD Full on the Entire Area is 0.038 ± 0.009 for the baseline versus 0.040 ± 0.008 for RadarGen, which the authors describe as expected because the baseline was trained using a similar loss objective.

• Stronger foreground object fidelity. For foreground objects (bounding boxes with more than 60% camera visibility, classes Car, Truck, Trailer), RadarGen achieves CD Loc. of 0.95 ± 0.65 versus 1.32 ± 0.79, CD Full of 0.069 ± 0.049 versus 0.075 ± 0.049, Density Similarity of 0.51 ± 0.41 versus 0.35 ± 0.43, and Hit Rate of 0.66 versus 0.37. Per-class MMD improves for all three classes across location, RCS, and Doppler.

• Generated radar is partially consumable by a real-data detector. A VoxelNeXt detector trained on MAN TruckScenes radar achieves an NDS of 0.48 on real data (50m range), 0.30 on RadarGen point clouds, and nearly zero on baseline point clouds. The paper notes the remaining gap while the hit rate within true bounding boxes is 0.66.

• Semantic conditioning matters most in ablations. Zeroing out the semantic map causes the most significant degradation, worsening geometric fidelity and notably increasing RCS MMD for both the Entire Area and per-object foreground classes.

• Appearance and velocity maps primarily help Doppler. Ablating either the appearance map or the velocity map mainly degrades the Doppler MMD. The authors argue that appearance details, even with segmentation present, help the model refine object class understanding and thus generate a more realistic motion profile.

• BEV conditioning beats direct multi-view conditioning on efficiency and geometry. A model conditioned directly on multi-view camera images (with Plücker and modality embeddings) improves MMD for radar attributes on the Entire Area, but the BEV-conditioned model gives better overall geometric fidelity, and the multi-view approach requires over 3× runtime: 9 days of training versus 2 days.

• Reconstruction trade-off drives a key hyperparameter. Testing σ ∈ {0.5, 1, 1.5, 2, 2.5, 3} with four recovery methods (random, peak, peak+random, deconv), larger σ reduces autoencoder reconstruction error because smoother maps are easier to reconstruct, but over-smooths structure and degrades point cloud recovery. The authors set σ = 2, and deconv consistently yields the best point cloud quality across all σ values.

• Scene editing propagates to radar. Using an off-the-shelf image editing tool (e.g., ChronoEdit) to remove or insert objects updates the radar response; in a car-to-truck replacement example the model correctly removes radar points from the area newly occluded by the truck.

Methodology in Plain English

Radar returns arrive as sparse, unordered 3D points with non-uniform sampling, because they come from peak-based target extraction rather than the grid-aligned measurements typical of LiDAR. That makes the usual range-image representation unusable, so RadarGen instead converts each radar point cloud into three dense bird's-eye-view images: a Point Density Map made by convolving the rasterized points with a Gaussian kernel, an RCS Map, and a Doppler Map, where each pixel in the latter two inherits the attribute of the nearest detection through a Voronoi tessellation. Elevation is discarded because automotive radar has limited vertical resolution. Each map is replicated across three channels to fit the autoencoder's RGB input space and encoded independently into latent representations that serve as diffusion supervision targets.

The generator is built on SANA, an efficient latent diffusion model that uses an autoencoder with 32x compression and linear-attention scaling in place of the quadratic self-attention typical of standard latent diffusion models. This choice matters because the multi-channel radar representation and conditioning cues produce a large number of tokens.

Rather than forcing the denoiser to infer depth, semantics, and motion from scratch, RadarGen conditions on BEV maps built from pretrained foundation models: an Appearance map colored from the original images, a Semantic map colored from segmented images (color-coded rather than one-hot, to preserve image-like statistics), and a Radial Velocity map derived from optical flow between consecutive frames plus metric depth, back-projected to 3D and reduced to the radial component. These conditioning tensors are spatially aligned with the radar targets and concatenated along the channel dimension.

The denoiser is a Diffusion Transformer operating jointly on the three radar map latents, concatenated into a single token sequence with shared self-attention and learnable modality identifiers so each channel keeps its own statistics while still sharing information. At inference, noise is denoised under BEV conditioning, decoded into BEV images, and then the sparse point cloud is recovered: because the Gaussian blur kernel used to build the density map is known and fixed, recovery is posed as an explicit L1-regularized non-negative deconvolution (LASSO) solved with an Iteratively Reweighted L1 scheme and a FISTA solver, then thresholded. RCS and Doppler values are retrieved from the corresponding map locations.

Implementation: modified SANA DiT with the pretrained v1.1 autoencoder, UnidepthV2 for metric depth, Mask2Former (trained on Cityscapes) for semantic segmentation, and UniFlow for flow prediction. Training ran for 2 days on 8 L40 (48GB) GPUs with each condition dropped with 10% probability; radar is filtered to ±50 m and a 512×512 grid is used.

Why This Matters

Impact on research. The paper argues that the imbalance between visual and radar simulation limits the fidelity of neural simulators, and shows a path to extending unified generative simulation across sensing modalities. It also contributes a benchmark suite for a task that previously had no standard evaluation, which the authors hope will support future progress in radar generation.

Real-world applications:

• Radar augmentation for camera-only or edited scenes. Camera-conditioned generation means existing visual datasets and simulation frameworks can be enriched with radar without re-recording radar data.
• Rare-scenario data generation. Editing input images lets practitioners add, remove, or replace objects and obtain corresponding radar returns, useful for long-tail conditions that are expensive to capture.
• Pre-training and testing radar-dependent perception. Generated radar can be consumed by detectors trained on real data; in the paper's example, a real vehicle replaced by a generated truck is still perceived as a truck by the detector.
• Simulation pipelines for autonomous driving. The camera-conditioned formulation is described as broadly compatible in principle with existing visual datasets and simulators, offering a scalable direction for multimodal generative simulation.

Industry relevance. Radar is already ubiquitous in production vehicles, providing low-cost, lightweight, weather-resilient perception, and the paper notes that commercial survey vehicles typically record only processed point clouds because raw radar storage is memory intensive. A framework that synthesizes usable radar point clouds from cameras targets exactly this data-scarcity bottleneck.

Future Directions

• Video input. The authors list extending the framework to video as future work.
• Text-based conditioning. Text conditioning is named as another planned extension.
• Multiple datasets and radar configurations. Training across more datasets and sensor setups is proposed; nuScenes evaluation is already reported in the supplementary material, and training on multiple radar configurations could broaden applicability.
• Modeling explicit radar mechanisms. The paper states it did not model explicit radar mechanisms because doing so requires large-scale raw data and detailed radar specifications and internal mechanisms that are often undisclosed, and calls this a fruitful next step if such data becomes available.
• Open question: the remaining detection gap. Generated radar reaches a hit rate of 0.66 within true bounding boxes yet achieves an NDS of 0.30 against 0.48 on real data; the authors suggest the detector is tailored to intricate properties of true data that the model may not fully capture, and state that detailed analysis of the subtle differences between real and generated radar is beyond the paper's scope.
• Open question: hallucination in occluded regions. The model can generate points in areas not directly visible to the cameras — desirable for filling in occluded objects, but potentially leading to uncontrolled hallucinations. Performance is also tied to upstream foundation models, which underperform in low-light night scenes, with strong reflections, or during camera occlusion.

Target Audience

Researchers and practitioners in autonomous driving perception, multimodal generative modeling, and simulation who need radar data without radar hardware. It will be most useful to readers already comfortable with latent diffusion models, diffusion transformers, and BEV representations, as well as to engineers building sensor-simulation or data-augmentation pipelines and to anyone benchmarking radar point cloud generation, since the paper defines the metric suite for that task.

Authors’ abstract

We present RadarGen, a diffusion model for synthesizing realistic automotive radar point clouds from multi-view camera imagery. RadarGen adapts efficient image-latent diffusion to the radar domain by representing radar measurements in bird's-eye-view form that encodes spatial structure together with radar cross section (RCS) and Doppler attributes. A lightweight recovery step reconstructs point clouds from the generated maps. To better align generation with the visual scene, RadarGen incorporates BEV-aligned depth, semantic, and motion cues extracted from pretrained foundation models, which guide the stochastic generation process toward physically plausible radar patterns. Conditioning on images makes the approach broadly compatible, in principle, with existing visual datasets and simulation frameworks, offering a scalable direction for multimodal generative simulation. Evaluations on large-scale driving data show that RadarGen captures characteristic radar measurement distributions and reduces the gap to perception models trained on real data, marking a step toward unified generative simulation across sensing modalities.