DyRAD: Radar Novel View Synthesis for Dynamic Driving Scenes

AI summary

Overview

Research area: Radar-based neural scene reconstruction and novel-view synthesis (NVS) for autonomous driving, sitting at the intersection of radar signal processing, differentiable rendering, and dynamic driving-scene simulation.

Technical level: Advanced — the paper assumes familiarity with neural rendering (NeRF, 3D Gaussian Splatting), radar signal-processing chains (range-FFT, azimuth beamforming, Doppler processing, CFAR detection), and standard reconstruction metrics (PSNR, SSIM, LPIPS, Pearson correlation, Chamfer distance, F1).

Scope: DyRAD is a differentiable, physics-grounded renderer that reconstructs dynamic driving scenes as static and motion-tracked point reflectors and synthesizes complete range–azimuth–Doppler (RAD) radar tensors from novel sensor poses, evaluated on RADIal, Boreas, and a new synthetic benchmark.

What This Paper Is About

Radar is the one automotive sensor that measures radial velocity directly through Doppler, yet existing radar novel-view synthesis methods throw this capability away: methods that handle dynamic scenes reconstruct only range–azimuth tensors, while methods that render Doppler assume everything in the scene is static. A second problem is that radar's signal-processing chain spreads each reflection across multiple range, azimuth, and Doppler bins, and prior methods let the scene representation absorb that spread as if it were real geometry, so it renders incorrectly once the viewpoint moves.

DyRAD's goal is to reconstruct a dynamic driving scene from recorded radar measurements in a way that (a) renders Doppler from learned object motion, (b) keeps sensor-induced blur separate from scene structure, and (c) still works when the virtual sensor is placed off the originally driven trajectory.

Key Contributions

1. Physics-grounded RAD rendering of dynamic scenes. The authors state this is the first radar NVS method to render Doppler for dynamic driving scenes. The scene is represented as static and dynamic point reflectors that follow rigid object tracks, and complete RAD tensors are rendered with Doppler derived from relative sensor–reflector motion.

2. Decoupling scene structure from sensor response. Each point reflector is rendered through a fixed, sensor-specific point-spread function (PSF) derived from the radar's signal-processing chain, separating scene structure from sensor-induced spread. This improves object-region reconstruction and detection and enables rendering the same scene under alternative radar configurations without refitting.

3. Off-path evaluation for radar NVS. Prior radar NVS evaluates only along the recorded trajectory, which the authors argue tests interpolation rather than spatial generalization. They introduce off-path evaluation via displaced ground-truth views in a synthetic benchmark and a cycle-consistency protocol adapted to real radar recordings.

Main Findings

• On-path RADIal reconstruction and detection: DyRAD reaches full-RAD Pearson correlation 0.272 (versus 0.068 for RadarSplat, the strongest baseline on that measure), object-region RAD correlation 0.658, and a foreground hit rate of 90.7% of reference-detected objects, compared with 26.9% for the strongest baseline. Chamfer distance drops to 4.01 m and joint detection F1 reaches 0.179.

• The static variant already beats the baselines, but misses vehicles: DyRAD-static (same representation and renderer, all reflectors static, no bounding-box separation) achieves full-measurement RA correlation 0.533 on RADIal versus 0.291 for RadarSplat and 0.055 for RadarFields, yet its object-region RAD correlation is −0.013 and its foreground hit rate is 7.0%. Adding motion modeling and Doppler raises those to 0.658 and 90.7%.

• Boreas (RA-only sensor): With the Navtech 360° spinning radar, which provides no Doppler, DyRAD's advantage is modest in full-measurement correlation (0.752 versus 0.666 for RadarSplat) but large within object regions (0.622 versus 0.178).

• Real-data off-path test (2 m lateral-shift round trip): DyRAD posts the best means across metrics, reaching foreground hit rate 91.7% versus 18.1% for the strongest prior baseline and 20.9% for the static model. On Boreas, RadarSplat nearly matches PSNR (24.29 versus 24.51 dB) but its object-region RA correlation is far lower (0.211 versus 0.641).

• Synthetic off-path evaluation (no refitting): Across 25 scene–offset pairs (lateral offsets of −1.75, +1.75, and +3.5 m; yaw offsets of +5° and +10°), object-region RAD correlation reaches 0.356, versus 0.161 for DyRAD-static and 0.086 for RadarFields. Chamfer distance is nearly tied with RadarSplat (5.13 versus 5.14 m) but joint detection F1 is much higher (0.227 versus 0.106). Foreground hit rate is 74.6%, versus 22.4% for DyRAD-static and 36.0% for RadarSplat — though the higher coverage brings more reference-empty annotated boxes containing predicted detections (4.9 on average, versus 0.7 and 3.4).

• Doppler supervision refines velocity, not coverage: Adding Doppler supervision raises joint detection F1 from 0.144 to 0.179 on-path and from 0.120 to 0.236 off-path, and reduces object Doppler peak error from 1.843 to 0.732 bins (on-path) and from 2.264 to 1.375 bins (off-path) — a 60% reduction relative to RA-only fitting. Foreground hit rate barely changes, and full-measurement RA correlation drops by 0.055 (on-path) and 0.033 (off-path).

• Interpolation consistency loss cuts both ways: On-path it improves RA, RD, and RAD correlation by 0.219, 0.143, and 0.051, and raises precision from 0.162 to 0.233. Off-path it improves RA PSNR by 1.50 dB and SSIM by 0.109 but lowers RAD correlation (from 0.447 to 0.363) and F1 (from 0.296 to 0.236), both still above all baselines. The authors keep it to balance temporal interpolation and RA fidelity against off-path RAD fidelity.

• Fixing the PSF matters more than learning spread: Compared with Gaussian primitives with learned extent and point reflectors with learned PSF bandwidths, the fixed sensor-derived PSF more than doubles detection recall. Off-path F1 is 0.236, versus 0.074 for learned extent and 0.124 for learned bandwidths.

• Sensor-configuration transfer without refitting: Fitting on a coarse radar configuration and then swapping in a finer PSF and sampling grid reduces Chamfer distance from 7.27 to 3.20 m and increases F1 from 0.101 to 0.195 relative to linear upsampling of the same model's coarse renders, although upsampling scores higher on most correlation and SSIM metrics.

Methodology in Plain English

DyRAD treats a radar scene as a cloud of zero-size "point reflectors" rather than blobs or occupancy fields. Each reflector stores a position, a base reflected power, and view-dependent reflectivity encoded with spherical harmonics. Reflectors are split into a static background and a set of dynamic objects; every object follows a planar rigid track with one learned point per training timestamp, linearly interpolated in between, with orientation estimated from the direction of travel. A dynamic reflector's world position is its object track's position plus its fixed offset in the object's own reference frame.

To render, each reflector is projected into the sensor frame to get range, azimuth, and viewing direction. Its velocity comes from the object track — translational velocity estimated as the slope of a linear fit to four neighboring track points, plus the component induced by rotation — and the sensor's own velocity is subtracted to get the radial velocity along the line of sight. That velocity is wrapped into the sensor's unambiguous Doppler interval, reproducing the aliasing real radars exhibit.

Instead of letting the network learn how wide each reflector's return should be, DyRAD multiplies each reflector's power by a fixed kernel along range, azimuth, and Doppler, precomputed from the radar's signal-processing chain (range-FFT leakage, beamformer response, Doppler-processing response) and frozen during optimization. This is the central design choice: measurement spread can no longer be explained away as broad scene geometry, and the same scene can be re-rendered for a different radar by swapping the kernel. Contributions are summed incoherently rather than alpha-composited, then converted to the recorded intensity scale with a fixed measurement floor.

Reflectors are initialized from peaks in the radar maps (background peaks accumulated in world coordinates, object peaks extracted from the strongest Doppler bin inside each annotated box) and optimized jointly with the track points using a reconstruction loss plus an interpolation-consistency loss applied only to the range–azimuth projection. New background reflectors are densified periodically from positive reconstruction residuals.

Why This Matters

Impact on research. The paper reframes a modeling question that radar NVS had largely sidestepped: whether a reconstruction has learned the scene or the sensor. By showing that fixing an analytic PSF more than doubles detection recall relative to learned extent or bandwidth, it argues that letting the renderer learn its own blur can silently compensate for errors in reflector placement and reflectivity. It also challenges the field's evaluation convention — held-out frames along the driven trajectory sit between near-identical training poses, so naive interpolation in measurement space can score as well as a correct forward model — and offers two concrete alternatives (synthetic displaced ground truth and a render-and-refit cycle-consistency protocol on real data).

Real-world applications.

• Closed-loop evaluation of autonomous driving stacks, where sensor observations must be synthesized for trajectories the vehicle never drove.
• Training and validating radar perception models (detection, tracking, Doppler-based classification) on data simulated under new ego paths.
• Sensor selection and specification studies: a scene reconstructed once can be re-rendered under different radar configurations instead of collecting new drives.
• Scene editing and augmentation for rare or safety-critical scenarios, demonstrated qualitatively in the paper's figures.

Industry relevance. Radar is prized for direct radial-velocity measurement and operation in adverse weather, but its sparsity and low spatial resolution make it the hardest automotive modality to simulate. A reconstruction that transfers across sensor configurations without refitting speaks directly to the cost of data collection and to hardware-in-the-loop pipelines, where the radar's own processing chain defines the measurement. The authors also state they will release the synthetic benchmark with off-path ground-truth measurements.

Future Directions

• Jointly refining reflector-to-object assignments. Assignments currently rely on bounding-box annotations and are frozen during optimization even as tracks and reflector positions are refined; the authors suggest correcting initialization errors by optimizing them together.
• Modeling multipath and speckle noise. The paper notes these are not modeled and that adding them could improve measurement realism.
• Extending to elevation. The formulation supports 3D reconstruction, but this work uses a ground-plane implementation because the evaluated measurements lack elevation. The authors propose evaluating elevation-resolving radar.
• Transfer across physical sensors. Beyond the processing configurations demonstrated here, the authors suggest exploring whether a reconstruction can move between actual, different radar hardware.

Target Audience

Researchers and engineers working on radar perception, sensor simulation, and neural scene reconstruction for autonomous driving will get the most from this paper, particularly those already familiar with NeRF or Gaussian-splatting pipelines and with radar processing chains. It is also relevant to practitioners who need synthetic radar data at off-trajectory viewpoints for closed-loop testing, and to readers interested in how explicitly modeling a sensor's point-spread function changes what a learned scene representation actually encodes. The paper is written for an expert audience: the method section is dense with radar-specific notation, and the dataset, training-hardware, and runtime details are largely deferred to the appendix, which is beyond the truncated content provided here.

Authors’ abstract

Reconstructing dynamic driving scenes from recorded sensor data supports closed-loop evaluation of autonomous driving systems by synthesizing observations beyond the original trajectory. Unlike cameras and LiDAR, radar measures radial velocity directly through Doppler. Yet existing radar novel-view synthesis fails to exploit this capability: methods addressing dynamic scenes reconstruct only range-azimuth tensors, while methods that render Doppler assume static scenes. Moreover, because radar processing spreads each reflection across multiple bins, existing representations absorb this spread into scene geometry, causing it to render incorrectly when the viewpoint moves. We present DyRAD, which models dynamic driving scenes using static background reflectors and motion-tracked dynamic point reflectors to render complete range-azimuth-Doppler (RAD) tensors. Reflector velocities are derived from object tracks and projected onto the line of sight, making Doppler both a rendered output and supervision for those tracks. Crucially, we render reflectors through a fixed analytic point-spread function (PSF) derived from the radar's signal-processing chain, preventing sensor-induced spread from being baked into the scene representation. Beyond improving scene reconstruction, this separation also enables zero-shot sensor-configuration transfer, allowing the same reconstructed scene to be rendered under different radar specifications without refitting. We evaluate DyRAD on RADIal, Boreas, and a synthetic benchmark across both on-path poses and displaced viewpoints untested by prior work. On RADIal, DyRAD recovers radar detections in 90.7% of reference-detected objects, compared with 26.9% for the strongest baseline.