JustDepth: Real-Time Radar-Camera Depth Estimation with Single-Scan LiDAR Supervision

AI summary

Overview

Research area: Computer vision for autonomous driving — radar-camera sensor fusion for dense metric depth estimation, with graph neural networks for global feature propagation.

Technical level: Advanced. The paper assumes familiarity with convolutional and transformer encoders, self-attention, graph neural networks (Max-Relative Graph Convolution, VIG layers), and depth-estimation loss design.

Scope: A single-stage radar-camera depth estimator, trained on single-sweep LiDAR without auxiliary annotations, that targets real-time on-board inference.

Links/authors: Wooyung Yun and Dongwook Kim (Department of Artificial Intelligence, Ajou University, Suwon, Republic of Korea); Soomok Lee (corresponding author, Department of Computer Science, Kennesaw State University, Marietta GA). Code: https://github.com/TPyun/JustDepth.

What This Paper Is About

Self-driving systems need dense, metrically accurate depth at low latency, but no single sensor provides it: cameras give rich appearance without metric scale, LiDAR gives accurate depth but degrades in adverse weather and is expensive to use densely, and automotive mmWave radar gives metric range but is sparse and noisy. Existing radar-camera depth pipelines are often multi-stage, depend on pretrained monocular depth models, or require auxiliary labels such as semantic/panoptic masks and 3D bounding boxes to build dense LiDAR ground truth. JustDepth instead performs single-stage fusion of one RGB image with one radar scan, trains only with raw single-scan LiDAR, and specifically targets the stripe artifacts that arise from that sparse supervision.

Key Contributions

1. A single-stage, constant-latency architecture. All radar returns in a frame are compressed into a fixed-width 1D representation, so the radar branch's computation does not scale with the number of points. Image and radar features are fused via height-wise self-attention in a Height Fusion Block, and depth cues are propagated globally by a GNN.
2. A training-only confidence decoder. An auxiliary head directly supervises the fusion module to localize radar-supported pixels, without producing an explicit intermediate depth map, and is discarded at inference so it adds zero test-time cost.
3. An LiDAR Distribution Leakage (LDL) mitigation strategy, using point upsampling plus a synchronized rotation with reflection padding applied to the image, radar, and LiDAR simultaneously.
4. A new metric, the Vertical-Horizontal Gradient Ratio (VHGR), to quantify scanline striping artifacts across the whole depth map, addressing limitations of the existing object-level MAE (OMAE) metric.

Main Findings

• Latency advantage: JustDepth processes a frame in 14.8 ms on an NVIDIA RTX 4070 Ti, about 39.7 times faster than GET-UP (587.6 ms). The abstract reports a 39.7× inference-time reduction versus recent state-of-the-art methods.
• Competitive accuracy at lower complexity: In Table I, JustDepth reaches MAE of 1324.2 mm (0–50 m), 1674.0 mm (0–70 m) and 1771.3 mm (0–80 m), with RMSE of 3285.5 / 4292.9 / 4602.0 mm, AbsRel 0.074 and log10 0.033 (relative metrics at 0–70 m). MAE is 8.54% higher than GET-UP (1541.0 mm at 0–70 m), but AbsRel is the best among compared methods (0.074 vs. 0.075 for GET-UP).
• Accuracy against the fastest prior method: Li et al. runs at about 51.1 ms per frame; JustDepth is more than 3× faster and reduces MAE by about 8.08% at the full 0–80 m range (1771.3 mm vs. 1927.0 mm), despite not using semantic segmentation guidance.
• Best stripe suppression: JustDepth attains the lowest VHGR (0.1767) in Table II, versus Li et al. (0.1868), Singh et al. (0.3813) and GET-UP (0.5210). The abstract summarizes this as a 66% reduction in stripe artifacts as measured by VHGR. JustDepth also has the lowest Vertical Gradient Magnitude (VGM, 0.1090), while Li et al. has the lowest OMAE (1881.0 mm vs. JustDepth's 2119.9 mm).
• OMAE vs. VHGR: The authors argue OMAE is restricted to LiDAR samples inside selected semantic masks, so classes with many returns dominate and striping in sparse or non-selected regions is weakly reflected; they therefore treat OMAE as a complementary object-level indicator and VHGR as the better whole-image LDL measure.
• Deeper GNN helps accuracy at higher cost: Table III shows FLOPs and parameters growing with GNN depth (33.67 G / 9.46 M at N=0; 42.58 G / 15.81 M at N=8; 51.49 G / 22.15 M at N=16), while MAE₇₀ falls from 1774.1 to 1674.0 to 1644.6 mm. At similar cost (43.29 G, 16.32 M), the 8-layer convolutional baseline is worse (MAE₇₀ 1768.8 mm), so N=8 is chosen as the balanced operating point.
• Every component contributes: Table IV shows removing the confidence decoder degrades MAE₇₀ to 1712.4 mm, removing self-attention to 1738.7 mm, and removing the GNN to 1774.1 mm (versus 1674.0 mm for the full model); RMSE₇₀ follows the same ordering.
• A deliberate accuracy/artifact trade-off: Table V shows that disabling rotation (Θ=0°) yields much lower MAE (794.1 mm) and RMSE (2701.6 mm) but VHGR of 0.8222 and strong horizontal striping; the full model trades that for 1674.0 mm / 4292.9 mm and VHGR 0.1767. Table VI shows VHGR decreasing with larger rotation ranges: 0.8222 at Θ=0, 0.2227 at 5, 0.1767 at 10, 0.1585 at 15, while MAE/RMSE increase slightly.
• Compared to Li et al.'s augmentation: Swapping in their crop-based Camera Intrinsics Disruption scheme lowers MAE₇₀ (1586.6 mm) but raises VGM and VHGR by roughly 35% (VHGR 0.2699 vs. 0.1767 for the authors' rotation + upsampling).
• Qualitative behavior: Predictions are reported as smoothly varying and stripe-free from day-time through dark night-time scenes, including where Li et al. still shows visible striping under low-light conditions.

Methodology in Plain English

JustDepth takes two inputs: one RGB image and one radar scan.

• Radar encoding: Calibrated radar points are projected into the image plane, but only the horizontal pixel coordinate and the range are kept — vertical position, Doppler, and RCS are discarded. Points are rasterized into a 1D grid of width W, storing the minimum range per column and setting empty columns to zero. Because this representation has a fixed width, the radar encoder's cost does not depend on how many radar returns exist in a frame.
• Fusion: Since camera and radar are horizontally aligned by calibration, fusion is done column by column. Each image column becomes an h×C token sequence, and the corresponding radar vector is copied down the column and concatenated, giving h×2C tokens. A self-attention block then reasons along the height dimension within each column, learning where in each column the radar depth prior should be placed. All columns share weights and are processed independently, giving complexity O(w·h²).
• Global propagation: The fused features are added to the final image features and reshaped into M = h·w tokens, one per pixel. A K-NN graph is built in feature space (K=9 neighbors, with a dilated schedule across depth), and Max-Relative Graph Convolution plus a feed-forward block exchanges messages between similar tokens — which are often on the same object even when far apart in the image. Stacking N=8 layers gives a large, data-adaptive receptive field.
• Decoding: A U-Net-style decoder upsamples through four stages, fusing GNN output with image-encoder skip features, to produce the dense depth map at input resolution.
• Training-only confidence head: A separate lightweight head predicts a per-pixel logit saying whether the depth at that pixel is supported by radar. Its target is built by checking whether a pixel lies inside a vertical strip around a radar column, has valid LiDAR, and agrees with the radar range within a threshold; it is trained with a BCEWithLogits loss. The head is thrown away at inference.
• Losses: Total loss = depth regression (masked Smooth L1/Huber over valid pixels) + edge-aware smoothness (first-order depth variation weighted by exp(−Sobel gradient), so depth can break at image edges) + confidence term, with λ_smooth = 0.1 and λ_conf = 10.0.
• LDL mitigation: A single LiDAR sweep leaves nearly horizontal scanlines with big gaps. The authors insert extra LiDAR samples at midpoints between pairs of projected points that are horizontally close (τ_x = 4 px), vertically separated (τ_y = 16 px), and depth-similar (τ_d = 0.2 m), found via a KD-tree with radius ρ = 48 px. They also rotate the image, radar, and LiDAR by the same random angle in [−10°, 10°], applying reflection padding to the image because zero padding creates black borders the network could exploit as a shortcut.
• Setup: nuScenes with 1000 scenes split 700/150/150 for train/validation/test; inputs at 900×1600 with 100 radar points; 200 epochs on two NVIDIA RTX 3090 GPUs, per-GPU batch size 8 (16 total); AdamW with a cosine-annealed learning rate.

Why This Matters

Impact on research: The paper argues that the field's trend toward multi-stage pipelines, pretrained monocular depth modules, and auxiliary label dependence raises latency and harms portability across datasets and sensors. JustDepth is a counterexample: single-stage, single-scan supervision, no auxiliary annotations. It also introduces VHGR as a whole-image measure of LiDAR Distribution Leakage, and shows empirically that MAE/RMSE and VHGR move in opposite directions under sparse single-sweep supervision — a measurement caveat for anyone benchmarking on this kind of ground truth.

Real-world applications:

• On-board perception for autonomous vehicles and ADAS, where 14.8 ms per frame on an RTX 4070 Ti-class device matters for time-critical control loops.
• Robotics and mobile platforms that already carry a camera and automotive radar, where adding dense metric depth without LiDAR or heavy annotations lowers sensor and labeling cost.
• Perception stacks that must keep operating when LiDAR degrades in adverse weather, since radar is weather-robust (though this paper does not report an adverse-weather benchmark of its own).
• Dataset and sensor portability, since the method avoids semantic/panoptic masks, 2D/3D boxes, ego-pose-dependent multi-frame accumulation, and pretrained monocular models.

Industry relevance: The reported trade-off — about 39.7× faster than GET-UP with only 8.54% higher MAE at 0–70 m and the best AbsRel, and more than 3× faster than the previous fastest method with roughly 8.08% lower MAE at 0–80 m — is directly relevant to automotive deployment constraints, where compute budget and annotation pipeline cost are often the binding limits.

Future Directions

• Validate the weather-robustness claim empirically. The motivation rests on radar working when LiDAR degrades, but no dedicated adverse-weather benchmark or evaluation is reported; nuScenes qualitative results cover day-time through dark night-time.
• Recalibrate evaluation under sparse supervision. The authors note that even an oracle model cannot reach zero MAE against sparse, scanline-structured LiDAR ground truth, so a small MAE/RMSE penalty accompanies stronger LDL suppression. This raises the question of what the right target metric or supervision signal should be.
• Push the VHGR/OMAE discussion further. Which whole-image artifact metrics should the community standardize on, and how do they correlate with downstream planning and control performance?
• Explore the accuracy-vs-artifact operating point. Table VI shows VHGR continuing to fall at Θ=15° while MAE/RMSE rise; the paper does not report whether a different balance, or combining the confidence branch with other supervision, could improve both simultaneously.
• Test transfer across datasets and sensor configurations. The paper's stated motivation is portability — no auxiliary labels, no monocular pretraining — but it evaluates only on nuScenes.

Target Audience

Researchers and engineers working on autonomous driving perception, radar-camera sensor fusion, and efficient depth estimation; practitioners who need real-time dense metric depth on constrained hardware; and anyone studying supervision-efficient training, since the paper is largely about achieving competitive depth from single-sweep LiDAR with no auxiliary annotations. Readers evaluating depth-estimation benchmarks will also find the OMAE-versus-VHGR discussion relevant.

Authors’ abstract

Accurate yet low-latency depth is essential for radar-camera perception in autonomous systems. Cameras provide rich appearance but lack metric scale, whereas automotive radar offers metric range but is sparse and noisy. Many pipelines are multi-stage or depend on auxiliary annotations, increasing latency and limiting portability. We introduce JustDepth, a single-stage radar-camera depth estimator trained only with radar, camera, and single-scan LiDAR. All radar returns are aggregated into a fixed-width 1D representation, decoupling runtime from point count. A Height Fusion Block fuses modalities, a lightweight GNN propagates depth globally, and a training-only confidence decoder stabilizes learning with zero test-time cost. We mitigate stripe artifacts via simple augmentations and quantify them using the Vertical-Horizontal Gradient Ratio (VHGR). On nuScenes, compared to recent state-of-the-art methods, JustDepth maintains accuracy while reducing inference time by 39.7x and stripe artifacts by 66% as measured by VHGR.