Towards Temporal Fusion Beyond the Field of View for Camera-based Semantic Scene Completion

AI summary

Overview

• Research area: Camera-based 3D semantic scene completion (SSC) for autonomous driving, specifically temporal fusion across frames.
• Technical level: Advanced (assumes familiarity with 3D perception, backprojection, deformable attention, and voxel-based scene completion).
• Scope: The paper introduces C3DFusion, a module that fuses 3D-lifted point features from current and historical frames in a shared metric space to reconstruct regions outside the current camera's field of view.

What This Paper Is About

Camera-based SSC methods predict a voxelized 3D scene with semantic labels from images, and recent work uses past frames to enrich the current one. However, these methods mostly improve regions already visible in the current frame and still fail to reconstruct out-of-camera-view blind spots near the ego-vehicle, even though earlier frames often observed those areas. The paper proposes C3DFusion, which explicitly aligns 3D point features lifted from both current and historical frames in the current frame's coordinate system to complete those unseen regions.

Key Contributions

1. The authors identify and address out-of-camera-view completion via temporal cues in camera-based SSC, which they describe as a previously overlooked factor in safety-critical driving applications.
2. They propose C3DFusion, a temporal geometry fusion method that performs perspective alignment between explicitly point-mapped historical and current features in the current frame's 3D metric space, rather than fusing LSS-style volumetric features.
3. They introduce two refinement techniques: historical context blurring, which attenuates historical point features according to depth to suppress noise from inaccurate warping, and current-centric feature densification, which increases the current frame's volumetric contribution.
4. Built on C3DFusion, their model reaches state-of-the-art results on SemanticKITTI (IoU 47.62, mIoU 18.98) and SSCBench-KITTI-360 (IoU 49.28, mIoU 21.74), and improves several other baseline architectures when integrated into them.

Main Findings

• SemanticKITTI test set: The model achieves IoU 47.62 and mIoU 18.98, which the authors report as state-of-the-art and a significant improvement over prior methods.
• Out-of-view (OOV) performance on SemanticKITTI: Evaluated on the validation set using the authors' implementation, the model reaches IoU 44.37 and mIoU 17.17, while compared methods perform significantly worse on the same regions.
• SSCBench-KITTI-360 test set: The model achieves IoU 49.28 and mIoU 21.74. For OOV regions on the test set, Table 2 reports IoU 52.41 and mIoU 23.72 (the running text states 52.41 IoU and 17.17 mIoU, which does not match the table).
• Ablation on components (SemanticKITTI validation): A temporal LSS fusion baseline reaches IoU 48.59 and mIoU 16.58. Replacing LSS alignment with point-based temporal fusion (TPFA) gives 49.09 IoU and 18.45 mIoU. Adding historical context blurring (HCB) gives 48.99 / 18.88, adding densification (CCFD) gives 48.87 / 18.86, and the full C3DFusion gives the best result at 49.53 / 19.31.
• Generalization across architectures: Adding C3DFusion to VoxFormer-S improves mIoU from 12.35 to 15.12, to OccFormer from 13.46 to 18.29, to ScanSSC from 17.12 to 18.73, and to CGFormer from 16.87 to 19.31. The authors note that OccFormer and ScanSSC with C3DFusion surpass the previous state-of-the-art mIoU of 18.22 reported for L2COcc.
• Temporal LSS fusion is unreliable: With VoxFormer-S and CGFormer, temporal LSS fusion changes mIoU by -2.79 and -0.29 respectively, while C3DFusion improves mIoU for every tested architecture.
• Densification intensity: Interpolation factors of 1x, 2x, 3x, and 4x yield mIoU of 18.88, 19.31, 19.08, and 18.86, with 2x best; the authors attribute degradation at higher factors to amplified depth estimation errors.
• Applying blurring to the current frame hurts: Applying context blurring only to the current frame gives IoU 48.74 and mIoU 18.29, and to both frames 49.22 / 18.76, versus 49.53 / 19.31 for history-only blurring.
• Number of frames: Performance improves from 1 frame (IoU 45.99, mIoU 16.37) through 4 frames (49.53 / 19.31) and then declines at 5 frames (49.20 / 18.58); 4 frames is used empirically.
• Efficiency: The full model uses 160 M parameters and 23987 MB of memory at mIoU 19.31, compared with HTCL-S at 182 M parameters, 34593 MB, and mIoU 17.13, and the temporal LSS fusion variant at 163 M, 22687 MB, and mIoU 16.58.
• Qualitative result: In cases where cars or pedestrians are invisible but very close to the ego-vehicle in the current frame, VoxFormer-T and HTCL-S perform comparably to single-frame CGFormer, while the proposed method reconstructs the missing structures.

Methodology in Plain English

The model follows the standard camera-based SSC pipeline of viewing transformation, voxel processing, and semantic prediction, with C3DFusion inserted at the viewing transformation stage.

Instead of lifting features into dense volumetric grids and fusing them across time, the method works directly with points. For each of the n input RGB frames, an image encoder produces 2D features and a pretrained depth estimator produces depth maps. Each pixel is backprojected with known camera parameters to a 3D point, and the corresponding 2D feature is attached to that point.

Historical point clouds are then warped into the current frame's coordinate system using relative poses from camera extrinsics, so that current and past geometry live in one shared metric space.

Two refinements are applied before fusion. Historical context blurring computes per-point weights as 1 minus the min-max normalized depth of each historical frame, so that farther points, whose depth estimates are less reliable, contribute less. Current-centric feature densification bilinearly upsamples the current frame's point features and depth map before backprojection (default 2x resolution), so the current frame is not outnumbered by historical points in overlapping regions.

All warped points are merged, points outside the target voxel grid are discarded, and the rest are voxelized by summing the point features in each voxel and dividing by the number of frames n. The resulting volume goes through 3D deformable cross-attention on occupied voxels and self-attention on unoccupied voxels in an MAE-like refinement scheme, then through a voxel branch (3D ResNet) and a TPV branch (2D Swin Transformer) whose outputs are combined by learned weights. A prediction head produces voxel-wise class logits, upsampled and softmaxed to the final volume. Training combines cross-entropy, two affinity losses, and a depth loss, with the depth loss weighted at 0.001 and the others at 1.

Training uses 25 epochs on 4 NVIDIA A6000 GPUs with batch size 4 and AdamW (beta1 = 0.9, beta2 = 0.99), with a cosine learning rate schedule.

Why This Matters

The work argues that safety-critical driving perception cannot rely only on what the current camera sees, and that earlier frames already contain usable information about nearby blind spots. It reframes temporal fusion in SSC from "smoothing what is visible" to "recovering what is not," and shows the idea transfers across multiple existing architectures, which makes it relevant to researchers building practical camera-only perception stacks rather than only LiDAR-based ones.

Real-world applications:

• Autonomous driving perception, where pedestrians, cyclists, or vehicles beside the ego-vehicle fall outside the camera's field of view.
• Advanced driver assistance and blind-spot warning systems built on low-cost camera suites.
• Robotics and mobile platforms using single-view RGB sensors for 3D occupancy estimation in cluttered environments.
• Low-cost 3D scene reconstruction where LiDAR is too expensive or unscalable.

Industry relevance: camera-based SSC is attractive because RGB sensors are cheap and scalable relative to LiDAR, and the demonstrated gains in out-of-view occupancy estimation target exactly the failure cases that matter for deployment. The reported efficiency numbers (160 M parameters, comparable memory to the temporal LSS baseline) make the approach plausible for integration into existing temporal SSC pipelines.

Future Directions

• Extending the method beyond single-view RGB, since the paper explicitly limits its setting to single-view RGB and notes that multi-camera nuScenes-based SSC is a parallel research line.
• Improving robustness to depth estimation error, which the authors identify as the source of degradation in historical points and in excessive densification upsampling.
• Determining the number of input frames adaptively rather than fixing it at 4, given that a fifth frame reduced IoU and mIoU in the ablation.
• Testing whether C3DFusion generalizes to the broader 3D perception tasks and architectures the authors mention in the conclusion, and to indoor or multi-camera benchmarks such as Occ3D-nuScenes.

Target Audience

Researchers and engineers working on 3D semantic scene completion, camera-based 3D perception, and autonomous driving perception. It is most valuable to readers already familiar with SSC benchmarks, backprojection-based 2D-to-3D lifting, and temporal fusion, and to practitioners looking for a module that can be dropped into existing single-frame SSC architectures.

Authors’ abstract

Recent camera-based 3D semantic scene completion (SSC) methods have increasingly explored leveraging temporal cues to enrich the features of the current frame. However, while these approaches primarily focus on enhancing in-frame regions, they often struggle to reconstruct critical out-of-frame areas near the sides of the ego-vehicle, although previous frames commonly contain valuable contextual information about these unseen regions. To address this limitation, we propose the Current-Centric Contextual 3D Fusion (C3DFusion) module, which generates hidden region-aware 3D feature geometry by explicitly aligning 3D-lifted point features from both current and historical frames. C3DFusion performs enhanced temporal fusion through two complementary techniques-historical context blurring and current-centric feature densification-which suppress noise from inaccurately warped historical point features by attenuating their scale, and enhance current point features by increasing their volumetric contribution. Simply integrated into standard SSC architectures, C3DFusion demonstrates strong effectiveness, significantly outperforming state-of-the-art methods on the SemanticKITTI and SSCBench-KITTI-360 datasets. Furthermore, it exhibits robust generalization, achieving notable performance gains when applied to other baseline models.