OccuFly: A 3D Vision Benchmark for Semantic Scene Completion from the Aerial Perspective

AI summary

Overview

Research area: 3D computer vision and robotic perception, specifically Semantic Scene Completion (SSC) from an aerial (UAV) viewpoint, plus a companion benchmark for metric monocular depth estimation.

Technical level: Advanced. The paper assumes familiarity with volumetric occupancy grids, Structure-from-Motion and Multi-View Stereo reconstruction, surface reconstruction, and monocular depth estimation metrics.

Scope: The paper introduces OccuFly, described as the first real-world, camera-based benchmark for aerial Semantic Scene Completion, together with a LiDAR-free data generation framework and baseline evaluations of state-of-the-art SSC and depth foundation models.

Note: the paper content provided is truncated before the conclusion, and several statistics (semantic class list, scene-wise depth distributions, group assignments) are stated to reside only in the supplementary material.

What This Paper Is About

Semantic Scene Completion jointly estimates dense volumetric occupancy of a scene and assigns a semantic label to every occupied voxel. This task is well studied for ground vehicles, but the aerial perspective has been largely unexplored because no dedicated real or synthetic SSC dataset exists for it.

The paper argues that the standard way of building SSC ground truth — fusing multiple LiDAR sweeps and manually labeling the resulting 3D points — is poorly suited to uncrewed aerial vehicles, which face flight regulations, strict mass and energy limits, and LiDAR point clouds that become sparser from elevated viewpoints. The authors therefore build a camera-only generation pipeline based on classical 3D reconstruction and release OccuFly, a multi-altitude, all-season aerial SSC benchmark with paired images, semantic voxel grids, and metric depth maps.

Key Contributions

1. A LiDAR-free, camera-based data generation framework. It relies on classical 3D reconstruction instead of LiDAR, avoids LiDAR-based point cloud sparsity, adheres to UAV mass and energy constraints by dropping LiDAR hardware, and reduces manual semantic labeling from tedious 3D annotation to efficient 2D image annotation.

2. OccuFly, described as the first real-world aerial SSC benchmark. It consists of 9 scenes with over 20,000 samples of nadir and oblique perspective images with corresponding 3D semantic voxel grids, 21 semantic classes, covering almost 200,000 m² at 50 m, 40 m, and 30 m altitude in urban, industrial, and rural environments across all seasons.

3. More than 20,000 per-frame metric depth maps. The authors train and release a Depth-Anything-v2 model on these depth maps, which enables state-of-the-art SSC in their benchmark.

4. Identification of a consistent domain gap. Terrestrial SSC models, 2D vision foundation models, and 3D vision foundation models all fail to generalize to aerial data, positioning OccuFly as a benchmark for addressing those limitations.

Main Findings

• Aerial SSC performance is very low. Across all altitudes, DISC achieves IoU 29.52% and mIoU 2.04%, while Symphonies achieves IoU 13.68% and mIoU 0.58%. At 50 m, DISC reaches IoU 31.10% and mIoU 2.20%; at 40 m, IoU 27.85% and mIoU 1.77%; at 30 m, IoU 26.88% and mIoU 2.23%. The authors state that state-of-the-art models recover coarse scene geometry but struggle to infer semantic structure.

• Off-the-shelf depth foundation models fail in zero-shot aerial settings. MapAnythingV1.1 records δ1 = 0.000, AbsRel = 0.799, RMSE = 30.068, MAE = 29.309. Metric3Dv2 records δ1 = 0.073, AbsRel = 0.471, RMSE = 19.578. DepthAnything2 records δ1 = 0.002, AbsRel = 0.729, RMSE = 28.382. DepthAnything3 records δ1 = 0.000, δ3 = 0.141, AbsRel = 0.591, RMSE = 22.615.

• Fine-tuning on OccuFly produces large improvements. DepthAnything2-OccuFly reaches δ1 = 0.834, δ2 = 0.976, δ3 = 0.999, AbsRel = 0.134, RMSE = 4.844, MAE = 4.193. Metric3Dv2-OccuFly reaches δ1 = 0.278, AbsRel = 0.381, RMSE = 13.643, MAE = 13.134. MapAnything cannot be fine-tuned because it is trained for 3D reconstruction and produces depth maps only as a byproduct, and DepthAnything3 provides no training scripts, so it was evaluated zero-shot only.

• A 3D foundation model shows large scale error. Evaluating DepthAnything3 on all OccuFly scenes yields an average metric scale deviation of 526%, which the authors cite as support for their choice of classical SfM+MVS reconstruction.

• The reconstruction pipeline is geometrically consistent. The SfM+MVS pipeline achieves an average RMSE reprojection error of 1.24 pixels across all scenes.

• The annotation strategy is highly efficient. Annotating fewer than 10% of images per scene automatically labels over 99% of reconstructed points. Empirically, ρ(J) > 0.99 while |J|/N < 0.1 on average. Image selection uses spatially stratified sampling, partitioning the ground plane into square cells with scene-dependent edge lengths of 23 to 28 meters.

• Automatic pseudo-labeling is not a viable substitute. Using Semantic Segment Anything for pseudo-labels produces poor segmentation quality, attributed to its SAM backbone underperforming in the aerial domain. A ConvNeXt model trained on a consolidation of seven established aerial segmentation datasets achieves only 17.58% mIoU against the manual annotations.

• Semantic and geometric accuracy are validated against held-out manual labels. Rendering the semantic point cloud into 135 additional manual annotations across all scenes and altitudes that were not used during data generation results in 92% pixel-wise agreement.

• OccuFly is substantially larger than existing resources. It provides more than 5 times the number of samples and over 6 times the number of labeled voxels compared with SemanticKITTI. It also has the largest class taxonomy among compared SSC benchmarks (21 classes) and is the only one in Table 2 that provides depth maps. For depth maps, OccuFly provides more than 13 times as many as WildUAV and more than 24 times as many as UseGeo.

• The benchmark has a defined evaluation structure. Training comprises 14,804 samples (scenes 01–05), validation 1,965 samples (scenes 06–07), and test 3,842 samples (scenes 08–09), totaling 20,611 samples over 193,938 m².

Methodology in Plain English

The framework has four modules.

1. 3D reconstruction. From calibrated, geo-referenced images and their recorded poses, the authors run Structure-from-Motion and Multi-View Stereo. This yields a dense metric 3D point cloud in world coordinates, per-image metric depth maps, and explicit 2D–3D correspondences linking image pixels to reconstructed points.

2. Semantic annotation. Instead of labeling points in 3D, the authors label only a small subset of images — on average fewer than 10% per scene — chosen by spatially stratified sampling. Semantic labels are then lifted from annotated pixels into the point cloud by back-projection. Because most points are observed by multiple cameras, per-point labels are assigned by unweighted majority voting, with ties broken by a fixed class-prior order derived from class frequencies. Unlabeled points are filled in with inverse-distance-weighted k-nearest neighbors, and a second kNN pass relabels every point to the dominant class in its neighborhood as a denoising step.

3. Class-aware densification and voxelization. Semantic classes are split into three disjoint groups. Instance classes (such as vehicles) are separated into objects with DBSCAN using class-specific parameters, then densified by placing 24 virtual cameras quasi-uniformly around each object, extracting silhouettes from projected points via α-shape boundaries, and performing multi-view silhouette carving inside a dilated bounding box. Ground classes (such as road) are densified with Poisson surface reconstruction into a watertight mesh that fills holes and enforces surface continuity, then voxelized. Other classes (such as constructions) are voxelized directly. The three groups are merged into a single scene-level grid using the precedence order instance over other over ground to resolve label conflicts.

4. Ground-truth sampling. Because the previous steps operate at scene level, per-frame grids are obtained by frustum culling with each frame's camera pose and intrinsics at a fixed metric specification. Each image is paired with a semantic voxel grid

Authors’ abstract

Semantic Scene Completion (SSC) is essential for 3D perception in mobile robotics, as it enables holistic scene understanding by jointly estimating dense volumetric occupancy and per-voxel semantics. Although SSC has been widely studied in terrestrial domains such as autonomous driving, aerial settings like autonomous flying remain largely unexplored, thereby limiting progress on downstream applications. Furthermore, LiDAR sensors are the primary modality for SSC data generation, which poses challenges for most uncrewed aerial vehicles (UAVs) due to flight regulations, mass and energy constraints, and the sparsity of LiDAR point clouds from elevated viewpoints. To address these limitations, we propose a LiDAR-free, camera-based data generation framework. By leveraging classical 3D reconstruction, our framework automates semantic label transfer by lifting &lt;10% of annotated images into the reconstructed point cloud, substantially minimizing manual 3D annotation effort. Based on this framework, we introduce OccuFly, the first real-world, camera-based aerial SSC benchmark, captured across multiple altitudes and all seasons. OccuFly provides over 20,000 samples of images, semantic voxel grids, and metric depth maps across 21 semantic classes in urban, industrial, and rural environments, and follows established data organization for seamless integration. We benchmark both SSC and metric monocular depth estimation on OccuFly, revealing fundamental limitations of current vision foundation models in aerial settings and establishing new challenges for robust 3D scene understanding in the aerial domain. Visit https://github.com/markus-42/occufly.