Spheriverse: 3D Scene Understanding from Spherical Observations in the Wild

AI summary

Overview

Research area: Computer vision for autonomous driving — specifically 3D scene understanding from 360° spherical (panoramic) imagery, covering semantic occupancy prediction, semantic mapping, and 3D object detection.

Technical level: Advanced. The paper assumes familiarity with voxel representations, BEV feature lifting, transformer-style query mechanisms, and equirectangular projection geometry.

Scope in one sentence: The paper introduces Spheriverse, a large real-world spherical image–LiDAR dataset with 3D benchmarks, and proposes SphereOcc, a camera-only occupancy framework that bridges the geometric mismatch between angular spherical observations and Cartesian voxel space.

What This Paper Is About

Spherical cameras see a full 360° around a vehicle, but the resulting images are stored in an angular (azimuth–elevation) coordinate system, while the 3D world that perception systems must reason about is a uniform Cartesian voxel grid. This mismatch — the paper calls it the "cross-space representation gap" — distorts object shapes, scales, and spatial relationships when image evidence is lifted into 3D, and existing datasets and methods largely ignore it because they assume perspective or narrow-FoV inputs.

The paper's goal is twofold: build a real-world dataset and benchmark suite that makes spherical 3D perception measurable, and design a model (SphereOcc) that explicitly accounts for spherical geometry when constructing voxel features.

Key Contributions

1. Spheriverse dataset. A real-world spherical perception dataset of 64,400 temporally aligned spherical image–LiDAR pairs organized into 644 sequences across 13 geographically and visually diverse regions. It covers 24-hour illumination (daytime, evening, dawn, night) and both clear and rainy weather, with fine-grained manual 3D semantic annotations including rare classes such as noise barriers, food vendors, electric scooters, and greenhouses. Sensing uses a DuxCam M4 spherical camera (5188×1979, 360° horizontal and 136.7° vertical FoV) plus a 128-beam Hesai OT128 LiDAR.

2. Three benchmarks with 30+ evaluated methods. Standardized benchmarks for semantic occupancy prediction, semantic mapping, and 3D object detection, with both overall and scene-wise comparisons across five major scene categories and thirteen fine-grained subcategories.

3. SphereOcc framework. A vision-only occupancy model built around two components: Cartesian–Spherical Representation Remodeling (CSRR), which injects range–azimuth spherical geometry into Cartesian voxel features via region-wise modulation, and Spherical Evidence Re-querying (SER), which conditions voxel queries on range–height–azimuth geometry to retrieve relevant semantic evidence from the source spherical image features.

4. State-of-the-art occupancy results. SphereOcc reaches 13.91% mIoU and 24.65% GeoIoU, improving over the best prior results (TPVFormer at 12.21% mIoU and SurroundOcc at 22.55% GeoIoU) by 1.70 and 2.10 percentage points, and ranks first in both metrics across all five scene categories.

Main Findings

• The cross-space gap is real and measurable. Spherical projection changes object shape, scale, and local spatial arrangement non-uniformly, and image-space proximity does not preserve metric neighborhoods in 3D. The paper visualizes these distortion regions explicitly.

• Existing 3D perception methods degrade under spherical input. Across 30+ adapted methods, performance varies sharply by scene type, with substantial drops in dense or structurally complex environments — evidence that Cartesian-centric architectures do not transfer cleanly to panoramic observations.

• SphereOcc leads on aggregate metrics. 13.91% mIoU (+1.70 pp over TPVFormer, a 13.9% relative gain) and 24.65% GeoIoU (+2.10 pp over SurroundOcc, 9.3% relative gain).

• Scene-wise consistency. SphereOcc ranks first in both mIoU and GeoIoU in all five scene categories, including Expressway/Peri-urban (12.82% mIoU vs. 11.29% best prior), Functional/Restricted Operational Areas (10.17% vs. 8.38%), Rural (11.32% vs. 10.22%), and Structurally Constrained Transportation Areas (16.60% vs. 13.66%).

• Weakest classes reveal remaining difficulty. Person IoU (3.02%) and Bike IoU (7.50%) stay low for nearly all methods, and some baselines score 0.00% on pedestrians — a signal that small, sparse classes remain unsolved under spherical observation.

• Robustness to spatial partitioning and reduced FoV. Region-wise evaluation across horizontal azimuth bins and vertical voxel layers, plus experiments with reduced spherical fields of view, show SphereOcc maintains its advantage rather than exploiting a single favorable configuration.

• The dataset has a strong built-in spatial class prior. Class composition varies systematically with azimuth and range: vehicle occupancy drops from 15.78% to 0.60% as radial distance increases, while vegetation rises from 22.74% to 47.95% and buildings climb from 12.75% to roughly 29%. Foreground participants become sparse at long range due to occlusion.

• Temporal coverage is balanced but daytime-heavy. Daytime accounts for 48.29% of samples, evening 29.97%, night 11.34%, and dawn 10.40%, with evening having the highest per-hour acquisition density.

Methodology in Plain English

The researchers first built the data foundation: they mounted a high-resolution spherical camera and a 128-beam LiDAR on a vehicle, synchronized them with precision time protocol, calibrated intrinsics and extrinsics, and collected roughly 89,674 raw spherical images across 13 regions. After temporal alignment and filtering, 64,400 image–LiDAR pairs remained. Point clouds were then manually annotated with semantic labels through an outsourced pipeline, and sequences were organized into five major and thirteen fine-grained scene categories plus a three-level semantic hierarchy. A database-oriented storage layout keeps the dataset interoperable with prior benchmarks.

For the model, SphereOcc starts conventionally: a spherical image encoder and feature pyramid produce multi-scale features, which are projected to matching channel dimensions and lifted into an initial 3D voxel representation using learnable voxel queries. This is standard practice and would, on its own, suffer from the cross-space gap. Two modules fix that.

CSRR (Cartesian–Spherical Representation Remodeling) addresses the fact that lifted voxel features have no explicit link to the spherical observation geometry. For each voxel, it computes the horizontal range and azimuth, then builds a six-element descriptor combining Cartesian coordinates with the sine and cosine of azimuth and the range. Elevation is deliberately omitted because it is already determined by the retained height and range. These descriptors are embedded by a small convolution and concatenated with the voxel features. A lightweight saliency estimator then scores every voxel, the scores are average-pooled into non-overlapping 3D regions, softmaxed across regions to produce relative importance weights, and upsampled back to full voxel resolution. The resulting gate modulates a learnable, zero-initialized channel residual that is added to the voxel features — so the modification starts as a no-op and learns how much spherical geometry to inject, where, and in which channels.

SER (Spherical Evidence Re-querying) takes the remodelled voxel features and asks each voxel to look back at the source image features. Using each voxel's range–height–azimuth geometry to mimic the 3D-to-2D spherical projection, plus learned sampling offsets and attention weights, it pulls semantically relevant image evidence into the voxel. This avoids relying on a single fixed projection ray, which would be brittle under spherical distortion.

The refined multi-scale features feed a standard occupancy decoder that produces four hierarchical predictions. All four are supervised during training with a weighted sum of cross-entropy, semantic, and geometric losses; only the highest-resolution output is used at inference. Because the whole pipeline uses only camera input, SphereOcc avoids the payload, power, and compute costs of LiDAR-assisted alternatives like SPHERE and EFFOcc.

Why This Matters

This work is among the first to treat spherical 3D semantic perception as a first-class research problem rather than an extension of perspective-camera pipelines. It supplies both the data and the evaluation infrastructure that the field previously lacked — no prior dataset paired wide vertical FoV spherical imagery with dense semantic 3D annotations across outdoor, dynamic, all-weather conditions — and it demonstrates a concrete architectural principle: spherical geometry should be injected into voxel representations explicitly, not left to be learned implicitly.

Real-world applications:

• Autonomous driving, where a single spherical camera can replace a multi-camera surround rig, reducing calibration complexity, cost, and overlap artifacts.
• Robotics and embodied agents, which need full-surround metric spatial representations to plan and navigate without blind spots.
• Survey and mapping vehicles, where dense semantic 3D reconstruction of road corridors supports map maintenance and infrastructure inventory.
• Assisted and automated fleet operation, particularly under low-light and adverse weather where the dataset provides dedicated night, dawn, and rain coverage.

Industry relevance: The paper benchmarks over 30 methods on a common platform, which gives engineers an empirical basis for choosing architectures. Its vision-only design directly addresses the cost and packaging constraints of deploying LiDAR on compact platforms, and the scene-wise breakdown exposes where current models fail — dense mixed-use intersections, construction zones, and long-range sparse classes — which is the information a deployment team actually needs.

Future Directions

• Improve small and sparse classes. Pedestrian and cyclist IoU values near or at zero across most methods indicate that current lifting and querying strategies do not preserve enough evidence for small, partially occluded objects at range.

• Extend beyond occupancy to full spherical 3D perception. The paper establishes semantic mapping and 3D object detection benchmarks but only proposes a new architecture for occupancy. Applying the CSRR/SER principle to mapping, detection, and motion forecasting is an open direction.

• Reduce dependence on dense 3D annotation. Annotation cost is likely the main barrier to scaling Spheriverse. Self-supervised, semi-supervised, or foundation-model-driven labeling could expand coverage to the full 89,674-frame raw collection.

• Test deployment constraints. Region-wise and reduced-FoV evaluations cover spatial robustness, but latency, memory, and real-time performance of the multi-scale voxel pipeline on embedded hardware remain unaddressed, as does cross-dataset generalization to spherical data collected with different camera models or projection formats.

Target Audience

Researchers and graduate students in 3D computer vision, autonomous driving perception, and embodied AI who work on occupancy prediction, BEV representation learning, or panoramic imagery. It is also relevant to practitioners evaluating sensor configurations for production driving or robotics stacks, and to dataset builders interested in temporal synchronization, multi-level semantic annotation hierarchies, and benchmark design across multiple 3D tasks. Readers without a background in voxel-based occupancy prediction or spherical projection geometry will find the methodology section demanding.

Authors’ abstract

Spherical observations provide global visual context for 3D scene understanding. However, visual information is encoded in an angular domain, whereas the physical world is represented in Cartesian coordinates. This cross-space representation gap complicates geometric correspondence and semantic evidence aggregation. To delve into this challenge, we introduce Spheriverse, comprising $64,400$ temporally aligned spherical image-LiDAR pairs organized into 644 sequences. The dataset spans diverse scenes, illumination, and weather conditions, with fine-grained semantic classes. We further establish benchmarks for semantic occupancy prediction, semantic mapping, and 3D object detection, evaluating 30+ methods through overall and scene-wise comparisons. For dense prediction, we propose SphereOcc, an occupancy framework that couples spherical geometry modeling with semantic evidence retrieval. Cartesian-Spherical Representation Remodeling (CSRR) incorporates spherical range-azimuth geometry into Cartesian voxel features through region-wise modulation. Spherical Evidence Re-querying (SER) then conditions queries on voxel content and range-height-azimuth geometry to adaptively retrieve relevant semantic evidence from source spherical image features. SphereOcc achieves 13.91% mIoU and 24.65% GeoIoU, outperforming the respective best-performing methods, TPVFormer and SurroundOcc, by 1.70 and 2.10 percentage points. It also ranks first in both metrics across all five scenes, with consistent advantages across the evaluated spatial partitions and reduced fields of view. The established benchmark and source code will be available at https://feit-feiteng.github.io/Spheriverse.