O3N: Omnidirectional Open-Vocabulary Occupancy Prediction

AI summary

Overview

Research area: Computer vision for embodied robotics — specifically 3D semantic occupancy prediction, omnidirectional (360°) panoramic perception, and open-vocabulary scene understanding.

Technical level: Advanced. The paper assumes familiarity with voxel-based 3D occupancy networks, vision-language embedding spaces (CLIP-style), Mamba/state-space sequence models, and open-vocabulary evaluation protocols (base vs. novel class splits).

Scope: The paper introduces O3N, a framework that predicts open-vocabulary 3D semantic occupancy from a single equirectangular omnidirectional RGB image, and evaluates it on the QuadOcc and Human360Occ benchmarks across two occupancy backbones.

What This Paper Is About

Existing camera-based 3D occupancy prediction methods are limited to a narrow field of view and to a fixed set of training categories, which is a problem for robots that must move through and reason about unpredictable real-world spaces. O3N addresses both limitations at once: it takes a single 360° equirectangular image as input and predicts 3D occupancy for categories that were never labeled during training, using text embeddings as the semantic interface. The goal is a perception module usable by "Consumer Embodied Intelligent Robotics" (CEIRs) — devices that must perceive, understand, and act in open-world environments.

Key Contributions

1. A new task setting and framework. The authors present O3N as the first framework for open-vocabulary occupancy prediction from a single omnidirectional RGB image, unifying 360° visual perception with 3D geometric-semantic prediction under open-ended semantics.

2. Polar-spiral Mamba (PsM). A dual-branch module that models cylindrical (polar) and Cartesian voxel representations jointly. It compresses cylindrical voxels into a BEV feature, scans the polar space in an outward spiral from the pole, then progressively fuses cylindrical features back into Cartesian cube voxels using a pre-computed Cartesian-to-cylindrical projection. The stated aim is to preserve angular-continuity and capture long-range dependencies while retaining metric geometry.

3. Occupancy Cost Aggregation (OCA). Instead of pulling voxel embeddings directly toward text embeddings, OCA builds a voxel-text similarity "occupancy cost volume," refines it with Atrous Spatial Pyramid Pooling (ASPP) per semantic class for intra-class spatial aggregation, then applies a linear transformer block for inter-class aggregation. It is supervised by a scene-class affinity loss over precision, recall, and specificity, computed only over valid voxels belonging to base classes.

4. Natural Modality Alignment (NMA). A gradient-free mechanism that aligns text embeddings with semantic prototypes (base-class prototypes updated via Exponential Moving Average over pixel embeddings, plus learnable prototypes for novel classes) through an iterative Random Walk process, which the authors show has a closed-form expression via the Neumann Series as the iteration count goes to infinity.

Main Findings

• QuadOcc gains over the open-vocabulary baseline (MonoScene backbone): O3N reaches 16.54 mIoU versus 14.33 for OVO, with novel-class mean mIoU of 21.16 versus 18.15 and base-class mean of 11.92 versus 10.52. The abstract reports this as +2.21 mIoU and +3.01 Novel mIoU over the baseline.

• QuadOcc gains transfer to a second backbone (SGN-S): 15.52 mIoU versus 13.81 for OVO (SGN-S), with novel-class mean 20.18 versus 17.60 and base-class mean 10.87 versus 10.02.

• Human360Occ (H3O) Homo split gains: O3N attains 24.25 mIoU versus 23.39 for OVO, and novel-class mean 15.36 versus 13.82. Base-class mean is 44.99 for O3N versus 45.73 for OVO. The abstract summarizes the H3O improvements as +0.86 and +1.54 over the baseline, matching the overall-mIoU and novel-mIoU differences.

• Performance against fully supervised methods: On QuadOcc, O3N's novel-class mean of 21.16 exceeds several fully supervised methods, which the authors cite as SSCNet, OccFormer, and VoxFormer-S. On H3O, O3N's 24.25 mIoU is below the fully supervised MonoScene (33.46) and OneOcc (37.29), and the paper describes it as "comparable to several fully supervised methods."

• Label efficiency: The QuadOcc comparison is reported for O3N trained with only 30% of base-class annotations.

• Qualitative behavior: Projected-view comparisons on H3O indicate the baseline produces fragmented predictions and semantic leakage at object boundaries, while O3N yields more coherent road and sidewalk layouts, sharper building structures, and better localized small foreground objects.

• Ablation study: The paper states it performs ablations on QuadOcc to dissect module contributions and that "introducing the PsM alone yields consistent improvements across all three metri—". The remaining ablation results (Table III) are not present in the provided content, so the individual and combined effects of PsM, OCA, and NMA are not reported here.

• Class-split statistics: On QuadOcc, the novel classes are vehicle, road, and building, accounting for approximately 68% of annotated non-empty voxels; base classes are person, vegetation, and terrain. On Human360Occ, novel classes are road, sidewalk, building, car, truck, bus, and two-wheeler, together approximately 75% of annotated non-empty voxels; base classes are vegetation, person, and pole.

• Dataset scale: QuadOcc is a real-world first-person 360° dataset collected on a quadruped robot in a campus environment, with 6 semantic classes plus an empty class on a 64×64×8 grid at 0.4 m resolution. Human360Occ is a CARLA-based human-ego 360° dataset with simulated gait, using 10 classes plus an empty class, with a "Homo" within-city split (per-map 8:2 train/val) and a "Heter" cross-city split (default 12/4 maps).

• Training setup: O3N is trained for 25 epochs on 4 NVIDIA RTX 3090 GPUs with a total batch size of 4. MonoScene is selected as the primary backbone and SGN as the second, with parameters kept consistent with each original model and with OVO's configuration.

Methodology in Plain English

The system takes one panoramic photograph and produces a 3D grid of voxels, each labeled with a category — including categories it was never trained on. Three ideas drive the design.

First, panoramic images distort space: equirectangular projection stretches the top and bottom of the image, and voxels far from the camera crowd into few pixels. To handle this, the researchers keep two voxel representations side by side — a cylindrical (polar) one, which naturally wraps around the camera, and an ordinary Cartesian cube grid. A Mamba-based sequence model scans the polar representation in a spiral that starts at the pole and works outward, matching how visual detail degrades with distance. The polar features are then resampled into the cube grid and added to it, so the network gets both continuous 360° coverage and standard metric geometry.

Second, instead of forcing each voxel embedding to look like its text label (which biases the model toward the few categories it has seen labels for), O3N computes similarity scores between every voxel and every class's text embedding, forming a cost volume. That volume is smoothed spatially across classes' own slices, then reasoned over across classes, and supervised with a loss that rewards precision, recall, and specificity at the scene level rather than voxel-by-voxel.

Third, the text embeddings themselves are nudged into the same region of the embedding space as the image/voxel features without any gradient updates. Prototypes for seen classes are tracked as running averages of pixel features, and text embeddings and prototypes are alternately refined in a Random Walk until they converge, which the authors derive in closed form. Novel-class prototypes are also introduced as learnable parameters.

During training, novel categories are collapsed into a single "unknown" class, so supervision uses L_b + 1 processed semantic labels. At inference, base classes are read directly from the occupancy head, while novel-class logits come from combining the distilled voxel embeddings' similarity to novel text embeddings with the OCA-predicted logits.

Why This Matters

Impact on research. The paper opens a task that sits at the intersection of two active lines of work — omnidirectional/panoramic 3D perception and open-vocabulary occupancy prediction — which had previously been studied separately. It also shows that a gradient-free alignment mechanism can carry semantic generalization, which is a distinctive alternative to the distillation-based approach used by prior open-vocabulary occupancy work such as OVO. The reported transfer across two structurally different backbones (MonoScene and SGN) suggests the modules are modular rather than architecture-specific.

Real-world applications:

• Quadruped and legged robots navigating campus or urban environments, which is the setting of the QuadOcc dataset.
• Consumer service and home robots that must recognize objects they were never explicitly trained to identify.
• Human-ego assistive or wearable perception systems, the setting simulated by Human360Occ.
• Autonomous navigation and interaction pipelines that need a dense 3D semantic map rather than a 2D bounding-box list.

Industry relevance. Full-surround awareness from a single camera lowers sensor cost relative to multi-camera or LiDAR rigs, which matters for consumer devices with tight power and compute budgets. The paper explicitly targets resource-constrained embodied platforms, arguing that transformers are too costly and that Mamba offers transformer-like long-sequence modeling at linear complexity.

Future Directions

• Extending and balancing the class splits. The authors note base-class distributions are highly imbalanced, with some categories under 1% of non-empty voxels, and describe their restricted base vocabulary as "amenable to future data and task expansion."
• Scaling the open vocabulary. Semantic scalability and cross-scene generalization are framed as demonstrated properties, but the number of novel categories evaluated remains tied to two datasets; broader vocabularies and free-form text queries are a natural next step.
• Applying O3N to real embodied platforms. The paper motivates deployment on resource-constrained CEIRs but evaluates only on QuadOcc and Human360Occ; on-robot inference results are not reported in the provided content.
• Full ablation and hyperparameter transparency. The provided content cuts off the ablation study and defers dataset details, class distributions, and hyperparameters to an appendix, leaving open questions about how much each of PsM, OCA, and NMA contributes individually.

Target Audience

Researchers and graduate students working on 3D occupancy prediction, panoramic/omnidirectional perception, open-vocabulary segmentation, and vision-language alignment. It is also relevant to robotics engineers building perception stacks for legged, service, or consumer embodied robots who need full-surround semantic 3D understanding, and to practitioners interested in Mamba-based alternatives to transformers for spatially structured data. Code is stated to be released at https://github.com/MengfeiD/O3N.

Authors’ abstract

Understanding and reconstructing the 3D world through omnidirectional perception is an inevitable trend in the development of autonomous agents and embodied intelligence. However, existing 3D occupancy prediction methods are constrained by limited perspective inputs and predefined training distribution, making them difficult to apply to embodied agents that require comprehensive and safe perception of scenes in open world exploration. To address this, we present O3N, the first purely visual, end-to-end Omnidirectional Open-vocabulary Occupancy predictioN framework. O3N embeds omnidirectional voxels in a polar-spiral topology via the Polar-spiral Mamba (PsM) module, enabling continuous spatial representation and long-range context modeling across 360°. The Occupancy Cost Aggregation (OCA) module introduces a principled mechanism for unifying geometric and semantic supervision within the voxel space, ensuring consistency between the reconstructed geometry and the underlying semantic structure. Moreover, Natural Modality Alignment (NMA) establishes a gradient-free alignment pathway that harmonizes visual features, voxel embeddings, and text semantics, forming a consistent "pixel-voxel-text" representation triad. Extensive experiments on multiple models demonstrate that our method not only achieves state-of-the-art performance on QuadOcc and Human360Occ benchmarks but also exhibits remarkable cross-scene generalization and semantic scalability, paving the way toward universal 3D world modeling. The source code will be made publicly available at https://github.com/MengfeiD/O3N.