OneOcc: Semantic Occupancy Prediction for Legged Robots with a Single Panoramic Camera

AI summary

Overview

Research area: Robotic 3D perception — specifically semantic scene completion (SSC) for legged and humanoid robots using omnidirectional (360°) vision.

Technical level: Advanced. The paper assumes familiarity with 3D occupancy prediction, camera projection models, voxelization schemes, and mixture-of-experts architectures.

Scope: The paper introduces OneOcc, a camera-only framework that predicts full-surround semantic occupancy for legged robots from a single panoramic camera, plus two new panoramic occupancy benchmarks (QuadOcc and Human360Occ).

What This Paper Is About

Most 3D semantic occupancy systems are built for cars: stable platforms with forward-facing sensors under benign motion. Legged and humanoid robots break these assumptions — they jitter violently with every footfall and must perceive in all directions at once to maintain balance and avoid obstacles. OneOcc addresses this by designing a panoramic occupancy pipeline that is explicitly robust to gait-induced body motion and delivers full 360° coverage from a single omnidirectional camera, rather than relying on heavy LiDAR stacks.

Key Contributions

1. OneOcc framework — A vision-only panoramic semantic scene completion system combining four components: Dual-Projection fusion (DP-ER) to jointly process raw annular and equirectangular views; Bi-Grid Voxelization (BGV) to reason in both Cartesian and cylindrical-polar space; a lightweight decoder with Hierarchical AMoE-3D (attention + mixture-of-experts 3D fusion); and Gait Displacement Compensation (GDC), a plug-and-play module that learns feature-level motion correction without extra sensors.

2. QuadOcc benchmark — A real first-person 360° dataset captured on a quadruped within a campus domain, with standardized day/dusk/night coverage. It contains 10 scenes and 24K frames, uses 6 semantic categories, and provides semi-automatic ground truth (multi-frame LiDAR aggregation plus Grounded-SAM initialization with manual fixes) on a 64×64×8 grid with 0.4 m voxels.

3. Human360Occ (H3O) benchmark — A CARLA-based human-ego 360° dataset with simulated gait, offering 160 sequences and 8K frames across 16 maps with varied weather and lighting. Each frame includes RGB, metric depth, semantic occupancy at two resolutions (64×64×8 and 128×128×16), and pose. It supports both within-city and cross-city splits.

4. State-of-the-art results and extensive evaluation — OneOcc sets new records on both benchmarks, including lighting-condition breakdowns, module ablations, FoV scaling studies, and measured efficiency on desktop and embedded hardware.

Main Findings

• QuadOcc performance: OneOcc reaches 20.56 mIoU, surpassing the best LiDAR baseline (LMSCNet, 18.44) by +2.12 mIoU and the best vision baseline (MonoScene, 19.19) by +1.37 mIoU. A camera-only pipeline rivals and even exceeds classical LiDAR stacks at this range and resolution.

• H3O generalization: OneOcc attains 37.29 mIoU on within-city and 32.23 mIoU on cross-city splits, improving over the best vision baselines (33.46/24.15) by +3.83 and +8.08 mIoU respectively — up to +33.5% relative. The larger margin under cross-city shift indicates strong distribution robustness.

• Additive ablation gains: Starting from an ER-only, single-grid, no-GDC, no-AMoE baseline (19.19 mIoU), each module contributes: GDC +0.39, DP-ER +0.31, BGV +0.41, and AMoE-3D +0.26. The gains are consistent and complementary across near-field and far-field structure.

• Lighting robustness: OneOcc leads on day (21.15 vs 18.58) and dusk (19.86 vs 15.14). At night it trails MonoScene on mIoU (13.50 vs 14.20) but achieves higher precision, attributed to frustum-artifact suppression.

• Expert count matters: For the AMoE-3D module, K=4 experts with 3D Gradient-Energy gating gives the best accuracy-efficiency trade-off. K=1 (a fused bottleneck) drops mIoU to 29.68 on H3O-Heter; K=8 slightly increases precision but reduces recall due to expert fragmentation.

• Qualitative advantages: On QuadOcc, OneOcc suppresses the "ghost elongation" frustum artifacts that MonoScene exhibits; on H3O it recovers pedestrians whose colors blend into the background, and preserves global layout and 360° continuity.

• Deployment efficiency: Inference on an RTX 4090 at 608×1216 takes 69.93 ms (~14.3 FPS) with 101.76M parameters and 1.82 GB peak CUDA allocation. Mixed precision (FP32+FP16) reduces latency to 52.84 ms (~18.9 FPS) and peak allocation to 1.49 GB, making it suitable for onboard legged perception.

Methodology in Plain English

The pipeline starts with a panoramic camera calibrated using a Taylor polynomial model. The raw annular image is unwrapped into an equirectangular format, and two encoders run in parallel — one on the raw ring and one on the unwrapped view — so the system gets both native fine texture from the ring and convolution-friendly azimuthal continuity from the equirectangular form (the DP-ER module).

Before lifting 2D features into 3D space, a small zero-initialized network predicts a 2D pixel displacement (dx, dy) from global-pooled features at each scale. This displacement warps the sampling coordinates used to gather features for voxels, effectively undoing the phase error caused by gait jitter before it gets baked into quantized voxels (this is GDC). Sampling is bilinear rather than an integer-index gather, which reduces aliasing.

Voxelization happens twice, simultaneously: once on a standard Cartesian grid (good for near-field foothold geometry) and once on a cylindrical-polar grid (where the horizontal axis matches azimuth, giving uniform ring sampling and less far-field aliasing). Features from the polar grid are resampled onto the Cartesian grid via pre-computed cross-grid indices and concatenated, giving a fused bi-grid volume.

A lightweight 3D UNet decoder processes this volume with trilinear upsampling at three levels. At each level, an AMoE-3D block applies channel attention and spatial attention (dual-path saliency), then routes voxels through a mixture of small 1×1×1 convolution experts. The routing weights come from a 3D gradient-energy measure, so high-contrast regions (class boundaries, thin structures like poles) get specialized treatment while flat ground is not over-smoothed. A final 1×1×1 head outputs per-voxel class logits, supervised only on valid voxels with deep supervision and standard SSC losses (cross-entropy, SCAL, frustum proportion — deliberately omitting the relation loss, which over-smooths azimuthal boundaries).

Why This Matters

Impact on research: This is among the first works to tackle dense 3D semantic occupancy specifically for legged/humanoid platforms under realistic gait-induced motion. It shows that a single panoramic camera — not a LiDAR stack — can rival or beat LiDAR baselines at practical voxel resolutions, and it provides two open benchmarks (QuadOcc and Human360Occ) that fill a clear gap, since existing SSC datasets target wheeled vehicles or stationary indoor scenes.

Real-world applications:

• Quadruped and humanoid navigation on uneven campus, indoor, or disaster-response terrain, where full-surround awareness is required to plan footholds and avoid collisions.
• Assistive and service robotics in human-shared spaces, where detecting pedestrians and thin obstacles (poles, two-wheelers) around the robot matters for safe interaction.
• Autonomous delivery and inspection with legged platforms that must operate day, dusk, and night in weather-diverse outdoor environments.
• Human-ego perception research via the H3O simulator, useful for wearables, AR/VR, and embodied AI agents that need to understand what surrounds a walking person.

Industry relevance: The efficiency profile (under 2 GB peak memory, ~19 FPS in mixed precision on a single GPU, and measured runtime on Jetson AGX Orin reported in the supplementary) fits the payload and power budgets of commercial quadruped and humanoid platforms such as those from Xiaomi and similar vendors. The authors propose robotic occupancy as an intermediate representation for world models and vision-language-action models, which points toward modular perception stacks that plug into broader embodied AI architectures.

Future Directions

• Online calibration robustness: OneOcc assumes accurate calibration with bounded drift. Self-calibration on the fly, optionally regularized by lightweight odometry priors, could make the system resilient to bumps and camera wear.
• Temporal aggregation: The main paper mentions temporal aggregation baselines only in the supplementary; strongly integrating multi-frame evidence could improve stability further under periodic gait.
• Occupancy as a world-model interface: The authors propose occupancy-sequence pretraining for cross-robot transfer and tokenized occupancy for language-conditioned control — an open research direction connecting perception to vision-language-action models.
• Nighttime and adverse-weather performance: OneOcc trails slightly at night on mIoU, and future work could target the low-albedo/specular-surface failure modes that degrade long-range completion under these conditions.
• Resolution scaling and narrower FoV trade-offs: The supplementary FoV study (90° to 360°) and resolution scaling results suggest further exploration of compute-vs-coverage trade-offs for specific robot tasks.

Target Audience

Robotics researchers working on perception for legged or humanoid platforms; SSC and 3D occupancy researchers interested in panoramic or non-Cartesian sensing setups; practitioners building onboard perception stacks for commercial quadrupeds and humanoids; and embodied AI researchers looking for occupancy-based intermediate representations for world models and vision-language-action systems. A background in 3D computer vision, camera calibration, and voxel-based scene representation is helpful for full engagement with the methodology.

Authors’ abstract

Robust 3D semantic occupancy is crucial for legged/humanoid robots, yet most semantic scene completion (SSC) systems target wheeled platforms with forward-facing sensors. We present OneOcc, a vision-only panoramic SSC framework designed for gait-introduced body jitter and 360° continuity. OneOcc combines: (i) Dual-Projection fusion (DP-ER) to exploit the annular panorama and its equirectangular unfolding, preserving 360° continuity and grid alignment; (ii) Bi-Grid Voxelization (BGV) to reason in Cartesian and cylindrical-polar spaces, reducing discretization bias and sharpening free/occupied boundaries; (iii) a lightweight decoder with Hierarchical AMoE-3D for dynamic multi-scale fusion and better long-range/occlusion reasoning; and (iv) plug-and-play Gait Displacement Compensation (GDC) learning feature-level motion correction without extra sensors. We also release two panoramic occupancy benchmarks: QuadOcc (real quadruped, first-person 360°) and Human360Occ (H3O) (CARLA human-ego 360° with RGB, Depth, semantic occupancy; standardized within-/cross-city splits). OneOcc sets a new state of the art on QuadOcc, outperforming strong vision baselines and remaining competitive with classical LiDAR baselines; on H3O it gains +3.83 mIoU (within-city) and +8.08 (cross-city). Modules are lightweight, enabling deployable full-surround perception for legged/humanoid robots. Datasets and code will be publicly available at https://github.com/MasterHow/OneOcc.