PanoAffordanceNet: Towards Holistic Affordance Grounding in 360° Indoor Environments

AI summary

Overview

Research area: Computer vision for embodied AI — specifically visual affordance grounding (localizing where in an image a person or robot can perform an action) extended from perspective images to 360° panoramic indoor scenes.

Technical level: Advanced. The paper assumes familiarity with transformer-based vision encoders, attention mechanisms, frequency-domain filtering, contrastive learning, and Equirectangular Projection (ERP) geometry.

Scope: The paper defines a new task, proposes one end-to-end framework for it, and contributes the first annotated dataset for panoramic affordance grounding.

What This Paper Is About

Existing affordance grounding research is primarily object-centric and evaluated on perspective-view images, but robots operate in a 360° physical action space, creating a mismatch between how models are trained and how agents actually perceive. The authors introduce "Holistic Affordance Grounding in 360° Indoor Environments," where a model must locate all functional interaction regions (for example, where one can sit, grasp, lean_back, or rest_arm) across an entire panoramic indoor scene rather than a single cropped object. Their goal is a framework that stays accurate despite the severe geometric distortion, sparse and scattered functional regions, and limited supervision that characterize 360° imagery.

Key Contributions

1. A new task formulation: Holistic affordance grounding in 360° indoor environments, shifting the paradigm from isolated object-level understanding to scene-level reasoning over the full panoramic field.
2. PanoAffordanceNet: An end-to-end one-shot framework containing a Distortion-Aware Spectral Modulator (DASM) for latitude-dependent spectral calibration, an Omni-Spherical Densification Head (OSDH) for restoring topological continuity from sparse activations, and a multi-level training objective combining pixel-wise, distributional, and region-text contrastive constraints.
3. The 360-AGD dataset: Described as the first high-quality indoor panoramic affordance grounding dataset, with an Easy Split of approximately 800 panoramic images and a Hard Split of approximately 1,200 panoramic images, annotated with keypoints across 19 affordance classes.
4. Benchmarking and validation: Extensive comparisons against adapted one-shot methods on 360-AGD, generalization results on the perspective AGD20K dataset, ablation studies, hyperparameter analysis, and real-world field experiments.

Main Findings

• Large gains on 360-AGD: On the Easy Split, PanoAffordanceNet reaches KLD 1.270, SIM 0.506, and NSS 4.490, versus OOAL at KLD 2.868, SIM 0.117, NSS 1.267 and OS-AGDO at KLD 2.853, SIM 0.124, NSS 1.299. On the Hard Split, it reaches KLD 1.306, SIM 0.474, NSS 4.398, versus OOAL at 3.067 / 0.097 / 1.484 and OS-AGDO at 2.965 / 0.115 / 1.484.
• Cross-domain generalization on AGD20K: On the Seen split, the method achieves KLD 0.739, SIM 0.616, NSS 1.750, compared with OOAL at 0.740 / 0.577 / 1.745. On the Unseen split it reaches KLD 1.185, SIM 0.475, NSS 1.419, compared with OOAL at 1.070 / 0.461 / 1.503. The authors attribute the strong SIM but slightly lower point-sensitive metrics to smoother affordance distributions produced by the panoramic-specific modules.
• Every component contributes: On the 360-AGD Hard Split, ablations show progressive improvement, with the full model (LoRA, DASM, and OSDH) achieving KLD 1.306, SIM 0.474, and NSS 4.398, compared with baseline values of KLD 1.475, SIM 0.416, NSS 4.196. DASM is credited with a critical correction for geometric warping and reduced KLD error; OSDH refines spatial consistency.
• Loss design matters: Training with only pixel-level BCE loss yields KLD 1.596, SIM 0.395, NSS 3.891. Adding KL and region-text contrastive losses yields the best result of KLD 1.306, SIM 0.474, NSS 4.398, with the contrastive loss particularly improving semantic-sensitive SIM and NSS.
• LoRA rank shows an inverted-U trend: Performance peaks at rank r = 16 (KLD 1.306, SIM 0.474, NSS 4.398), degrades at low ranks (r = 4 gives KLD 1.343, r = 8 gives 1.341), and deteriorates at high ranks (r = 24 gives KLD 1.356; r = 32 gives 1.403).
• Robustness to top-k seed selection: Across top-k in [5, 20], KLD fluctuates by only 0.006 (from 1.306 to 1.312), indicating the OSDH does not require precise seed tuning. Values at top-k = 1 and top-k = 30 are slightly worse (KLD 1.315 and 1.320).
• Real-world viability: In field experiments using an Insta360 X4 panoramic camera mounted on a head-mounted cap in office and domestic settings, the model localized functional regions such as sit and display despite varying illumination and geometric distortion.
• Qualitative failure of perspective methods: Heatmaps from OOAL and OS-AGDO on panoramic inputs show erratic, fragmented activations and semantic drift. For different tasks on the same object in the same scene (for example lean_back and rest_arm), those methods produce highly similar, confused response maps, whereas PanoAffordanceNet produces clearly differentiated predictions.

Methodology in Plain English

The framework starts from two frozen pre-trained models: DINOv2 (ViT-B/14) for visual features and a CLIP text encoder (ViT-B/16) combined with a CoOp-based prompt learner for affordance class text embeddings. To adapt the visual backbone without overfitting on the one-shot annotation regime, the authors insert Low-Rank Adaptation (LoRA) matrices into the transformer attention layers.

Text information is injected into the visual features through multi-head attention, activating semantically relevant regions. Those features are then split into high-frequency and low-frequency components — high frequencies via a Laplacian operator to capture interaction boundaries, low frequencies via Gaussian smoothing to retain global structure. Because ERP stretches content near the poles and preserves sharpness near the equator, each branch gets its own correction: a High-Frequency Enhancement Module sharpens boundaries near the equator and suppresses polar artifacts, while a Low-Frequency Stabilization Module keeps global structure consistent near the poles. A language-driven channel gate and a self-adaptive spatial gate then fuse the branches through gated residual addition, followed by contextual re-aggregation.

A lightweight transformer decoder uses the text embeddings as queries to cross-attend to the refined visual features, producing an initial affordance map. This map is fragmented, so the Omni-Spherical Densification Head projects visual features onto a unit hypersphere, builds a cosine-similarity affinity matrix between tokens, selects high-confidence seeds by top-k, suppresses noise with a confidence map based on sigmoid-normalized activations, and propagates seed confidence through the affinity matrix with a learnable residual scalar to fill in topologically continuous regions.

Training combines three losses: binary cross-entropy for pixel accuracy, KL divergence to match the predicted heatmap to the ground-truth distribution's shape and intensity, and an InfoNCE region-text contrastive loss that pools visual features over each affordance region and aligns them with the matching text embedding, disambiguating multiple affordances on the same object.

For the dataset, images came from 360-Indoor and Gibson for the Easy Split (original resolutions approximately 512×1024) and from PanoContext and Sun360 for the Hard Split (original resolutions reaching 4552×9104). Annotators placed multiple keypoints inside all valid, non-occluded interaction areas for each of 19 affordance classes; heavily occluded regions with ambiguous boundaries were ignored. Keypoints were converted into continuous heatmaps with a Gaussian kernel.

Training used the AdamW optimizer with an initial learning rate of 1e-5 and cosine annealing, on two NVIDIA A6000 GPUs for 20k iterations with batch size 4. Panoramic inputs were resized to 560×1120. Augmentations included random flipping, color jittering, random rotations of ±3°, random scaling of ±5%, and horizontal wraparound shifts, with a Gaussian blur applied to binary annotation masks to produce soft supervision heatmaps. Evaluation used KLD, SIM, and NSS.

Why This Matters

Impact on research. The paper reframes affordance grounding from an object-centric, perspective-view problem to a scene-level, omnidirectional one, and shows that directly transferring existing one-shot methods produces severe degradation. It supplies both a benchmark dataset and a baseline, and demonstrates that panoramic-aware design can also remain competitive on the standard perspective AGD20K benchmark, suggesting the modules generalize across view types.

Real-world applications.

• Service robots that need to identify where in a room they or a person can sit, grasp, lean, or place objects, using a single 360° sensor rather than searching with a narrow field of view.
• Assistive and wearable systems, such as the head-mounted panoramic camera setup the authors tested, that provide functional scene understanding from an egocentric viewpoint.
• Home and office automation, where graspable, sittable, or usable surfaces must be identified for task planning.
• Mobile platforms and telepresence systems requiring global spatial awareness rather than repeated perspective scans.

Industry relevance. Consumer 360° cameras (the authors used an Insta360 X4) are inexpensive and widely available, so omnidirectional perception is practical to deploy. A method that localizes functional regions reliably in real homes and offices under varying illumination provides directly usable functional priors for robot decision-making and manipulation planning. The authors state that source code and the benchmark dataset will be made publicly available.

Future Directions

• Extending the framework to temporal reasoning for dynamic scenes, which the authors explicitly name as future work.
• Cross-modal synergy with 3D spatial representations, also named by the authors.
• Reducing reliance on the custom keypoint annotation pipeline — the paper does not report whether the approach can scale to affordance classes or object categories beyond the 19 annotated affordances.
• Investigating the trade-off the authors observe on AGD20K, where smoother panoramic-aware predictions improve SIM but slightly degrade point-sensitive metrics such as NSS, to determine whether panoramic design can be made strictly beneficial on perspective benchmarks.

Target Audience

Researchers and graduate students in computer vision, embodied AI, and robotics who work on affordance grounding, panoramic or omnidirectional perception, or scene-level visual reasoning. It is also relevant to engineers building perception stacks for service robots and wearable systems, and to dataset builders interested in annotation strategies that replace dense pixel-wise segmentation with keypoint-based supervision. Readers without background in attention mechanisms, frequency decomposition, or contrastive objectives will find the methodology section demanding.

Authors’ abstract

Global perception is essential for embodied agents in 360° spaces, yet current affordance grounding remains largely object-centric and restricted to perspective views. To bridge this gap, we introduce a novel task: Holistic Affordance Grounding in 360° Indoor Environments. This task faces unique challenges, including severe geometric distortions from Equirectangular Projection (ERP), semantic dispersion, and cross-scale alignment difficulties. We propose PanoAffordanceNet, an end-to-end framework featuring a Distortion-Aware Spectral Modulator (DASM) for latitude-dependent calibration and an Omni-Spherical Densification Head (OSDH) to restore topological continuity from sparse activations. By integrating multi-level constraints comprising pixel-wise, distributional, and region-text contrastive objectives, our framework effectively suppresses semantic drift under low supervision. Furthermore, we construct 360-AGD, the first high-quality panoramic affordance grounding dataset. Extensive experiments demonstrate that PanoAffordanceNet significantly outperforms existing methods, establishing a solid baseline for scene-level perception in embodied intelligence. The source code and benchmark dataset will be made publicly available at https://github.com/GL-ZHU925/PanoAffordanceNet.