RGBD20K: A Large-Scale Benchmark for RGB-D Semantic Segmentation

AI summary

Overview

Research area: Computer Vision — RGB-D (color plus depth) semantic segmentation, and the datasets/benchmarks that support it.

Technical level: Intermediate. The dataset and benchmark discussion is accessible to a general reader; the proposed fusion method involves transformer attention terminology.

Scope: This paper introduces RGBD20K, a 20,000-pair RGB-D semantic segmentation benchmark with 160 categories and refined annotations, plus a new fusion model called score-purified fusion (SPF).

What This Paper Is About

RGB-D semantic segmentation models are limited less by architecture than by the datasets they are trained and evaluated on, which are small, cover few object categories, and contain noisy labels. The authors build a much larger and more carefully annotated benchmark, RGBD20K, and propose a fusion method that exploits this higher-quality multimodal data to set new results across several benchmarks.

Key Contributions

1. A large-scale RGB-D dataset. RGBD20K contains 20,000 RGB-D image pairs, compared with 2,347 for NYUv1, 1,449 for NYUv2, and 10,335 for SUN RGB-D.
2. Expanded semantic space and scene diversity. The dataset covers 160 fine-grained categories and 75 scene types, versus 40 categories for NYUv2 and 37 for SUN RGB-D. It contains 368,312 masks, an average of 18.4 per image, and follows a natural long-tailed distribution.
3. High-fidelity annotation refinement. Each RGB-D pair underwent multiple rounds of manual inspection and refinement, with depth-aware ordering used to resolve occlusions and a flexible taxonomy allowing new classes to be added during annotation.
4. A new fusion model. The score-purified fusion (SPF) network follows a "purify-then-attend" principle and achieves state-of-the-art performance on NYUv2, SUN RGB-D, SUN RGB-D† (re-annotated), and RGBD20K.

Main Findings

• SPF leads on every benchmark evaluated. SPF reaches 60.5 mIoU on NYUv2, 55.0 on SUN RGB-D, 62.2 on SUN RGB-D†, and 46.3 on RGBD20K, surpassing GeminiFusion (60.2 / 54.6 / 61.5 / 45.4), DFormer v2 (58.4 / 53.3 / 59.7 / 44.1), and DPLNet (59.3 / 52.8 / 59.0 / 28.0). SPF uses a Swin-L-384 backbone with 416.4M parameters.
• RGBD20K is substantially harder than prior benchmarks. Nearly all evaluated methods score far lower on RGBD20K than on NYUv2 or SUN RGB-D. For example, DPLNet drops from 59.3 mIoU on NYUv2 to 28.0 on RGBD20K, and PGDENet drops from 53.7 to 22.4.
• Annotation noise in SUN RGB-D suppresses measured performance. Refining only the test set annotations improved a pretrained DPLNet from 52.8 to 55.0 mIoU. Retraining representative models on the fully refined dataset (SUN RGB-D†) raised DPLNet to 59.0, and all evaluated methods improved consistently under that setting.
• Cross-modal projection matters. Removing the RGB and depth alignment projections reduces mIoU from 46.3 to 45.7 (−0.6) on RGBD20K.
• Purification must be bidirectional and applied to both Key and Value. Removing the score entirely drops performance to 45.5 (−0.8). Applying it only to the Key gives 46.0 (−0.3), only to the Value 45.9 (−0.4), only to RGB 45.8 (−0.5), and only to Depth 46.0 (−0.3).
• Layer-specific learnable noise works best. Learnable noise with addition achieves 46.3 mIoU; switching to multiplication gives 46.1 (−0.2), and random Gaussian noise performs worse (45.3 with addition, 45.5 with multiplication).
• A 2-layer MLP with Sigmoid is the strongest Relation Arbiter. It reaches 46.3 mIoU, ahead of 2-layer MLP with Softmax (45.8), 1×1 CNN with Sigmoid (45.5), and 3×3 CNN with Sigmoid (45.4).

Methodology in Plain English

The authors assembled RGBD20K by pooling 20,000 depth-aligned image pairs from heterogeneous sources: SUN RGB-D (10,335 images), DIML RGB-D (8,365), RGB-D Mirror (824), RGBD1K (235), VidSOD (100), DepthTrack (98), and ARKitTrack (43). These were unified under a single 160-class taxonomy through manual cleaning and re-annotation using an interactive labeling interface and a hierarchical labeling scheme, from coarse categories such as furniture and appliances down to fine-grained classes. Depth-aware ordering places background regions like walls and floors at the farthest layer to keep foreground–background relationships consistent. The data is split into 18,000 training and 2,000 testing pairs (10 percent for testing), stratified to preserve distributions of scene types, object categories, and depth characteristics; all 160 categories appear in both subsets.

The proposed SPF model builds on GeminiFusion with a four-stage hierarchical encoder similar to SegFormer and a SegFormer head decoder. Its central idea is that standard cross-attention suffers from "attention dilution" because it must simultaneously resolve cross-modal inconsistencies such as sensor noise and misaligned depth boundaries while aggregating contextual information. SPF instead purifies the Key and Value representations before attention runs. A Score Head projects RGB and depth features into aligned embeddings, and a Relation Arbiter cross-examines them to produce reliability scores, split into intra-modal and cross-modal components. These scores are used to weight and blend the features (and learnable noise terms) into purified Keys and Values before standard multi-head attention. Training used a SwinTransformer Large backbone, weight decay of 0.01, 300 epochs divided into three 100-epoch stages with learning rates of 6×10⁻⁵, 3×10⁻⁵, and 1.5×10⁻⁵ respectively, poly decay with a power of 0.01, and four NVIDIA H100 GPUs. Evaluation uses mean Intersection-over-Union (mIoU), with input resolutions of 480×640 for NYUv2 and RGBD20K and 480×480 for SUN RGB-D. NYUv2 provides 795 training and 654 testing images across 40 categories; SUN RGB-D provides 5,285 training and 5,050 testing images across 37 categories, roughly 7× larger than NYUv2.

Why This Matters

Impact on research. The paper argues that progress in RGB-D segmentation is bottlenecked by dataset scale, semantic coverage, and label quality rather than model design alone. Its annotation-quality experiment provides concrete evidence: re-annotating SUN RGB-D changed measured DPLNet performance from 52.8 to 59.0 mIoU, meaning that prior evaluations may have systematically underestimated models. RGBD20K's 160-class taxonomy and long-tailed distribution also make it suitable for supervised, open-vocabulary, and zero-shot perception research.

Real-world applications (as identified in the paper):

• Robotics, where pixel-level recognition of complex indoor scenes is required.
• Autonomous systems operating in indoor environments.
• Intelligent perception systems that must combine visual and geometric cues.
• General-purpose multimodal scene understanding, where RGB-only approaches struggle in complex scenes.

Industry relevance. The paper positions RGBD20K as a training resource for data-hungry vision transformers and as a reliable evaluation foundation for comparing fusion architectures, and it frames efficient fusion (as in DFormer's channel allocation and DPLNet's prompt-based design) as an active engineering concern.

Future Directions

• Closing the performance gap on RGBD20K. Scores on the new benchmark remain low (SPF leads at 46.3 mIoU), leaving substantial room for stronger models.
• Exploiting the long-tailed distribution. The dataset mirrors real-world object frequencies, raising the question of how to generalize well on both common and infrequent classes.
• Open-vocabulary and zero-shot segmentation. The authors explicitly note the dataset supports these settings, and the flexible taxonomy permits new classes to be added.
• Better use of annotation-quality insights. The SUN RGB-D† results suggest further study of how label noise in training data impairs feature learning and generalization, and whether the refinement approach should be extended to other benchmarks.

Target Audience

Researchers and practitioners working on multimodal semantic segmentation, RGB-D perception, and benchmark construction; robotics and autonomous systems engineers who need indoor scene understanding; and graduate students or applied scientists interested in how dataset scale, category taxonomy, and annotation quality affect model performance. Readers focused purely on image segmentation architectures will also find the score-purified fusion ablations relevant, while readers new to the field can follow the dataset comparison without deep background in transformers.

Authors’ abstract

In this paper, we propose RGBD20K, a novel dataset for facilitating the development of more robust and general RGB-D semantic segmentation by encompassing abundant categories and high-quality annotations. RGBD20K possesses several attractive properties: (1) Expanded Semantic Space. In particular, it covers 160 fine-grained categories, largely surpassing the category diversity of existing popular RGB-D benchmarks (e.g., NYUv2 with 40 classes and SUN RGB-D with 37 classes). With such enriched semantic coverage, we expect to promote the learning of more generalizable segmentation models. (2) Larger Scale. Compared with current benchmarks, RGBD20K offers 20,000 RGB-D image pairs, providing a substantially larger training resource that benefits the development of more powerful deep models. (3) High-Fidelity Annotation. We perform rigorous re-evaluation and correction of existing labels to resolve long-standing annotation noise, resulting in a clean and reliable ground-truth foundation. Furthermore, we propose a novel score-purified fusion (SPF) method, which achieves state-of-the-art performance across all evaluated benchmarks, demonstrating the effectiveness of our approach in leveraging high-quality multimodal information for RGB-D semantic segmentation. The dataset is here: https://github.com/ShaohuaDong2021/RGBD20K/.