IndustryShapes: An RGB-D Benchmark dataset for 6D object pose estimation of industrial assembly components and tools

AI summary

Overview

• Research area: Computer vision, specifically 6D object pose estimation for industrial robotics, plus object detection and segmentation benchmarking.
• Technical level: Intermediate. The paper assumes familiarity with pose estimation terminology (RGB-D input, CAD models, instance-level versus novel-object methods, BOP-style metrics).
• Scope: The paper introduces IndustryShapes, an RGB-D benchmark dataset of five industrial tools and components captured in realistic assembly settings and laboratory setups, with annotations for instance-level and novel-object 6D pose estimation, and it reports baseline benchmark results for representative state-of-the-art methods.

What This Paper Is About

Most existing 6D pose estimation datasets focus on household or consumer objects, use controlled laboratory setups, or target bin-picking scenarios. Industrial assembly workstations contain objects that are textureless, metallic, reflective, thin, symmetric or geometrically similar, and models trained on cleaner data often fail to generalize to these conditions. The paper's goal is to provide a realistic, industrially relevant RGB-D benchmark with challenging object properties, rich annotation, and support for both instance-level and novel-object pose estimation pipelines, including the onboarding sequences required by model-free methods.

Key Contributions

1. A challenging industrial object set in realistic conditions: Five industrial objects (Objects 1 and 3 are tools; Objects 2, 4 and 5 are components) chosen to exhibit lack of texture, complex geometries, reflective surfaces, thin structures and symmetries. Data spans single-object setups through cluttered scenes with multiple objects, multiple instances of the same object, and occlusions. Data were captured with an Intel RealSense D455 RGB-D camera at 640 x 480 resolution, with object-to-camera distances from 200 mm to 1000 mm and the majority between 400 and 800 mm.
2. Two complementary sets: A classic set intended primarily for instance-level methods, containing 21 scenes (13 training, 8 test) and a total of 4,623 images with approximately 6k annotated poses, and an extended set tailored to novel-object pose estimation. The extended set includes 10 RGB-D static onboarding sequences (two per object, averaging 650 frames each, for a total of 6.3k RGB-D frames) and three test scenes with over 2k images and approximately 10.3k annotated object instances, captured with a hand-held Intel RealSense D405. The authors state that IndustryShapes is the first dataset to offer RGB-D static onboarding sequences.
3. A benchmarking framework: Evaluation of instance-level baselines (EPOS, DOPE, ZebraPose), novel-object baselines (FoundPose, FoundationPose in model-based and model-free configurations), and detection/segmentation methods (CNOS, SAM-6D), using the BOP challenge protocol.
4. BOP-formatted annotations: All data include RGB images, depth maps, annotated 6D object poses and segmentation masks, formatted according to BOP specifications, with annotation accuracy verified through depth comparison.

Main Findings

• Most methods perform poorly on the dataset: The authors describe generally low performance across most methods, which they present as evidence of the dataset's challenging nature. In the classic set, AR scores were EPOS 0.51, ZebraPose 0.50, DOPE 0.08, FoundPose 0.30, FoundationPose model-based 0.67 and FoundationPose model-free 0.25.
• Novel-object methods are competitive with instance-level methods: FoundPose and FoundationPose were evaluated without retraining on the classic and extended sets, and the authors note that recent novel-object approaches achieve comparable performance to instance-level methods.
• FoundationPose (model-based) is the strongest baseline reported: It achieved the highest AR on both sets, 0.67 on classic and 0.69 on extended, with ADD 0.78 and 0.81 respectively. FoundationPose (model-free) scored AR 0.25 on classic and 0.33 on extended.
• Large per-object variability: AR per object varies substantially (Table VI). Object 5, described as reflective with a thin structure, consistently shows lower performance than Object 1. EPOS and DOPE both scored 0.00 AR on Object 5 in the classic set; ZebraPose scored 0.84 on Object 1 but 0.11 on Object 2 and 0.11 on Object 5.
• DOPE's weak result is attributed to its design: The authors attribute its low scores to reliance on image data and projected bounding cuboids alone, without using CAD models. They deliberately chose not to add synthetic data for DOPE in order to keep evaluation consistent across methods.
• Detection and segmentation are only moderate: CNOS reached BBox mAP 0.240 (classic) and 0.574 (extended), with segmentation mAP 0.203 and 0.512. SAM-6D reached BBox mAP 0.270 and 0.527, with segmentation mAP 0.345 and 0.453. Both methods scored higher AP on the extended dataset than on the classic set, and SAM-6D had a clear advantage in segmentation on the classic set.
• Annotation accuracy was measured via depth comparison: The absolute depth error between annotated poses and captured depth was less than 12 mm for the classic set and approximately 5 mm for the extended set, described as a relative error of less than 5% against the objects' mean diameter of 254 mm. Table IV reports the depth difference statistics: for the classic set (D455), mean delta -1.85 mm, standard deviation 11.6, mean absolute delta 9.3, median absolute delta 7.15; for the extended set (D405), mean delta -1.01 mm, standard deviation 5.2, mean absolute delta 10.2, median absolute delta 8.95.
• Noted dataset asymmetry: The paper reports that some classic test scenes contain fewer frames than others, and that training scene complexity is uneven, with some objects trained on cluttered realistic setups and others on simpler ones, which the authors say affects instance-level method performance.
• Annotation count discrepancy in the paper: The contributions section states that the dataset encompasses over 26k annotations, while Table II reports a total of 22.4k annotations across 34 scenes (classic train 3.9k, classic test 1.9k, onboarding 6.3k, extended test 10.3k). The paper does not reconcile these two figures.

Methodology in Plain English

The authors built a dataset rather than a new algorithm. They selected five industrial tools and components with properties known to be difficult, produced high-quality 3D CAD models with photorealistic textures, and then captured data in two settings: a realistic industrial assembly environment and a laboratory turn-table setup. The classic set combines 1,217 images from 4 laboratory scenes, 1,122 images from 8 industrial assembly scenes, and 1,361 synthetically rendered images for Object 3, totaling 3,700 training frames, with 923 test images from challenging industrial scenes. The extended set adds hand-held close-range RGB-D onboarding sequences and three office-environment test scenes with all five objects, unconstrained lighting, distractors and occlusions.

Poses were annotated in two ways. For laboratory data and where marker installation was feasible in the industrial setting, ArUco markers were used to estimate camera poses. Where markers could not be placed, the team built a semi-automatic pipeline combining Structure-from-Motion reconstruction with manually defined anchor points whose 3D coordinates were known from the CAD models, establishing 2D–3D correspondences that were tracked across frames and used to solve the PnP problem. Annotation quality was then checked by comparing captured depth against depth rendered from the ground-truth poses.

For benchmarking, the authors followed the BOP challenge protocol, using VSD, MSSD, MSPD and ADD error metrics, and reporting Average Recall (AR) as the mean recall of VSD, MSSD and MSPD, plus mAP for detection and segmentation. Instance-level methods were trained and tested on the classic set; novel-object methods were evaluated directly on classic and extended sets without retraining. Two candidate methods were excluded: GDR-Net because its accuracy is tightly coupled to bounding-box quality and its per-object training time was impractical, and Gen6D and OnePose++ because of manual adaptation effort and reliance on an iOS-only capture app, respectively.

Why This Matters

Impact on research. The dataset targets a gap the authors identify between lab-based research and real manufacturing deployment. It provides a benchmark in which the reported baselines mostly score low, which the authors present as evidence that current methods have room for improvement in this domain. It also provides the first RGB-D static onboarding sequences in a pose estimation dataset, which enables evaluation of model-free pipelines that need an onboarding phase to build an object representation.

Real-world applications:

• Robotic pick-and-place in assembly line workstations, where objects may be placed on holders, boxes or trolleys.
• Precise assembly and disassembly tasks requiring accurate object orientation, not just position.
• Human-robot collaboration settings with cluttered, mixed-use environments and occlusions.
• Bin-picking and part-handling scenarios involving reflective, textureless or symmetric industrial parts.

Industry relevance. The object set was drawn from assembly line workstations, the dataset collection was supported by Stellantis—Centro Ricerche FIAT (CRF), and the work was funded by the EU Horizon Europe programmes SOPRANO (GA No 101120990) and PANDORA (GA No 101135775). The camera distances were chosen to reflect typical assembly line distances that allow robotic manipulation while respecting collision-safety constraints, and the 640 x 480 resolution was selected to balance spatial detail against real-time processing demands.

Future Directions

• Increasing the number of frames in underrepresented classic test scenes to address the uneven object representation the authors identify as a limitation.
• Diversifying training data across all objects so that scene complexity is more balanced, since some objects were trained on cluttered realistic setups and others on simpler ones.
• Ensuring fuller pose coverage across the dataset, which the authors list alongside the other limitations as a target for improvement.
• Open question raised by the results: how to close the performance gap between the strongest baseline (FoundationPose model-based, AR 0.67 classic and 0.69 extended) and reliable deployment-level accuracy, particularly for difficult objects such as the reflective, thin Object 5.

Target Audience

Researchers and engineers working on 6D object pose estimation, especially those studying instance-level and novel-object methods, model-free onboarding-based pipelines, or industrial robotic perception. It is also relevant to practitioners in manufacturing automation who need realistic benchmarks for evaluating perception systems, and to dataset and benchmark designers interested in annotation methodology for RGB-D industrial data. Readers without prior exposure to pose estimation metrics such as VSD, MSSD, MSPD, ADD and AR would benefit from supplementary background reading.

Authors’ abstract

We introduce IndustryShapes, a new RGB-D benchmark dataset of industrial tools and components, designed for both instance-level and novel object 6D pose estimation approaches. The dataset provides a realistic and application-relevant testbed for benchmarking these methods in the context of industrial robotics bridging the gap between lab-based research and deployment in real-world manufacturing scenarios. Unlike many previous datasets that focus on household or consumer products or use synthetic, clean tabletop datasets, or objects captured solely in controlled lab environments, IndustryShapes introduces five new object types with challenging properties, also captured in realistic industrial assembly settings. The dataset has diverse complexity, from simple to more challenging scenes, with single and multiple objects, including scenes with multiple instances of the same object and it is organized in two parts: the classic set and the extended set. The classic set includes a total of 4,6k images and 6k annotated poses. The extended set introduces additional data modalities to support the evaluation of model-free and sequence-based approaches. To the best of our knowledge, IndustryShapes is the first dataset to offer RGB-D static onboarding sequences. We further evaluate the dataset on a representative set of state-of-the art methods for instance-based and novel object 6D pose estimation, including also object detection, segmentation, showing that there is room for improvement in this domain. The dataset page can be found in https://pose-lab.github.io/IndustryShapes.