Retrieving Objects from 3D Scenes with Box-Guided Open-Vocabulary Instance Segmentation

AI summary

Overview

Research area: Computer vision, specifically open-vocabulary 3D instance segmentation (OV-3DIS) and multimodal 3D retrieval.

Technical level: Intermediate. The paper assumes familiarity with 2D detection, point clouds, and open-vocabulary segmentation pipelines, but its core idea — using 2D detection boxes to find objects that a 3D segmenter misses — is conceptually straightforward.

Scope: The paper proposes a box-guided mask generation method that supplements a pretrained 3D instance segmenter with RGBD-based proposals from an open-vocabulary 2D detector, improving retrieval of rare object categories while keeping per-scene inference to under one minute.

What This Paper Is About

Robots and augmented-reality systems need to find specific objects in 3D scenes based on natural-language queries, a task framed as open-vocabulary 3D instance segmentation. State-of-the-art methods are accurate but slow because they rely on SAM for mask generation and CLIP for classification, often taking several minutes per scene. The paper targets a faster method, Open-YOLO 3D, which removes SAM and CLIP but often fails on object categories that are rare in 3D training data — the authors address this by using 2D detector boxes to build extra 3D masks for those missed objects.

Key Contributions

1. A method that uses 2D bounding boxes from an open-vocabulary detector to guide the discovery of novel object masks in 3D point clouds, improving generalization to unseen categories.

2. A mask generation strategy that avoids SAM entirely, instead assembling 3D instances from superpoints produced by an efficient graph-based segmentation algorithm (Felzenszwalb and Huttenlocher 2004), enabling inference in under one minute per scene.

3. Demonstrated improvements on rare (tail) object categories across two benchmarks — ScanNet200 and Replica — while maintaining high efficiency, using the same pretrained models as Open-YOLO 3D.

4. A qualification of the result: the paper reports that its approach does not improve every metric, noting a small decrease in one ScanNet200 measure, and attributes its larger gains to lower IoU thresholds because superpoint-based masks can be noisy.

Main Findings

• ScanNet200 overall mAP: The proposed method reaches 24.9 mAP versus 24.7 for Open-YOLO 3D (+0.2), 23.7 for Open3DIS, 25.8 for Any3DIS, 15.4 for OpenMask3D, and 13.0 for OVIR-3D. The supervised Mask3D is reported at 26.9 mAP.

• ScanNet200 tail classes: The method reports 22.4 mAP on tail classes versus 21.6 for Open-YOLO 3D (+0.8), 21.8 for Open3DIS, 26.4 for Any3DIS, 14.9 for OpenMask3D, and 11.7 for OVIR-3D.

• ScanNet200 head/common classes: Results are 27.6 mAP on head (-0.2 versus Open-YOLO 3D's 27.8) and 24.3 on common (+0.0 versus Open-YOLO 3D's 24.3), so the reported gain is concentrated in tail categories.

• ScanNet200 runtime: The method runs at 55.9 seconds per scene versus 21.8 seconds for Open-YOLO 3D, but far below OVIR-3D (466.8 s), OpenMask3D (553.87 s), and Open3DIS (360.12 s). Any3DIS runtime is listed as N/A.

• Replica overall mAP: The method reaches 24.0 mAP versus 23.7 for Open-YOLO 3D (+0.3), 18.5 for Open3DIS, 13.1 for OpenMask3D, and 11.1 for OVIR-3D.

• Replica runtime: 43.7 seconds per scene versus 16.6 for Open-YOLO 3D, but well below OpenMask3D (547.32 s), Open3DIS (187.97 s), and OVIR-3D (52.74 s).

• Larger gains at looser IoU: On Replica the method improves mAP50 to 31.8 (+3.2) and mAP25 to 37.4 (+2.6) over Open-YOLO 3D. The authors state these larger gains come from relying solely on superpoints to form novel masks, which can introduce noise and produce imperfect masks that are penalized at higher IoU thresholds.

• Qualitative evidence: In scene scene0353_00 of ScanNet200, Open-YOLO 3D fails to retrieve a "calendar" instance because it depends entirely on the pretrained 3D network for proposals, while the proposed method uses the 2D detector to form extra masks covering low-frequency classes.

Methodology in Plain English

The pipeline starts with a 3D point cloud, multi-view RGB images, depth information, and camera intrinsics/extrinsics. Two parallel sources of instance proposals are created and then merged.

First, the point cloud is passed through a pretrained Mask3D segmenter to produce class-agnostic point-based instance masks. Second, the point cloud is segmented by a graph-based algorithm into geometrically coherent regions called superpoints, and each RGB frame is run through the open-vocabulary detector YOLO-World to get bounding boxes with class labels.

For each 2D box, all pixels inside it are projected to 3D using depth and camera parameters, and Open3D is used to fit a 3D oriented box around the projected points. Boxes that overlap at least τ_box percent with an existing point-based mask are discarded as redundant. For the remaining boxes, superpoints are assigned to the box if at least τ_spp percent of their points fall inside it, forming a coarse mask. These coarse masks are merged sequentially across frames: a new mask joins an existing candidate when it has IoU of at least τ_merge and shares the same prompt label, otherwise it starts a new candidate. A final filter discards any RGBD-based mask whose IoU with a point-based mask exceeds τ_filter, keeping the point-based mask because it usually has better geometry. The surviving masks are merged with the point-based masks into the final instance set.

Classification reuses the 2D detector instead of CLIP. Label maps are built per frame by initializing all pixels to -1 and painting bounding boxes with their predicted class labels, processing boxes from largest to smallest so that smaller, closer objects take precedence. Point cloud projections, visibility masks, and an occlusion check using a depth tolerance τ_depth determine which projected pixels are actually visible in each frame. Each 3D mask's label distribution is then aggregated from the top-k most visible frames, and the instance is assigned the most frequent class label. This follows the Open-YOLO 3D classification scheme.

Implementation details: the YOLO-World extra-large variant is run on the first frame of every 10-frame interval for ScanNet200 and on all frames for Replica, to keep the comparison with Open-YOLO 3D fair. Processed frames are downsampled by a factor of 5 during box-guided proposal generation. The code is stated to be released at https://github.com/ndkhanh360/BoxOVIS.

Why This Matters

Impact on research. The paper isolates a specific weakness of efficiency-first OV-3DIS pipelines — poor generalization to rare classes — and shows a targeted fix that reuses the same pretrained models rather than adding new heavy components. It also frames a direction for follow-up work: getting more recognition power out of 2D foundation models without paying the SAM-and-CLIP latency cost.

Real-world applications:

• Service robots that must locate everyday or unusual household items on request in indoor environments.
• Augmented reality systems that highlight or annotate queried objects in a live 3D scan.
• Warehouse or logistics assistants that retrieve specified items from cluttered shelves.
• Interactive search over 3D scans of buildings, where users ask for objects that are rare in training data.

Industry relevance. The runtime comparison matters commercially: existing SAM-and-CLIP pipelines at 360–554 seconds per scene are impractical for interactive use, while the proposed method stays under a minute. The paper is also candid that it is slower than Open-YOLO 3D (55.9 s versus 21.8 s on ScanNet200; 43.7 s versus 16.6 s on Replica) — an explicit cost/accuracy trade-off that practitioners must weigh.

Future Directions

• Building a GPU-based implementation of the 2D-to-3D box uplifting step, which the authors identify as the main runtime bottleneck because Open3D is slow at producing oriented 3D bounding boxes.
• Refining only the final mask candidates, potentially with SAM, to improve mask quality without giving up the overall efficiency of the pipeline.
• Using open-world 3D object classifiers such as OpenShape or DuoMamba as an alternative route for classifying 3D instances.
• Investigating whether superpoint-based novel masks can be made less noisy, since the reported gains are larger at IoU 25% and 50% than at higher thresholds.

Target Audience

Researchers and practitioners working on 3D scene understanding, open-vocabulary segmentation, and embodied AI or robotics perception. It is most useful to readers already familiar with OV-3DIS benchmarks and two-stage mask-generation pipelines, and to engineers evaluating whether an efficiency-first retrieval system can meet a real-time or near-real-time budget. Readers new to point cloud segmentation would need background reading on Mask3D, superpoints, and open-vocabulary detection to follow the method section in full.

Note on reporting: the paper states its code will be made available, but the experiments do not report per-category breakdowns beyond the head/common/tail split, and the Any3DIS runtime is listed as N/A. The paper's text contains a typographical inconsistency in the metric description, referring to the IoU 25% metric as "mAP 20" while the tables label it mAP 25.

Authors’ abstract

Locating and retrieving objects from scene-level point clouds is a challenging problem with broad applications in robotics and augmented reality. This task is commonly formulated as open-vocabulary 3D instance segmentation. Although recent methods demonstrate strong performance, they depend heavily on SAM and CLIP to generate and classify 3D instance masks from images accompanying the point cloud, leading to substantial computational overhead and slow processing that limit their deployment in real-world settings. Open-YOLO 3D alleviates this issue by using a real-time 2D detector to classify class-agnostic masks produced directly from the point cloud by a pretrained 3D segmenter, eliminating the need for SAM and CLIP and significantly reducing inference time. However, Open-YOLO 3D often fails to generalize to object categories that appear infrequently in the 3D training data. In this paper, we propose a method that generates 3D instance masks for novel objects from RGB images guided by a 2D open-vocabulary detector. Our approach inherits the 2D detector's ability to recognize novel objects while maintaining efficient classification, enabling fast and accurate retrieval of rare instances from open-ended text queries. Our code will be made available at https://github.com/ndkhanh360/BoxOVIS.