Seg2Grasp: A Robust Modular Suction Grasping in Bin Picking

AI summary

Overview

Research area: Robotics — specifically robotic bin picking, suction-based grasp pose estimation, class-agnostic instance segmentation, and open-vocabulary object classification.

Technical level: Intermediate. The paper assumes familiarity with instance segmentation architectures (Mask2Former-style mask transformers), RANSAC plane fitting, surface normals from point clouds, and CLIP-style vision-language models.

Scope: The paper proposes and evaluates Seg2Grasp, a three-module pipeline (Segmentation, Grasping, Classification) for suction grasping in cluttered, dynamic bin-picking scenarios, tested in real-robot experiments against DexNet 4.0 and SuctionNet.

What This Paper Is About

Bin picking — pulling objects out of a cluttered container — is hard for robots when lighting, camera positions, bin types, and the objects themselves vary from what a model saw during training. Existing systems mostly use end-to-end learning that maps sensor input directly to grasp output, and these tend to lose accuracy on unfamiliar objects or novel configurations.

The goal of this paper is to replace that monolithic approach with a modular pipeline in which segmentation, grasp-point selection, and object identification are handled by separate specialized components, so that each can remain reliable under conditions the overall system has not seen before.

Key Contributions

1. A modular three-stage pipeline (Seg2Grasp) that decomposes bin picking into class-agnostic object segmentation, suction grasp-pose estimation from surface normals, and open-vocabulary object classification, rather than mapping sensor input directly to grasps end-to-end.

2. A depth-weighted segmentation input and RANSAC-based mask filtering scheme. The RGB image is multiplied by a normalized-and-inverted depth image to improve object contrast, and a RANSAC routine estimates the largest planar area within each proposed mask so that the mask with the highest-centroid inlier plane is selected as the target object.

3. A training-free suction scoring formulation. The grasping module evaluates candidate suction points using three normalized and weighted criteria — surface angle, distance from the centroid, and count of graspable points — combined into a single score, relying only on RGB-D imagery rather than task-specific training data.

4. A fine-tuned Mask-CLIP classifier with mask prompt tuning for open-vocabulary labeling of masked object images, addressing the domain shift that occurs when CLIP is given masked inputs containing large blank regions.

Main Findings

• Seg2Grasp leads under optimal conditions, with the gap widening as difficulty increases. On the EASY setup (single layer, trained objects) Seg2Grasp reached a pick success rate (pr) of 0.89 and an object success rate (or) of 0.96, versus DexNet 4.0 at pr 0.85 / or 0.94 and SuctionNet at pr 0.72 / or 0.92 / sr 0.93.

• The advantage is largest on complex, novel objects. On the HARD setup (complex, novel), Seg2Grasp recorded pr 0.79, or 0.86, and sr 0.83, while DexNet 4.0 fell to pr 0.28 / or 0.31 (segmentation not reported) and SuctionNet to pr 0.29 / or 0.23 / sr 0.26. The paper states this is a drop of both baselines below 31% in or and sr.

• The medium-difficulty setting shows the same pattern. On MEDIUM (double layer, mixed), Seg2Grasp scored pr 0.87 / or 0.91 / sr 0.89, compared with DexNet 4.0 at 0.41 / 0.61 and SuctionNet at 0.51 / 0.53 / 0.43.

• Seg2Grasp is far more tolerant of camera height changes. With camera heights tested at 50 cm (optimal), 100 cm, and 150 cm over 10 experimental sets with up to 75 attempts per set, both baselines degraded at higher positions; DexNet 4.0 succeeded on only 10 to 15 objects, SuctionNet dropped notably, and Seg2Grasp maintained high performance across all heights.

• Seg2Grasp clears bins regardless of bin type. Across a large yellow bin (baseline), a shopping basket, and a small white box — with 25 objects per set, none seen in training, and up to 75 attempts per set — neither baseline fully cleared the bin, while Seg2Grasp maintained consistently high success rates.

• Diagnosed failure modes of the baselines. DexNet 4.0 struggles with object overlap, particularly when heavier items are stacked on top; SuctionNet is highly sensitive to its environment and objects and frequently misidentified the yellow background as a suction point.

• Mask-CLIP beats standard CLIP for masked-input classification. On the 'mask-category' evaluation dataset, Mask-CLIP achieved Top-1 accuracy of 73.8% and Top-3 accuracy of 84.7%, compared with CLIP at 66.1% and 78.9%.

Methodology in Plain English

Segmentation. The system first builds a combined input image by multiplying the RGB image by a depth image that has been normalized and inverted, so that closer objects stand out more strongly. A Mask Transformer model (the standard Mask2Former-style architecture) processes this fused input and outputs up to 100 class-agnostic masks per image. To decide which mask is the target, the system runs RANSAC on each mask's points to estimate its largest planar region, computes the centroid of the plane's inliers, and picks the mask whose centroid sits highest in z — the most elevated, and therefore most accessible, object.

Grasping. For the selected target, the module works on a filtered and downsampled point cloud. It computes surface normals for every point, discards candidate points whose angle exceeds a threshold, and around each remaining candidate defines a neighborhood sized to the suction cup radius. Within that neighborhood it keeps only points whose normals lie within a tolerance angle of the center normal. If enough such points exist, a plane normal is fitted. Each candidate is then scored on three things — how well it aligns angularly, how close it is to the object's centroid, and how many graspable points it has — and these are normalized and combined with weights that sum to 1. The candidate with the highest final score becomes the suction point, along with its normal.

Classification. In parallel, the mask proposals are passed to a Mask-CLIP model that has been fine-tuned. Because masked images contain large blank regions, tokenizing them for CLIP can produce zero tokens — a domain shift. The authors address this with mask prompt tuning, adding learnable prompt tokens derived from the binary mask to preserve boundary information. Classification is then done by cosine similarity between the masked-image features and text descriptors of the categories.

Training and data. Segmentation was trained on UOAIS-SIM (50,000 photorealistic RGB-D bin images) using a Tversky loss plus a classification loss, with the AdamW optimizer, learning rate 1e-4, batch size 4, and five epochs. The classifier was built on a 'mask-category' dataset of 720,000 product mask images across 53 main categories generated from photographs of 10,000 objects from 72 different angles (derived from the Product Image Dataset), fine-tuned using OpenCLIP with a ViT-L/14 CLIP variant, 90% of data for training and 10% for evaluation, over 10 epochs.

Real-robot evaluation. Experiments were done with a UR5e arm fitted with a Robotiq AirPick Vacuum Gripper and an Azure Kinect DK camera positioned at an elevated angle, across objects including boxes, cylinders, spheres, and irregular shapes. Success was measured with three metrics: pick success rate (pr = successful grasps / total attempts), object success rate (or = objects handled / total objects), and segmentation success rate (sr = items segmented within ±20% of ground-truth area / total objects).

Why This Matters

Impact on research. The paper is an argument that modularity, rather than larger end-to-end models, is what buys robustness in unstructured manipulation. It provides quantitative evidence that specialized components degrade more gracefully than monolithic grasp predictors when the test conditions drift away from training conditions, and it shows that an off-the-shelf segmentation backbone plus a training-free geometric grasp scorer can outperform dedicated learned grasp networks on novel objects.

Real-world applications:

• Warehouse and logistics order fulfillment, where robots pick mixed items from totes or bins that change content constantly.
• E-commerce and retail back-of-house sorting, matching the paper's own use of a 53-category product taxonomy (snack, drink, dairy, and so on).
• Manufacturing and assembly feed stations, where parts must be lifted from bins and placed in known orientations based on their identity.
• Recycling and waste sorting, where objects are class-agnostic and unpredictable and open-vocabulary labels could map items to material streams.

Industry relevance. Vacuum grippers are cheap, fast, and mechanically simple compared with multi-fingered hands, so a pipeline that works with suction and does not require retraining for every new SKU is directly commercially useful. The authors frame Seg2Grasp as suited to industrial automation where lighting, camera placement, and bin geometry cannot be held constant.

Future Directions

• Extending beyond suction-compatible objects. The authors explicitly note the experiment was designed to focus on objects compatible with the vacuum gripper. Generalizing the grasp module to parallel-jaw or multi-fingered grippers is left open.
• Learning the scoring weights instead of fixing them. The composite grasp score uses weights w_θ, w_d, and w_g that sum to 1, but the paper does not report how these weights were chosen or whether they were tuned. Learning or adaptively setting them per object could be a next step.
• Wider open-vocabulary coverage. The classification module was evaluated on 53 main categories; whether Mask-CLIP performance holds at much larger taxonomies, or on categories outside the product domain, is not reported.
• Closing the simulation-to-reality and timing gaps. The segmentation model was trained on simulated UOAIS-SIM data, and the paper reports no inference-time or throughput measurements — both would matter for industrial deployment.

Target Audience

Robotics researchers and engineers working on manipulation, bin picking, and industrial automation; practitioners building pick-and-place systems who need to choose between modular and end-to-end architectures; and students interested in how class-agnostic segmentation, geometric grasp scoring, and open-vocabulary vision-language classification can be combined into a working robotic pipeline. Reviewers and researchers focused on robustness and adaptability benchmarks in unstructured environments will also find the camera-height and bin-variation experiments directly relevant.

Authors’ abstract

Current bin picking methods that rely heavily on end-to-end learning often falter when confronted with unfamiliar or complex objects in unstructured environments. To overcome these limitations, we introduce Seg2Grasp, a modular pipeline designed for robust suction grasping in dynamic and cluttered bin scenarios. Seg2Grasp is built on a three-step process: Segmentation, Grasping, and Classification. The Segmentation module employs a Transformer-based model to generate class-agnostic object masks from RGB-D images, ensuring accurate detection across various conditions. The Grasping module uses surface normals and mask proposals to determine the optimal suction points, enhancing grasp success. Finally, the Classification module leverages fine-tuned open-vocabulary Mask-CLIP for precise object identification, enabling versatile handling of diverse objects. Real-world robotic experiments demonstrate that Seg2Grasp outperforms existing methods in success rates and adaptability, establishing it as a powerful tool for automated bin picking in industrial settings.