TransBiolab: A Real-World Multi-View Dataset of Cluttered Transparent Biomedical Objects

AI summary

Overview

• Research area: Computer vision datasets and benchmarks for robotic perception, specifically transparent-object segmentation, depth estimation/completion, and 6D pose estimation in biomedical laboratory settings.
• Technical level: Advanced.
• Scope: The paper introduces TransBiolab, a real-world calibrated multi-view RGB-D dataset of cluttered transparent biomedical laboratory objects, together with benchmarks for segmentation, depth, 6D pose estimation, and a real-robot manipulation evaluation.

What This Paper Is About

Autonomous biomedical laboratories need robots to recognize, localize, and manipulate transparent plasticware such as well plates, Petri dishes, centrifuge tubes, culture flasks, pipettes, reagent bottles, and bioreactors. These objects are weakly textured, strongly view-dependent, and difficult because reflection, refraction, and transmission entangle them with the background, especially when many similar transparent instances appear together and occlude one another. Existing transparent-object datasets (KeyPose, TODD, StereOBJ-1M, ClearPose, TransCG, Trans10K-v2) advanced segmentation, depth, and pose estimation but typically use household or chemical objects rather than biomedical plasticware, and they do not center evaluation on the combined setting of multi-object clutter, occlusion, and calibrated multi-view capture. TransBiolab was built to fill that gap.

Key Contributions

1. A real-world RGB-D dataset of cluttered transparent biomedical objects. TransBiolab contains 161,315 frames from 98 scenes and about 1.03M instance annotations across 15 laboratory object types, including 6D poses, full masks, visible masks, depth, and per-frame camera calibration. It comprises 88 controlled collection scenes and 10 held-out real-world laboratory scenes.

2. A capture and annotation design organized around operational difficulty. Data is organized along three axes: object category, the total number of objects in a frame, and camera viewpoint. Controlled scenes use 11 arrangement patterns, 4 tabletop backgrounds, and 2 lighting settings; sequences are captured as calibrated multi-view trajectories with a robot-mounted camera.

3. Dataset-centric benchmarks for three perception tasks. The paper defines and reports segmentation (SAM 3, TransLab), depth estimation and completion (Depth Anything 3, ClearGrasp), and 6D pose estimation (FoundationPose, MegaPose-6D) benchmarks, including results on held-out laboratory scenes (TBiolab-HO).

4. A system-level real-robot manipulation evaluation. Using released segmentation and pose annotations, the authors run grasping trials with a Franka parallel-jaw gripper and a LinkerHand 10-DoF dexterous hand, and provide BOP-style format-conversion utilities for existing pose estimation toolchains.

Main Findings

• Segmentation remains unsaturated in the biomedical lab setting. On the mask-level protocol, SAM 3 scores IoU 0.662, precision 0.881, recall 0.703, and F1 0.764 on TransBiolab-all, versus IoU 0.699 and F1 0.789 on ClearPose-all and IoU 0.704 and F1 0.824 on Trans10K. TransLab scores IoU 0.812 and F1 0.881 on Trans10K but drops to IoU 0.363 and F1 0.496 on TransBiolab-all.
• Held-out scenes are harder for segmentation. On TBiolab-HO, SAM 3 drops to IoU 0.477, precision 0.656, recall 0.648, F1 0.632; TransLab reaches IoU 0.344, precision 0.475, recall 0.665, F1 0.491.
• Instance-matching segmentation numbers. On the evaluated subset with a text-prompt protocol, SAM 3 attains overall mIoU 0.685, a match rate of 84.8%, and matched-instance mF1 of 0.881.
• Object geometry drives segmentation difficulty. Best per-category results include the Grex 1L bioreactor (100.0% match rate, mIoU 0.914) and the 90 mm dish (97.1%, 0.845); hardest are the 15 ml centrifuge tube (71.7%, 0.484), 50 ml centrifuge tube (75.6%, 0.495), 75 ml flask (77.1%, 0.574), and 125 ml flask (78.6%, 0.587). By geometric family: plate-like mIoU 0.789 (89.2% match), bottle-like 0.700 (86.3%), tube-like 0.536 (76.1%).
• Segmentation is relatively stable across most viewpoints. mIoU stays between 0.681 and 0.693 for VP1 through VP8, then falls to 0.609 at VP9 and 0.583 at VP10, where match rate drops to 71.2% and 68.0% while matched-instance mF1 remains high (0.919 and 0.915), indicating missed detections dominate over matched-mask quality.
• Depth errors are larger on TransBiolab than on ClearPose. For Depth Anything 3 (metric depth, no test-time scale or shift alignment) on TransBiolab-all: AbsRel 0.371, RMSE 0.224 m, MAE 0.220 m, raw missing 0.349. ClearGrasp on TransBiolab-all: AbsRel 0.393, RMSE 0.473 m, MAE 0.349 m, raw missing 0.313. On ClearPose-all, DA3 gives AbsRel 0.161, RMSE 0.221 m, MAE 0.141 m, raw missing 0.390, and ClearGrasp gives AbsRel 0.327, RMSE 0.404 m, MAE 0.276 m, raw missing 0.299.
• ClearGrasp performs better on the held-out laboratory scenes. On TransBiolab held-out: ClearGrasp AbsRel 0.179, RMSE 0.132 m, MAE 0.086 m, raw missing 0.178, versus DA3 AbsRel 0.302, RMSE 0.180 m, MAE 0.174 m, raw missing 0.189.
• 6D pose accuracy drops sharply from ClearPose to TransBiolab. FoundationPose reaches ADD-S AUC 80.80, ADD AUC 53.90, mean ADD-S 23.20 mm, and success 67.86% on TransBiolab-all, compared with ADD-S AUC 91.05, ADD AUC 49.26, mean ADD-S 9.77 mm, and success 89.53% on ClearPose-all. On TBiolab-HO it reaches ADD-S AUC 75.35, ADD AUC 47.06, mean ADD-S 34.53 mm, success 62.31%.
• Without depth, pose estimation is much weaker. MegaPose-6D (RGB, ground-truth box, CAD) scores ADD-S AUC 31.76, ADD AUC 31.49, mean ADD-S 67.76 mm, success 34.14% on TransBiolab-all; on TBiolab-HO, ADD-S AUC 55.50, ADD AUC 25.17, mean ADD-S 96.46 mm, success 26.59%. The authors note that because the two methods take different inputs, this is not a controlled depth ablation.
• Symmetry and geometry, not clutter alone, explain pose difficulty. FoundationPose results by geometric family: rack/elongated ADD-S AUC 93.6 (10.4 mm), plate-like 92.2 (7.9 mm), dish-like 90.4 (10.4 mm), tube-like 78.2 (22.2 mm), bottle-like 65.9 (45.6 mm), and large/irregular 56.9 (53.5 mm). Large ADD-S/ADD gaps for tube racks, Petri dishes, and microplates indicate rotational identity uncertainty under symmetry, whereas the pipette shows a smaller gap (ADD-S AUC 93.0, ADD AUC 88.2, 10.8 mm) due to its asymmetric geometry.
• Clutter alone does not produce a monotonic trend. Grouping by total object count and distinct category count shows that some moderate-clutter bins outperform sparser ones, which the authors attribute to geometry and symmetry still dominating difficulty.
• Shallow viewpoints are the hardest for pose. FoundationPose at VP1 gives ADD-S AUC 72.2 with mean ADD-S 38.7 mm, rising to ADD-S AUC 87.8 with mean ADD-S 12.5 mm at VP10.
• Real-robot manipulation is a system-level evaluation. With a Franka parallel-jaw gripper, 98 of 150 trials succeeded (65.3%); with a LinkerHand 10-DoF dexterous hand, 85 of 150 trials succeeded (56.67%). The authors state the lower dexterous-hand rate may reflect grasp synthesis and control complexity in addition to perception error.
• Annotation geometry is internally consistent. Projected CAD silhouettes versus visible masks give a mean IoU of 0.949 (median 0.971); median symmetric contour error is 0.62 pixels for same-view reprojection and 0.91 pixels after cross-view transfer, with the 95th-percentile error below one pixel in both cases. The paper notes this audit is not independent metrological validation.

Methodology in Plain English

The authors first selected 15 biomedical laboratory objects and grouped them into five functional categories: well plates (6-, 12-, 24-, and 96-well), cell culture dishes (35 mm, 60 mm, 90 mm), liquid-handling objects (15 ml and 50 ml centrifuge tubes, pipette, tube rack), cell culture flasks (25 ml, 75 ml, 125 ml), and a 1 L bioreactor. Seven objects are fully transparent, six combine transparent bodies with opaque caps or bases, and two are opaque auxiliary tools.

Data was captured with an Intel RealSense D435i RGB-D camera at 1280 × 720 resolution and 30 FPS, mounted on a 7-DoF Franka Emika Panda robot arm so the camera could travel a repeatable continuous multi-view path above each scene. Each sequence lasts about 50 seconds and typically contains 1,300–1,800 frames, with a mean sequence length of about 1,646 frames; full trajectories are released as calibrated videos. The 11 arrangement patterns were designed to reflect recurring laboratory procedures such as sample preparation, liquid handling, mixing, sampling, liquid transfer, transport between workstations, and cell expansion or harvest; scene difficulty increases from 4–5 objects to 9–12 objects with heavier mutual occlusion. Controlled scenes vary by four tabletop backgrounds (white tablecloth, grey tablecloth, bare wooden tabletop, magazine-covered tabletop) and two lighting settings (top light about 190 lx, side light about 75 lx). Ten additional held-out real-world laboratory scenes were captured handheld with new layouts, more distractors, and different backgrounds and lighting.

Annotation is sequence-centric rather than single-frame, because depth alone is unreliable for transparent objects. Each object was measured at millimeter precision and reconstructed as a 3D mesh in Rhino, and all 15 models are released in OBJ format. Image sequences and models were imported into the Blender-based ProgressLabeller workspace; camera trajectories were estimated with ORB-SLAM3, and the depth stream was fused into a scene-level point cloud using KinectFusion-style reconstruction. Annotators aligned meshes in the multi-view workspace by jointly checking RGB reprojection, depth-generated point clouds, and plane consistency across views, producing object identity, 6D pose, full mask, visible mask, aligned depth, camera intrinsics, and per-frame extrinsics. Sequence-level annotation takes about 40 minutes for a sequence of roughly 2,000 images.

For benchmarking, the authors use scene-disjoint splits and ten sampled viewpoints per trajectory for within-dataset diagnostics, while keeping full sequences available for video and multi-view methods. Sampled setups all contain more than three objects: 161,059 frames (99.84%) contain more than three in-frame objects, and 256 frames (0.16%) contain exactly three when camera motion briefly moves other objects out of view; those frames are retained for sequence continuity. The 6D pose baseline FoundationPose is run on RGB-D frames with the visible mask, camera intrinsics, and CAD model, using the standard register–refine pipeline with 15 refinement iterations, with ADD, ADD-S, and AUC up to 0.1 m reported.

Why This Matters

Impact on research. TransBiolab targets a boundary case where general-purpose perception models remain weak. It provides a common testbed that separates difficulty sources that are usually confounded: object category, clutter level, and viewpoint. Because each scene is a calibrated multi-view sequence, the dataset supports both frame-based evaluation and multi-view or video-based methods for resolving occlusion of transparent objects, and the BOP-style conversion utilities let existing pose-estimation toolchains be adapted with limited effort. The comparison table positions TransBiolab (RGB-D, 15 objects, 98 scenes, 161,315 images, 4 backgrounds, 2 lighting settings, multi-instance, multi-view, distractors) alongside transparent-object datasets that differ in domain, modality, and scale.

Real-world applications:

• Autonomous biomedical laboratories that must recognize, localize, and transfer lab vessels across liquid handling, sample preparation, and cell-culture workflows.
• Robotic grasping and placement of transparent plasticware, including dexterous-hand manipulation driven by segmentation and 6D pose output.
• Depth sensing pipelines for transparent objects, where missing or distorted depth returns degrade downstream planning.
• View planning and multi-view fusion for robot-mounted cameras, informed by the finding that shallow viewpoints are the hardest for pose estimation.

Industry relevance. The dataset and benchmarks speak directly to laboratory automation vendors and robotics integrators building perception-driven workcells, particularly those relying on commercial RGB-D cameras and standard pose-estimation toolchains. The real-robot trials on a Franka arm with two end-effectors show that a perception-to-manipulation pipeline is feasible end to end while also quantifying the gap between benchmark accuracy and system success. The authors state the dataset contains no faces, personal identities, or patient data.

Future Directions

• Improving transparent-object segmentation on held-out laboratory scenes, where SAM 3 falls to IoU 0.477 and TransLab to IoU 0.344 on TBiolab-HO.
• Extending depth estimation and completion methods to the biomedical lab domain, where both DA3 and ClearGrasp show higher object-region errors on TransBiolab-all than on ClearPose-all.
• Exploiting the multi-view and video structure of the data for view planning and multi-view fusion, given the finding that shallow views (VP1, ADD-S AUC 72.2) are much harder than steep views (VP10, ADD-S AUC 87.8).
• Designing pose methods that resolve rotational ambiguity under symmetry, since tube racks, Petri dishes, and microplates keep high ADD-S AUC but much lower ADD AUC, and large irregular containers such as the 125 ml flask remain difficult (ADD-S AUC 41.0, mean ADD-S 75.1 mm with FoundationPose).
• Separating perception error from grasp synthesis and control in manipulation evaluation, since the dexterous-hand success rate (56.67%) is lower than the parallel-jaw rate (65.3%) and may reflect grasp-type selection, joint mapping, and contact-rich closure.

Target Audience

Researchers and engineers working on transparent-object perception, 6D pose estimation, RGB-D depth completion, and robotic manipulation; developers building autonomous laboratory or biomedical workcell systems; and benchmark designers interested in how clutter, object geometry, symmetry, and camera viewpoint affect segmentation, depth, and pose models. The paper is best suited to readers with prior familiarity with RGB-D datasets, pose estimation metrics such as ADD and ADD-S, and segmentation evaluation.

Authors’ abstract

Autonomous biomedical laboratories increasingly rely on visual perception to recognize, localize, and manipulate transparent plasticware, yet high-quality real-world datasets for this setting remain limited. The scarcity of domain-relevant data is particularly restrictive in cluttered multi-object scenes, where mutual occlusion and view-dependent appearance changes remain challenging even for contemporary visual foundation models. Existing transparent-object datasets have advanced segmentation, depth, and pose estimation, but they usually do not evaluate the combined setting of multi-object clutter, occlusion, and calibrated multi-view capture that characterizes real laboratory manipulation scenes. To address this gap, we present TrainsBiolab, a real-world RGB-D dataset of cluttered transparent biomedical objects captured as calibrated multi-view sequences. TrainsBiolab contains 161,315 frames from 98 scenes and 1.03M instance annotations over 15 laboratory object types, including 6D poses, full and visible masks, depth, and per-frame camera calibration. The dataset is organized along three axes that reflect operational difficulty: object category, the total number of objects in a frame, and camera viewpoint. We further define dataset-centric benchmarks for segmentation, depth estimation and completion, and 6D pose estimation, and report a system-level robot manipulation evaluation enabled by the released annotations and calibrations. By focusing on repeated transparent instances, clutter, and multi-view laboratory capture, TrainsBiolab provides a resource for segmentation, depth estimation, 6D pose estimation, and multi-view reasoning in autonomous laboratory manipulation. Project page: https://dualtransparency.github.io/TransBiolab/.