Industrial Dexterity Benchmark: A Hardware-Software Benchmarking Platform for Industrial Dexterous Manipulation

AI summary

Overview

• Research area: Robotics — industrial dexterous manipulation, end-to-end imitation learning, multimodal perception (RGB, point cloud, proprioception, force/torque), and standardized benchmarking.
• Technical level: Advanced. The paper assumes familiarity with diffusion policies, behavior trees, ROS2, teleoperation, and force/torque control, though its framing of the problem is straightforward.
• Scope in one sentence: The paper introduces three industrial benchmark boards, a ROS2-based imitation-learning infrastructure, and a multimodal diffusion policy (AG-iDP3), then compares six sensor/policy configurations on a datacenter cable-cleaning and re-insertion task.

What This Paper Is About

Industrial dexterous tasks such as cable routing, connector insertion, and precision assembly are still largely manual, and the authors' earlier classical vision-plus-control pipeline for the NIST Assembly Task Board #4 (ATB4) harness-insertion challenge worked only under controlled conditions and was brittle and hard to scale. The paper's goal is to replace that modular pipeline with end-to-end learned policies, and to provide a reproducible benchmark for measuring how well different sensor suites solve industrially relevant manipulation. The study reports results exclusively on one board — datacenter cable manipulation — with the other two board designs described for completeness.

Key Contributions

1. The Industrial Dexterity Benchmark (IDB) Boards — three custom benchmarking boards inspired by the NIST Assembly Task Board set: datacenter fiber switches (IDB Board #1), automotive cable harnesses (IDB Board #2), and planetary gearbox assembly (IDB Board #3). They are 3-D printed, laser cut, and built from off-the-shelf hardware at a total cost in the low hundreds of USD, with CAD drawings, bill of materials, fabrication instructions, and scoring metrics published online.
2. An imitation-learning framework — a ROS2-based infrastructure tying together teleoperation, sensor and sub-component integration, data collection, and real-time robotic arm control, deployed as modular Docker containers.
3. AG-iDP3 — a multimodal diffusion-based policy framework whose models fuse RGB images, point clouds, joint positions, and wrist-frame wrench data, deployed and chained together using a PyTrees-based behavior tree architecture.
4. Evaluator nodes and a head-to-head evaluation — behavior-tree evaluator nodes determine the completeness of each learned task, and six hardware/sensor configurations are compared on the cable-cleaning task using 48 trials per configuration.

Main Findings

• Best configuration reaches 76% total task success: The multi-view RGB configuration (RGB wrist D405 + RGB scene D435i, each encoded by R3M) achieved a 76% grasp + insert combined success rate. All three phases required roughly 100 demonstrations each to train a well-performing policy across every ablation.
• The single-camera baseline is far behind: The single-camera RGB DP baseline (configuration 6) reached 36% total. All four multimodal expansions (configurations 1–4) outperformed it.
• The egocentric point-cloud policy beats egocentric RGB: The point-cloud-only iDP3 configuration (configuration 5) reached 52% total, replicating the iDP3 finding that an egocentric point-cloud policy beats an egocentric RGB policy.
• 3-D context drives grasping: Grasp success ranged from 88% to 98% across the five configurations that include 3-D context (point cloud, or two RGB cameras providing different perspectives). Only in the single-RGB configuration, where 3-D context is entirely removed, did grasp success drop to 48%.
• Insertion differentiates total scores: Across configurations 1–5, insertion score varied by 48 percentage points while grasp success varied by only 10 percentage points.
• ToF beats passive stereo in the head-to-head scene camera comparison: In the direct comparison between the two scene cameras, the EVAL-ADTF3175 ToF outperformed the RealSense D435i by 7 percentage points, suggesting indirect ToF is better suited to industrial datacenter conditions.
• Cleaning was solved by every configuration: The cleaning phase succeeded on 100% of trials across every configuration after a successful grasp, which is why it is excluded from the total score.
• Wrench input helps only during insertion: Across the three behavior-tree phases, wrench input only helped during the contact-rich insert phase; for the grasp and clean phases it provided no measurable benefit and was gated off.
• Deployment efficiency versus classical pipelines: Deploying the learned policy on a new task requires only roughly 100 teleoperated demonstrations, whereas the classical pipeline demands thousands of labeled images for perception training, extensive per-stage parameter tuning, and reprogramming of motion planning and force/torque steps for each new task.
• The learned policy remains brittle to incidental scene changes: Removing a background object (a red polymer block) present at training time caused failure, and changing the routing of a cable along the robot arm (loose vs. tied) also caused failure. These were mitigated by training and deploying on cropped images and by adopting pulley-based cable management.

Methodology in Plain English

The authors started with a classical pipeline for the NIST ATB4 harness-insertion task, using an xArm6 robot with a wrist camera and a force/torque sensor. That pipeline identified the board with fiducial markers, matched depth images against a 2-D CAD connector layout, ran a circular connector and clip search up to five times, then used YOLO for segmentation, FoundationStereo for depth, and FoundationPose for 6-DOF object pose. The output fed inverse kinematics and motion planning, followed by visual servoing and a force/torque classification with a closed-loop spiral search on failed insertions. Because that approach was lighting-sensitive, struggled with in-hand connector pose, and required recalibration and reprogramming for even minor task-space changes, the authors moved to learned end-to-end policies.

For the new benchmark, they built three boards and selected the IDB Board #1 cable-cleaning and re-insertion task on the SC simplex connector as the baseline. The board is assembled from aluminum extrusion (about 270 mm tall, 250 × 340 mm footprint) with three 3-D-printed patch panels housing RJ45, SC simplex, and LC duplex fiber ports, plus a 3-D-printed circular pad mimicking an industrial connector contact cleaner. Trials were run at the board's lowest difficulty setting: every port empty except the single target connector, appropriate for the simple two-finger Robotiq 2F-85 parallel gripper used. The task was split into three phases — grasp, clean, and insert.

Demonstrations were collected by teleoperation with a GELLO-style leader arm. A data aggregator produced time-aligned RGB, point cloud, joint, and wrist force/torque streams at 15 Hz, with a master data stream that all others align to, a capture GUI, a validity checker, and an automated conversion to training-ready Zarr format. A CRISP impedance controller ran at 50 Hz, driving a Franka FR3 at 1 kHz via the Franka Control Interface. Inference ran on a dedicated GPU node (RTX 5090) that published action-chunk outputs at 15 Hz. The host was an Ubuntu 24.04 RT-kernel machine with ROS2 Control and URDF integration; all runs were logged in MCAP and visualized in Foxglove.

In AG-iDP3, wrist RGB and optionally scene RGB are each encoded by an R3M (ResNet18) backbone — fine-tuned on the collected demonstrations, which performed better than a frozen backbone. Point clouds are downsampled and encoded by a lightweight multi-stage PointNet encoder, while joint positions and wrist wrench pass through unencoded. All vectors are concatenated into one observation, fed to a diffusion U-Net that outputs an action chunk of T = 15 joint actions. Chunks are blended by temporal ensembling with decay k = 0.01 and interpolated by cubic splines to a 50 Hz command stream, roughly 3× denser than the inference rate, with only the first N = 3 steps of each chunk committed before re-inference. A PyTrees behavior tree orchestrates the task, pairing each learned policy node in a Parallel node with an evaluator node that stops the policy on success or after a 1-minute timeout. The behavior tree combines the learned policies in grasp, clean, and insert with classical motion primitives for move to home, extract cable, return to rack, and return home.

Why This Matters

The paper quantifies the shift from hand-engineered perception and control to learned multimodal policies under industrial constraints, using a fixed protocol and 48 trials per configuration so that the effect of each input modality can be isolated rather than confounded with task variation. It also supplies a shareable, low-cost benchmark family that other groups can fabricate and reproduce.

Real-world applications:

• Datacenter cable management: Cable installation, routing, replacement, and connector cleaning in densely populated rack switches, where up-time requirements exceed 99.99% and human intervention is currently common.
• Automotive cable harness assembly: Routing two 6-pin cable harnesses through bungee routing clips and connecting eight connectors (two 6-pin, six 2-pin) in enclosed, partially occluded regions, with LED feedback for verified electrical connection.
• Gearbox assembly: Picking planetary gearbox components from cutout holders, aligning gear meshes, and tightening M3 fasteners in the correct sequence on a NEMA 17 stepper motor-driven assembly.
• High up-time industrial environments generally: The authors argue the results justify a shift toward scalable robotic automation wherever retraining must be fast and inexpensive.

Industry relevance: The work originates from an industrial lab (all authors are with Analog Devices, Inc., USA), compares Analog Devices' EVAL-ADTF3175 ToF sensor against an Intel RealSense D435i scene camera, and emphasizes on-site data collection economics: about 100 demonstrations per phase is the binding constraint for industrial deployment, which is why the authors retain the iDP3 convolutional diffusion U-Net rather than a transformer policy.

Future Directions

• Complete the remaining IDB boards: Only the datacenter cable manipulation board produced results; the automotive cable harness and gearbox assembly boards are described for completeness and stated to be the subject of future work.
• Fix the robustness gap: The visual encoder appears to latch onto incidental scene features rather than task-relevant ones. The authors point to data augmentation, domain randomization, and broader scene diversity in training.
• Improve sensing for tight-clearance insertion: The resolution of both point-cloud scene cameras, further downsampled in the PointNet encoder, was not fine enough to resolve the receptacle features needed for tight-clearance insertion — likely why the dual-RGB configuration outperformed both RGB-wrist + PC-scene configurations. The authors note ToF may still be more reliable in industrial conditions where lighting varies and equipment discolors.
• Scale to more demanding setups: The advanced routing option on IDB Board #2 would likely require bimanual coordination, and the current two-finger parallel gripper lacks the dexterity to manipulate a connector among tightly packed neighbors, so board difficulty in this study was fixed at its lowest setting.

Target Audience

Robotics researchers and engineers working on manipulation, imitation learning, and multimodal diffusion policies; industrial automation teams evaluating sensor suites for contact-rich assembly; and benchmarking groups interested in reproducible, low-cost task boards. Readers wanting an entry point to the topic will find the problem framing accessible, but the policy architecture, behavior-tree orchestration, and control-stack details assume a robotics background.

Authors’ abstract

Dexterous manipulation remains a critical bottleneck in industrial automation; tasks such as cable routing, connector insertion, and precision assembly still rely heavily on manual labor despite decades of robotics research. This work presents a progression from classical, modular robotics pipelines toward an end-to-end multimodal imitation-learning framework for industrial dexterous manipulation. As a part of this work, we introduce three key contributions: a set of Industrial Dexterity Benchmark (IDB) boards aimed to mimic datacenter cable management, automotive cable harnesses, and gearbox assembly tasks; a scalable imitation learning framework (DAG-ROS); and a multimodal diffusion-based policy framework (AG-iDP3) that creates models fusing RGB images, point clouds, joint positions, and wrist-frame wrench data. Focusing on the datacenter cable manipulation board, we evaluate the performance of a task involving cleaning a single cable over variations of an end-to-end AI policy using 48 trials per configuration. The best performing configuration, a multimodal expansion Diffusion Policy (DP), includes a multi-view RGB image source passed through an R3M encoder and reaches a 78% grasp and insert combined task success rate. This performance marks a significant improvement over the 36% observed from the single-camera RGB DP baseline. Each of the tested configurations requires only approximately 100 teleoperated demonstrations per task phase. These results indicate that the correct learned policy can outperform classical vision and control robotic methods in robustness, generalization, and deployment efficiency, justifying a shift toward scalable robotic automation for high up-time industrial environments.