Give me scissors: Collision-Free Dual-Arm Surgical Assistive Robot for Instrument Delivery

AI summary

Overview

Research area: Surgical robotics, specifically dual-arm manipulator motion planning and real-time collision avoidance for operating-room assistance.

Technical level: Advanced. The paper combines vision-language-model task planning, learned distance-prediction networks, and constrained quadratic-programming control.

Scope: The paper designs, implements, and experimentally validates a dual-arm robot that takes surgical instruments from a sterile drape and hands them to a surgeon on verbal request, while avoiding environmental obstacles and self-collisions in real time.

What This Paper Is About

Scrub nurses repeatedly hand instruments to surgeons throughout an operation, which causes fatigue and can reduce focus, and staffing shortages can slow a surgical team down. Existing robotic scrub nurses can deliver instruments, but the instrument categories and delivery paths must be predefined, which limits generalization and leaves the robot without real-time collision avoidance in a dynamic room. This paper builds a dual-arm robot that uses a vision-language model to plan grasps and handovers zero-shot from the surgeon's instructions, and executes those plans through a unified quadratic-programming framework that keeps the arms clear of obstacles and of each other.

Key Contributions

1. A dual-arm surgical assistive robot for instrument delivery that uses a vision-language model to automatically generate grasping and delivery motion from the surgeon's instructions, without fine-tuning or predefined operations.
2. A unified real-time quadratic-programming framework that achieves reactive obstacle avoidance and dual-arm self-collision avoidance simultaneously during autonomous movement, acting as a safety filter alongside the task objectives.
3. A real-time obstacle minimum-distance perception method that estimates the minimum distance from robot links to environmental obstacles using a capsule approximation of the arms plus a distance-prediction neural network, with no visual markers or prior environment modeling.
4. Experimental validation showing an 83.33% success rate in real-world instrument delivery with no collisions in any trial.

Main Findings

• Instrument delivery success: Across 30 real-world trials covering all 6 pairings of four instruments (scalpel, tweezer, scissors, hemostat), each instrument was tested 15 times, and the average success rate was 83.33%. No collisions occurred in any trial.
• Per-instrument results (Table II): Scalpel — 0 collisions, detection 15/15, grasp 13/15, delivery 13/13, success 86.67%. Tweezer — 0, 15/15, 15/15, 15/15, success 100.0%. Scissors — 0, detection 14/15, grasp 12/14, delivery 12/12, success 80.00%. Hemostat — 0, detection 14/15, grasp 11/14, delivery 10/11, success 66.67%.
• Detection and grasping failure modes: No detection errors occurred for the scalpel and tweezers, while one error occurred for the scissors and one for the hemostat, attributed to their similar shapes causing VLM misjudgment. Grasping the tweezers succeeded completely (15/15); grasp success for the other three instruments was 86.67%, 85.71% and 78.57%. The cited cause of failed grasps is that these instruments are thin and smooth and therefore difficult to grasp on a flat desktop.
• Delivery failure mode: Delivery was fully successful except for one instance in which the surgeon failed to catch the hemostat.
• Simulation comparison: Three state-of-the-art reactive methods (DawnIK, CollisionIK, CBF-QP) were compared with all methods extended to cover both obstacle and self-collision avoidance. CollisionIK, CBF-QP and the proposed method accomplished obstacle avoidance, while DawnIK failed and encountered an obstacle collision at 4.7 s. All four methods achieved self-collision avoidance. CollisionIK became trapped in a local optimum during self-collision avoidance, and CBF-QP exhibited unstable oscillations.
• Reported comparison values: DawnIK 0.034, 0.116, 31.94; CollisionIK 0.038, 0.136, 32.10; CBF-QP 0.035, 0.055, 39.47; the proposed method 0.022, 0.054, 17.77. The paper states the proposed method had the shortest optimization time, the minimum mean position error, and the smallest maximum acceleration (17.77 m/s² versus 31.94, 32.10 and 39.47 m/s²). In the provided excerpt the table lists three numeric values per method under four numeric column headers (Opt Cost, Time, Mean Pos Error, Max Accel), so the column-by-column assignment of the two smaller values per row is ambiguous.
• Real-world collision avoidance: Ten trials per method were run with the surgeon approaching the left and right arms separately between 18 s and 30 s, with the same trajectory each time, and no visual markers used. Both the proposed QP framework and CBF-QP successfully avoided collisions in all trials; both ran at 40 Hz. The proposed method required less optimization time and showed less jittering in the end-effector's Cartesian acceleration.
• System behavior: In simulation the dual-arm robot achieved both self-collision avoidance and obstacle avoidance, and returned smoothly to the desired trajectories after each avoidance maneuver.

Methodology in Plain English

Perceiving obstacles in real time. Three Intel RealSense D435i RGB-D cameras, with overlapping views, build point clouds of the space around the robot. The complex arm geometry is approximated by capsules — cylinder-like safety volumes around each link — which makes it cheap to find the few obstacle points that are close to the arms. Points that belong to the robot itself are removed: a 2D segmentation mask of the robot is generated and mapped onto the synchronized depth map, so the system knows which points to ignore and cannot mistake the robot for an external obstacle. A neural network then predicts the minimum distance between the robot and nearby obstacles directly from the joint configuration and the filtered point cloud, along with the gradient of that distance; this avoids the high computational cost of measuring distance against the full robot mesh.

Staying safe while moving. The predicted distances feed a quadratic program that acts as a safety filter. Its cost has three parts: one that pushes joint increments toward the desired Cartesian velocity of each arm, one that pulls the robot toward the desired joint configuration, and one that keeps the joint increments small. The constraints apply a logarithmic term on the ratio between the predicted minimum distance and a safety threshold, so that when the robot is farther than the threshold the constraint relaxes and when the robot approaches an obstacle the constraint strongly pushes the joint increments along the gradient of the distance. The same structure is used for the distance between the two arms to prevent self-collision, plus joint-limit and joint-velocity constraints.

Understanding the surgeon. The surgeon's spoken instruction is converted to text. Camera images are processed with DINOv2 for pixel features and SAM to segment the objects of interest; each object's 3D keypoint is computed from its masked point cloud and features, then projected back onto the image as a numbered visual marker. The text, the marked-up image, and a prompt template are passed to a vision-language model (GPT-4o was applied), which returns the stage-by-stage sub-goals, expressed as L2-norm objective functions tied to those keypoints, and determines the grasp and release stages. MediaPipe hand landmarks locate the keypoint of the surgeon's hand near the robot for the handover. The sub-goals are minimized and interpolated into desired Cartesian positions, which are converted into desired joint configurations by inverse kinematics and passed to the quadratic program.

Hardware and settings. The platform used two Franka Research 3 robotic arms. Low-level control ran on two MIC-770-V2 (Advantech, China) industrial computers with Intel Core i7-10700 CPUs, 8 GB memory, Ubuntu 22.04 LTS with a 1 kHz PREEMPT_RT kernel. High-level task planning and QP optimization ran on a workstation with an Intel Core i9-14900KF CPU and NVIDIA RTX 4090 GPU, and a second workstation handled RGB-D acquisition and real-time perception. ROS2 connected the devices. The Flexible Collision Library (FCL) generated ground-truth minimum distances, and the SLSQP solver in SciPy performed the numerical QP optimization.

Authors’ abstract

During surgery, scrub nurses are required to frequently deliver surgical instruments to surgeons, which can lead to physical fatigue and decreased focus. Robotic scrub nurses provide a promising solution that can replace repetitive tasks and enhance efficiency. Existing research on robotic scrub nurses relies on predefined paths for instrument delivery, which limits their generalizability and poses safety risks in dynamic environments. To address these challenges, we present a collision-free dual-arm surgical assistive robot capable of performing instrument delivery. A vision-language model is utilized to automatically generate the robot's grasping and delivery trajectories in a zero-shot manner based on surgeons' instructions. A real-time obstacle minimum distance perception method is proposed and integrated into a unified quadratic programming framework. This framework ensures reactive obstacle avoidance and self-collision prevention during the dual-arm robot's autonomous movement in dynamic environments. Extensive experimental validations demonstrate that the proposed robotic system achieves an 83.33% success rate in surgical instrument delivery while maintaining smooth, collision-free movement throughout all trials. The project page and source code are available at https://give-me-scissors.github.io/.