MiDAS: A Multimodal Data Acquisition System and Dataset for Robot-Assisted Minimally Invasive Surgery

AI summary

Overview

Research area: Surgical robotics — specifically Robot-Assisted Minimally Invasive Surgery (RAMIS), multimodal surgical data acquisition, non-invasive motion sensing, and surgical gesture recognition.

Technical level: Intermediate. The system engineering and sensor setup are accessible, but the validation metrics (correlation of signal, normalized RMSE, temporal IoU, gesture-recognition benchmarks) assume some familiarity with machine learning and signal processing.

One-sentence scope: The paper introduces MiDAS, an open-source, platform-agnostic system that collects time-synchronized, non-invasive multimodal data from surgical robots without touching proprietary telemetry, validates it against internal robot kinematics, and releases two annotated datasets (Peg Transfer on Raven-II and hernia-repair suturing on da Vinci Xi).

What This Paper Is About

RAMIS research depends on multimodal data such as stereo video, robot kinematics, and pedal/interaction events, but the most valuable signals — the low-level telemetry inside clinical robots like the da Vinci — are locked behind proprietary interfaces, institutional privacy constraints, and limited access to clinical platforms. The authors build MiDAS, a system that reconstructs comparable information using only external sensors (electromagnetic hand tracking, a depth camera, and force sensors on the foot pedals), and they validate how closely those external signals approximate the robot's own internal measurements.

Key Contributions

1. An open-source, non-invasive acquisition system. MiDAS integrates electromagnetic hand tracking, RGB-D depth-camera hand tracking, stereoscopic video capture, and foot-pedal force sensing through a unified client–server framework, collecting fully synchronized data in real time without modifying or interfacing with proprietary control systems.

2. New proxy modalities for robot kinematics. The authors propose surgeon hand and foot motion signals as alternatives to proprietary kinematics, and demonstrate through correlation, error, and temporal-alignment analysis that these externally sensed signals closely approximate surgeon and robot kinematics and are usable for downstream tasks such as activity recognition.

3. Cross-platform evaluation on two representative tasks. MiDAS was deployed for dry-lab Peg Transfer on the open-source Raven-II platform and for hernia repair suturing on KindHeart hybrid porcine tissue models using the clinical da Vinci Xi during a 2024 training bootcamp at the University of Virginia Hospital. The work benchmarks state-of-the-art transformer-based (MTRSAP) and CNN-based (MS-TCN++) gesture recognition models across different modality combinations.

4. Public release of the system and annotated datasets. The data collection system and the resulting multimodal, multi-platform datasets with gesture annotations are released, including what the authors describe as the first multimodal dataset of hernia repair suturing on simulation models.

Main Findings

• External hand tracking tracks internal robot motion closely. Over 15 Peg Transfer trials, EmHT–MTM and EmHT–PSM mean correlation (CoS) exceeded 0.8 across all positional axes (X, Y, Z). EmHT showed NRMSE of 17.5% (X), 24.4% (Y), and 16.2% (Z) against MTM, and 17.0% (X), 21.0% (Y), 16.4% (Z) against PSM.

• Depth-camera hand keypoints are a weaker but still usable positional proxy. HandKP–MTM and HandKP–PSM achieved mean CoS exceeding 0.7 across all axes, with 0.88–0.89 alignment on the Z-axis. NRMSE values were higher than EmHT (e.g., 26.13% for HandKP–MTM on X, 30.15% on Y).

• Orientation estimation is the weak point. EmHT–MTM orientation reached a CoS of 0.90 for Yaw but only 0.61 for Pitch and 0.65 for Roll. EmHT–PSM orientation was substantially lower, ranging from 0.32 (Roll) to 0.44 (Pitch). Orientation from HandKP did not achieve strong correlations and is reported in an appendix.

• Grasper angle estimation from EmHT is moderate. IoU was approximately 0.53–0.54 with accuracy of approximately 0.80–0.81 for both MTM and PSM grasper angles.

• Foot pedal sensing performs well on both platforms. On Raven-II, FPS achieved F1 0.85, precision 0.78, recall 0.95, Temporal IoU 0.74, and a lag of 166.67 ms. On da Vinci Xi, it achieved F1 0.78, precision 0.69, recall 0.86, Temporal IoU 0.67, and a lag of 133.33 ms. Both platforms show a bias toward high recall at the cost of some false positives.

• Pedal usage varies by procedure and console configuration. In suturing trials considering energy pedals only, Primary Energy (Right) mapped to FPS Pedal 4 accounted for roughly 70.8% of actuations with a detection F1 of 0.78. The Camera pedal was the most frequently used pedal overall. The Clutch pedal was infrequently actuated because clutching was available on the hand manipulators; it was not detected using FPS because finger clutch was used instead of the foot pedal clutch.

• Non-invasive signals approach proprietary telemetry for gesture recognition. On the Raven-II Peg Transfer task, MTRSAP trained on internal telemetry (MTM and PSM) reached an F1 of 0.87 for both inputs. EmHT with MTRSAP achieved accuracy 0.87 and F1 0.86 — closely matching the proprietary baseline. HandKP performed poorly (F1 0.38), and image-only models using DINOv2 features reached F1 0.48, which the authors attribute to the limited dataset size of 15 trials.

• Suturing gesture recognition results on da Vinci Xi are not fully reported in the provided content, which is truncated at the start of that table; therefore no numbers for that experiment can be stated here.

• Participants found the system non-intrusive. Of 40 participants, mostly 2nd and 3rd year surgical residents, 92.3% reported overall satisfaction with the training experience, 86.1% rated the KindHeart tissue models as realistic, and 80% disagreed or strongly disagreed that the sensors negatively affected their psychomotor performance.

• Dataset scale. The Raven-II Peg Transfer dataset contains 7 gesture classes and 345 gesture samples across 15 trials, totaling approximately 36 minutes (about 2.4 minutes per trial). The da Vinci Xi suturing dataset contains 8 gesture classes and 1,724 gesture samples across 17 trials, totaling approximately 212 minutes (about 12.5 minutes per trial), with roughly 3.5 hours of annotated suturing segments collected from the bootcamp.

Methodology in Plain English

The team built a piece of software that acts as a central server, coordinating several off-the-shelf sensors and synchronizing everything into a single time-aligned record.

Hand motion. Two miniature electromagnetic sensors are mounted on each master manipulator at the thumb and middle-finger contact pads, logging 3D position and orientation at 270 Hz using an NDI trakSTAR device. Raw sensor readings are mapped into the robot's reference frames with a calibrated rigid-body transformation, plus a learned residual correction modeled by a multilayer perceptron. Separately, a ZED Mini RGB-D camera is mounted above the console, producing synchronized 720p color video and depth maps at 30 Hz to extract thumb and middle-finger keypoints per hand. Gaps of up to one second in keypoint detection are filled with cubic spline interpolation, and longer gaps are forward-filled.

Video. Open Broadcaster Software captures stereoscopic streams at 30 Hz — via Blackmagic SDI recorders on the da Vinci Xi and via a ZED camera on Raven-II — and is controlled programmatically through the OBS WebSocket API.

Foot pedals. A compact module built around an Arduino-class microcontroller and thin-film force-sensitive resistors is affixed directly to the pedal surfaces. It supports up to nine pedals and records binary states and timestamped analog signals at 30 Hz.

Data collection. For Raven-II, 15 Peg Transfer trials were collected (three pegs moved per trial) and labeled with the existing DESK seven-gesture taxonomy; ground-truth MTM and PSM kinematics and clutch presses were available for comparison. For da Vinci Xi, data were collected at a 2024 UVA Hospital bootcamp on 7 inguinal and 10 ventral hernia repair trials using two da Vinci Xi units and KindHeart porcine tissue models. Because the da Vinci Xi exposes no internal kinematics,

Authors’ abstract

Background: Robot-assisted minimally invasive surgery (RMIS) research increasingly relies on multimodal data, yet access to proprietary robot telemetry remains a major barrier. We introduce MiDAS, an open-source, platform-agnostic system enabling time-synchronized, non-invasive multimodal data acquisition across surgical robotic platforms. Methods: MiDAS integrates electromagnetic and RGB-D hand tracking, foot pedal sensing, and surgical video capturing without requiring proprietary robot interfaces. We validated MiDAS on the open-source Raven-II and the clinical da Vinci Xi by collecting multimodal datasets of peg transfer and hernia repair suturing tasks performed by surgical residents. Correlation analysis and downstream gesture recognition experiments were conducted. Results: External hand and foot sensing closely approximated internal robot kinematics and non-invasive motion signals achieved gesture recognition performance comparable to proprietary telemetry. Conclusion: MiDAS enables reproducible multimodal RMIS data collection and is released with annotated datasets, including the first multimodal dataset capturing hernia repair suturing on high-fidelity simulation models.