Sensorium: An Open-Source Multimodal Data Collection and Curation Library for Embodied Surgical AI

AI summary

Overview

Research area: Surgical robotics and embodied AI data infrastructure — multimodal data collection, curation, and dataset tooling for robot learning.

Technical level: Intermediate. The paper assumes familiarity with ROS/ROS 2, robot teleoperation, and dataset formats (CSV, LeRobot), but the motivation and evaluation are readable without deep expertise.

Scope (one sentence): The paper presents Sensorium, an open-source Python library built on ROS and ROS 2 for recording, aligning, annotating, and exporting multimodal surgical robot data, validated over 1,516 recordings across phantom, ex vivo, and in vivo porcine experiments.

What This Paper Is About

Collecting the multimodal data needed to train embodied surgical AI — stereo video, robot kinematics, pedal input, force sensing, and language annotations — is expensive, hard to repeat, and tightly coupled to whatever hardware a given lab happens to own. Existing open-source tools either target one specific robot (e.g., SurgSync on the da Vinci Research Kit) or reconstruct kinematics indirectly when a robot exposes no telemetry (MiDAS). The authors build Sensorium to fill the gap: a general, configuration-driven library that records from any ROS-integrated robot or sensor, then aligns, annotates, and exports those recordings as reusable episodic datasets for downstream surgical AI training.

Key Contributions

1. A flexible, configuration-driven collection and curation library. Sensorium records from heterogeneous ROS/ROS 2 devices — any device that publishes a ROS topic can be added through configuration without modifying the recording core — and keeps all post-recording steps (alignment, processing, annotation, export) ROS-independent so they can run later on a separate workstation.

2. An operator-oriented browser GUI. A single interface unifies live recording supervision (camera view, sensor readiness, recording state, elapsed episode time), post-hoc episode review, surgical-phase and language annotation, action-denoising preview, and dataset generation, with both a command-line and a browser-based interface for local and remote operation.

3. Real-world validation across surgical settings. The system was used for 1,516 recordings on the UR5e robotic platform, spanning 12 cohorts of phantom, ex vivo, and in vivo porcine experiments totaling 13 h 04 min of data, plus a temporal-alignment analysis and preliminary usability feedback from seven operators.

4. Downstream use in published surgical AI work. Datasets collected and curated with Sensorium were used to train the behavioral cloning policies evaluated in prior work, the vision-based force-estimation models, and preliminary world-model training in ongoing research.

Main Findings

• Deployment breadth: 1,516 recordings across 12 cohorts, all on the UR5e robotic platform, covering bowel grasping and retraction, endoscope guidance, liver lifting, thread pulling, tissue displacement, and open-surgery bowel retraction. Total recorded duration was 13 h 04 min.

• Sensor coverage: Recorded streams include stereo video (V), robot state (R), gripper (G), camera–robot transform (T), pedal (P), force (F), and IMU (I), in varying combinations per cohort. Cohort sizes ranged from 4 episodes (open-surgery bowel retraction, in vivo) to 426 episodes (bowel grasping/retraction policy rollouts, mixed).

• Temporal alignment accuracy: The median nearest-neighbor residual between reference-camera timestamps and robot-state samples was at most 8.60 ms across all cohorts, and the 95th percentile was at most 17.68 ms. Two cohorts were far tighter: tissue displacement (in vivo) at 0.49 ms median / 0.99 ms 95th percentile, and liver lifting (phantom) at 0.82 ms / 15.58 ms.

• Reference-stream detail: The combined stereo stream was used as the alignment reference when available; 256 rollout recordings instead used the left stream.

• Operator usability: Seven users completed an eight-item 5-point Likert questionnaire. Overall ease of use, fit into the procedural workflow, visibility of sensor readiness, and willingness to use the system in future sessions each scored a mean of 5.0/5. Starting and stopping recordings, identifying when recording was active, and confidence that data were saved each scored 4.86/5. Understanding and responding to warnings or sensor failures scored 4.33/5 (n = 6, one unanswered item).

• Error-message caveat: The two respondents who gave neutral ratings on error messages had experienced no recording errors, so their scores reflect the absence of an opportunity to assess error messages rather than poor messaging.

• Qualitative feedback: Respondents highlighted the simple interface, rapid access to recorded episodes, quick removal of unwanted episodes, and access from devices on the local network. Suggested improvements were keyboard shortcuts, voice control for start/stop, clearer feedback during saving, and more spacing between episode-deletion controls.

• No comparative benchmark: The paper reports no head-to-head performance comparison against SurgSync, MiDAS, or other acquisition systems, and no quantitative downstream model accuracy for the policies or force-estimation models trained on Sensorium data; those results are attributed to separate prior publications.

Methodology in Plain English

Sensorium is organized around episodes. During an experiment, the recording application runs alongside the robot and subscribes to ROS topics from the endoscope, robot controller, pedal board, force sensor, haptic devices, and so on. Across the authors' deployments, all workstations running the robot, camera, and other sensors had their clocks synchronized using the Precision Time Protocol (PTP), and the recorder always preserves the publication timestamp from each message header. A raw episode is a self-contained folder of timestamped video and state streams; full ROS bag recording can optionally be enabled to retain the original message streams. Before recording starts, the system checks that required streams and the output location are available, so failures surface while the setup can still be corrected.

Afterwards, a ROS-independent curation pipeline runs — either inside the recording app or standalone over an existing recording directory. Streams are first restricted to their shared time interval and aligned to a chosen reference stream by nearest-neighbor timestamp matching (for each reference timestamp, the closest sample from each other stream is selected); episodes that exceed a configurable alignment tolerance are rejected. Configurable transforms such as stereo rectification, resizing, and coordinate-frame changes are applied after acquisition, so multiple representations can be derived from one raw episode.

Annotation happens in the browser: users split an episode into temporal segments, with each phase linked to one or more paraphrased language instructions via a dataset-configurable mapping, or provide a single dataset-level task description for coarse tasks. A configurable schema then defines which observations, robot states, actions, and annotations go into the final dataset, which is exported either as per-episode CSV tables with optional losslessly encoded PNG frames or as LeRobot v3 datasets with Parquet-backed features and image or video observations. Actions can be taken from recorded streams or derived from observations, with optional magnitude-based filtering (an example configuration zeroes translation vectors below the 30th percentile of nonzero norms and roll changes below the 20th percentile). Each episode is validated independently, so failing recordings are excluded and reported without aborting the rest of the build.

Why This Matters

Impact on research. Multimodal surgical datasets are the bottleneck for behavioral cloning, surgical vision-language models, and generative surgical world models, yet collection protocols demand clinical input, ethical approval, and scarce in vivo time. A reusable, hardware-agnostic recording and curation layer makes such data cheaper to acquire, easier to re-represent at different sampling rates or annotation protocols, and more reproducible across labs — a prerequisite for the field's move toward shared datasets like those aggregated in Open-H-Embodiment.

Real-world applications:

• Recording surgical robot demonstrations for imitation-learning policies for tasks such as bowel grasping and retraction.
• Building language-annotated phase datasets for surgical vision-language model training and surgical phase recognition.
• Capturing synchronized force and video streams for vision-based force estimation in teleoperated surgery.
• Equipping new laparoscopic robotics testbeds with a ready-made recording application, as done for the open-source UR5e-based testbed cited in the paper.

Industry relevance. Surgical robotics groups, hospital research labs, and medical-device companies typically run heterogeneous hardware — dVRK, KUKA, Franka Emika Panda, UR5/UR5e — and cannot standardize on a single vendor's acquisition stack. A configuration-driven, ROS-native recorder that exports to a widely adopted format (LeRobot v3) lowers the cost of assembling training corpora and lets raw data be retained for reuse as annotation conventions and model requirements change.

Future Directions

• Cross-platform validation. The authors plan to evaluate Sensorium across different robotic platforms, experimental setups, and research sites; all 1,516 reported recordings used a single UR5e platform, so generality beyond it is untested.
• Broader and larger usability studies. The current evaluation covers seven participants from one laboratory, which the authors identify as limiting generalizability to other workflows and hardware configurations.
• Reduced dependence on ROS wrapping. Sensors without native ROS support must currently be wrapped in ROS nodes, a limitation the authors list explicitly.
• Resampling and reference-stream flexibility. Export is tied to the selected reference stream with no support for resampling to a user-defined frequency — a natural extension for pipelines that need fixed-rate control data.
• Community-driven extension. The authors intend to grow the library based on feedback from the broader open-source community, including the interface improvements suggested by their operators (shortcuts, clearer saving feedback, safer deletion controls).

Target Audience

Robotics and AI engineers who need to build surgical or manipulation datasets from ROS-based hardware; researchers in surgical robotics, imitation learning, and vision-language-action modeling who want a reusable acquisition pipeline rather than a bespoke one; lab operators and study coordinators responsible for running recording sessions; and groups standing up new robotic testbeds who want an off-the-shelf recording, annotation, and LeRobot-export stack. Readers looking for new model architectures or benchmark-beating learning results will not find them here — this is infrastructure work whose value lies in reproducibility and practical deployment.

Authors’ abstract

Embodied surgical AI requires reliable multimodal data, yet surgical data collection remains costly, difficult to repeat, and strongly dependent on the hardware setup. We present Sensorium, an open-source Python library for recording, processing, annotating, and curating multimodal surgical data from configurable robot and sensor streams using the Robot Operating System (ROS) and ROS2 middleware. The library combines operator-facing data acquisition, configurable processing, semantic annotation, and episodic dataset generation. We evaluate Sensorium across 1516 recordings of phantom, ex vivo, and in vivo porcine experiments involving multiple surgical tasks and sensing configurations, and report preliminary usability feedback.