Where It Moves, It Matters: Referring Surgical Instrument Segmentation via Motion

AI summary

Overview

• Research area: Surgical computer vision, specifically referring video object segmentation and vision-language grounding applied to endoscopic and robotic surgical video.
• Technical level: Intermediate. The paper assumes familiarity with transformer-based segmentation architectures, vision-language encoders, and video object segmentation benchmarks, but its central idea (motion as the grounding signal) is intuitive.
• Scope: This one sentence covers the paper: it introduces a motion-guided framework (SurgRef) and a new annotated dataset (Ref-IMotion) for segmenting surgical instruments in video based on free-form natural language that describes how tools move, rather than what they look like.

What This Paper Is About

Referring surgical instrument segmentation means finding and outlining a specific tool in a surgical video based on a spoken or written description, such as "the grasper pulling the gallbladder to the left." Existing methods anchor these descriptions in static appearance cues like instrument names or left/right position, which breaks down when tools look alike, are partially hidden by tissue or blood, or are described with unfamiliar terminology across hospitals. This paper argues that motion, meaning the trajectories, retraction patterns, and tool-tissue interactions that define surgical procedures, is a more consistent and interpretable signal for grounding language than appearance, and builds both a model and a dataset around that idea.

Key Contributions

1. Constructed the Ref-IMotion dataset. A multi-institutional surgical video dataset built by annotating EndoVis-17, EndoVis-18, CholecSeg8k, and GraSP with dense spatiotemporal masks and manually curated motion-centric referring expressions. It contains 319 clips, 21,350 annotated frames, and 718 expressions (358 of them motion-based), making it the largest referring video segmentation dataset in surgical computer vision to date.

2. Proposed SurgRef, a motion-guided referring video segmentation framework. The model grounds both static and motion-centric language in surgical video by leveraging a language-guided transformer decoder based on Mask2Former, a Swin Transformer visual backbone, and a frozen RoBERTa text encoder, plus a novel key-frame attention module that selects expression-relevant frames.

3. Introduced language-guided key-frame selection. Instead of processing every frame, a lightweight MLP scores each frame's relevance to the referring expression using decoder-level object queries, and only the top-scoring frames are passed to the segmentation decoder. This suppresses temporal redundancy and improves both efficiency and accuracy.

4. Demonstrated state-of-the-art accuracy and cross-domain generalization. SurgRef outperforms prior surgical and general-purpose methods (VIS-Net, VISA, MPG-SAM 2) and transfers zero-shot from robot-assisted prostatectomy data to laparoscopic cholecystectomy data without dataset-specific tuning.

Main Findings

• Motion supervision improves all expression styles, not just motion ones. On EndoVis-IM17, training with motion expressions raised J&F from 79.03 to 89.42 overall, and even appearance-based test expressions improved from 74.25 to 85.17. Spatial-based expressions improved from 76.64 to 87.33. Similar gains held on EndoVis-IM18 and GraSP-IM.

• State-of-the-art results across benchmarks. With key-frame selection, SurgRef reached 89.42 J&F on EndoVis-IM17 (vs. 89.41 for MPG-SAM 2, but with better J and Dice), 84.48 J&F on EndoVis-IM18 (vs. 83.03 for MPG-SAM 2), and 84.97 J&F on GraSP-IM.

• Strong zero-shot cross-procedure generalization. A model trained on GraSP-IM (robot-assisted radical prostatectomy) transferred to CholecSeg8k-IM (laparoscopic cholecystectomy) with 70.62 J&F and 68.13 Dice, despite differences in anatomy, tool sets, imaging modality, and institution.

• Key-frame selection beats uniform sampling and cosine similarity. At T'=8, the proposed selection achieved 89.42 J&F versus 82.16 for uniform sampling and 85.71 for cosine similarity. Performance saturated beyond T'=8, indicating the module identifies the most informative frames rather than simply benefiting from more data.

• Spatial cues matter more than name cues in cross-style generalization. A model trained without instrument names still generalized well to original expressions (79.65 J&F), suggesting motion and spatial context can compensate for missing lexical identifiers. Removing location information caused severe degradation (down to 38.64 J&F when both name and location were absent).

• Motion is inferred reliably through structured annotation. For GraSP-IM, the authors estimated motion direction by tracking bounding-box centers across frames, divided frames into a 3×3 grid, and used an LLM to convert structured instrument-action-object-spatial inputs into natural language descriptions.

Methodology in Plain English

The authors approached the problem in three stages. First, they assembled training data: they took four existing surgical video datasets, selected clips with high motion complexity (tools appearing, disappearing, or interacting with tissue), and hand-annotated them with natural language expressions that describe how tools move over time, such as "the tool enters from the top and pulls the gallbladder to the left." For one dataset they used an LLM to turn structured motion data into natural language.

Second, they adapted a standard image segmentation architecture to video and language. A Swin Transformer extracts visual features from each frame, while a frozen RoBERTa model encodes the text expression into a semantic embedding. This embedding initializes a set of language-driven queries in a Mask2Former-style transformer decoder, which then produces a segmentation mask for each candidate object, keeping only those with high classification confidence.

Third, they added two motion-aware components. A key-frame attention module scores each frame in the clip based on how well it aligns with the referring expression, and only the top-scoring frames (T'=8 in experiments) are passed forward for final mask prediction. An inter-frame attention layer then lets queries exchange information across those selected frames, capturing object motion. The model is trained with a composite loss combining classification, mask prediction (binary cross-entropy and Dice), and temporal consistency objectives. It runs for 100,000 iterations on two NVIDIA A100 GPUs, taking about 23 hours.

Why This Matters

This work shifts the grounding signal in surgical vision-language systems from static appearance to dynamic motion, which is more robust to occlusion, similar-looking tools, and the terminological inconsistency that plagues real clinical settings across institutions.

Impact on research: It establishes the first large-scale benchmark for motion-centric referring segmentation in surgery, providing a shared evaluation target for future vision-language work. It also shows that motion supervision transfers to static expression types, suggesting that motion may be a stronger supervisory signal than previously assumed for surgical grounding.

Real-world applications:

• Surgical training and AR overlays: Trainees could query instruments or actions verbally ("show me where the grasper retracts"), and the system highlights the relevant tool in real time.
• Intelligent intraoperative assistants: Systems could answer verbal queries from surgeons, highlight key targets, and flag when a referred instrument is being mishandled or occluded.
• Natural human-robot collaboration: Robotic surgery systems could accept procedural commands like "cut where the grasper is holding," enabling fine-grained semantic control.
• Cross-institutional deployment: Because the model generalizes zero-shot across procedures and institutions, it could be deployed at a new hospital without retraining on that hospital's terminology or tool set.

Industry relevance: Surgical robotics companies (e.g., Intuitive), surgical AI platform developers, and hospital IT vendors building OR integration systems have a direct stake, since language-driven interfaces are a natural path toward semi-autonomous surgical assistance.

Future Directions

• Scaling and diversifying the dataset. Ref-IMotion currently covers cholecystectomy, prostatectomy, and porcine procedures. Extending it to more procedures, institutions, and imaging modalities would test whether motion-centric grounding holds broadly.

• Real-time performance. The current model uses two A100 GPUs and takes 23 hours to train, and inference with key-frame selection is not benchmarked for latency. Meeting intraoperative constraints is a practical open problem.

• Improving robustness to missing spatial cues. The ablation shows severe drops (down to 38.64 J&F) when both instrument names and location information are absent from expressions. Developing models that rely more purely on motion dynamics remains an open challenge.

• Motion representation beyond query scoring. The framework currently infers motion implicitly through temporal attention. Explicit motion modeling (e.g., optical flow, trajectory prediction, or scene-graph reasoning) could further improve grounding under heavy occlusion.

• Clinical validation and safety. Moving from benchmark metrics to prospective clinical studies, including how surgeons actually phrase commands in the OR and how segmentation errors affect downstream decisions, is a necessary next step.

Target Audience

This paper benefits surgical AI researchers, computer vision researchers working on referring video segmentation and vision-language grounding, and medical robotics engineers developing language-driven OR assistants. It is also relevant to surgical data scientists building benchmarks and to clinicians interested in how verbal commands could integrate with intraoperative computer vision systems. Readers without prior exposure to transformer-based segmentation or vision-language models will find the central intuition accessible, but will need background reading to follow the architectural details.

Authors’ abstract

Enabling intuitive, language-driven interaction with surgical scenes is a critical step toward intelligent operating rooms and autonomous surgical robotic assistance. However, the task of referring segmentation, localizing surgical instruments based on natural language descriptions, remains underexplored in surgical videos, with existing approaches struggling to generalize due to reliance on static visual cues and predefined instrument names. In this work, we introduce SurgRef, a novel motion-guided framework that grounds free-form language expressions in instrument motion, capturing how tools move and interact across time, rather than what they look like. This allows models to understand and segment instruments even under occlusion, ambiguity, or unfamiliar terminology. To train and evaluate SurgRef, we present Ref-IMotion, a diverse, multi-institutional video dataset with dense spatiotemporal masks and rich motion-centric expressions. SurgRef achieves state-of-the-art accuracy and generalization across surgical procedures, setting a new benchmark for robust, language-driven surgical video segmentation.