SUREON: A Benchmark and Vision-Language-Model for Surgical Reasoning

AI summary

Overview

• Research area: Surgical computer vision and vision–language models (medical AI), specifically video question answering and reasoning over surgical footage.
• Technical level: Intermediate. The paper is readable without deep surgical background, but assumes familiarity with vision–language models, supervised fine-tuning, and reinforcement learning from feedback (GRPO).
• Scope: The paper introduces SUREON, a large-scale video QA dataset and expert-validated benchmark for surgical reasoning harvested from narrated surgical lecture videos, together with two adapted 8B-parameter models (SureonVLM and SureonVLM-R1) trained on it.

What This Paper Is About

Existing surgical AI systems are trained on fixed annotation ontologies, so they can label what is visible — tools, phases, actions, anatomy — but cannot explain why a maneuver is performed, what risk it poses, or what comes next. Annotating such reasoning at scale is extremely difficult, yet expert-narrated surgical lecture videos already contain explanations of intent, rationale, and anticipation. The paper's goal is to systematically extract that reasoning signal into a structured video QA dataset and benchmark, then train and evaluate vision–language models that can answer reasoning-oriented questions about surgery.

Key Contributions

1. SUREON dataset and benchmark. A large-scale video QA dataset built from public expert-narrated surgical lecture videos, spanning 134.7K clips and 170 procedure types and yielding 206.8k QA pairs, plus an expert-validated benchmark of 354 examples reviewed by two expert surgeons.
2. A 12-category surgical question taxonomy. The taxonomy covers perception (Entity Existence, Entity Attribute, Entity Localization, Instrument–Action Interaction, Procedure-Agnostic Action Description), reasoning and temporal understanding (Action Description, Local Action Reasoning, Decision Reasoning, Sequence Summarization, Temporal Ordering, Forecast), and safety (Safety Practice Identification), with structured Chain-of-Thought rationales for reasoning-intensive types.
3. A multi-agent data curation pipeline. Paired specialized generator and validator GPT-5 agents identify "Semantic Grounding Moments" in transcripts and convert them into validated question–answer samples, with generators restricted to transcript text to avoid hallucinated visual annotations.
4. Two adapted surgical VLMs. SureonVLM, built on Qwen3-VL via a progressive three-step supervised fine-tuning schedule using SUREON plus 18 public labeled datasets, and SureonVLM-R1, further optimized with Group Relative Policy Optimization (GRPO) to produce explicit reasoning traces.

Main Findings

• Multiple-choice performance on the SUREON benchmark: SureonVLM reaches 0.85 average accuracy and SureonVLM-R1 0.84, versus 0.66 for the Qwen3-VL (8B) base model, 0.68 for GPT-5.1, and 0.60 for Gemini 3.1 Pro. The paper reports the frontier commercial models trailing by roughly 20% and 29% respectively.
• Open-ended performance narrows the gap: With an LLM judge (Opus 4.6), GPT-5.1 scores 0.34 and Gemini 3.1 Pro 0.31, compared with 0.32 for SureonVLM and 0.24 for SureonVLM-R1. The paper attributes SureonVLM-R1's lower open-ended score to its correct-answer reward and multiple-choice-only Stage 2 training.
• Largest gains in clinically critical categories: Safety Action Identification reaches 0.92 (SureonVLM) and 0.93 (SureonVLM-R1), versus 0.62 for GPT-5.1 and 0.47 for Gemini 3.1 Pro. Decision Reasoning reaches 0.98 and 1.00. Entity Localization is the exception: SureonVLM at 0.53 marginally trails GPT-5.1 and Gemini 3.1 Pro, both at 0.55.
• Full test set results: On the full SUREON test set (20K examples), SureonVLM and SureonVLM-R1 both average 0.85, against 0.65 for GPT-5.1, 0.71 for Gemini 3.1 Pro, and 0.62 for Qwen3-VL. Commercial models were evaluated on a 20% random subsample because of API costs.
• Ablation findings: Progressive adaptation (SFT Steps 1–2) plus SUREON fine-tuning produces the largest gain, lifting accuracy from 0.66 to 0.83. Adding public datasets gives a marginal improvement to 0.84. Open-ended training raises generative scores substantially (Exact Match 0.09 to 0.15, LLM-as-Judge 0.25 to 0.32) with no drop in multiple-choice accuracy, making that configuration (SureonVLM) the best SFT setup. Explicit CoT supervision does not improve metrics but is described as essential for stable GRPO training, since models trained without CoT during SFT failed to generate thinking tokens afterward.
• No loss on standard surgical perception tasks: On phase and action recognition, critical view of safety (CVS) assessment, and tool detection, SureonVLM reaches a Phase Cholec80 F1 of 0.63, Phase HeiChole 0.41, Phase MultiBypass140 0.40, and CVS Endoscapes 0.32, above the comparison models on those tasks. The reported table shows exceptions: on Action HeiChole F1, SureonVLM scores 0.04 versus 0.18 for GPT-5.1, 0.21 for Gemini 3.1 Pro, and 0.17 for Qwen3-VL; on Tool Endoscapes mAP@.5:.95, SureonVLM scores 0.22 versus 0.61 for Gemini 3.1 Pro, with GPT-5.1 and Qwen3-VL both at 0.00.
• Qualitative reasoning behavior: Analysis of SureonVLM-R1's thinking tokens shows reasoning beyond pattern matching — for example distinguishing cold scissors from activated monopolar cautery through the absence of cautery sparks, recognizing tissue blanching as evidence of energy application, and inferring that a vessel branch was sacrificed because an enlarged interlobar lymph node required en bloc removal. The paper notes these traces have not been validated by surgeons.
• Answer leakage mitigation: Optical character recognition and a surgery-specific text detection pipeline are used to blur textual overlays before evaluation.

Methodology in Plain English

The researchers start from public surgical lecture videos that already carry expert narration. They define "Semantic Grounding Moments" as segments where the narration explicitly anchors something visual, a decision, or a rationale. A multi-agent pipeline processes each video and its transcript: for each of the 12 question types, a generator agent reads only the transcript (never the video pixels, to avoid inventing visual details) and proposes candidate question–answer pairs, each tied to a clip, optional multiple-choice options, and an optional Chain-of-Thought rationale. A matching validator agent then filters out samples that are not grounded in the narration, that use the wrong temporal window, or that describe a scene the narration is not actually commenting on. Human experts reviewed randomly selected test samples per category, discarding about 20% of reviewed samples until roughly 30 validated examples per type remained.

For the models, the team adapts Qwen3-VL through three progressive SFT stages on a mixed corpus (30% SUREON clips, 50% images from standard datasets, 20% videos from standard datasets, 3 epochs per stage): first only the projection layer, then the vision encoder and projection together, then the projection and language model with the vision encoder frozen. In the final stage, multiple-choice options are removed with probability 0.5 to expose the model to open-ended questions. A subsequent GRPO stage uses multiple-choice inputs and rewards correctness, format compliance with the thinking/answer template, tag validity, and task-specific reasoning for Temporal Ordering and Forecasting; reasoning traces are not supervised here, so the model explores its own reasoning paths.

Why This Matters

• Research impact: The work reframes the bottleneck in surgical AI as data availability rather than model capability, showing that an 8B model trained on lecture-derived supervision can outperform much larger general-domain systems on safety-critical surgical questions. It also provides a reusable taxonomy, benchmark, and curation pipeline for reasoning-oriented surgical datasets.
• Real-world applications:
  • Intra-operative decision support that flags safety-relevant practices and rationale rather than only naming instruments.
  • Surgical training and coaching systems that explain why a step is performed and what should come next.
  • Automated generation of understandable rationales for surgical video review and quality assurance.
  • Workflow anticipation tools that forecast the next procedural step for operating-room coordination.
• Industry relevance: The work comes from Intuitive Surgical Inc., situating it directly in the surgical robotics and operating-room technology sector, where interpretability and safety-critical reasoning are prerequisites for deployment. The paper also notes that proprietary-API and prompt-engineering approaches limit scalability, arguing for in-house adapted open models.

Future Directions

• Reducing dependence on LLM-based judgement in evaluation, given the paper's own limitation that fluent answers may be favored over clinically accurate ones.
• Validating the models' reasoning traces with surgeons, since the paper states these traces are currently unvalidated and may be sensitive to hallucinations.
• Addressing the pedagogical selectivity of lecture narration, which underrepresents routine operative steps and biases the dataset toward teaching moments.
• Extending the approach beyond the single narrated-lecture source used here, and improving open-ended response quality for reasoning-optimized models, which currently trail on Exact Match and LLM-judged metrics.

Target Audience

Researchers and engineers working on medical and surgical computer vision, vision–language models, or multimodal reasoning; clinicians and surgical educators interested in AI-assisted training and decision support; and industry teams building surgical robotics or operating-room intelligence systems who need to understand what current surgical VLMs can and cannot reason about.

Authors’ abstract

Surgeons don't just see -- they interpret. When an expert observes a surgical scene, they understand not only what instrument is being used, but why it was chosen, what risk it poses, and what comes next. Current surgical AI cannot answer such questions, largely because training data that explicitly encodes surgical reasoning is immensely difficult to annotate at scale. Yet surgical video lectures already contain exactly this -- explanations of intent, rationale, and anticipation, narrated by experts for the purpose of teaching. Though inherently noisy and unstructured, these narrations encode the reasoning that surgical AI currently lacks. We introduce SUREON, a large-scale video QA dataset that systematically harvests this training signal from surgical academic videos. SUREON defines 12 question categories covering safety assessment, decision rationale, and forecasting, and uses a multi-agent pipeline to extract and structure supervision at scale. Across 134.7K clips and 170 procedure types, SUREON yields 206.8k QA pairs and an expert-validated benchmark of 354 examples. To evaluate the extent to which this supervision translates to surgical reasoning ability, we introduce two models: SureonVLM, a vision-language model adapted through supervised fine-tuning, and SureonVLM-R1, a reasoning model trained with Group Relative Policy Optimization. Both models can answer complex questions about surgery and substantially outperform larger general-domain models, exceeding 84% accuracy on the SUREON benchmark while outperforming general-domain models on standard surgical perception tasks. Qualitative analysis of SureonVLM-R1 reveals explicit reasoning behavior, such as inferring operative intent from visual context.