MedGRPO: Multi-Task Reinforcement Learning for Heterogeneous Medical Video Understanding

AI summary

Overview

Research area: Medical video understanding with large vision-language models (VLMs), combining a new benchmark (MedVidBench) with a reinforcement-learning training framework (MedGRPO).

Technical level: Advanced. The paper assumes familiarity with vision-language models, supervised fine-tuning, and policy-gradient RL methods such as GRPO.

Scope in one sentence: The paper builds a 531,850-sample instruction-following benchmark over 8 medical video sources and 8 tasks, then introduces a reward-normalization scheme and an LLM-judge reward that let a single multi-task RL run improve on supervised fine-tuning without the training collapse that standard RL causes.

What This Paper Is About

Medical video understanding demands fine-grained spatial precision (for example, telling a "grasper" apart from a generic "tool"), temporal reasoning over multi-phase procedures, and correct clinical terminology. Existing medical video datasets hold rich annotations such as bounding boxes, action triplets, and phase labels, but these are not in the conversational question-answer format that VideoLLMs require. The authors convert those annotations into instruction-following data at scale, and then show that applying standard RL on the resulting heterogeneous mixture fails catastrophically because datasets differ wildly in difficulty, so they design a reward scheme that equalizes learning across easy and hard sources.

Key Contributions

1. MedVidBench: a benchmark of 531,850 (also written as 532K) video-instruction pairs across 8 medical sources and 8 tasks spanning video-level, segment-level, and frame-level understanding, built through a multi-perspective quality-assurance pipeline that uses source-specific prompting strategies and dual-model validation.
2. Cross-dataset reward normalization: logistic functions centered on dataset-task-specific percentile statistics that map each dataset's median performance to a common reward value (0.5), giving fair optimization regardless of task difficulty or metric scale.
3. A medical LLM judge: a GPT-4.1-based comparative-similarity scorer that rates generated captions on five clinical dimensions (medical terminology precision, instrument and anatomy identification, specificity versus vagueness, clinical procedure context, and action and state accuracy) using 1-5 rubrics, combined with semantic similarity in a hybrid reward.
4. Comprehensive evaluation and ablations: the full MedGRPO pipeline on Qwen2.5-VL-7B and Qwen3-VL-4B improves over supervised fine-tuning baselines across most tasks, outperforms closed-source models such as GPT-4.1 and Gemini-2.5-Flash, and ablations show that removing reward normalization causes training collapse with unstable entropy.

Main Findings

• SFT on MedVidBench beats closed-source models. Qwen2.5-VL-7B after supervised fine-tuning reaches 0.894 CVS accuracy and 0.177 STG mIoU, versus 0.018 CVS and 0.014 STG for GPT-4.1 and 0.101 CVS and 0.047 STG for Gemini-2.5-Flash. The medically fine-tuned Qwen2.5VL-7B_Surg-CholecT50 scores 0.000 on both CVS and STG, and the RL-trained general-domain VideoChat-R1.5-7B also scores 0.000 on both, showing single-dataset surgical specialization and general video RL do not transfer.

• MedGRPO improves over SFT on most tasks. On Qwen2.5-VL-7B, CVS rises by 0.002 to 0.896, STG by 0.025 to 0.202, SA by 0.036 to 0.254, TAG@0.3 by 0.074 to 0.216, TAG@0.5 by 0.065 to 0.156, DVC LLM score by 0.132 to 3.797, DVC F1 by 0.049 to 0.214, VS LLM score by 0.588 to 4.184, and RC LLM score by 0.685 to 3.442.

• One task regresses. NAP accuracy drops from 0.442 (SFT) to 0.405 under MedGRPO, because NAP was not among the tasks used for reward optimization (rewards were computed for TAG, STG, VS, and RC).

• Reward normalization is what prevents collapse. Removing it (ablation Row B) drops CVS from 0.894 to 0.020, STG from 0.177 to 0.010, and TAG@0.3 from 0.142 to 0.004, with training entropy becoming highly unstable and spiking; the normalized version keeps entropy stable.

• The difficulty gap motivating the design is large in absolute terms. The paper cites a median spatiotemporal grounding mIoU around 0.5 for CoPESD versus around 0.12 for EgoSurgery.

• The medical LLM judge adds value beyond semantic similarity. With the judge (Row D), VS reaches 3.824 and RC 3.235; without it (Row E), VS reaches only 3.733 and RC 2.984. The full four-task configuration (Row A) achieves the best VS (4.184) and RC (3.442), described as improvements of 16.4% and 24.8% over the SFT baseline.

• Multi-task training produces synergy. Adding captioning tasks (VS + RC) to grounding tasks (TAG + STG) improves STG from 0.193 to 0.202 (+4.7%), TAG@0.3 from 0.202 to 0.216 (+6.9%), and TAG@0.5 from 0.142 to 0.156 (+9.9%).

• The framework transfers across model scales. Applying the same pipeline to Qwen3-VL-4B takes SFT results of 0.895 CVS, 0.465 TAG@0.3, 0.403 TAG@0.5 up to 0.898, 0.504, and 0.441 respectively under MedGRPO, with gains on STG (+0.043), TAG@0.3 (+0.039), and DVC F1 (+0.045).

• 2026 off-the-shelf models still trail fine-tuned ones on grounding. GPT-5.4 reaches 0.164 CVS (versus 0.018 for GPT-4.1) and 3.976 VS (versus 2.490), but grounding remains low (STG: 0.004 for GPT-5.4, 0.059 for Gemini-3.1-Flash-Lite, 0.051 for Qwen3.5-4B; TAG@0.3: 0.086, 0.072, 0.074 respectively). SFT on Qwen3.5-4B gives the highest NAP accuracy across all models at 0.576, with TAG@0.3 of 0.482 and TAG@0.5 of 0.429. The paper states MedGRPO results for Qwen3.5-4B will be added in a future update.

• A scaling trend is reported. Performance on DVC F1 and VS LLM judge score improves consistently as training samples from the Large-Scale version increase from 0 to 461K.

• Human expert preference. Medical professionals preferred captions generated using the annotation-enriched prompts over baseline generation from raw frames alone, rated on clinical accuracy and terminology precision; the paper does not report a numeric preference rate in the main text.

Methodology in Plain English

The authors start from 8 existing medical video datasets covering laparoscopic surgery (CholecT50, CholecTrack20, Cholec80-CVS, CoPESD), open surgery (AVOS, EgoSurgery), robotic surgery (JIGSAWS), and nursing procedures (NurViD). Instead of writing new annotations, they translate existing ones into question-answer pairs. For densely annotated surgical datasets they overlay bounding boxes and labels on frames and add procedure-specific context to the prompt. For web-sourced videos they extract transcripts with Whisper-X and add video metadata such as titles. Two models, GPT-4.1 and Gemini-2.5-Flash, each generate captions independently, and a quality filter keeps only pairs whose outputs have sentence-transformer similarity at or above 0.3, sampled at 50-180 frames with adaptive FPS of 0.1 to 3.0, with video-level train/test splits at a 0.15 test ratio.

Training happens in two stages. First, supervised fine-tuning adapts Qwen2.5-VL-7B to the medical data. Second, GRPO reinforcement learning refines it using groups of 8 sampled responses and a clipped surrogate objective with asymmetric clipping (epsilon_low of 0.2 and epsilon_high given as 0.3 in the method section and 0.28 in the implementation details), removing the KL penalty following DAPO.

The central fix is the reward. For each dataset-task pair, a logistic function with slope k = 3.0 maps the raw metric onto (0,1), centered on that pair's median and scaled by its interquartile range, computed from SFT baseline predictions on the training set. Any dataset performing at its own median therefore earns exactly 0.5, so easy and hard sources contribute comparable gradient signal, gradients never die, and the bounded output fits GRPO's group normalization. For grounding tasks, the reward multiplies a logistic-normalized mIoU by a format term that applies a 0.6 penalty for invalid output. For captioning, the final reward is an equal-weighted average of normalized semantic similarity and the LLM judge's mean score across five clinical dimensions.

Why This Matters

Impact on research. The paper isolates a failure mode — reward-scale imbalance across heterogeneous datasets — that is easy to hit when combining medical video corpora for RL, and shows that a median-centered logistic normalization plus a domain-specific judge fixes it without architectural changes. It also provides a benchmark and a reproducible curation recipe for converting legacy medical annotations into instruction-following data.

Real-world applications:

• Surgical training and skill assessment, where the SA task and CVS task target procedural safety and competency.
• Intraoperative decision support, using next-action prediction and temporal action grounding to anticipate the next step in a procedure.
• Automated surgical documentation and reporting, via dense video captioning and video summarization of long procedures.
• Nursing and clinical workflow monitoring, drawing on NurViD-sourced procedure understanding.

Industry relevance. Clinical documentation, surgical robotics, and hospital IT vendors all depend on being able to fine-tune compact open models (4B-7B parameter range) rather than paying for closed APIs; this paper shows such models beat GPT-4.1, Gemini-2.5-Flash, and the 2026 models GPT-5.4 and Gemini-3.1-Flash-Lite on grounding and most captioning metrics after domain adaptation. The reported hardware footprint of 8× H100 GPUs and a 5,000-update GRPO run is a practical budget signal for teams considering similar pipelines.

Future Directions

• Completing the MedGRPO results for Qwen3.5-4B, which the paper says will be added in a future update, to confirm whether the reward-normalization gains hold on the newest base model.
• Applying GRPO rewards to the remaining tasks, particularly NAP, which regressed (0.442 to 0.405) precisely because it was excluded from the reward set.
• Extending the framework beyond the four rewarded tasks (TAG, STG, VS, RC) to cover DVC, CVS, SA, and NAP, and testing whether the same median-centered normalization works for modalities beyond video.
• Addressing the still-modest grounding numbers on the hardest datasets — even the best reported STG mIoU is 0.202 — and exploring whether the reward function, the judge, or the underlying visual representation is the limiting factor.

Target Audience

Researchers and engineers working on medical or clinical vision-language models, video-language RL and preference optimization, and multi-dataset or multi-task training. It is also relevant to surgical data science groups that hold annotated procedure datasets and want to convert them into instruction-tuning data. Readers need a working background in VLMs and policy-gradient RL to follow the method sections, though the benchmark description and results tables are accessible to a broader applied machine-learning audience.

Authors’ abstract

Large vision-language models struggle with medical video understanding, where spatial precision, temporal reasoning, and clinical semantics are critical. To address this, we first introduce \textbf{MedVidBench}, a large-scale benchmark of 531,850 video-instruction pairs across 8 medical sources spanning video, segment, and frame-level tasks, curated through a rigorous quality assurance pipeline with expert-guided prompting and dual-model validation. While supervised fine-tuning on MedVidBench yields noticeable gains, standard Reinforcement Learning (RL) fails due to imbalanced reward scales across datasets, which destabilizes optimization and leads to training collapse. To overcome this, we introduce \textbf{MedGRPO}, a novel RL framework for balanced multi-dataset training with two key innovations: (1) \emph{cross-dataset reward normalization} that maps each dataset's median performance to a common reward value, ensuring fair optimization regardless of difficulty, and (2) a \emph{medical LLM judge} that evaluates caption quality on five clinical dimensions through comparative similarity scoring. Supervised fine-tuning Qwen2.5-VL-7B on MedVidBench outperforms GPT-4.1 and Gemini-2.5-Flash across all tasks, while MedGRPO further improves the SFT baseline on grounding and captioning. Our work establishes a foundational benchmark and training methodology for advancing medical video understanding with VLMs. Our project website is available at: https://uii-america.github.io/MedGRPO/.