Med-Scout: Curing MLLMs' Geometric Blindness in Medical Perception via Geometry-Aware RL Post-Training

AI summary

Overview

Research area: Computer Vision / Medical Multimodal Large Language Models (MLLMs), specifically reinforcement-learning post-training for geometric visual grounding in radiology.

Technical level: Advanced. The paper assumes familiarity with MLLM architectures, reinforcement learning, Group Relative Policy Optimization (GRPO), and medical imaging modalities (CT, MRI, X-ray).

Scope: This paper diagnoses a failure mode it calls "geometric blindness" in medical MLLMs, then proposes a reinforcement-learning post-training framework (Med-Scout) and a benchmark (Med-Scout-Bench) to cure and quantify that blindness.

What This Paper Is About

Multimodal Large Language Models can produce fluent, clinically plausible medical descriptions while failing to respect the actual geometric facts of the image — relative anatomical scale, correct topology, and pixel-level structural coherence. The authors call this "geometric blindness" and trace it to training paradigms that optimize linguistic fluency over geometric fidelity. Med-Scout is a geometry-aware RL post-training framework that derives verifiable supervision from unlabeled medical images instead of costly expert annotations, aiming to ground model outputs in objective geometric constraints.

Key Contributions

1. Unveiling geometric blindness. Through a pilot study on Qwen3-VL-8B-Instruct and Lingshu-7B, the authors identify a critical gap where MLLMs fail to ground outputs in geometric constraints despite semantic fluency, and show that Chain-of-Thought prompting does not fix it.

2. Geometry-aware RL post-training (Med-Scout). A reinforcement-learning framework that cures geometric blindness using three geometric proxy tasks and a Dense Geometric Reward (DGR) mechanism integrated within GRPO.

3. Med-Scout-Bench. A new benchmark built from the intrinsic geometric properties of medical images, constructed from a synthesized pool of 108,000 VQA cases with a balanced 10,800-case (10%) subset, spanning CT, MRI, and X-ray.

4. Substantial performance improvements. Med-Scout improves geometric perception by over 40% on the new benchmark and achieves state-of-the-art generalization on radiological and comprehensive medical VQA benchmarks.

Main Findings

• Scale blindness. Out of 200 positive samples where models correctly identified lesions in cropped local views, the models failed to detect the same lesions in over 20% of the corresponding global views.

• Topology blindness. When 200 images with correctly identified pathology locations were rotated by 180 degrees, models failed to adapt their spatial descriptions in 80% of cases — indicating reliance on rigid priors rather than actual topological positions.

• Anomaly blindness. Under a "cut-paste" protocol applied to 200 samples, models failed to identify the artificially introduced structural artifacts in over 90% of cases, instead producing standard reports that overlooked the perturbations.

• Chain-of-Thought does not cure it. CoT prompting produced only marginal performance fluctuations across all three pilot experiments (e.g., Qwen3-VL-8B-Instruct went from 69.50% to 67.00% on Experiment 1, 18.00% to 20.50% on Experiment 2, and 9.00% to 12.00% on Experiment 3; Lingshu-7B went from 80.50% to 77.50%, 20.50% to 22.00%, and 8.50% to 7.00%).

• Large benchmark gains. On Med-Scout-Bench, Qwen3-VL-8B-Instruct improved average accuracy from 39.7% to 83.6%, and Lingshu-7B rose from 31.9% to 71.9%. Open-source models outperformed the proprietary models GPT-5 and Gemini-3-Flash, and general-domain models improved more than medical specialists.

• Gains on radiological VQA. Qwen3-VL-4B gained 4.2 points on RadImageNet-VQA. Lingshu-7B with Med-Scout surpassed Gemini-3-Flash on VQA-RAD (71.0% vs. 70.2%).

• Gains on report generation. Med-Scout boosted Qwen3-VL-4B's CIDEr score by 4.3 points on MIMIC-CXR, and Lingshu-7B with Med-Scout achieved a state-of-the-art CIDEr (105.2 on MIMIC-CXR and 183.3 on IU-Xray, compared with its baseline 104.9 and 180.8).

• Generalization to broader medical VQA. Consistent improvements appeared across PMC-VQA, OmniMedVQA, and MedXpertQA. HuatuoGPT-Vision-7B gained 2.9 points on PMC-VQA, and Lingshu-7B further elevated its already state-of-the-art results.

• Dense rewards matter more than domain-specific tasks alone. Even when restricted to standard sparse rewards (disabling the Dense Geometric Reward mechanism), Med-Scout still outperformed general-domain proxy tasks Jigsaw-R1 and ViCrit: Qwen3-VL-4B-Instruct baseline was 58.3 (Rad-VQA) and 38.4 (generalization), rising to 60.8 and 40.2 with Med-Scout, while Jigsaw-R1 and ViCrit slightly degraded or matched baseline.

• Direct vs. Reasoning Mode. The performance gap between Direct Mode and Reasoning Mode was negligible, which the authors suggest may partly stem from the reasoning structure reward prioritizing structural constraints over logical validity.

• Data scaling. Performance on Med-Scout-Bench improved consistently as training data increased from 20% to 100%, without saturation. A strong positive correlation emerged between Med-Scout-Bench scores and average accuracy on six external benchmarks.

• Energy landscape separation. Using 800 factual-counterfactual report pairs from MIMIC-CXR, baseline models showed overlapping energy distributions for factual and perturbed descriptions, while Med-Scout established a distinct energy barrier separating spatial hallucinations from factual descriptions.

• Attention concentration. Attention-map visualization showed the baseline Qwen3-VL-4B-Instruct with scattered attention drifting to background noise, while Med-Scout concentrated on critical regions such as lung apices, ankle joints, and dense mass in liver.

Methodology in Plain English

The authors avoid expensive expert annotation by turning unlabeled medical images into three automatically verifiable puzzle-like tasks, then reward the model for solving them correctly.

1. Hierarchical Scale Localization. Three local patches are cropped from an image at two scale levels — Level 1 covering 20% of the image area and Level 2 covering 6.25% — with crop centers restricted to the central [0.2, 0.8] region to avoid non-informative background. The model must identify each patch's scale level and predict its normalized bounding box in the original image.

2. Topological Jigsaw Reconstruction. The image is divided into a 2×2 grid and shuffled with a random permutation. The model must recover the original index order, reasoning about global anatomical layout.

3. Anomaly Consistency Detection. A 4×4 grid is used, and one patch in the central region is replaced with a reference patch from a different image. The reference is chosen by modality-specific protocols: adjacent slices for CT/MRI, or the top-1 most similar image retrieved via BiomedCLIP for X-ray. The model must output the grid index of the inconsistent patch.

Answers are posed as open-set VQA, either in Direct Mode or Reasoning Mode (with a Chain-of-Thought path). Training uses GRPO with a total reward of R = R_acc + R_fmt + I_CoT · R_reason, where accuracy reward is capped at 1.0 and the format and reasoning rewards at 0.5 each, giving a maximum total reward of 2.0. Instead of pass/fail, the Dense Geometric Reward gives continuous credit: average classification correctness plus average IoU for localization, element-wise alignment for the jigsaw, and an exponential distance-decay reward based on Euclidean distance between predicted and true grid coordinates (with temperature hyperparameter tau) for anomaly detection.

Training runs for 7,200 steps on a node with 6 × NVIDIA RTX PRO 6000 GPUs, using AdamW with a peak learning rate of 1×10⁻⁶, cosine decay, warm-up ratio 0.01, global batch size 192, group size G = 8, and KL divergence coefficient beta = 0.04. The 97,200-sample split excluding Med-Scout-Bench was used for training. Evaluation uses the LLM-as-a-Judge paradigm with Gemini-3-Flash for open-ended responses.

Why This Matters

Impact on research. The paper reframes a class of medical MLLM errors — plausible but geometrically unfaithful outputs — as a measurable, curable deficit rather than a prompting artifact. It shows that CoT alone does not address it, and it releases a benchmark and an annotation-free training signal that other groups can build on. It also provides evidence that general-domain vision-language foundations absorb geometric training better than some medical specialists.

Real-world applications (logical extensions of the evaluated tasks):

• Radiological report generation, where geometric consistency with the scan is clinically critical — the paper reports SOTA CIDEr for Lingshu-7B with Med-Scout on MIMIC-CXR and IU-Xray.
• Medical visual question answering in clinical decision support, evaluated across RadImageNet-VQA, VQA-RAD, SLAKE, PMC-VQA, OmniMedVQA, and MedXpertQA.
• Multi-scale lesion review workflows, mirroring the "zoom-in" diagnostic process in the Hierarchical Scale Localization task.
• Anomaly screening, where the Anomaly Consistency Detection task trains sensitivity to pixel-level structural discontinuities that violate anatomical coherence.

Industry relevance. The framework extracts supervision from unlabeled data rather than expert annotations, which the authors frame as a cost-efficient and scalable pathway to clinical AI. Improvements transfer to both proprietary-model comparisons and open-source backbones across four backbones (Qwen3-VL-4B/8B-Instruct, Lingshu-7B, HuatuoGPT-Vision-7B).

Future Directions

1. Reasoning Mode needs better reward design. The negligible gap between Direct Mode and Reasoning Mode may stem from the reasoning structure reward prioritizing structural constraints over logical validity, implying that validating the logical content of reasoning chains is an open problem.

2. Pushing the data scaling curve. Performance improved from 20% to 100% of training data without saturation, suggesting models have not reached their capacity limits — further scaling and possibly larger backbones remain untested.

3. Extending beyond the three proxy tasks and three modalities. The benchmark and training data cover CT, MRI, and X-ray; whether other modalities and clinical constraints (e.g., temporal consistency) can be formulated as verifiable proxy tasks is not yet reported.

4. Closing the remaining gap to saturation. Med-Scout-Bench scores topped out at 83.6% for Qwen3-VL-8B-Instruct with Med-Scout, leaving headroom; the authors do not report what residual failure modes remain.

Target Audience

Researchers and practitioners working on multimodal large language models, medical image analysis, reinforcement learning post-training, and clinical AI reliability. It is most useful to those already comfortable with GRPO-style RL, vision-language model training pipelines, and radiological benchmarks, and to teams that need a concrete, annotation-free method and benchmark for diagnosing spatial grounding failures in medical models.

Authors’ abstract

Despite recent Multimodal Large Language Models (MLLMs)' linguistic prowess in medical diagnosis, we find even state-of-the-art MLLMs suffer from a critical perceptual deficit: geometric blindness. This failure to ground outputs in objective geometric constraints leads to plausible yet factually incorrect hallucinations, rooted in training paradigms that prioritize linguistic fluency over geometric fidelity. This paper introduces Med-Scout, a novel framework that "cures" this blindness via Reinforcement Learning (RL) that leverages the intrinsic geometric logic latent within unlabeled medical images. Instead of relying on costly expert annotations, Med-Scout derives verifiable supervision signals through three strategic proxy tasks inspired by the systematic reading and reasoning patterns of clinicians: Hierarchical Scale Localization, Topological Jigsaw Reconstruction, and Anomaly Consistency Detection. To rigorously quantify this deficit, we present Med-Scout-Bench, a new benchmark specifically designed to evaluate geometric perception. Extensive evaluations show that Med-Scout significantly mitigates geometric blindness, outperforming leading proprietary and open-source MLLMs by over 40% on our benchmark. Furthermore, this enhanced geometric perception generalizes to broader medical understanding, achieving superior results on radiological and comprehensive medical VQA tasks.