Beyond Fluency: A Clinical Benchmark and Anomaly-Enhanced Baseline for Spine MRI Report Generation

AI summary

Overview

Research area: Automated radiology report generation for lumbar spine MRI, bridging medical image analysis, vision-language models (VLMs), and clinical evaluation methodology (computer vision / medical AI).

Technical level: Intermediate to Advanced. Readers should be comfortable with vision-language models, segmentation architectures (U-Net++), parameter-efficient fine-tuning (QLoRA), and text-generation metrics (BLEU, ROUGE-L, METEOR, BERTScore).

Scope: The paper benchmarks five state-of-the-art VLMs on lumbar spine MRI report generation and structured disc-level diagnosis, shows that standard language metrics fail to penalize clinically serious errors, and proposes an architecture-agnostic framework that feeds disc-level anomaly heatmaps into VLMs as auxiliary visual input.

What This Paper Is About

Low back pain affects approximately 619 million individuals worldwide, and lumbar spine MRI is the gold standard for assessing it, but producing radiology reports is time-consuming and subject to inter-rater variability. Automated report generation is an obvious target, yet existing benchmarks focus mainly on chest X-ray and CT, leaving musculoskeletal MRI—especially lumbar spine MRI—underexplored. The core problem this paper attacks is that a model can produce a fluent, well-structured report that scores highly on BLEU or BERTScore while still misassigning a finding to the wrong vertebral level, which in clinical practice is a severe failure. The authors build a clinical benchmark to expose this gap and then propose an anomaly-detection-guided grounding approach to partially close it.

Key Contributions

1. A multi-protocol VLM benchmark on lumbar spine MRI with limited data. Five multimodal architectures (BiomedGPT, ChatGPT-5.0, MAIRA-2, MedGemma, VILA-M3) are evaluated both on free-text report generation against clinical reference reports and on structured per-finding, per-IVD-level diagnostic classification against radiological gradings. The authors also test whether supervised fine-tuning on a limited spine MRI dataset improves diagnostic accuracy.

2. A robustness analysis of models and metrics under clinically representative variability. The study varies MRI contrast (T1w, T2w, T1w+T2w), central slice positioning (center ± 1), image resolution (native versus resampled to 0.5 mm in-plane voxel spacing), and prompt verbosity. It adds a controlled perturbation analysis injecting negations, anatomical swaps, terminology variations, and structural changes into reference reports to test whether standard language metrics penalize critical clinical errors.

3. An anomaly-guided report generation framework via explicit visual grounding. A disc-level anomaly detection module built on semi-supervised U-Net++ produces a six-channel spatial heatmap tensor over the IVD levels T12-L1 through L5-S1, which is converted into an RGB overlay and passed to the VLM as auxiliary input. The approach is architecture-agnostic and doubles as an independent, inspectable interpretability output for clinical oversight.

4. A demonstration that metric-driven optimization can be actively unsafe. Fine-tuning on report text degraded performance and produced anatomically implausible references to non-existent vertebral levels (e.g., L6–L7), despite such labels never appearing during training.

Main Findings

• Fluency does not imply diagnostic correctness: MedGemma-27B and ChatGPT-5.0 achieved the highest scores on both LSMRI and SPIDER in zero-shot report generation, yet their IVD-level pathology detection remained near random, with balanced accuracy reported as 50–57% in the per-level classification analysis and as 50–56% in the discussion.

• Structured detection is driven by a negative-prediction bias: Per-level classification was near random across all findings and models, with high specificity (74–83%) but low sensitivity (2–34%). Level stratification showed near-zero sensitivity at upper levels and elevated sensitivity at L4-L5 (70–85%) with sharply reduced specificity (30–40%), plus an analogous pattern at L5-S1 for MedGemma-27B—consistent with anatomical priors learned from training-data frequency rather than image evidence.

• Input ablations produced no consistent improvement: No ablation—prompt length, MRI contrast, slice selection, or resampling—yielded consistent gains (BERTScore Δ ≤ ±0.2, BLEU Δ ≤ ±3.7), with gains on one metric coinciding with losses on another.

• Standard metrics penalize style more than diagnostic error: Paraphrasing caused modest changes (BERTScore −6.6%, BLEU −14.2%), while superficial structural changes induced larger penalties (BERTScore −11.5%, BLEU −12.6%). Clinically critical perturbations produced smaller penalties: negation reversals (−6.6%), anatomical location swaps (−5.3%), and terminology exchanges (−5.5%).

• Fine-tuning results split by supervision type: QLoRA fine-tuning of MedGemma-4B with finding-based supervision (per-IVD tokens) consistently improved metrics over the pre-trained baseline across all findings, despite having no explicit report-generation objective. Fine-tuning directly on report text degraded performance and increased output variability, including references to non-existent vertebral levels (e.g., L6–L7).

• Degenerate outputs in smaller models: BiomedGPT and smaller VILA-M3 variants showed near-zero BLEU, consistent with degenerate outputs. BiomedGPT-XLarge produced notably low scores on T1w inputs (BERTScore 22.41 on LSMRI; 33.56 on SPIDER).

• Anomaly heatmaps localize reliably: The U-Net++ heatmaps achieved an overall DICE of 82.3 ± 2.0 and RMSE of 0.4 ± 0.1 on SPIDER (positive ground-truth pixels only, RMSE after [0,1] normalization). Per-level DICE ranged from 87.0 ± 1.8 (L1-L2) to 79.3 ± 3.1 (L5-S1); RMSE ranged from 0.1 ± 0.1 (L4-L5) to 0.6 ± 0.2 (T12-L1 and L5-S1). Upper levels (T12-L1, L1-L2) had lower absolute scores, reflecting lower pathology prevalence and weaker supervision.

• Heatmap overlays improve the structured localization task most: Sensitivity at upper levels (T12-L1 through L3-L4), near zero under zero-shot prompting, increased for several findings when the overlay highlighted anomalous disc regions, with specificity largely maintained. For MedGemma-27B, sensitivity at L1-L2 rose from 2.1 to 22.6 and at L2-L3 from 1.8 to 17.0, while specificity dropped from 100.0 to 88.8 and 88.3 respectively; at L3-L4 sensitivity rose from 1.5 to 87.5 with specificity falling from 100.0 to 17.5.

• Integration depends on multi-image support: Models with multi-image conditioning (ChatGPT) consistently improved METEOR and ROUGE-L with the overlay, indicating spatially localized anomaly signals promote more anatomically specific language. BERTScore gains were modest, aligning with its insensitivity to localization precision. Models without native multi-image support (MedGemma) showed no consistent benefit, confirming effective integration requires attending to both inputs jointly. Text-guided report generation from AD-based localization could match or surpass zero-shot performance.

Methodology in Plain English

Benchmarking. The authors take three public lumbar spine MRI datasets with different characteristics: LSMRI (515 patients with symptomatic low back pain and expert-written reports), SPIDER (218 patients with structured IVD-level grading labels, converted into synthetic reference reports using predefined sentence templates with limited lexical variation to reduce bias toward any single phrasing), and LumbarDISC (1,976 patients annotated with the location and severity of five conditions related to stenosis in five IVD levels). They feed a single central sagittal slice plus a task prompt to each of five VLMs and compare the resulting text against reference reports.

Testing robustness two ways. First, they change the inputs—contrast weighting, slice position, resolution, prompt length—to see whether the models actually notice. Second, they deliberately corrupt the reference reports in ways that mimic or exaggerate clinical variation: benign changes (paraphrasing, boilerplate insertion, length inflation) and harmful ones (negation reversals, swapping vertebral levels, swapping clinically distinct terminology). If the metrics penalize the harmless changes more than the harmful ones, the metrics are not measuring clinical correctness.

Structured localization as a separate test. Rather than relying on free text, they derive binary presence/absence labels for each finding at each IVD level (T12-L1 through L5-S1) and prompt models to output only affected levels in a constrained format. This isolates localization ability from writing style.

Anomaly detection and grounding. Because pixel-level anomaly annotations do not exist at scale for lumbar spine MRI, the authors derive weak spatial targets from structured grading labels. Using vertebrae detection landmarks from SpineNetV2 and SPIDER gradings, they build two supervision signals: localization heatmaps covering the disc space and adjacent endplates, and anomaly heatmaps encoding pathology-specific spatial distributions (e.g., posterior disc margin for herniation, central canal for stenosis). A UNet++ predicts a six-channel heatmap from three central sagittal slices. Training uses a channel-balanced MSE plus a channel-wise regularization term that penalizes cross-channel co-activation between IVD levels, and proceeds in two phases—localization first, then anomaly targets—via linear interpolation over training epochs.

Feeding the heatmap back into the VLM. The six channels are collapsed into one RGB overlay by giving each pixel the color of its maximally activated disc level, scaled by activation magnitude, then alpha-blended with the original MRI slice. ChatGPT and VILA-M3, which support multi-image conditioning, receive both the original slice and the overlay. MedGemma, which does not, receives a structured finding report extracted from the heatmap as textual context.

Fine-tuning setup. MedGemma-4B is adapted with QLoRA—base weights frozen in 4-bit NF4 form, low-rank adapters trained in bfloat16 (20.4M parameters, 0.47% of total). Training uses AdamW with cosine scheduling, linear warmup over 3% of steps, gradient accumulation of 16 steps, lr = 2⁻⁴, r = 32, α = 64, an 80/20 patient-level split on SPIDER with both T1w and T2w scans, and evaluation on the held-out LSMRI test set. The anomaly detector is pre-trained for disc localization on 85% of LumbarDISC and 30% of LSMRI for 351 epochs, then fine-tuned on anomaly heatmaps using SPIDER for 501 epochs, with the weighting term ramped linearly from 0 to 1 over 1,000 training steps, using AdamW at a fixed learning rate of 10⁻³.

Evaluation. Report generation uses BLEU-4, ROUGE-L, METEOR, and BERTScore F1. Localization uses sensitivity, specificity, balanced accuracy, F1, and accuracy, reported per-finding and per-IVD-level. Anomaly detection uses DICE (threshold 0.1) and RMSE on [0,1]-normalized heatmaps, stratified by IVD level.

Why This Matters

Impact on research. The paper argues that the field's default evaluation toolkit is misaligned with clinical reality. Because fluent-but-wrong reports score well and clinically severe perturbations cost less metric penalty than surface-level rewording, metric-driven optimization can systematically reward the wrong behaviour. This reframes benchmark design for medical report generation: evaluation must probe localization and image grounding, not just text similarity. The authors also release code, training and evaluation scripts, and reproducibility instructions, and their anomaly-grounding method is architecture-agnostic, requiring no task-specific VLM retraining.

Real-world applications.

• Clinical triage support: Highlighting disc-level anomalies as an inspectable overlay on the MRI so a radiologist can verify where the model is looking, decoupled from the generated text.
• Quality control in reporting workflows: Providing a separate image-derived evidence trace that supports structured review and human oversight, rather than trusting a text output at face value.
• Prioritization of lumbar spine MRI reads: Assisting with the rapid triage of structural pathologies such as disc herniation and stenosis at the correct vertebral level, which clinical guidelines emphasize.
• Low-annotation deployment: Weakening the dependency on large, densely annotated spine MRI datasets by deriving spatial supervision from existing structured grading labels rather than pixel-level annotation.

Industry relevance. Any vendor building radiology foundation models or reporting copilots for musculoskeletal imaging should treat this as a cautionary result: high BERTScore is not evidence of diagnostic competence, and models appear largely invariant to clinically meaningful input variation such as contrast, resolution, or slice placement. The heatmap-overlay approach offers a relatively cheap, model-agnostic way to add grounding—and to expose to regulators and clinicians what evidence the output is actually based on.

Future Directions

1. Scaling anomaly-target supervision. The authors note that pixel-level anomaly annotations are not available at scale for lumbar spine MRI and that the anomaly targets are weakly derived from grading labels; upper IVD levels (T12-L1, L1-L2) score lower because of lower pathology prevalence and weaker supervision, which better supervision could address.

2. Closing the specificity-sensitivity trade-off. The largest sensitivity gains from heatmap integration at L3-L4 came with specificity dropping from 100.0 to 17.5, and gains were model- and finding-dependent. Calibrating this trade-off, and extending gains to models lacking multi-image conditioning, is an open problem.

3. Overcoming the anatomical prior bias. Models systematically over-predict lower lumbar levels (L4-L5, L5-S1), apparently reflecting training-data frequency rather than image-based reasoning. Whether stronger grounding, more balanced data, or explicit level conditioning can remove this bias is untested here.

4. Establishing whether grounding survives metric-driven optimization. Fine-tuning on report text degraded performance and invented non-existent vertebral levels (e.g., L6–L7), while finding-based supervision helped. How best to combine anomaly-guided grounding with fine-tuning without reintroducing hallucinated anatomy remains unresolved, as does whether these results transfer beyond MRI.

Target Audience

Medical AI researchers working on radiology report generation, vision-language models, or clinical evaluation methodology; radiologists and clinical informaticists interested in what current VLMs can and cannot reliably do on lumbar spine MRI; machine learning engineers building or auditing clinical reporting systems who need to know why standard text metrics are insufficient evidence of diagnostic correctness; and benchmark designers in medical imaging who want a template for evaluating anatomical localization rather than linguistic fluency.

Authors’ abstract

Radiology reporting is time-consuming and subject to inter-rater variability, making automated report generation an attractive clinical application for Vision-Language Models (VLMs). We benchmark state-of-the-art VLMs on lumbar spine MRI with a focus on diagnostic accuracy and demonstrate that standard lexical and semantic metrics poorly reflect clinical correctness: fluent, well-structured reports can score highly while containing clinically meaningful diagnostic errors. To address this failure mode, we propose an architecture-agnostic framework that augments VLM inputs with spatially localized, disc-level anomaly heatmaps generated by a semi-supervised U-Net++ model. These heatmaps both improve anatomical sensitivity through explicit visual grounding and provide an independent interpretability output for clinical oversight, moving us closer to diagnostically reliable, visually grounded VLMs for lumbar spine MRI interpretation.