CURE: Curriculum-guided Multi-task Training for Reliable Anatomy Grounded Report Generation

AI summary

Overview

Research area: Medical computer vision and vision-language models (VLMs), specifically grounded radiology report generation from chest X-rays.

Technical level: Intermediate. The paper assumes familiarity with vision-language models, multi-task learning, bounding-box grounding metrics (IoU), and curriculum learning; the conceptual core is accessible, but the experimental tables require domain background.

Scope: The paper proposes a curriculum-guided multi-task fine-tuning framework (CURE) that improves visual grounding, report quality, and factual reliability in medical VLMs using only existing public chest X-ray datasets.

What This Paper Is About

Medical vision-language models can write radiology reports from chest X-rays, but they frequently hallucinate findings that are not present in the image and fail to link their textual statements to the correct anatomical regions. CURE addresses this by reorganizing existing public datasets into a unified instruction format spanning three grounding tasks, then training the model with an error-aware curriculum that automatically shifts sampling weight toward whichever datasets and anatomical categories the model is currently handling worst. The goal is a more trustworthy, interpretable medical VLM without collecting any new data.

Key Contributions

1. An error-aware curriculum learning framework that dynamically reweights training samples at two levels of granularity (dataset-level and class/anatomy-level) based on the model's measured error rate, rather than relying on fixed heuristic loss weights.
2. A task reformulation strategy that converts heterogeneous supervision (bounding boxes, clinical phrases, anatomy labels, report sentences) into a unified triplet format of (image, instruction, response), including a new Anatomy-Grounded Report Generation (AGRG) task derived from Chest ImaGenome scene graphs.
3. Demonstration that grounding capability can be transferred to a model that lacks it entirely. The base MedGemma-4B-IT has no innate grounding mechanism; CURE instills one, surpassing MAIRA-2 on localization across in-domain and zero-shot benchmarks while training on a fraction of the available instances.
4. A targeted hallucination analysis using natural language inference to quantify the reduction in false-positive abnormal findings, contradictions, and improved entailment relative to ground-truth reports.

Main Findings

• Large localization gains on anatomy-grounded generation: On Chest ImaGenome, CURE reaches 0.601 mean IoU versus MAIRA-2's 0.249 — an improvement of +0.35 IoU, roughly doubling localization accuracy, despite MedGemma-4B-IT having no grounding capability before training.

• Consistent phrase-grounding improvements: CURE outperforms MAIRA-2 on both micro- and macro-averaged IoU on MS-CXR, PadChest-GR, and — importantly — zero-shot VinDr-CXR, which was never seen during training. The largest relative gains appear on PadChest-GR, suggesting the curriculum reweighting helps most on tasks with many fine-grained categories.

• Better report quality on in-domain grounded tasks: On Chest ImaGenome AGRG, CURE achieves the best CheXbert F1-micro (0.529), cosine similarity (0.691), and CXRFEScore (0.549). On the MIMIC-CXR report generation test set, the combined AGRG+GRG variant achieves the highest recall (R-Ma 0.582, R-Mi 0.781), CheXbert cosine similarity (0.792), RaTEScore (0.597), and CheXbert F1-macro (0.415).

• Sharp reduction in hallucinations: Across six anatomical regions, abnormal-finding hallucination drops from 26.50% (MAIRA-2) to 8.78% (CURE) — an 18.6% reduction. Contradiction rates fall from 33.22% to 17.44%, and entailment more than doubles from 15.94% to 39.50%. Clavicle hallucination falls from over 59% to 1.00%.

• Anatomy training beats phrase training for reliability: The authors attribute the hallucination reduction to a key data-design difference. Standard phrase grounding is biased toward abnormal findings, whereas AGRG exposes the model to both normal and abnormal descriptions for each anatomical location, producing a more balanced generative process.

• Honest limitation on one benchmark: On PadChest-GR grounded report generation, MAIRA-2 still leads on most text-based metrics because it was pretrained on the proprietary USMix dataset (193K text-only, 69K grounded reports). CURE still leads on grounding IoU there.

• Ablations confirm each component matters: Starting from a plain multi-task baseline, data augmentation gives small consistent gains, curriculum reweighting at a 3000-step interval outperforms more frequent updates, and a dedicated 3000-step Chest ImaGenome pretraining stage produces the largest jump — enabling the model to match MAIRA-2 on MS-CXR phrase grounding before the final hyperparameter search.

• Data efficiency: Due to compute limits, only about 1.74% of the 12.9M available training instances are actually processed (225,000 instances over 9,000 steps at effective batch size 25).

Methodology in Plain English

The researchers took a general medical vision-language model (MedGemma-4B-IT) that can describe images but cannot point to anything, and fine-tuned it with LoRA adapters using three task types drawn from public chest X-ray datasets:

• Phrase grounding — "Ground the phrase: {finding}" → output the phrase plus bounding-box coordinates.
• Grounded report generation — "Generate a grounded report" → output the report with coordinates embedded inline after each phrase.
• Anatomy-grounded report generation — split into three prompts: "Locate the {location}", "Describe the {location}", and "Locate and describe the {location}". Because a single X-ray can yield 9 to 36 such instances, this expands one dataset into millions of training samples.

The curriculum is the distinctive part. The training runs in stages. Between stages, the model is evaluated on small validation subsets, and two metrics are computed: IoU (does the box land in the right place?) and CXRFEScore (does the text read clinically correctly?). These are blended into a single performance score. The error (1 minus that score) becomes the sampling weight for the next stage — both for which dataset to draw from and, within a dataset, which anatomical region or phrase category to emphasize. This means the model automatically spends more time on whatever it is currently worst at, without anyone hand-tuning loss weights. An initial phase pretrains specifically on Chest ImaGenome for 3,000 steps to build grounding ability before the curriculum begins.

Why This Matters

Impact on research: The paper argues that factuality and visual grounding in medical VLMs are not problems you solve with more data — they are problems you solve with better training schedules. By showing that a curriculum driven purely by measured error can close a large grounding gap without proprietary data, it offers a reproducible alternative to the standard "collect a bigger medical corpus" playbook. It also introduces hallucination reduction as a first-class metric rather than a side note.

Real-world applications:

• Radiologist workflow support: A model that grounds each finding to a specific anatomical region lets clinicians verify the model's claim against the image directly, turning an opaque text suggestion into an auditable annotation.
• Quality assurance and triage: Lower false-positive rates for abnormalities (especially on bone structures like clavicles) reduce alert fatigue in systems that pre-screen chest X-rays.
• Regulatory and audit trails: Grounded outputs provide evidence for each statement, which matters for documentation requirements in clinical deployment and post-market surveillance.
• Low-resource and rural settings: Because CURE trains efficiently on public data with parameter-efficient fine-tuning, it lowers the barrier for institutions that cannot afford to curate private labeled datasets.

Industry relevance: The model weights are released on Hugging Face and the code on GitHub, positioning this as an open alternative to proprietary grounded report generators. For medical AI vendors, the core technique — error-aware reweighting over imbalanced multi-task data — is directly transferable to other modality mixes (CT, MRI, mammography) and other multi-task pipelines where rare classes are clinically important but statistically starved.

Future Directions

1. Closing the text-quality gap on grounded report generation. CURE still trails MAIRA-2 on PadChest-GR text metrics due to the latter's proprietary USMix pretraining. Evaluating whether the curriculum can substitute for that data, or whether a modest amount of curated grounded reports closes the gap, is an open question.

2. Extending the curriculum beyond chest X-rays. The framework is modality-agnostic in principle but was validated only on chest radiographs. Transferring it to CT, MRI, or multi-view 3D imaging — where anatomy categories and spatial structure are far more complex — is untested.

3. Replacing sequential staged training with continuous adaptation. The current method retrains in discrete stages with validation pauses. A smoother online version that updates sampling weights continuously could reduce the computational overhead of repeated evaluation cycles.

4. Standardizing hallucination evaluation for medical VLMs. The paper's NLI-based protocol using an LLM judge (Gemini 2.5 Flash Lite) is practical but not yet a community standard; validating it against human radiologist judgments and broadening it beyond six anatomical regions would strengthen the claim.

Target Audience

This paper is most valuable to medical AI researchers and graduate students working on vision-language models, grounded report generation, or multi-task learning with imbalanced data. It also serves applied machine learning engineers in health tech who need practical, compute-efficient fine-tuning recipes for domain adaptation, and clinical informatics teams evaluating whether current medical VLMs are reliable enough for deployment. Readers with a general ML background will follow the core idea easily; the evaluation tables require familiarity with report-generation metrics such as CheXbert, RadGraph F1, and CXRFEScore.

Authors’ abstract

Medical vision-language models can automate the generation of radiology reports but struggle with accurate visual grounding and factual consistency. Existing models often misalign textual findings with visual evidence, leading to unreliable or weakly grounded predictions. We present CURE, an error-aware curriculum learning framework that improves grounding and report quality without any additional data. CURE fine-tunes a multimodal instructional model on phrase grounding, grounded report generation, and anatomy-grounded report generation using public datasets. The method dynamically adjusts sampling based on model performance, emphasizing harder samples to improve spatial and textual alignment. CURE improves grounding accuracy by +0.35 IoU, boosts report quality by +0.192 CXRFEScore, and reduces hallucinations by 18.6%. CURE is a data-efficient framework that enhances both grounding accuracy and report reliability. Code is available at https://github.com/PabloMessina/CURE and model weights at https://huggingface.co/pamessina/medgemma-4b-it-cure