Boosting Medical Visual Understanding From Multi-Granular Language Learning

AI summary

Overview

Research area: Medical vision-language pretraining — specifically contrastive image-text representation learning for retinal fundus and chest X-ray images.

Technical level: Advanced. The paper assumes familiarity with contrastive learning (CLIP), vision transformers, BERT-style text encoders, and Kullback–Leibler divergence.

Scope: The paper introduces Multi-Granular Language Learning (MGLL), a plug-and-play contrastive framework that adds multi-label and cross-granularity supervision to CLIP-style pretraining, and validates it on two newly constructed datasets and eleven downstream benchmarks.

What This Paper Is About

Standard image-text pretraining such as CLIP aligns each image with a single text label at a single level of detail, which is a poor match for medical imaging where one image can carry multiple disease labels described at several levels (for example a disease category plus a clinical explanation). The authors build MGLL to align images against multiple labels and multiple annotation granularities at once, without adding granularity-specific encoders or extra computational cost. The goal is stronger, more transferable medical visual features that improve downstream classification and also serve as a vision encoder for multimodal large language models.

Key Contributions

1. MGLL framework: A contrastive learning framework using multi-granular language that enables simultaneous multi-label and cross-granularity alignment between images and text.
2. Architecture-agnostic objectives: A set of multi-label, multi-granularity learning objectives (soft CLIP loss, point-wise loss, and smooth KL divergence loss) that can be integrated into vision-language models and foundation models.
3. Two large-scale datasets: A structured multi-granular, multi-label labeling system and the resulting datasets — MGLL-Fundus (246,389 fundus image–text pairs drawn from 49 public datasets and covering more than 50 disease categories) and MGLL-Xray (190,882 X-ray images from the MIDRC repository).
4. Broad empirical validation: Experiments on more than ten downstream datasets (eleven are listed in the experimental setup) showing consistent improvements over state-of-the-art methods, plus benchmarks showing MGLL as a drop-in vision encoder inside multimodal large language models.

Main Findings

• Fundus performance: Across nine downstream fundus datasets, MGLL shows improvements in both linear probing and full fine-tuning, with the largest gains in linear probing. On the multi-label dataset RFMiD, MGLL outperforms other methods by at least 16.6% in linear probing and 6.7% in full fine-tuning.

• X-ray performance (MIDRC-XR and MIDRC-XR-Portable): In linear probing, MGLL improves AUC over the second-best method by 2.23% on MIDRC-XR (over UniChest) and 3.81% on MIDRC-XR-Portable (over UniMed-CLIP). MGLL reaches 61.25 AUC / 56.57 ACC / 21.19 mAP linear probe and 99.08 / 90.06 / 73.33 fully fine-tuned on MIDRC-XR, and 83.86 / 89.06 / 30.62 linear probe and 99.75 / 98.80 / 89.87 fully fine-tuned on MIDRC-XR-Portable.

• Transfer to ChestX-ray14: With multi-granular labels built from MIMIC-CXR, MGLL achieves 82.94% AUC, 90.41% accuracy and 28.53% mAP in linear probing, surpassing the second-best method (CARZero) by 5.62%, 6.47% and 1.65% respectively.

• Gains inside multimodal large language models: On a multiple-choice benchmark of 2,233 clinical cases spanning ten ophthalmological conditions (four options per question), replacing standard vision encoders with MGLL produced average accuracy gains ranging from 4.6% (InternVL) to 34.1% (LLaVA-Med). Med-Flamingo improved by 31.7%, Janus-Pro by 10.88%, LLaVA by 7.3%, Qwen-VL by 9.5%, Mini-Gemini by 10.4% and InstructBLIP by 14.7%. The paper states that seven advanced MLLMs were tested while listing eight (InstructBLIP, Mini-Gemini, Qwen-VL, InternVL, LLaVA, LLaVA-Med, Med-Flamingo, Janus-Pro).

• Loss ablation (RFMiD): Standard CLIP performs worst (44.66 linear-probe AUC). The point-wise loss alone gives 70.34, the soft CLIP loss alone gives 67.86, combining both gives 75.73, and adding the smooth KL loss gives the best result at 79.62 linear-probe AUC and 92.83 fully fine-tuned.

• Granularity count matters (MIDRC-XR-Portable): Performance rises monotonically with the number of granularity levels — MGLL 1 (one combined granularity) reaches 80.54 linear-probe AUC, MGLL 2 reaches 82.92, and MGLL 3 (modality, study description, series description) reaches 83.86. MGLL 3 beats CLIP by +12.43% AUC, +10.84% ACC and +8.31% mAP, and beats MGLL 1 by +3.32% AUC, +2.09% ACC and +2.30% mAP.

• Encoder choices (RFMiD): ViT-L/14 performs best among image encoders (79.62 linear-probe AUC), ahead of ViT-H/14 (79.18), ConvNext-Large (78.34), ViT-B/16 (75.53) and ConvNext-Base (74.45). Bidirectional BERT is the best text encoder (79.62), ahead of LLaMA (74.89) and the CLIP text encoder (68.93).

• Robustness to degraded inputs (MIDRC-XR-Portable): Higher resolution helps monotonically — Standard-Resolution (512×512) reaches 83.86 linear-probe AUC versus 80.66 for Low-Resolution (128×128) and 78.82 for Ultra Low-Resolution (64×64), yet MGLL still beats CLIP even at Ultra Low-Resolution. For text quality, standard descriptions (83.86) beat 20% partial omissions (81.14), which beat 20% partial errors (80.02), and both degraded conditions still outperform baseline CLIP.

• Qualitative evidence: Class activation maps show CLIP assigning nearly uniform attention across the fundus image, while MGLL localizes regions of interest such as hard exudates for chorioretinitis and the retinal pigment epithelium for age-related macular degeneration.

Methodology in Plain English

The authors start from the standard CLIP setup: an image encoder and a text encoder trained to pull matching image-text pairs together and push mismatched pairs apart. They change three things about the training objective.

First, instead of pairing each image with one caption, they treat an image as matching several text labels and weight each label by how often it co-occurs with the image (computed by normalizing a co-occurrence matrix). This "soft CLIP loss" lets the image converge toward a weighted centre of several text features rather than a single one. Second, they add a point-wise binary cross-entropy loss over individual image-text pairs, which raises similarity for valid labels and suppresses similarity for invalid ones. Third, because different granularities (for example disease category versus clinical explanation, or modality versus study description versus series description) are encoded separately, they add a smooth Kullback–Leibler divergence loss that pulls each granularity's predicted distribution toward the average of all of them, so the granularities agree with each other. The final objective is a weighted sum with weights 0.5, 1 and 1 for the soft CLIP, point-wise and smooth KL terms respectively.

Data-wise, the authors assemble MGLL-Fundus from 49 public sources with two granularity levels (disease category and clinical explanation derived from dataset label explanations and EyeWiki), and MGLL-Xray from MIDRC with three granularity levels (modality, study description, series description, using LOINC coding to standardize the heterogeneous DICOM descriptions, where over 138,000 studies fall into only 97 unique descriptions while the rest use 1,300 different descriptions). Default encoders are ViT-L/14 and BiomedicalBERT.

Evaluation uses AUC, mAP and accuracy on eleven downstream datasets (FIVES, IDRiD, OIA-DDR, ADAM, PALM, REFUGE, RIM-ONE, RFMiD, MIDRC-XR, MIDRC-XR-Portable, ChestX-ray14) under linear probing and full fine-tuning, with official data splits and test sets never accessed during pretraining. All baselines were pretrained on the authors' own multi-granular datasets for fair comparison.

Why This Matters

The work argues that medical images carry hierarchical, multi-label semantics that single-label contrastive pretraining cannot capture, and shows that restructuring the training objective — rather than adding parameters — yields consistent gains across many datasets and even inside general-purpose and medical multimodal language models.

Impact on research: It offers a general recipe for multi-label and cross-granularity contrastive learning that the authors present as architecture-agnostic and applicable beyond the two modalities tested, along with theoretical arguments (gradient conditions and KL non-negativity) for why each loss term helps.

Real-world applications:

• Retinal screening programs, where a single fundus image must be read for co-occurring conditions such as diabetic macular edema and diabetic retinopathy at multiple severity levels.
• Radiology reporting workflows, where study and series descriptions are heterogeneous; MGLL-Xray uses LOINC standardization to organize this metadata for cohort selection.
• Clinical decision support and diagnostic assistants built on multimodal large language models, where swapping in an MGLL-pretrained vision encoder improved multiple-choice diagnostic accuracy by 4.6% to 34.1% in the reported benchmark.
• Deployment in settings with imperfect documentation or lower-quality images, since the ablations show MGLL still beats CLIP with erroneous text, missing text, or 64×64 images.

Industry relevance: Because MGLL is described as a plug-and-play module that introduces no granularity-sensitive encoders and no additional computational cost, it can be retrofitted into existing vision-language pipelines and medical foundation models. The reported transfer to ChestX-ray14 using MIMIC-CXR-derived labels suggests value for institutions that already hold large but loosely annotated archives.

Future Directions

• Extending MGLL beyond retinal fundus and chest X-ray images to the other modalities the introduction mentions, such as biosignals and genomics, where multi-label and multi-granularity structure also occurs.
• Testing how far the framework scales — the encoder ablation found ViT-H/14 slightly underperformed ViT-L/14, which the authors attribute to possible overfitting or diminishing returns, leaving the scaling question open.
• Determining how to apply MGLL when annotations are incomplete or inconsistently formatted across sources, since the granularity ablation shows that more granularity levels help but real-world datasets often lack them.
• Investigating clinical deployment under noisy conditions more thoroughly, given that text errors degraded performance more than omissions in the reported ablations.
• The conclusion is truncated in the available text, so any additional directions the authors state there are not reported.

Target Audience

Researchers and engineers working on medical imaging AI, vision-language pretraining, or multimodal foundation models who want a method for handling multi-label and hierarchical annotations without architectural changes. It is also relevant to clinical AI teams looking to improve diagnostic assistants through better vision encoders, and to readers interested in how contrastive objectives can be adapted from single-label natural-image settings to complex structured medical data. Readers without background in contrastive learning or information theory will find the objectives section demanding.

Authors’ abstract

Recent advances in image-text pretraining have significantly enhanced visual understanding by aligning visual and textual representations. Contrastive Language-Image Pretraining (CLIP) has played a pivotal role in multimodal learning. However, its focus on single-label, single-granularity alignment limits its effectiveness in complex domains such as medical imaging, where images often correspond to multiple high-level labels (e.g., disease categories) across different annotation granularities (e.g., diagnostic description, clinical explanation). To address this, we propose Multi-Granular Language Learning (MGLL), a contrastive learning framework designed to improve both multi-label and cross-granularity alignment. MGLL leverages structured multi-label supervision, integrates textual descriptions across granularities, and introduces soft-label supervision with point-wise constraints to enhance alignment. MGLL employs smooth Kullback-Leibler (KL) divergence to ensure cross-granularity consistency while maintaining computational efficiency as a plug-and-play module for vision-language models. Pretrained on our constructed large-scale multi-granular datasets and evaluated across multiple datasets, MGLL outperforms other state-of-the-art methods in downstream tasks. The code is available at https://github.com/HUANGLIZI/MGLL.