MV-MLM: Bridging Multi-View Mammography and Language for Breast Cancer Diagnosis and Risk Prediction

AI summary

Overview

• Research area: Medical computer vision — vision-language pre-training (CLIP-style contrastive learning) applied to multi-view mammography for breast cancer classification and image-based risk prediction.
• Technical level: Advanced. The paper assumes familiarity with contrastive learning objectives, CLIP-style dual encoders, linear probing versus fine-tuning, and evaluation metrics such as AUC and C-index.
• Scope: The paper proposes MV-MLM (referred to as MV-CLIP in its result tables), a multi-view mammography-language model trained on paired mammograms and synthetic pseudo-reports, and evaluates it on mass, calcification, and malignancy classification plus breast cancer risk prediction.

What This Paper Is About

Computer-aided diagnosis systems for breast cancer need large, finely annotated datasets, which are expensive and slow to collect, and mammography in particular lacks the large-scale paired image-and-report datasets that exist for chest X-rays. This paper asks whether a CLIP-style vision-language model can be trained on mammograms using synthetic text generated from tabular exam metadata instead of real radiology reports, and whether adding multi-view (CC and MLO) supervision improves generalization and data efficiency. The goal is a single pre-trained model that transfers to several downstream breast-screening tasks: mass classification, calcification classification, malignancy classification, and future cancer risk prediction.

Key Contributions

1. Multi-view vision-language contrastive learning model: A VLM training scheme that aligns high-resolution, multi-view mammogram images with synthetic text reports generated from tabular annotations, enabling learning from sparsely labeled data without real clinical text reports. It uses feature-map tokenization and transformer modules on top of standard ConvNet backbones to keep computation manageable at high resolution.
2. Synthetic report generation: A method that converts structured tabular annotations from mammography exams (for example BI-RADS scores, mass size, calcification type, plus patient-level metadata such as age, gender, race) into text descriptions that simulate radiology reports, used to pretrain the vision-language model.
3. Improved performance across multiple downstream tasks: Reported state-of-the-art results on malignancy classification, mass and calcification classification, and breast cancer risk prediction, with reported improvements over other CLIP-based models, self-supervised learning approaches, and fully supervised models.
4. Data efficiency and generalization: With contrastive learning on synthetic reports, the model is reported to generalize across datasets, reduce forgetting during fine-tuning, and require fewer training parameters and fewer labeled examples than traditional supervised methods.

Main Findings

• Pre-training scale without real reports: The model was pre-trained on a proprietary dataset of 134,500 mammography exams comprising 540,000 images from four standard views per exam (CC and MLO for both breasts). This data contains no clinical report text, only tabular patient and exam information.
• Mass classification (VinDr-Mammo, AUC): The best configuration, MV-CLIP + Tr with EfficientNet-B5, scored 0.7083 at linear probing with 10% of data, 0.7421 at 50%, 0.7649 with the full training set, and 0.8614 after fine-tuning, versus 0.6040 / 0.6418 / 0.6228 / 0.8312 for Mammo-CLIP (EN.B5) and 0.5090 / 0.5796 / 0.5734 / 0.8103 for the supervised RN.34 baseline. The paper reports almost 10% improvement at 10% and 50% of the data and over 14% gains with the whole dataset, mainly on linear probing.
• Calcification classification (VinDr-Mammo, AUC): MV-CLIP + Tr with EfficientNet-B5 reached 0.8558 (LP 0.1), 0.9288 (LP 0.5), 0.9393 (LP 1), and 0.9812 (FT), compared with 0.6399 / 0.6748 / 0.7318 / 0.9746 for Mammo-CLIP. The authors note possible saturation in the dataset or that calcification is not too difficult to solve during fine-tuning.
• Malignancy classification (RSNA-Mammo, AUC): MV-CLIP + Tr with EfficientNet-B5 scored 0.6863 (LP 0.1), 0.7406 (LP 1), and 0.7753 (FT), versus 0.5411 / 0.6017 / 0.7257 for Mammo-CLIP and 0.5136 / 0.6077 / 0.7271 for the supervised EN.B5 baseline. Comparisons to ViT-B-14 transformer models are reported only for fine-tuning: MaMa-CLIP 0.73, MGCA 0.687, MM-MIL 0.65.
• Comparison with SSL and Open-CLIP (private test data, malignancy, fine-tuning AUC): MV-CLIP (RN.34) reached 0.845 on linear probing and 0.939 after fine-tuning, versus (Custom-)CLIP 0.826 / 0.937, SimCLR 0.669 / 0.908, SwaV 0.671 / 0.907, DINO 0.665 / 0.909, BYOL 0.659 / 0.909, and Open-CLIP (RN.50) 0.915 after fine-tuning (linear probing not reported for Open-CLIP).
• Risk prediction (internal risk-mammo dataset): MV-CLIP with RN+Tr achieved a C-index of 0.73, a 2-year AUC of 0.76, and a 5-year AUC of 0.69, compared with 0.69 / 0.74 / 0.65 for the supervised RN+Tr model, 0.71 / 0.73 / 0.64 for RN+Tr with (Custom-)CLIP, 0.65 / 0.69 / 0.64 for supervised RN-34, and 0.57 / 0.62 / not reported for Mirai. The authors suggest the model learned visual cues related to other risk factors from synthetic text, such as age and breast density.
• Transformer module benefit: Adding a transformer on top of the CNN and its tokenized feature map improved performance further, with the paper reporting 3.7 percentage points gained by (Custom-)CLIP with fine-tuning. The multi-view framework gained smaller additional improvements from the transformer, which the authors interpret as evidence that multi-view pre-training already yields generalizable, near-optimal representations.
• Frozen text encoder: BioClinicalBERT is used as the text encoder and frozen, because the authors found its representations empirically sufficient and freezing reduces computation while keeping the focus on the vision model.
• Fair-comparison caveat: The authors state that the best-performing Mammo-CLIP model was also trained with one of the evaluation datasets and actual clinical reports, whereas MV-CLIP was not.

Methodology in Plain English

Each mammography exam is treated as a set of four images (left/right breast, CC and MLO views) plus tabular metadata. The metadata is split into patient-level fields and laterality-level fields that are shared across the two views of the same breast.

The researchers first turn the tables into text. A large language model, LLaMa-3-7B-instruct, is given a short prefix instruction plus a suffix that summarizes the tabular keys and values, and it writes a synthetic pseudo-report describing the findings. No few-shot examples are supplied, to save input tokens and because the authors hypothesized that simple, noisy text supervision would be sufficient. A post-processing step strips repeated prompt text and truncates the output after the first generated paragraph. Because clinical findings are assumed to be shared across the two views of one breast, both views of a laterality get the same report.

The images are preprocessed by zeroing out pixel values below 40 (background), cropping to the breast region using Sobel edge detection and connected component analysis, and resizing to 1520 × 912. Augmentations include affine transforms with rotations up to 20 degrees, a minimum translation of 0.1%, scaling between 0.8 and 1.2, shearing by 20 degrees, and elastic transformations with α = 10 and σ = 5.

A ConvNet (ResNet-34 or EfficientNet-B5) produces a feature map, which is reshaped into pseudo-tokens and projected to 256 tokens; text tokens are projected to the same size. Each modality then passes through its own transformer with 4 blocks, 8 self-attention heads per block, and MLPs projecting to 1024 hidden dimensions. Global max pooling gives a single embedding per image and per text. Two loss terms are combined, with no weighting between them: a standard CLIP image-text contrastive loss that pulls matching image-report pairs together (temperature τ1 = 0.007), and a multi-view contrastive loss that matches the MLO and CC embeddings of the same breast (temperature τ2 = 0.1), forcing the model to focus on visual attributes visible from both views. The two views are swapped with probability 0.5 during training.

Optimization used AdamW with a learning rate of 5e-5 and weight decay of 1e-4, a cosine-annealing scheduler with one epoch of warm-up, mixed precision, and distributed data parallelism across 8 H100 GPUs. Pre-training ran for 10 epochs (ResNet-34 models with per-device batch size 32; the EfficientNet-B5 CLIP model with 18, all other EfficientNet-B5 models with 8). Classification training ran 30 epochs with per-device batch sizes of 96 for ResNet-34 and 16 (fine-tuning) or 40 (linear probing) for EfficientNet-B5. Risk prediction was fine-tuned with a per-device batch size of 8 for 20 epochs.

Evaluation used public VinDr-Mammo (5,000 exams, 20,000 images, from Vietnam) and RSNA-Mammo (11,913 exams) for mass, calcification, and malignancy classification, plus internal private data for risk prediction (risk-mammo: 16,867 training exams and 2,245 test exams). Baselines included a fully supervised model trained on the private data with weighted cross-entropy, a "(Custom-)CLIP" model trained on the same data, resolution, and image-text pairs as MV-CLIP, the released Mammo-CLIP checkpoint, and runs with and without the transformer module.

Why This Matters

Impact on research. The paper argues that the main bottleneck for medical vision-language models outside chest X-ray is the absence of large paired image-report datasets, and that synthetic reports derived from structured tabular labels can substitute for real clinical text at scale. It also provides evidence that multi-view supervision — using the geometric relationship between CC and MLO views as an auxiliary contrastive signal — produces representations that transfer better than single-view pre-training, at least on the tasks and datasets tested.

Real-world applications (as grounded in the paper):

• Computer-aided diagnosis for screening mammography, supporting radiologists in interpreting challenging exams where cancers are missed that were visible in hindsight.
• Malignancy and lesion-type (mass, calcification) classification to flag abnormal views.
• Image-based risk assessment that predicts whether a patient will develop breast cancer in the next 2 or 5 years, which the paper notes can be used to tailor screening recommendations such as a shorter interval or an additional exam.
• Data-efficient deployment in settings with scarce labels: linear probing results approach fine-tuned performance in some experiments, which the authors connect to clinical applicability where data is scarce and fine-tuning is expensive.

Industry relevance. The work is a collaboration involving LMU University Hospital and Lunit Inc., and it targets a practical obstacle for commercial CAD and screening products: the cost of curating paired image-report datasets and the difficulty of fine-tuning large models per site. A model that learns from metadata-derived text and generalizes across public datasets such as VinDr-Mammo and RSNA-Mammo could lower the barrier to deploying and adapting screening AI in new populations.

Future Directions

• Reducing reliance on paired data further: The paper identifies the availability of large-scale paired datasets as a remaining bottleneck and calls for more research on generating synthetic data or leveraging weak supervision from unpaired datasets.
• Interpretability for clinical adoption: The authors describe spatially aligned explanation methods such as Mammo-FActOR as promising but requiring further validation across diverse populations and imaging conditions.
• Extension to three-dimensional imaging: The paper notes that Digital Breast Tomosynthesis offers better lesion visibility than two-dimensional mammography but is limited by higher cost and longer acquisition time, leaving room for adapting the approach to DBT.
• Filling out the ablation evidence: The provided content truncates the ablation study, listing only "Encoder," "Private Data 1," "Private Data 2," and "Private Data," with the corresponding results table not included, so the detailed ablation conclusions are not reported here.

Target Audience

Researchers and engineers working on medical imaging AI, particularly vision-language models, self-supervised learning, and mammography CAD; machine learning practitioners interested in cross-modal pre-training with synthetic or weak text supervision; and clinical AI teams evaluating data-efficient pre-training strategies for breast cancer screening and risk prediction. Readers without a background in contrastive learning or transformer architectures will find the methodological section demanding, but the results and framing remain accessible.

Authors’ abstract

Large annotated datasets are essential for training robust Computer-Aided Diagnosis (CAD) models for breast cancer detection or risk prediction. However, acquiring such datasets with fine-detailed annotation is both costly and time-consuming. Vision-Language Models (VLMs), such as CLIP, which are pre-trained on large image-text pairs, offer a promising solution by enhancing robustness and data efficiency in medical imaging tasks. This paper introduces a novel Multi-View Mammography and Language Model for breast cancer classification and risk prediction, trained on a dataset of paired mammogram images and synthetic radiology reports. Our MV-MLM leverages multi-view supervision to learn rich representations from extensive radiology data by employing cross-modal self-supervision across image-text pairs. This includes multiple views and the corresponding pseudo-radiology reports. We propose a novel joint visual-textual learning strategy to enhance generalization and accuracy performance over different data types and tasks to distinguish breast tissues or cancer characteristics(calcification, mass) and utilize these patterns to understand mammography images and predict cancer risk. We evaluated our method on both private and publicly available datasets, demonstrating that the proposed model achieves state-of-the-art performance in three classification tasks: (1) malignancy classification, (2) subtype classification, and (3) image-based cancer risk prediction. Furthermore, the model exhibits strong data efficiency, outperforming existing fully supervised or VLM baselines while trained on synthetic text reports and without the need for actual radiology reports.