Structure is Supervision: Multiview Masked Autoencoders for Radiology

AI summary

Overview

Research area: Medical computer vision — self-supervised representation learning for chest radiography (X-rays).

Technical level: Intermediate. The paper assumes familiarity with masked autoencoders (MAE), vision transformers (ViT), and contrastive/alignment losses in self-supervised learning. The clinical framing is accessible, but the method builds directly on recent SSL literature.

Scope (one sentence): The paper proposes MVMAE and MVMAE-V2T, pretraining frameworks that turn the multi-view structure of radiology studies — and optionally the accompanying reports — into self-supervision for chest X-ray representation learning, evaluated on large-scale public datasets.

What This Paper Is About

Radiology datasets are small (hundreds of thousands, not billions of images) and labels are expensive and noisy, so purely supervised models underperform and generic vision-language pretraining often needs costly adaptation. Most existing self-supervised methods treat each radiograph as an independent image, ignoring that a clinical study typically contains several projections of the same patient — frontal, lateral, sometimes repeated or of undetermined type — plus a report.

The paper's goal is to exploit this structure as a free supervision signal: learn representations that are simultaneously detail-rich (through masked reconstruction) and consistent across views of the same study (through cross-view alignment), with radiology text used only as an optional auxiliary signal during pretraining.

Key Contributions

1. MVMAE (Multiview Masked Autoencoder): The first multi-view masked autoencoder for radiology. It combines per-view masked image reconstruction with a study-level cross-view alignment loss, and adds learnable modality embeddings for frontal, lateral, and unknown projections so a single shared encoder can handle all views.

2. MVMAE-V2T (Vision-to-Text): An extension that adds an autoregressive captioning objective over the study report as auxiliary supervision during pretraining. Text is never used at inference, so the deployed model stays vision-only and fair to compare against pure vision baselines.

3. A study-centric evaluation protocol: A unified "FLU" (Frontal–Lateral–Unknown) policy that retains single-view exams, includes all projections of multi-view studies, and keeps uncertain views rather than discarding them — mirroring how radiologists actually read exams and allowing analysis of how performance scales with the number of views.

4. A unified cross-institutional benchmark: MIMIC-CXR, CheXpert Plus, PadChest, and Chest X-ray harmonized into a shared 14-label space (CheXpert labeler), with Chest X-ray used only for unsupervised pretraining because its reports and label ontology differ. Includes per-label behavior analysis and calibration analysis.

Main Findings

• MVMAE leads overall classification. On the Combined benchmark across 14 labels, MVMAE reaches 87.41 AUROC / 41.27 F1, ahead of the Independent ablation (86.60 / 35.48), CheXagent (86.25 / 33.56), fully Supervised (84.83 / 29.33), and BiomedCLIP (83.97 / 29.02). Gains hold per-dataset and across institutions.

• Alignment is doing real work. The "Independent" ablation removes the cross-view term (β = 0) and reconstructs each projection separately. It underperforms MVMAE, showing that masked reconstruction alone does not explain the gains — the study-level consistency constraint does.

• MVMAE-V2T adds semantic grounding, mostly when labels are scarce. It achieves the best F1 on the Combined set (38.96) and best Top-5 results, and its advantage over MVMAE grows in low-label finetuning regimes, though the gap narrows as more labeled data becomes available.

• Strong label efficiency. Reaching roughly 80% AUROC requires about 5K labeled studies for MVMAE variants, 10K for the Independent baseline (2x more), and 50K for the fully supervised baseline (10x more).

• Better calibrated predictions. Under linear probing on 20K samples, MVMAE variants produce lower Brier scores than baselines on MIMIC-CXR, CheXpert, PadChest, and Combined — relevant for clinical deployment where probability estimates matter.

• Per-label differences follow clinical intuition. Frequent, visually localized findings score higher; rare or textually ambiguous labels (e.g., Pneumonia, Pleural Other) remain difficult. Apparent gaps on CheXpert partly stem from severe imbalance in its official validation split (some conditions have only one positive study), and disappear when model selection is done per label rather than jointly.

Methodology in Plain English

The approach has three moving parts, all applied during unsupervised pretraining:

1. Mask and reconstruct. Each radiograph is split into 16×16 patches. A very high fraction of patches (90%) is masked out, and only the visible patches are fed to a ViT-Base encoder. A lightweight decoder then tries to reconstruct the hidden patches. This forces the encoder to learn fine-grained visual detail. In MVMAE, this happens for every view separately.

2. Pull views of the same study together. For each study, encoder outputs for its different projections (frontal, lateral, unknown) are compared pairwise, and their mean squared difference is minimized. This makes representations view-invariant while remaining spatially detailed — unlike contrastive methods, which discard reconstruction, or pure MAEs, which ignore cross-view correspondence. The alignment weight β is annealed from zero upward to stabilize early training. Small learnable embeddings tell the encoder which projection type it is seeing, analogous to segment embeddings in language models.

3. Optionally predict the report. MVMAE-V2T adds a transformer decoder that generates report tokens one at a time, conditioned on the view's encoder output, trained with cross-entropy. Reports act purely as a training signal — no text is needed at test time. This differs from CLIP-style image-text alignment, which treats text as a co-equal modality.

Downstream, the pretrained encoder is finetuned end-to-end on 14 diagnostic labels derived from the CheXpert labeler, with negative/not-mentioned/uncertain collapsed into a single 0 class. Predictions from single-view models are averaged as a late-fusion ensemble. Comparisons include a fully supervised baseline, an ablated version without alignment, BiomedCLIP, and the CheXagent vision encoder.

Why This Matters

This work argues that the path to better medical foundation models is not simply more data, but better use of the structure already present in clinical data. It shows that a well-chosen inductive bias — cross-view consistency — can be more valuable than scaling a general-purpose biomedical vision-language model, at a fraction of the model size (ViT-Base vs. ViT-Large for CheXagent).

Real-world applications:

• Triage and screening in under-resourced settings, where few radiologists are available and models must be trained with minimal labeled data.
• Cross-institution deployment, since joint pretraining across MIMIC-CXR, CheXpert, and PadChest improves per-dataset performance rather than overfitting to one hospital's acquisition protocol.
• Reducing annotation cost, with label-efficiency results suggesting pretrained encoders can reach usable accuracy with an order of magnitude fewer labeled studies.
• Generalization to other structured imaging, such as longitudinal MRI or multi-sequence CT, which share the property of repeated or complementary scans per patient.

Industry relevance: Imaging AI vendors, hospital systems, and regulators all care about label efficiency, calibration, and robustness across sites. A pretraining recipe that improves all three, and that does not require text at inference, is directly compatible with existing vision-only deployment pipelines and avoids dependence on report availability or report quality.

Future Directions

• Extending beyond chest X-ray. The same multi-view alignment idea applies to any domain with repeated or complementary acquisitions — longitudinal MRI, multi-sequence CT, or repeated pathology slides — but this remains untested.

• Scaling the view analysis. The current view-scaling experiment is restricted to studies with exactly two available views. How performance behaves with three or more projections, or with repeated acquisitions of the same view type, is an open question.

• Improving rare and ambiguous labels. Performance still lags on infrequent or textually vague findings. Better handling of label imbalance, uncertain labels, and report-derived semantics is needed.

• Richer use of reports. MVMAE-V2T avoids explicit image-text alignment by design. Whether stronger text objectives — without sacrificing vision-only inference — could yield further gains, especially in low-label regimes, is unresolved.

Target Audience

This paper is most useful for medical imaging ML researchers working on self-supervised pretraining and foundation models; practitioners building clinical decision-support systems who need label-efficient, well-calibrated models; and clinical AI teams evaluating whether to adopt X-ray-specific pretrained encoders over general-purpose biomedical VLMs. Readers with a background in self-supervised vision learning will find the method straightforward, while those primarily from a clinical background can extract the evaluation protocol and benchmark results without the technical details.

Authors’ abstract

Building robust medical machine learning systems requires pretraining strategies that exploit the intrinsic structure present in clinical data. We introduce Multiview Masked Autoencoder (MVMAE), a self-supervised framework that leverages the natural multi-view organization of radiology studies to learn view-invariant and disease-relevant representations. MVMAE combines masked image reconstruction with cross-view alignment, transforming clinical redundancy across projections into a powerful self-supervisory signal. We further extend this approach with MVMAE-V2T, which incorporates radiology reports as an auxiliary text-based learning signal to enhance semantic grounding while preserving fully vision-based inference. Evaluated on a downstream disease classification task on three large-scale public datasets, MIMIC-CXR, CheXpert, and PadChest, MVMAE consistently outperforms supervised and vision-language baselines. Furthermore, MVMAE-V2T provides additional gains, particularly in low-label regimes where structured textual supervision is most beneficial. Together, these results establish the importance of structural and textual supervision as complementary paths toward scalable, clinically grounded medical foundation models.