Epistemic-aware Vision-Language Foundation Model for Fetal Ultrasound Interpretation

AI summary

Overview

Research area: Medical computer vision and multimodal foundation models, specifically vision-language models (VLMs) applied to fetal ultrasound interpretation.

Technical level: Intermediate. Readers should be comfortable with vision-language model architecture, multi-image/multi-view reasoning, and reinforcement-learning-style preference optimization (DPO/GRPO). The clinical terminology is explained well enough for non-clinicians.

Scope: The paper introduces FetalMind, a clinically guided vision-language model that performs both full report generation and multi-label disease diagnosis from variable numbers of fetal ultrasound views, trained on a new 12-center, million-image dataset.

What This Paper Is About

Medical vision-language models have advanced on tasks like visual question answering and report generation, but they are mostly built around structured adult imaging such as chest X-rays. Fetal ultrasound is fundamentally different: a single examination produces dozens of images from inconsistent anatomical views, more than 300 fine-grained diseases exist, and disease-relevant views overlap heavily across conditions. The paper identifies two failure modes in existing models: information collapse (roughly ten text tokens in a diagnosis must be aligned against potentially 10^4 visual tokens across views, drowning out the salient evidence) and disease confusion (coexisting conditions with overlapping views become hard to distinguish). The goal is a model whose inference path mirrors how obstetricians actually work — survey all views, then progressively narrow attention to disease-bearing planes.

Key Contributions

1. FetalMind, described as the first model for fetal ultrasound that jointly handles report generation and diagnosis with a variable number of input views. It is released in 1B and 7B configurations and combines class-wise spatial alignment, fetal token injection, Salient Epistemic Disentanglement (SED), and Salient View Preference Optimization (SVPO).

2. FetalSigma-1M, the first large-scale fetal ultrasound report corpus: approximately 20K patient reports and over 1.19M images from 12 medical centers and 20,566 patients, spanning early, mid, and late gestation, covering all standard views and more than 300 disease categories, organized as image–report–diagnosis triplets.

3. Salient Epistemic Disentanglement (SED), which injects an expert-curated bipartite disease-to-view knowledge graph into the model. Salient views are treated as perturbation variables and swapped between fetuses with different diseases to construct preference pairs, with four clinically motivated swap recipes and hard anatomical constraints.

4. Extensive multi-center evaluation showing gains over open-source, specialized medical, and proprietary models (including GPT-5 and Gemini 2.5 Pro), with 98% accuracy on a curated set of nine major congenital anomalies.

Main Findings

• Broad benchmark gains: FetalMind-M7 reaches an average score of 38.2 across natural-language generation and classification metrics, versus 24.1 for GPT-5 and 24.2 for Gemini 2.5 Pro. Over strong baselines it improves BLEU-1 by roughly 5.6 points, BLEU-4 by 14.2, METEOR by 20.5, and ROUGE-L by 17.8.

• Diagnosis improvements: Binary abnormal/normal accuracy rises by 9.7 points over prior models, with F1 up 13.5 points and recall up 9.3 points. Mapping diseases to 20 anatomical categories yields a 23.8-point gain, evidence that SED grounds diseases to the correct images.

• Robustness across gestational stages: FetalMind-M7 outperforms all baselines in early, mid, and late pregnancy, with gains between 2.2% and 24.9%. The early-trimester multi-label disease metric improves by 20.5 points, which matters because earlier detection gives clinicians more time for follow-up.

• Critical congenital anomalies: On 153 clinically confirmed cases spanning nine major anomalies across three centers, FetalMind achieves 98% diagnostic accuracy. GPT-5 and Gemini 2.5 Pro consistently failed these cases, often misclassifying them as normal. The paper's qualitative example shows GPT-5 missing a ventricular septal defect by underusing 2D and Doppler blood-flow signals.

• Attention correlates with correctness: Diagnostic accuracy correlates positively (above 0.3) with the relative attention the model gives to disease-related views over non-disease views, measured as mean attention weight across query tokens, layers, and heads.

• Ablation results: Removing SED produces the largest degradation (BLEU-4 drops from 23.1 to 13.7, average score from 45.2 to 40.5). Removing spatial alignment hurts report generation disproportionately (BLEU-4 falls to 16.3) while barely affecting diagnosis. Removing fetal token injection causes the smallest but consistent drop. Substituting standard DPO (28.7) or GRPO (37.3) for SVPO performs worse than even vanilla training (38.0).

• Task-dependent temperature: Diagnosis benefits from near-deterministic decoding, while report generation peaks at a small nonzero temperature around beta = 0.1, where exploratory diversity helps narrative coverage without sacrificing factual stability.

• Generalists versus specialists: General-purpose models such as GPT-5 and Gemini 2.5 Pro outperform narrow medical specialists like LLaVA-Med and Med-Flamingo, suggesting that tight specialization can erode reasoning ability acquired during large-scale pretraining.

Methodology in Plain English

The pipeline follows the clinical workflow in three stages.

First, align images to views. Since a correct anatomical view is a prerequisite for any reliable reading, the authors train two separate classifiers — one for early gestation covering 9 view categories, one for mid/late gestation covering 41 — on about 10,000 view-annotated images. This lets the model take an arbitrary set of ultrasound images and tag each with the anatomical plane it shows. Splitting by trimester accounts for the substantial appearance differences across pregnancy.

Second, inject domain tokens. Many congenital anomalies have nearly identical textual descriptions but are clinically distinct (ventricular versus atrial septal defect, for example), and the 40-plus standard views share overlapping wording. The model maps key disease and view terms to special tokens, forcing explicit separability between near-synonyms so the language model does not conflate clinically independent entities.

Third, teach the model which views matter. The authors hand-build a bipartite graph linking each disease to its clinically salient views, curated from textbooks and expert input. Using this graph, they perform view swapping: for a given disease, the salient views of one fetus are swapped with those of a different fetus. Four recipes cover the plausible cases — replacing a diseased region with normal images, replacing normal with diseased, exchanging disease regions between two abnormal fetuses, and merging disjoint disease sets to synthesize multi-disease cases. Hard constraints keep the construction anatomically plausible: images not part of the swap stay with the original fetus, prompts are updated whenever the image count changes, and swaps only happen within matched gestational stages.

Fourth, optimize preferences. Each swap produces a triplet of (swapped input, donor's disease label as the chosen answer, receiver's original label as the rejected answer). The model is trained with Salient View Preference Optimization, an offline preference objective in the DPO family. The gradient analysis shows the loss gives a strong, non-negligible learning signal even when chosen and rejected answers are very close — exactly the hard cases involving negation, laterality, units, and anatomical loci that matter clinically. Roughly 10,000 such samples are generated, evenly split across the four swap types and restricted to within-center pairs so that institution-specific formatting and device effects do not contaminate the signal.

Training ran on NVIDIA A800 GPUs: one epoch of alignment, three of instruction tuning, and one epoch of SVPO reinforcement learning at a learning rate of 5e-5 with 224×224 images. The 1B model is built on InternVL3 and the 7B on Qwen2.5-VL. Evaluation uses a 7:1:2 split across nine centers with three held out entirely for external validation.

Why This Matters

Impact on research. The paper reframes medical VLM design around clinical workflow rather than generic image-text alignment. Its central claim — that structured clinical priors and knowledge-graph-guided preference construction beat end-to-end scaling on a specialized domain — is a direct challenge to the assumption that general-purpose models will simply absorb everything. The release of FetalSigma-1M as a 1.19M-image, 12-center, all-trimester corpus with expert-verified reports and diagnoses gives the field a benchmark that did not previously exist, and the multi-center plus multi-device evaluation protocol is a template for assessing generalization honestly.

Real-world applications:

• Prenatal screening support in regions with shortages of trained sonographers, where the model could flag likely anomalies for review and reduce missed diagnoses.
• Report drafting, automatically producing structured reports from an examination's image set, saving clinician documentation time.
• Anomaly triage, prioritizing high-risk cases for senior review, especially early-trimester findings where follow-up windows matter.
• Second-reader or training tool, letting residents compare their interpretation against a model that exposes which views drove its conclusion, which also supports auditability.

Industry relevance. The work is directly relevant to medical imaging device manufacturers, PACS vendors, and health IT companies building ultrasound workflow software. The efficiency results matter commercially: the 1B variant ranks second on most generation metrics while being far cheaper to serve, and the finding that a small amount of structured clinical knowledge can beat proprietary frontier models on this task is a strong argument for domain-specific deployment over API calls to general models. The multi-device, multi-center validation also speaks to the regulatory evidence typically expected for diagnostic tools.

Future Directions

• Prospective clinical validation. The current evaluation is retrospective. Establishing real-world safety and utility requires prospective studies in live clinical settings, including how clinicians interact with and override the model.

• Splicing artifacts and synthetic data risk. The swap-based training data is synthesized, and the authors acknowledge the theoretical risk that the model learns artifacts of the splicing process rather than genuine disease-view associations. Better swap realism or verification mechanisms are open problems.

• Deployment integration and uncertainty. Connecting the system directly to PACS and ultrasound consoles, plus adding calibrated uncertainty estimation and case triage, would make the model usable as an oversight tool rather than a standalone predictor.

• Rare anomalies, domain shift, and privacy. Broader coverage of rare conditions, robustness to new devices and populations through active and continual learning, and privacy-preserving federated training across hospitals are all identified as priorities.

• Temporal and multimodal extension. The disease-to-view knowledge graph is currently static. Extending it across time — tracking how findings evolve over the course of a pregnancy, or integrating Doppler and other signal modalities more fully — is a natural next step, particularly given that GPT-5's failure case involved underusing Doppler flow information.

Target Audience

This paper is most valuable to medical AI and multimodal foundation model researchers, especially those working on domain adaptation, preference optimization, and multi-image reasoning. It is also directly relevant to clinicians and clinical informatics teams in obstetrics and prenatal imaging, to regulatory and health-technology evaluation groups assessing diagnostic AI, and to engineers building ultrasound or PACS-integrated decision-support products. Readers without a machine learning background will find the clinical motivation and evaluation results accessible, while the methodology sections assume familiarity with vision-language architectures and reinforcement learning from human feedback.

Authors’ abstract

Recent medical vision-language models have shown promise on tasks such as VQA, report generation, and anomaly detection. However, most are adapted to structured adult imaging and underperform in fetal ultrasound, which poses challenges of multi-view image reasoning, numerous diseases, and image diversity. To bridge this gap, we introduce FetalMind, a medical AI system tailored to fetal ultrasound for both report generation and diagnosis. Guided by clinical workflow, we propose Salient Epistemic Disentanglement (SED), which injects an expert-curated bipartite graph into the model to decouple view-disease associations and to steer preference selection along clinically faithful steps via reinforcement learning. This design mitigates variability across diseases and heterogeneity across views, reducing learning bottlenecks while aligning the model's inference with obstetric practice. To train FetalMind at scale, we curate FetalSigma-1M dataset, the first large-scale fetal ultrasound report corpus, comprising 20K reports from twelve medical centers, addressing the scarcity of domain data. Extensive experiments show that FetalMind outperforms open- and closed-source baselines across all gestational stages, achieving +14% average gains and +61.2% higher accuracy on critical conditions while remaining efficient, stable, and scalable. Project Page: https://hexiao0275.github.io/FetalMind.