CARDIUM: Congenital Anomaly Recognition with Diagnostic Images and Unified Medical records

AI summary

Overview

Research area: Medical computer vision and multimodal machine learning, specifically prenatal detection of Congenital Heart Diseases (CHDs) from fetal ultrasound/echocardiography combined with maternal clinical records.

Technical level: Intermediate. The dataset and clinical framing are approachable, but the model section assumes familiarity with Vision Transformers, self-attention/cross-attention, Weight of Evidence encoding, and class-imbalance strategies.

Scope: The paper introduces the CARDIUM dataset — described as the first publicly available multimodal dataset for prenatal CHD detection — and a multimodal transformer baseline that fuses fetal images with maternal tabular records.

What This Paper Is About

Prenatal detection of Congenital Heart Diseases is critical but difficult: the conditions are rare (roughly 8 in every 1,000 live births globally each year), the fetal heart is small and constantly moving, and detection rates can be as low as 30% in low- and middle-income countries. Existing AI methods rely on private, imbalanced, image-only datasets with no clinical context and little released code, which blocks reproducibility and fair comparison. This paper addresses that gap by releasing a public multimodal dataset linking fetal ultrasound and echocardiographic images to maternal clinical records, along with an open-source multimodal baseline model.

Key Contributions

1. The CARDIUM dataset — the first publicly available multimodal dataset for prenatal CHD detection, combining fetal echocardiographic and ultrasound images with maternal clinical records from 1,103 patients, collected retrospectively on Colombian women between 2013 and 2024.
2. A multimodal transformer architecture (the CARDIUM model) that uses self-attention for intra-modality dependencies and a dual cross-attention mechanism to fuse image and tabular feature representations, followed by a three-layer MLP classifier.
3. Class-imbalance strategies for clinical realism, including weighted random sampling, loss weighting with a factor of 1.2 on the positive class, image augmentation, and hard positive mining applied to the tabular encoder.
4. Open release of dataset and code at the project's GitHub and website, plus a benchmark comparison against multimodal state-of-the-art methods (TIP and MMCL) and generalization tests on an external ultrasound dataset.

Main Findings

• Multimodal performance: The full model achieves an F1-score of 0.798 ± 0.048, CHD precision of 0.876 ± 0.173, CHD recall of 0.757 ± 0.104, and AUC of 0.974 ± 0.012 on the CARDIUM dataset.
• Gain over single modalities: Combining images with clinical data improves over image-only by 11% and over clinical-data-only by 50%. Image-only reached F1 0.689 ± 0.066 (precision 0.659 ± 0.135, recall 0.742 ± 0.119, AUC 0.955 ± 0.0154); tabular-only reached F1 0.294 ± 0.019 (precision 0.192 ± 0.019, recall 0.634 ± 0.049, AUC 0.794 ± 0.028).
• Trimester-dependent performance: F1 was 0.222 ± 0.314 in the first trimester, 0.603 ± 0.092 in the second, and 0.732 ± 0.072 in the third. Only five CHD-positive cases exist in the first trimester, so the authors caution against firm conclusions about early-stage detection.
• Feature-space effect of fusion: t-SNE visualizations show that after multimodal fusion, CHD and non-CHD clusters become more compact and distinct than in the tabular encoder or image encoder outputs alone.
• Image-only cohort evaluation: On 11 CHD patients (144 CHD images) and 113 non-CHD patients (767 non-CHD images) with no available clinical records, the model achieved an F1-score of 0.8528 ± 0.106.
• Generalization to an external dataset: On Fetal-Planes-DB, ViT Small scored 0.900, the CARDIUM model trained from scratch scored 0.914, and the CARDIUM model pre-trained on CARDIUM scored 0.918. MedMamba-B achieved the best result at 0.933, followed by VMamba-B at 0.927; Swin Transformer-B (0.854), ConvNext-B (0.855), and EfficientNetV2-B (0.885) scored lower than the CARDIUM variants.
• Outperformance of multimodal SOTAs: TIP reached 0.459 ± 0.027 and MMCL 0.349 ± 0.090, both far below the CARDIUM model's 0.798 ± 0.048. The authors attribute this to contrastive learning struggling with class imbalance.
• Data quantity matters: Training on half of the CARDIUM dataset produced a 13% decrease in F1-score.
• Fusion strategy comparison: MLP Fusion 0.454 ± 0.067; Transformer Encoder Fusion 0.686 ± 0.086; Transformer Decoder Fusion 0.607 ± 0.091; Transformer Encoder with Cross-Attention Fusion 0.681 ± 0.048; the authors' double transformer decoder fusion 0.798 ± 0.048 — outperforming all others by at least 11%.
• Imbalance handling is decisive: Adding a weighted random sampler raised performance by 39.6% (from 36.1% to 75.7%), and combining it with a loss factor of 1.2 on the positive class added 4.1% more, reaching the final 79.8%.
• Image encoder comparison: Among ResNet 18, ResNet 50, ViT Tiny, ViT Small, and MedViT, ViT Small outperformed all others by at least 6%.

Methodology in Plain English

The researchers built a dataset from a single clinical site by taking routine fetal ultrasound and echocardiographic images, having an expert echographer discard inconsistent or very low-quality images, and pairing those images with the mother's clinical record. Because images and clinical notes are not captured on the same day, they collapsed all clinical events from a pregnancy into one tabular entry per patient: stable variables kept one value, binary fields were set to 1 if any record was positive, and ordinal fields took the highest reported level.

For the tabular data, they standardized units, corrected out-of-bounds values, applied z-score normalization, fixed typographical errors, merged semantically similar disease and medication names (for example, grouping vaginitis, candidiasis, and acute vaginitis under vaginal infections), and rolled categories with fewer than four occurrences into an "Others" label. Categorical variables were encoded using Weight of Evidence encoding with a five-fold cross-encoding scheme so that each fold's encoding is computed only from the other four folds, preventing data leakage.

The model has three parts. An image encoder (a fine-tuned ViT pre-trained on ImageNet, twelve layers, six attention heads, dropout 0.3 on the transformer path and 0.2 on the classification head) and a tabular encoder (a two-layer transformer encoder with eight attention heads that treats each feature as a token) produce modality-specific embeddings. These embeddings enter two parallel stacks of transformer decoder layers, where each modality first refines itself with self-attention and then exchanges information with the other modality through cross-attention. The refined features are concatenated and passed to a three-layer MLP for CHD versus non-CHD classification.

Training used a binary cross-entropy loss, the AdamW optimizer, a learning rate of 5 × 10⁻⁷, 100 epochs, and an NVIDIA Quadro RTX 8000. Image and tabular encoders were trained separately, frozen, and then transferred into the fusion module. To handle imbalance, they used loss weighting, image augmentation, weighted random sampling, and hard positive mining on the tabular encoder (oversampling false negatives every 20 epochs). Evaluation uses three-fold cross-validation with per-image training but per-patient inference, averaging outputs across all images belonging to a patient. Reported metrics are F1-score, precision, recall for the CHD class, and AUC.

Why This Matters

Impact on research: The paper opens a field that has been bottlenecked by private data. Prior CHD detection models (ResNet, residual CNN, DenseNet21 approaches reporting up to 99% AUC, 93% accuracy, and 92%/100% inter-/intra-patient accuracy respectively) all rely on private datasets, and only one of them releases code. ImageCHD, the earlier open-access option, is restricted to postnatal cases and uses imaging only. CARDIUM gives the community a public, multimodal, prenatal benchmark with fixed folds for fair comparison.

Real-world applications:

• Assisting specialists in under-resourced settings where access to fetal cardiology expertise and equipment is limited, directly targeting the low detection rates reported in low- and middle-income countries.
• Triage support during routine obstetric ultrasound, where the standard four cardiac views (four-chamber, three-vessel trachea, left and right ventricular outflow) are already acquired.
• Risk stratification using maternal clinical records in settings where imaging is unavailable or inconclusive, since the tabular-only model still reached an AUC of 0.794 ± 0.028.
• Second-opinion or screening tools that generalize to other ultrasound tasks, as demonstrated on the external Fetal-Planes-DB dataset covering six anatomical planes.

Industry relevance: The work targets a genuine unmet need in maternal-fetal medicine, a market where regulatory, privacy, and data-scarcity barriers have historically slowed AI adoption. The release of a multimodal benchmark with a reproducible baseline lowers the barrier for medical device companies and clinical AI teams, and the strong generalization results suggest the learned representations transfer to adjacent ultrasound classification tasks rather than being locked to one hospital's protocol. Its explicit handling of class imbalance reflects the practical reality of clinical deployment, where positive cases are far rarer than negative ones.

Future Directions

1. Expand the dataset, especially early-stage data: only five CHD-positive cases exist in the first trimester, and training on half the data already cost 13% in F1-score, so dataset size is a direct lever on performance.
2. Multi-center and multi-population validation to address the demographic and geographic bias introduced by the dataset's exclusive focus on Colombian women.
3. Robustness to image quality and protocol variability, since differences in how specialists apply clinical protocols may affect real-world deployment.
4. Extend the generalization evaluation beyond the external Fetal-Planes-DB experiment — where MedMamba-B still achieved the best F1 of 0.933 — to determine where the multimodal framework's advantages transfer and where stronger unimodal backbones remain preferable.

Target Audience

Medical AI researchers and computer vision practitioners working on multimodal learning, particularly those interested in clinical data fusion with imbalanced datasets. Also relevant to fetal cardiologists, maternal-fetal medicine specialists, and clinical informatics teams evaluating AI-assisted prenatal screening; to dataset and benchmark builders interested in privacy-preserving clinical data release; and to regulators or health technology assessors who need to understand the limitations of single-site, single-population training data before deployment.

Authors’ abstract

Prenatal diagnosis of Congenital Heart Diseases (CHDs) holds great potential for Artificial Intelligence (AI)-driven solutions. However, collecting high-quality diagnostic data remains difficult due to the rarity of these conditions, resulting in imbalanced and low-quality datasets that hinder model performance. Moreover, no public efforts have been made to integrate multiple sources of information, such as imaging and clinical data, further limiting the ability of AI models to support and enhance clinical decision-making. To overcome these challenges, we introduce the Congenital Anomaly Recognition with Diagnostic Images and Unified Medical records (CARDIUM) dataset, the first publicly available multimodal dataset consolidating fetal ultrasound and echocardiographic images along with maternal clinical records for prenatal CHD detection. Furthermore, we propose a robust multimodal transformer architecture that incorporates a cross-attention mechanism to fuse feature representations from image and tabular data, improving CHD detection by 11% and 50% over image and tabular single-modality approaches, respectively, and achieving an F1 score of 79.8 $\pm$ 4.8% in the CARDIUM dataset. We will publicly release our dataset and code to encourage further research on this unexplored field. Our dataset and code are available at https://github.com/BCV-Uniandes/Cardium, and at the project website https://bcv-uniandes.github.io/CardiumPage/