CARDIAG: A Dense Segment Classification Benchmark of Deep Learning Architectures for Coronary Angiography

AI summary

Overview

Research area: Medical computer vision, specifically dense pixel-level segmentation of coronary X-ray angiography images using deep learning.

Technical level: Advanced. The paper assumes familiarity with segmentation architectures (U-Net variants, Transformers, state-space models), loss functions, and clinical scoring conventions such as the SYNTAX score.

Scope: The paper introduces CARDIAG, a multi-center coronary angiography dataset with SYNTAX segment labels, and uses it to benchmark 24 deep learning architectures for dense segment classification, including generalization, data efficiency, calibration, and clinical metadata analyses.

What This Paper Is About

Automatically outlining the coronary arteries and their named segments in angiograms is essential for measuring stenosis severity and plaque burden, but the field has lacked a shared, well-documented evaluation protocol. Existing public data such as ARCADE is limited in size, lacks metadata, and raises concerns about label quality and patient-level leakage, making fair comparison between models difficult. This paper builds a new multi-center dataset (CARDIAG) with SYNTAX class labels, catheter masks, uncertainty masks, supplementary frames and acquisition metadata, then uses it to systematically compare 24 deep learning architectures and nominate a strong baseline.

Key Contributions

1. Release of CARDIAG, a multi-center dataset (five medical centers) with SYNTAX segment labels, explicit catheter masks, uncertainty annotations, intermediate frames, and non-sensitive DICOM metadata, split at both center and patient level to prevent leakage. The bundle consists of 644 data points from 114 patients.

2. A broad architectural benchmark of 24 models spanning classic CNNs, Vision Transformers, modern large-kernel CNNs, state-space models, and domain-pretrained baselines, evaluated with metrics covering overlap, topology, diameter error, calibration, and boundary distance.

3. Robustness and efficiency analysis, including leave-one-center-out generalization for the top model in each architecture family, data efficiency curves over training-set fractions of 0.2, 0.4, 0.5, 0.8 and 1.0, and computational cost comparisons.

4. Clinical and acquisition meta-analysis plus uncertainty quantification, examining how patient demographics, vessel side, and projection angle affect performance, and evaluating calibration via inference-time dropout.

Main Findings

• Best single architecture: The ConvNeXt V2 encoder with a DeepLab V3 Plus decoder achieved the highest macro F1, F1(m) = 0.456 ± 0.027, and led every F1 category (critical 0.694 ± 0.038, high 0.464 ± 0.045, low 0.273 ± 0.031) while tying for the best clDice (0.810 ± 0.016).

• Ensembling improves results modestly: Combining ConvNeXt V2, Feature Pyramid Network and Mamba U-Net raised macro F1 to 0.479 ± 0.026 with hard voting, 0.475 ± 0.025 with soft voting, and 0.468 ± 0.025 with entropy-weighted voting. Voting strategy was not decisive; entropy weighting was slightly worse.

• State-space models are competitive but not dominant: VM-UNet (0.438 ± 0.023), FPN (0.437 ± 0.025), DeepLab V3+ (0.432 ± 0.024), SegFormer Big (0.428 ± 0.025) and the Mamba family (Swin-UMamba 0.427, Mamba-UNet 0.427) clustered closely together, performing in the same range as strong classical baselines rather than surpassing them.

• Transformers did not beat the leading modern CNNs: Scaled-up SegFormer reached 0.428 ± 0.025, while SwinUNETR fell to 0.372 ± 0.024, near the lower end of the benchmark.

• Domain-specific pretraining gave mixed results: XRayVision weights improved U-Net macro F1 from 0.350 ± 0.022 to 0.360 ± 0.024 with a notable clDice gain, but RadImageNet weights on DeepLab slightly underperformed standard DeepLab (0.407 ± 0.024 vs. 0.417 ± 0.025).

• A specific architectural incompatibility surfaced: RepLKNet with a DeepLab head scored 0.280 ± 0.017, worse than basic U-Net, but pairing the same backbone with a Hamburger decoder recovered performance to 0.425 ± 0.024.

• Diameter estimation favored Mamba-UNet: It achieved the lowest diameter error, DE = 2.176 ± 0.111, whereas boundary accuracy was best for ConvNeXt FPN (HD95 = 52.108 ± 4.749).

• Scaling up did not guarantee accuracy: The heaviest models, such as ConvNeXt UPerNet (0.315 ± 0.022) and ConvNeXt UPerNet-class compute, produced sub-par metrics, while the best model sat in the middle of the compute range. Mamba U-Net was described as exceptionally efficient, combining low diameter error with low cost.

• Models were well calibrated: Calibration scores (AUC-ROC of uncertainty as an error predictor) were high across the board, with the strongest values near 0.981 ± 0.003 for the ensembles, and the weakest listed at 0.937 ± 0.007 for DeepLab.

• Annotation quality was measured: Mean Dice agreement for full vascular tree segmentation was 0.87 across a randomly selected subset of 10 images, rising to 0.9072 after uncertainty masks excluded ambiguous distal fragments. Fleiss' kappa for stenosis labeling was 0.7611.

• Dataset imbalance is documented: There are more left-side than right-side samples, distal segments appear less often (segment 15 only for left-dominant hearts), the sample skews toward ages 50-80, and men comprise 58 percent of the cohort versus 40 percent women.

Methodology in Plain English

The team collected angiographic images from 114 anonymized patient examinations performed on a Siemens Axiom Artis system in cardiology centers in Northern Poland. Labeling ran from 2019 to 2021 using in-house AngioTagger software, carried out by three interventional cardiology specialists with at least three years of experience, after training sessions and consensus meetings to align criteria. Annotations were generated automatically first and then manually corrected. Task definitions followed the SYNTAX score convention, and each pixel is assigned one of 26 classes or background, giving a dense prediction problem with a long-tailed class distribution.

To keep the evaluation honest, the data was grouped first by medical center and then by patient ID, and whole patients were assigned to training (70 percent), validation (10 percent) and test (20 percent) sets, so no patient appears in more than one split and center proportions are preserved.

Every architecture was trained identically, using a combined cross-entropy plus multi-class Dice loss, the AdamW optimizer with beta values of 0.5 and 0.999, weight decay of 10^-2 and learning rate of 10^-4, cosine annealing with warm restarts (T0 = 20, multiplier 1, minimum learning rate 10^-6), exponential moving average of weights with alpha_max = 0.999, and early stopping with patience of 100 epochs up to a maximum of 700 epochs.

Evaluation used a family of metrics: several variants of F1 (overall macro, critical, high and low diagnostic importance), 95th-percentile Hausdorff distance, clDice for centerline topology, a newly proposed diameter error computed at skeleton pixels, and a calibration score defined as the AUC-ROC of the uncertainty map as a classifier of model errors, with uncertainty obtained from five dropout-enabled inference samples. Additional experiments covered leave-one-center-out cross-validation (300 epochs for the top model per family) and data efficiency training at fixed iteration budgets of 8,000 with early stopping disabled.

Why This Matters

Impact on research: The paper directly addresses the reproducibility gap in SYNTAX segmentation, where many prior studies relied on unreleased in-house data. By publishing the dataset with metadata, patient and center identifiers, uncertainty masks and supplementary frames, and by evaluating a wide cross-section of architecture families under one protocol, it gives future work a common yardstick. It also introduces a diameter-error metric aimed at clinical relevance rather than pure overlap.

Real-world applications:

• Automated quantification of lumen geometry and stenosis severity to support catheterization lab decisions.
• 3D reconstruction of coronary arteries from 2D projections using the released SID, SOD and angle metadata.
• Frame interpolation across supplementary frames, using up to 30 preceding and 10 following images per projection.
• Robotic or algorithmic catheter localization via the released catheter binary masks, and lesion detection workflows.

Industry relevance: Medical device and imaging software vendors build quantitative coronary angiography tools; a standardized, calibrated benchmark with uncertainty estimates supports regulatory arguments about reliability, and the finding that mid-sized models match the largest ones has direct implications for deployment cost on hospital hardware.

Future Directions

• Extend the benchmark beyond SYNTAX segmentation to the other tasks the dataset supports, such as stenosis detection and frame interpolation, which the authors explicitly list as possible uses.
• Investigate why certain encoder-decoder pairings fail, such as RepLKNet with DeepLab, to establish general rules for architecture compatibility rather than case-by-case fixes.
• Improve generalization evaluation, since the leave-one-center-out analysis was restricted to only the top model from each family and the dataset covers five centers with a demographically narrow cohort.
• Develop better uncertainty metrics, as the authors themselves express doubt that the calibration score used here is the best indicator of whether models are genuinely decorrelated, and the findings on high-resolution versus low-resolution features invite further architectural study.

Target Audience

Researchers in medical image analysis and computer vision working on segmentation benchmarks; clinical scientists and interventional cardiologists interested in automated coronary assessment; engineers building or validating angiography analysis software; and machine learning practitioners who want a realistic, leak-controlled medical dataset for testing architectures under distribution shift and class imbalance.

Authors’ abstract

Accurate pixel-level classification of coronary angiograms is critical for cardiovascular disease assessment, yet the field lacks standardized evaluation protocols. In this work we demonstrate a new benchmark for the assessment of deep learning models which densely classify pixels of coronary angiograms to one of SYNTAX classes (or background). The evaluation covers 24 distinct architectures starting with classic convnets to recent state-space-based vision algorithms. We release CARDIAG - a multi-center, multi-label dataset which we carefully split to reliably compute metrics, accounting for diameter error, overlap, centerline quality and calibration. The data contains SYNTAX labels, binary, uncertainty and segmentation masks as well as intermediate frames together with the selected non-sensitive DICOM metadata. From the multitude of algorithms, we nominate ConvNeXt V2 encoder with DeepLab V3 Plus decoder as the best performing, achieving macro $F_1=0.456$, which we then ensemble with Mamba U-Net and Feature Pyramid Network, for an increased $F_1=0.479$. We demonstrate all the architectures to be well calibrated and determine the generalization of the top 5 methods, together with the data efficiency of these architectures. We highlight the importance of both high-resolution and low-resolution features in encoding. We also demonstrate the model correctness in the context of patient demographic, vessel sides and projection angle configurations. Overall the released benchmark allows for future studies to robustly and rigorously assess the proposals, not only for SYNTAX segmentation, but lesion detection and many more.