SignalMC-MED: A Multimodal Benchmark for Evaluating Biosignal Foundation Models on Single-Lead ECG and PPG

AI summary

Overview

Research area: Machine learning for biosignals — specifically benchmarking foundation models on single-lead electrocardiogram (ECG) and photoplethysmogram (PPG) data for clinical prediction.

Technical level: Intermediate. Familiarity with foundation models, linear probing, and common evaluation metrics (AUROC, Pearson correlation) helps, but the benchmark framing and task descriptions are readable without deep domain expertise.

Scope: The paper introduces SignalMC-MED, a benchmark built from 22,256 emergency department visits with synchronized 10-minute ECG and PPG recordings, and uses it to systematically evaluate seven foundation models plus hand-crafted domain features across 20 clinical prediction tasks.

What This Paper Is About

Most biosignal foundation models are evaluated on short (typically 10-second), single-modality signals, which does not reflect how clinicians actually monitor patients — continuously, over minutes to hours, with ECG and PPG recorded together. This paper addresses that gap by releasing a standardized benchmark for long-duration, synchronized ECG + PPG signals and using it to run a controlled head-to-head comparison of general time-series models, ECG-specific models, a PPG-specific model, a multimodal model, and traditional hand-crafted signal features.

Key Contributions

1. A large-scale multimodal benchmark. SignalMC-MED is derived from the MC-MED dataset and contains 22,256 adult emergency department visits with synchronized 10-minute single-lead ECG and PPG signals, drawn from 17,841 unique patients. It uses chronological train/val/test splits (15,832 train, 2,030 val, and 1,995 test visits) with no patient appearing in more than one split.

2. A 20-task clinical suite. The benchmark spans five categories: age regression, sex classification, emergency department disposition classification, eight laboratory value regression tasks, and nine prior ICD-10 diagnosis classification tasks (including atrial fibrillation, heart failure, cardiac device implantation, chronic kidney disease, diabetes mellitus, sleep-related breathing disorder, anemia, respiratory compromise, and venous thromboembolism).

3. A systematic head-to-head model comparison. The authors evaluate two general time-series foundation models (MOMENT, Chronos-Bolt), four unimodal biosignal foundation models (D-BETA, ECGFounder, xECG, PaPaGei), one multimodal biosignal foundation model (CSFM), and hand-crafted ECG and PPG domain features — each in ECG-only, PPG-only, and ECG + PPG settings. None were pretrained on MC-MED.

4. Empirical guidance on fusion, scale, and duration. The paper quantifies how modality specialization, multimodal fusion, signal length, model size, and hand-crafted features each affect downstream performance under a frozen, linear-probing protocol.

Main Findings

• Domain-specific models beat general time-series models. CSFM-base achieves the best overall performance and mean rank, while xECG (an ECG-only-pretrained model) is consistently second. On the test set with ECG + PPG inputs, CSFM-base reaches Pearson correlation 0.818 ± 0.002 for age, AUROC 0.823 ± 0.002 for sex, 0.721 ± 0.005 for ED disposition, 0.374 ± 0.003 for laboratory regression, and 0.735 ± 0.003 for ICD-10 classification.

• Multimodal ECG + PPG fusion helps almost universally. Averaging independently extracted ECG-only and PPG-only representations (late feature-level fusion) yields consistent improvements over unimodal inputs across tasks, train percentages, and signal lengths — even for models pretrained only on ECG.

• Simple late fusion slightly outperforms CSFM's internal multimodal mechanism. The authors attribute this potentially to the relative scarcity of paired ECG + PPG segments versus ECG-only data in CSFM's pretraining, and to the absence of explicit cross-modality alignment losses.

• A PPG-specific model performs poorly, including on PPG. PaPaGei ranks last for ECG-only (mean rank 23.8), second-last for PPG-only (20.0), and last for ECG + PPG (20.6). The authors note it was pretrained on 20M 10-second PPG segments from only 13.5K subjects, whereas CSFM, ECGFounder, and xECG used data from more than 1.5 million individuals, suggesting subject-level diversity matters more than segment count.

• Hand-crafted features remain a strong baseline. ECG domain features rank fourth across all modalities and outperform multiple foundation models, and concatenating them with learned FM representations consistently improves ECG performance — though gains are less consistent for PPG.

• Longer signals beat bigger models. Using the full 10-minute signal consistently outperforms shorter segments (10 seconds to 5 minutes), while larger variants within a model family (e.g., CSFM-tiny/base/large) do not reliably outperform smaller ones. Projecting features to 256 dimensions with PCA preserves these trends, so the result is not explained by feature dimensionality alone.

• Performance tracks physiological signature strength, not label frequency. Cardiac device detection (2.4% test prevalence) and atrial fibrillation detection (7.6%) achieve among the highest AUROC values, while more prevalent conditions such as anemia (11.1%) and respiratory compromise (13.1%) show more modest performance. The authors note anemia has no specific electrocardiographic signature.

• Latent structure aligns with clinical labels. UMAP projections show clearer clustering by age group and atrial fibrillation label for CSFM-base, xECG-10min, and ECGFounder than for general time-series foundation models or PaPaGei.

Methodology in Plain English

The researchers took the MC-MED dataset — 118,385 emergency department visits from 70,545 unique adult patients at Stanford Health Care between 2020 and 2022 — and identified 54,024 visits with both ECG and PPG recordings. Of these, 24,567 had at least 10 minutes of temporally overlapping signals, and after preprocessing and quality control, 22,256 visits remained.

Each 10-minute signal is resampled to the sampling frequency each model expects (250 Hz as the default where applicable), cleaned with NeuroKit2, and Z-score normalized. Signals are chopped into non-overlapping 10-second segments. A frozen foundation model — never fine-tuned — turns each segment into a feature vector, and those vectors are averaged into a single visit-level representation. For multimodal runs, the ECG-only and PPG-only visit vectors are simply averaged. Linear models are then trained on top: ridge regression for continuous targets and regularized logistic regression for binary classification, with one regularization hyperparameter per task type chosen on the validation split.

To test data efficiency, the downstream models are trained on 10%, 25%, 50%, and 100% of the training visits, with training visits sampled with replacement and the whole downstream training plus hyperparameter selection repeated five times. Regression is scored by Pearson correlation and classification by AUROC. Models are ranked by mean rank across the five aggregated task categories.

Why This Matters

Impact on research. Prior biosignal benchmarks focused largely on unimodal, 10-second ECG or PPG segments. SignalMC-MED supplies a controlled, reproducible setting for long-duration, synchronized multimodal evaluation, and its findings challenge two common assumptions: that bigger models are better, and that a purpose-built multimodal architecture necessarily beats simple feature averaging. The benchmark is also constructed so that no evaluated model was pretrained on MC-MED, avoiding pretraining–evaluation overlap.

Real-world applications:

• Continuous bedside monitoring in emergency departments, where ECG and PPG are already captured together by patient monitors.
• Triage support, via ED disposition prediction (discharge versus inpatient, observation, or ICU admission).
• Non-invasive estimation of laboratory values from signals, such as potassium, calcium, eGFR, glucose, hemoglobin, albumin, BUN, and sodium.
• Opportunistic screening for prior diagnoses such as atrial fibrillation, heart failure, and cardiac device implantation from routine monitoring data.

Industry relevance. The results give concrete deployment guidance for anyone building biosignal models: prefer domain-specific pretraining over general time-series pretraining, use both modalities when available, favor longer signal windows over larger parameter counts, and keep hand-crafted ECG features as a validation baseline. The finding that a 6M-parameter PPG model underperformed models pretrained on far more subjects also signals that pretraining data diversity, not just scale, drives generalization.

Future Directions

• Fine-tuning and nonlinear adaptation. All results come from frozen representations with linear downstream models; the authors note outcomes may differ under end-to-end fine-tuning or nonlinear adaptation.
• Longer time horizons. The benchmark uses only the first 10 minutes of recording. Extending to hours or days per patient would test long-term temporal modeling for continuous monitoring.
• More models and better fusion. Seven representative foundation models plus domain features were compared; many additional models could be included, and more advanced multimodal fusion and temporal aggregation strategies remain to be explored.
• Cross-institution generalization. Although chronological splits prevent temporal leakage and patient overlap, all data come from a single healthcare system, so generalizability across institutions, acquisition devices, and patient populations is untested — as is robustness under distribution shift and prospective clinical integration.

Target Audience

Machine learning researchers working on biosignals and clinical foundation models; biomedical engineers designing ECG or PPG pipelines; clinicians and health informatics teams evaluating whether these models are ready for bedside use; and benchmark or evaluation researchers interested in how frozen-feature linear probing, modality fusion, signal duration, and model scale interact on realistic clinical data.

Authors’ abstract

Recent biosignal foundation models (FMs) have demonstrated promising performance across diverse clinical prediction tasks, yet systematic evaluation on long-duration multimodal data remains limited. We introduce SignalMC-MED, a benchmark for evaluating biosignal FMs on synchronized single-lead electrocardiogram (ECG) and photoplethysmogram (PPG) data. Derived from the MC-MED dataset, SignalMC-MED comprises 22,256 visits with 10-minute overlapping ECG and PPG signals, and includes 20 clinically relevant tasks spanning prediction of demographics, emergency department disposition, laboratory value regression, and detection of prior ICD-10 diagnoses. Using this benchmark, we perform a systematic evaluation of representative time-series and biosignal FMs across ECG-only, PPG-only, and ECG + PPG settings. We find that domain-specific biosignal FMs consistently outperform general time-series models, and that multimodal ECG + PPG fusion yields robust improvements over unimodal inputs. Moreover, using the full 10-minute signal consistently outperforms shorter segments, and larger model variants do not reliably outperform smaller ones. Hand-crafted ECG domain features provide a strong baseline and offer complementary value when combined with learned FM representations. Together, these results establish SignalMC-MED as a standardized benchmark and provide practical guidance for evaluating and deploying biosignal FMs.