Leveraging Foundational Models and Simple Fusion for Multi-modal Physiological Signal Analysis

AI summary

Overview

Research area: Machine learning for physiological signal analysis — specifically self-supervised foundation models and multi-modal fusion of electrocardiogram (ECG) and electroencephalogram (EEG) signals.

Technical level: Intermediate. Readers should be comfortable with transformer architectures, self-supervised pretraining, and masked reconstruction objectives, though the paper's core idea is explained in accessible terms.

Scope: This paper adapts an existing EEG transformer (CBraMod) into a symmetric ECG encoder pretrained with a dual-masking objective, then fuses ECG and EEG embeddings by simple concatenation for emotion recognition on the DREAMER dataset.

What This Paper Is About

Analyzing ECG and EEG together can reveal complementary information about human health and emotion, but there is very little labeled data where both signals are recorded simultaneously, and the two signal types look and behave very differently. The authors ask whether carefully pretrained, modality-specific foundation-model encoders — combined with a deliberately simple concatenation fusion step — can overcome the shortage of multi-modal supervision. They test this on emotion recognition, predicting valence, arousal, and dominance from paired ECG and EEG.

Key Contributions

1. A symmetric ECG foundation encoder. The authors adapt the CBraMod criss-cross transformer (12 layers, 8 attention heads) — originally built for EEG — to ECG, keeping the architecture identical across modalities so the two representation spaces are comparable.

2. A dual-masking pretraining strategy for ECG. Pretraining masks both individual patches (Bernoulli-sampled) and entire channels (leads), forcing the encoder to learn intra-lead temporal patterns and inter-lead dependencies.

3. A composite reconstruction loss. The paper adds a channel reconstruction loss to the existing patch reconstruction loss, computed over both masked and unmasked channels with equal weighting, summed as 1/2 patch loss + 1/2 channel loss.

4. Evidence that simple fusion suffices. Rather than complex cross-modal alignment, the authors average embeddings channel-wise, concatenate ECG and EEG embeddings, and pass them to a lightweight feed-forward classifier — achieving near state-of-the-art emotion recognition results.

Main Findings

• Near state-of-the-art on DREAMER. On the DREAMER dataset, the model scores Valence Acc. 69.44 / F1 81.14 / AUC 68.69; Arousal Acc. 75.00 / F1 84.21 / AUC 84.79; Dominance Acc. 79.63 / F1 88.30 / AUC 86.69.

• Strong AUC in arousal and dominance. The paper reports the arousal AUC of 84.79 as best among compared methods and the dominance AUC of 86.69 as second-best. (Note that MiniRocket is also listed at 86.69 dominance AUC.)

• Best valence F1 and second-best valence accuracy. The model achieves the best F1-score (81.14) for valence and the second-best accuracy (69.44%), approaching Brant-X while using what the authors describe as a more efficient architecture.

• Pretraining loss drops sharply. Reconstruction MSE during self-supervised ECG pretraining fell from approximately 30 to a final value of 0.1136, indicating progressive learning of ECG structure without labels.

• Pretraining scale. The ECG encoder was pretrained on the PhysioNet/Computing in Cardiology Challenge 2021 dataset, comprising 88,253 12-lead ECG recordings drawn from seven distinct datasets across three continents, with recordings up to 60 seconds and sampling rates of 500–1000 Hz, on dual NVIDIA A100 GPUs.

Methodology in Plain English

The authors treat each physiological signal type as needing its own specialist encoder before any combination happens.

For ECG, they take the CBraMod architecture — a criss-cross transformer with 12 layers and 8 attention heads — and pretrain it from scratch. Signals go through a pipeline: a Butterworth bandpass filter (0.5–40 Hz) to remove baseline wander and high-frequency noise, a notch filter (50 Hz or 60 Hz depending on region) to remove powerline interference, and multi-rate resampling to 100, 200, 500, and 1000 Hz so the model sees many sampling resolutions. Each recording is cut into 200-sample windows and reshaped to (12, 10, 200) — 12 clinical leads, 10 temporal patches, 200 samples per patch — with leads reordered into conventional clinical configuration.

During pretraining, two things get hidden from the model at once. Patch masking hides individual time patches according to a Bernoulli draw. Channel masking independently hides entire leads. The model must reconstruct what was hidden, and the loss averages two equally weighted parts: patch reconstruction (as in CBraMod) and channel reconstruction, which is computed over both fully masked and unmasked channels. The masking ratio values themselves are not reported in the paper.

For the downstream task, the ECG encoder uses the authors' self-supervised weights and the EEG encoder uses CBraMod weights pretrained on the Temple University Hospital EEG Corpus (TUEG). Each encoder produces embeddings that are averaged across channels, concatenated, and fed to three parallel binary classification heads (one each for valence, arousal, dominance).

Evaluation uses the DREAMER dataset: 23 subjects, 18 video clips each, with 14-channel EEG at 128 Hz and 2-channel ECG at 256 Hz. Continuous self-assessed valence, arousal, and dominance scores are discretized at a threshold of 3 into Low (<3) and High (≥3). Training uses a subject-independent 3:1:1 train/validation/test split with the Adam optimizer at a learning rate of 10⁻³ for 10 epochs. Full hyperparameters are reported in an appendix table: 10 epochs, batch size 8, initial learning rate 1×10⁻³, classifier dropout 0.2, Adam optimizer, StepLR scheduler with gamma 0.1 and step size 4 epochs, and BCE loss.

Why This Matters

Impact on research. The paper argues that multi-modal physiological machine learning does not necessarily require elaborate cross-modal alignment machinery. If a strong per-modality foundation encoder plus plain concatenation gets near state-of-the-art results, that redirects effort toward encoder quality and pretraining data rather than fusion complexity. It also extends an existing EEG architecture to ECG in a symmetric way, suggesting the architecture transfers across biosignal types and that representation spaces from different modalities can be combined without explicit alignment.

Real-world applications (as suggested by the paper's framing):

• Healthcare diagnostics, where paired cardiac and neural signals could support richer clinical assessment.
• Brain-computer interfaces, which depend on interpreting neural activity in context.
• Affective computing and emotion-aware systems, the task directly evaluated here.
• Label-efficient deployment in settings where expert annotations are expensive — the pretraining is self-supervised and needs no diagnostic labels.

Industry relevance. The approach is designed for data-scarce regimes: pretraining on unlabeled recordings (88,253 ECG records here) then fine-tuning with limited multimodal labels. That is attractive for wearables and remote monitoring, where unlabeled signal data is abundant but labeled multimodal datasets are rare. The paper also frames the work as scalable and generalizable across patient populations, though it only demonstrates this on one dataset.

Future Directions

1. Richer fusion mechanisms. The authors state that simple embedding concatenation may not fully exploit fine-grained temporal or cross-modal dependencies, and list advanced fusion as future work.

2. Broader downstream tasks. Evaluation is currently limited to emotion recognition; generalization to other physiological applications and more diverse populations remains an open question.

3. Mitigating data scarcity further. The paper proposes weak supervision or synthetic data augmentation as candidate strategies beyond self-supervised pretraining.

4. Unspecified pretraining settings. The masking ratios for patch masking and channel masking are not stated in the provided content, so the sensitivity of pretraining to these hyperparameters is an open question this paper does not answer.

Target Audience

This paper suits researchers and graduate students working on physiological signal processing, self-supervised representation learning, or multi-modal fusion — particularly those interested in adapting foundation-model recipes (masked pretraining, transformer encoders) to biosignals. It is also relevant to applied practitioners in affective computing or clinical monitoring who need label-efficient methods, and to readers following the CBraMod line of work who want to see whether its architecture extends from EEG to ECG.

Authors’ abstract

Physiological signals such as electrocardiograms (ECG) and electroencephalograms (EEG) provide complementary insights into human health and cognition, yet multi-modal integration is challenging due to limited multi-modal labeled data, and modality-specific differences . In this work, we adapt the CBraMod encoder for large-scale self-supervised ECG pretraining, introducing a dual-masking strategy to capture intra- and inter-lead dependencies. To overcome the above challenges, we utilize a pre-trained CBraMod encoder for EEG and pre-train a symmetric ECG encoder, equipping each modality with a rich foundational representation. These representations are then fused via simple embedding concatenation, allowing the classification head to learn cross-modal interactions, together enabling effective downstream learning despite limited multi-modal supervision. Evaluated on emotion recognition, our approach achieves near state-of-the-art performance, demonstrating that carefully designed physiological encoders, even with straightforward fusion, substantially improve downstream performance. These results highlight the potential of foundation-model approaches to harness the holistic nature of physiological signals, enabling scalable, label-efficient, and generalizable solutions for healthcare and affective computing.