BRIDGE-EEG: Bridging Self-Supervised Pretraining and Efficient Deployment for Cross-Dataset EEG Classification

AI summary

Overview

Research area: Efficient deep learning for electroencephalography (EEG) — specifically the intersection of self-supervised pretraining, knowledge distillation, and edge-hardware deployment for brain-computer interfaces.

Technical level: Intermediate. Readers should be comfortable with convolutional neural networks, self-supervised contrastive learning, knowledge distillation, and basic signal-processing concepts such as the Short-Time Fourier Transform. The EEG-specific material is explained adequately for non-neuroscientists.

Scope: The paper presents and empirically validates a three-stage pipeline (unified preprocessing, SimCLR pretraining, knowledge distillation) that produces compact EEG classifiers matching or exceeding much larger foundation models on several benchmarks while being measurably cheaper to run on GPU, CPU, and edge hardware.

What This Paper Is About

EEG analysis faces a trade-off: large "foundation models" pretrained on thousands of hours of brain data generalize well across patients and tasks but are far too big to run on wearables, while small task-specific models deploy easily but must be retrained from scratch for every new dataset and rarely transfer. BRIDGE-EEG closes that gap by pretraining one large model on unlabeled EEG from five heterogeneous datasets, then compressing it into models small enough for edge devices without giving up most of the accuracy. The authors also do something the field has largely skipped — they measure actual energy per inference on real hardware rather than reporting theoretical FLOP counts.

Key Contributions

1. An SSL-to-KD pipeline for multi-task EEG classification. An SE-ResNet18 teacher (11.84 M parameters) pretrained with SimCLR is compressed into SE-ResNet8 (1.56 M) and SE-ResNet4 (0.48 M) students via two distinct distillation strategies — one task-agnostic, one task-specific — yielding a family of deployable models rather than a single architecture.

2. A unified 62-channel spectrogram preprocessing scheme. Heterogeneous recordings with different channel counts, montages, and sampling rates are mapped into a common 62-channel, 128×128 time-frequency representation, allowing one shared backbone to handle fundamentally different acquisition setups.

3. Cross-tier inference profiling with measured energy. Latency and dynamic energy per inference are reported for a server GPU (NVIDIA L4), a desktop CPU (Intel i7-8700), and an edge device (NVIDIA Jetson Orin Nano), extending the evaluation beyond theoretical complexity.

4. A six-benchmark multi-task evaluation spanning three application categories — abnormality detection, motor imagery, and emotion recognition — with honest reporting of where the approach succeeds and where a representation gap remains.

Main Findings

• Compressed students beat far larger foundation models on clinical abnormality detection. On TUAB, the SE-ResNet8 (Strategy B) student reaches 90.35% accuracy with 1.56 M parameters, above REVE (83.15%, 69 M) and LaBraM-Huge (82.58%, 369 M) — roughly 44× and 237× fewer parameters. Even the 0.48 M SE-ResNet4 student reaches 88.00%.

• Distillation sometimes outperforms the teacher. On TUAB (87% teacher vs. 90% student) and SIENA (98.4% vs. 99.5%), students exceed the teacher, consistent with the known regularizing effect of knowledge distillation on binary tasks.

• Emotion recognition results depend heavily on dataset size. On SEED, Strategy B (task-specific distillation) wins at 80.00%, above LaBraM-Base (73.18%) and NeuroLM-XL (60.34%, 1,696 M parameters). On the 7-class, 8-subject EmoEEG set the pattern reverses: Strategy A (77.32%) beats both Strategy B (60.29%) and the teacher (75.89%), suggesting task-specific distillation inherits teacher bias when labeled data is scarce.

• Motor imagery reveals a genuine representation gap. On BCI-IV-2a the best model reaches only 38.66%, well below REVE (63.96%) and NeuroGPT (58.60%). The authors attribute this to the pretraining pool containing only one MI dataset and to severe channel mismatch (22-channel and 3-channel MI datasets mapped into a 62-channel input, leaving most channels zero-filled). On the simpler binary BCI-IV-2b the gap narrows: teacher 70.86%, close to EEGPT (72.12%, 25 M).

• Knowledge distillation preserves the teacher's representations even when the teacher is weak. Across both MI datasets the students stay within a few points of the teacher, indicating that the bottleneck lies in the pretrained features, not in the compression step.

• Accuracy alone can mislead on imbalanced tasks. On TUAB the students have higher accuracy than the teacher but lower macro F1 (0.83 vs. 0.87), implying a majority-class bias. On SEED, Strategy A yields more class-balanced predictions (78% accuracy, 0.77 F1) than Strategy B (80% accuracy, 0.60 F1) despite lower accuracy.

• Energy savings are real and hardware-dependent. On the Jetson Orin Nano, compression cuts energy per inference from 46.67 mJ to 15.64 mJ (3.0×); on the NVIDIA L4 GPU the same comparison yields a 7.9× reduction (24.43 mJ to 3.11 mJ). Latency drops from 12.93 ms to 5.73 ms on the edge device.

• Squeeze-and-excitation blocks cost latency, not power. SE-ResNet8 and plain ResNet8 draw nearly identical dynamic power (1.42 W vs. 1.32 W on GPU) but differ in latency (2.19 ms vs. 1.47 ms), so the SE overhead shows up in energy per sample rather than instantaneous power. Oddly, on the CPU the non-SE ResNet8 was slower (26.35 ms) than both SE-ResNet8 (12.22 ms) and the larger ResNet18 (17.64 ms), showing that realized latency cannot be predicted from parameter count.

Methodology in Plain English

The pipeline has three stages. First, everything is forced into a common shape. Raw EEG from any dataset is mapped onto a standard 62-electrode layout; missing electrodes are filled with zeros rather than interpolated, on the theory that the model's attention mechanism will learn to ignore uninformative channels. Recordings are cut into 4-second windows, filtered to 1–50 Hz, notch-filtered, baseline-corrected, and z-score normalized per channel to erase differences in electrode impedance and amplifier gain. Each window becomes a log-scaled 128×128 spectrogram per channel via the Short-Time Fourier Transform.

Second, the authors train one big encoder — a ResNet18 with squeeze-and-excitation blocks inserted after each residual stage — using SimCLR, a contrastive learning method. Labels are thrown away. Each spectrogram is augmented twice (random channel dropout simulating detached electrodes, time-bin and frequency-bin masking simulating motion artifacts, and additive Gaussian noise), and the network is trained to make the two views of the same signal land close together in embedding space while pushing apart embeddings from different signals. Because every sample in a batch acts as a negative regardless of which dataset it came from, the loss implicitly forces the encoder to discriminate across recording conditions, devices, and populations.

Third, the 11.84 M-parameter teacher is compressed. Strategy A distills the still-label-free teacher into a 1.56 M SE-ResNet8 on unlabeled data, producing a shared starting point that is then fine-tuned per task. Strategy B instead fine-tunes the teacher separately on each labeled task first, then distills each fine-tuned teacher into its own student using the standard Hinton loss (a mix of cross-entropy with ground truth and KL divergence against the teacher's softened outputs). Strategy A is cheaper and more robust when labels are scarce; Strategy B extracts more accuracy when they are plentiful.

Evaluation covers six held-out datasets, none of which overlap with the five pretraining datasets. Inference is profiled at batch size 1 — single-sample, real-time conditions — with idle power subtracted from average inference power to isolate dynamic energy per prediction.

Why This Matters

Impact on research. The paper reframes EEG foundation-model evaluation around measured energy rather than parameter count or FLOPs, which has been the norm. It also demonstrates that a modest pretrained backbone plus distillation can be competitive with models one to three orders of magnitude larger on some tasks, which challenges the assumption that scaling is the only path forward in this domain. The negative motor imagery result is equally useful: it isolates pretraining data diversity and input spatial resolution as the binding constraints, not model size.

Real-world applications:

• Wearable seizure and abnormality monitoring. Continuous at-home EEG screening requires inference on battery-powered hardware; the 15.64 mJ per inference on the Jetson Orin Nano, and the authors' stated ambition of MCU-class deployment, make this plausible.

• Consumer neurotechnology and sleep staging. Emotion recognition and sleep-staging tasks map naturally onto earbuds, headbands, and other consumer form factors where a 0.48 M to 1.56 M parameter model is feasible.

• Remote clinical screening in low-resource settings. A single shared backbone that ingests 3-channel to 64-channel recordings with varying sampling rates reduces the need for site-specific model retraining.

• Assistive and rehabilitation BCIs. Motor-imagery-driven prosthetics and rehabilitation interfaces need low-latency, on-device inference, though this paper shows the MI accuracy is not yet sufficient for that use case.

Industry relevance. The measured cross-tier profiling is directly actionable for hardware teams deciding whether an EEG workload belongs on a GPU, an embedded ARM SoC, or a microcontroller. The finding that non-SE and SE variants have nearly identical power draw but different latency is the kind of detail that matters for thermal and battery budgeting in a shipping product.

Future Directions

• Widening pretraining diversity for motor imagery. The authors explicitly attribute the MI gap to the pretraining corpus containing only one MI dataset (PhysioNet MI) while abnormality and emotion each contribute two. Adding MI data to the pool is the obvious next experiment.

• Input representations that preserve spatial resolution. Mapping 3-channel or 22-channel MI recordings into a 62-channel grid leaves most inputs zero-filled, which likely destroys the fine-grained spatial patterns MI decoding depends on. Alternatives such as channel-aware architectures, graph-based electrode modeling, or learned interpolation are open questions.

• Extending to true microcontroller-class hardware. The paper demonstrates edge-device (Jetson Orin Nano) deployment and motivates MCU deployment but does not demonstrate it. Quantization, pruning, and fixed-point inference are the natural follow-ups.

• Better distillation strategies for small, many-class datasets. The EmoEEG result — where task-specific distillation collapsed to 60% while task-agnostic transfer held at 77% — suggests that teacher soft-label quality degrades sharply with few samples per class. Adaptive or confidence-weighted distillation could address this.

• Closing the accuracy–F1 gap on clinical tasks. Students beat the teacher on TUAB accuracy but trailed on macro F1, meaning they miss more abnormal recordings. For a screening tool that trade-off is unacceptable, and it deserves explicit treatment.

Target Audience

Researchers and engineers working at the intersection of biosignal machine learning and embedded systems — particularly those building EEG-based BCIs, wearable health monitors, or clinical screening tools who need to know what accuracy is achievable at a given energy budget. It is also relevant to the broader efficient-ML community as a case study in combining self-supervised pretraining with knowledge distillation and measuring the result on real hardware. Readers interested in EEG foundation models will find the cross-tier energy profiling and the honest motor-imagery failure analysis the most valuable parts; readers looking for clinical-grade MI decoding will find this paper's results, as the authors themselves acknowledge, insufficient.

Authors’ abstract

The growing use of electroencephalography (EEG) motivates automated analysis that is accurate, transferable, and deployable on constrained hardware. Recent EEG foundation models learn general representations from large-scale pretraining, but their size and computational cost limit edge and wearable deployment. We introduce BRIDGE-EEG, an efficient multi-task EEG classification pipeline that preserves the benefits of pretraining while reducing model size. A unified preprocessing scheme maps heterogeneous recordings with different channel counts, montages, and sampling rates to a device-agnostic 62-channel time--frequency representation. We pretrain an SE-ResNet18 teacher (11.84 M parameters) with SimCLR on unlabeled EEG from five heterogeneous datasets, then compress it into SE-ResNet8 (1.56 M) and SE-ResNet4 (0.48 M) students using task-agnostic and task-specific distillation. We evaluate six benchmarks spanning abnormality detection, motor imagery, and emotion recognition. For abnormality detection and emotion recognition, the students achieve accuracy comparable to or better than several recent EEG foundation models with 10--1,000$\times$ more parameters. Motor imagery shows a remaining representation gap, highlighting the importance of pretraining diversity. Inference profiling on a server GPU, desktop CPU, and NVIDIA Jetson Orin Nano shows up to 3.0$\times$ lower edge energy per inference (15.64 mJ vs. 46.67 mJ). The compact models further support future deployment on MCU-class wearables.