EEG-Bench: A Benchmark for EEG Foundation Models in Clinical Applications

AI summary

Overview

Research area: Machine learning for clinical neuroscience, specifically benchmarking EEG (electroencephalography) foundation models against classical machine learning baselines on diagnostic and event-detection tasks.

Technical level: Intermediate.

Scope: The paper introduces a unified, reproducible benchmark of 11 clinical EEG tasks drawn from 14 public datasets, and uses it to compare three EEG foundation models (BENDR, Neuro-GPT, LaBraM) against two classical pipelines (SVM, LDA) across Parkinson's disease, epilepsy, schizophrenia, OCD, mild traumatic brain injury, artifact detection, sleep staging, and seizure detection.

What This Paper Is About

EEG foundation models are increasingly proposed for clinical use, but they are usually evaluated on a narrow set of tasks (typically brain-computer interface paradigms, TUEG epilepsy, TUAB abnormal EEG detection, and sleep staging), leaving conditions such as mild traumatic brain injury, Parkinson's disease, schizophrenia, and OCD underrepresented. The authors build a single benchmarking framework covering 14 publicly available EEG datasets and 11 diagnostic tasks with minimal preprocessing and standardized subject-level splits, then run a side-by-side comparison of classical baselines and modern foundation models under a strict cross-subject setting. The goal is to find out whether general-purpose EEG models actually generalize better than simple feature-based models in realistic, heterogeneous clinical conditions.

Key Contributions

1. Standardized and extensible framework: A unified platform that parses clinical EEG datasets into a common format, applies minimal preprocessing, and supports evaluation of both classical and modern models, released at https://github.com/ETH-DISCO/EEG-Bench under the GNU GPL v3.0 license or later.
2. Well-defined clinical tasks: A curated suite of 11 diagnostic and event-based tasks spanning epilepsy, Parkinson's disease, schizophrenia, OCD, mild traumatic brain injury, abnormal EEG detection, artifact detection (binary and multiclass), sleep stage classification, and seizure detection, all evaluated in a strict cross-subject setting.
3. Comprehensive performance evaluation: The first side-by-side comparison, to the authors' knowledge, of EEG foundation models (BENDR, Neuro-GPT, LaBraM) and classical baselines (SVM, LDA with Brainfeatures-derived features) across this clinical task suite, including a held-out cross-dataset Parkinson's disease setting.
4. Open data and code release: All prepared data and code are released in a reproducible, plug-and-play format, with automated download scripts for all 14 datasets.

Main Findings

• LaBraM is often strongest, but not always: LaBraM reached a balanced accuracy of 0.838 on abnormal EEG detection (substantially outperforming all other models) and 0.740 on OCD classification, showing it can model long, noisy, heterogeneous clinical recordings.
• Epilepsy is a clear exception: BENDR (0.740) and Neuro-GPT (0.734) strongly outperformed LaBraM (0.565) on epilepsy detection, while SVM and LDA both scored 0.531. The paper attributes this to the highly imbalanced epilepsy dataset and suggests LaBraM may be more sensitive to label imbalance or prone to overfitting.
• A simple baseline won the mild traumatic brain injury task: LDA reached 0.813 balanced accuracy on mTBI, beating every foundation model (LaBraM 0.740, Neuro-GPT 0.646, BENDR 0.640, SVM 0.626). The paper describes this as evidence that in low-data regimes, classical models with strong inductive biases and low capacity can be not only viable but superior.
• Schizophrenia favored a classical model: SVM was best at 0.679, ahead of LDA (0.547), Neuro-GPT (0.545), LaBraM (0.543), and BENDR (0.471). The paper suggests schizophrenia-related EEG markers are subtle and distributed and hard to capture without tailored inductive structure or more specialized training data.
• Sleep staging was a near-failure for foundation models: LDA (0.671) and SVM (0.652) produced stable results around 0.66 balanced accuracy, while LaBraM scored 0.192, BENDR 0.169, and Neuro-GPT 0.166.
• Channel mismatch may explain weak transfer: The paper argues that BENDR's and Neuro-GPT's inability to learn beyond random guessing on sleep stages and seizure tasks suggests these models cannot use previously unseen channels meaningfully, since the datasets in these tasks share no channels with either model. LaBraM, which is more flexible with channel configurations, reached at least a few percentage points above random guessing despite also not being trained on those channels.
• Baselines stayed close on multi-label tasks: Across the multi-label tasks, at least one baseline model was never more than 2% worse than LaBraM on balanced accuracy.
• Remaining task scores (balanced accuracy): PD (All) — Neuro-GPT 0.687, LDA 0.658, LaBraM 0.656, SVM 0.648, BENDR 0.529. PD (Held-Out) — Neuro-GPT 0.673, LaBraM 0.673, LDA 0.654, BENDR 0.615, SVM 0.596. Binary artifact — LaBraM 0.756, SVM 0.745, Neuro-GPT 0.711, LDA 0.705, BENDR 0.535. Multiclass artifact — SVM 0.437, LaBraM 0.430, LDA 0.325, Neuro-GPT 0.226, BENDR 0.192. Seizure — LaBraM 0.588, SVM 0.572, LDA 0.529, BENDR 0.501, Neuro-GPT 0.500.
• Weighted F1 tells a different story on seizure detection: Weighted F1 scores for the seizure task were 0.995 (Neuro-GPT), 0.994 (BENDR), 0.987 (LDA), 0.986 (LaBraM), and 0.974 (SVM), far above the corresponding balanced accuracies of roughly 0.5, reflecting the extreme class imbalance (3,515,547 no-seizure samples vs. 11,525 seizure samples in the task). The paper reports this table without additional commentary.
• Compute cost of the evaluation: Each experiment was repeated five times with different random seeds, requiring 270 hours on an A100 GPU and 16 AMD EPYC 7742 CPUs in total.

Methodology in Plain English

The authors gathered 14 publicly available EEG datasets that were collected in medical contexts for diagnosis or monitoring, covering a wide range of hardware (categories in the paper include Unknown, Easycap, actiCAP, NeuroScan SynAmps2, and Brain Vision system), channel counts (3 to 64 for the clinical research datasets, 18–19 for the Temple University Hospital and CHB-MIT datasets), sampling rates, recording lengths, and subject demographics spanning ages 1 to 80.

From these datasets they defined 11 tasks. Diagnostic tasks (abnormal vs. normal, epilepsy vs. no epilepsy, Parkinson's vs. control, OCD vs. control, mTBI vs. control, schizophrenia vs. control) operate on full-length recordings. Event-based tasks (seizure detection, binary artifact detection, multiclass artifact detection, sleep stage classification) operate on shorter segments. All tasks are split by subject, never by sample, so a model is always tested on people it never saw during training. Because multiple Parkinson's datasets exist, the authors also test cross-dataset transfer on a held-out Parkinson's dataset.

Two families of models are compared. The classical baselines are Linear Discriminant Analysis and Support Vector Machines operating on handcrafted features from the Brainfeatures toolbox: spectral power in standard bands (delta 0.5–4 Hz, theta 4–8 Hz, alpha 8–13 Hz, beta 13–30 Hz, gamma 30–100 Hz), statistical moments, complexity measures such as entropy and fractal dimension, and continuous/discrete wavelet and discrete Fourier transforms. Signals for these baselines are resampled to 200 Hz, restricted to the subset of channels common across datasets, and truncated to a consistent length.

The foundation models are BENDR, Neuro-GPT, and LaBraM, which learn from raw signals. All three receive signals bandpass filtered to 0.1–75 Hz, notch filtered to suppress line noise, and resampled to 200 Hz. BENDR and Neuro-GPT require a fixed channel layout, so the authors keep the pretrained channels, zero-pad missing ones, and map datasets with incompatible layouts where necessary — in the extreme case where no channels overlap at all (Sleep-Telemetry and CHB-MIT), dataset channels are assigned to arbitrary model input channels. Each released model gets a linear classification head and is fine-tuned; for Neuro-GPT only the encoder is fine-tuned, following the authors' own findings. LaBraM-Base is fine-tuned end-to-end because it tolerates heterogeneous channel configurations. Because clinical recordings run from minutes to hours and exceed these models' input limits of 4 to 60 seconds, each recording is split into non-overlapping chunks, encoded separately, and the resulting embeddings are averaged before classification.

Performance is reported as balanced accuracy and weighted F1 on fixed test splits, chosen for robustness to class imbalance and for wide adoption in the EEG literature respectively. Results are means over five random seeds. The benchmarking tool is run from a conda environment via a single command, and the paper documents how to add new datasets, tasks, and models through base classes and abstract interfaces.

Why This Matters

Impact on research. The paper challenges the assumption that larger pretrained EEG models automatically dominate in clinical settings. Its head-to-head results show that classical models remain competitive or superior on several tasks (mTBI, schizophrenia, sleep staging) and that foundation models can fail badly outside the channel layouts and data regimes they were pretrained on. By releasing prepared data, standardized subject splits, and extensible base classes, it gives the field a common yardstick instead of scattered per-paper evaluation protocols.

Real-world applications:

• Hospital EEG monitoring: Automated flagging of abnormal EEG in long inpatient recordings, the setting behind TUAB (2,383 subjects, 2,993 recordings, 47.5 days of data).
• Seizure detection: Continuous detection in long ambulatory or ICU recordings, as represented by CHB-MIT (23 subjects, 686 recordings, 41 days of data).
• Sleep staging: Automatic scoring of sleep phases, as represented by the Sleep-Telemetry dataset (22 subjects, 44 recordings), where the paper shows foundation models currently lag well behind simple baselines.
• Artifact cleanup: Detecting eye movement, muscle, electrode, chewing, and shivers artifacts so that clinicians and downstream models see cleaner signal, as represented by TUAR.
• Support for neurological and psychiatric diagnosis: Parkinson's disease, OCD, schizophrenia, and mild traumatic brain injury classification, which the paper notes are underrepresented in prior evaluation efforts.

Industry relevance. The results are directly relevant to developers of EEG foundation models, who now have a public benchmark that can reveal channel-transfer and class-imbalance weaknesses before deployment; to medical device and clinical decision-support companies weighing a lightweight, interpretable classical pipeline against a heavier deep model; and to anyone who needs reproducible, licensed evaluation infrastructure, since the benchmark code is released under the GNU GPL v3.0 license or later and includes automated downloads for all datasets, with TUAB, TUEP, and TUAR requiring acceptance of a data use agreement through the NEDC portal.

Future Directions

1. Integrate task-specialized models. The authors note that the current model suite includes only a few standard ML pipelines and foundation models, and that adding specialized models built for individual tasks such as seizure detection, sleep staging, and schizophrenia diagnosis would give a fuller picture of capabilities and limitations.
2. Expand datasets, tasks, and models. The authors plan to continuously grow the benchmark and invite community contributions of evaluations of emerging EEG models under the same unified setup.
3. Diagnose channel-transfer limits. The observation that BENDR and Neuro-GPT perform near random guessing on tasks whose channels are unseen raises the open question of how to make pretrained EEG models usable on arbitrary electrode layouts, which the paper identifies as a key differentiator for LaBraM.
4. Address low-data and class-imbalance regimes. The mTBI result (LDA at 0.813 beating all foundation models) and the epilepsy result (BENDR and Neuro-GPT well above LaBraM) point to open questions about regularization and imbalance handling for high-capacity models on small clinical cohorts and skewed label distributions.

Target Audience

This paper is most useful to machine learning researchers working on biosignal and EEG foundation models, clinical neurophysiologists and neurologists interested in what current automated decoding can and cannot do, benchmark and reproducibility engineers who need standardized subject-level evaluation infrastructure, and industry practitioners in medical devices or clinical decision support deciding whether a classical pipeline or a pretrained deep model is the better fit for a given diagnostic task.

Authors’ abstract

We introduce a unified benchmarking framework focused on evaluating EEG-based foundation models in clinical applications. The benchmark spans 11 well-defined diagnostic tasks across 14 publicly available EEG datasets, including epilepsy, schizophrenia, Parkinson's disease, OCD, and mild traumatic brain injury. It features minimal preprocessing, standardized evaluation protocols, and enables side-by-side comparisons of classical baselines and modern foundation models. Our results show that while foundation models achieve strong performance in certain settings, simpler models often remain competitive, particularly under clinical distribution shifts. To facilitate reproducibility and adoption, we release all prepared data and code in an accessible and extensible format.