EEG-DLite: Dataset Distillation for Efficient Large EEG Model Training

AI summary

Overview

Research area: Machine learning for physiological signals — specifically data-efficient pre-training of large EEG (electroencephalography) foundation models.

Technical level: Intermediate. The paper sits at the intersection of self-supervised learning, dataset distillation, and brain-computer interfaces. Familiarity with autoencoders, transformers, and core-set selection helps, but the core ideas are accessible.

Scope: This paper introduces EEG-DLite, a data distillation framework that compresses a 2,500-hour EEG pre-training corpus down to 5% of its original size while preserving downstream model performance.

What This Paper Is About

Training large EEG foundation models is expensive because the raw datasets are enormous, noisy, and highly redundant across time and subjects. EEG-DLite addresses this by intelligently selecting a small, diverse, high-quality subset of EEG segments that can substitute for the full dataset during pre-training. The goal is to make EEG foundation model training dramatically cheaper without sacrificing—and sometimes even improving—generalization to downstream tasks.

Key Contributions

1. First EEG-specific data distillation framework. EEG-DLite is the first framework designed to distill pre-training data for large EEG foundation models, achieving comparable or better performance using only 5% of the original 2,500-hour corpus.

2. First head-to-head comparison of generative vs. selection-based distillation for EEG. The authors benchmark synthetic sample generation (M3D) against core-set selection and show that generative methods struggle with EEG's low signal-to-noise ratio and high dimensionality, whereas selection-based distillation works well.

3. A three-stage pipeline combining SSL compression, outlier removal, and diversity sampling. The framework uses a self-supervised multi-view autoencoder to embed EEG segments, HBOS-based outlier filtering to prune artifacts, and greedy k-center selection to maximize representativeness.

4. Systematic analysis of how pre-training data quantity affects generalization. The paper reports downstream performance across distillation ratios of 1%, 5%, 10%, and 25%, showing that 5% distilled data matches what random sampling needs roughly 25% to achieve.

Main Findings

• 5% is enough. Training on 5% of the dataset selected by EEG-DLite achieves performance comparable to, and in some cases better than, training on the full 2,500 hours. Random sampling requires about 25% to reach a similar level.

• Massive compute savings. GPU pre-training time drops from 30 hours to 2 hours under identical hardware (four NVIDIA RTX 4090 GPUs), a 15× reduction.

• Outlier removal matters. Removing the top ~0.05–1% of samples with the highest out-of-distribution scores improves downstream accuracy; manual inspection showed these are often noise or artifact-contaminated segments.

• Generative distillation fails on EEG. The M3D synthesis baseline underperformed even random sampling and produced unrealistic signals with plateaus, flat transitions, and blocky repetitive patterns. It also became computationally infeasible at 25% ratios due to GPU memory demands.

• SSL beats PCA for embeddings. Self-supervised autoencoder representations produced more stable and discriminative latent spaces than incremental PCA, leading to better distilled subsets on TUEV, MoBI, and TUAB.

• Distilled models can surpass full-data models. On TUEV, MoBI, and TUAB, models pre-trained on distilled subsets sometimes outperformed those trained on the complete dataset, suggesting careful sample selection can beat brute-force scaling.

• Subject imbalance emerges after diversity sampling. After k-center selection, some subjects contribute far more segments than others (visualized on SEED and SEED-VII), reflecting inherent inter-subject variability in EEG.

Methodology in Plain English

The researchers start with EEG segments that are high-dimensional (many channels × time points) and noisy, making direct selection expensive and unreliable. Their pipeline has three stages:

1. Compress. They train a lightweight autoencoder with self-supervised objectives to map each EEG segment—plus its magnitude and phase information from a Fourier transform—into a compact 64-dimensional vector. The autoencoder uses a reconstruction loss (to preserve signal content) and an inter-instance discrimination loss (to encourage diverse features). This compresses millions of segments into something tractable.

2. Filter. In the compressed space, they compute an out-of-distribution score for each segment using histogram-based outlier scoring (HBOS). The top τ% (typically 0.05–1%) get discarded as noise, artifacts, or corrupted recordings.

3. Select. From the surviving pool, they run a greedy k-center algorithm that picks points to minimize the maximum distance from any segment to its nearest selected center. This guarantees the final subset is spatially spread out—i.e., diverse and representative of the original distribution.

The resulting subset is then used to pre-train a LaBraM-base foundation model (12 transformer layers, ~400M parameters) and fine-tune on TUEV, TUAB, SEED-V, and MoBI. A separate pilot study uses EEGNet on SEED in a supervised cross-subject setting.

Why This Matters

Impact on research. This is the first systematic study of data distillation for EEG foundation models. It demonstrates that data quality and diversity matter more than raw volume for generalization—a finding that shifts the field's focus from "collect more data" toward "curate better data." It also provides the first evidence that generative distillation, which works in vision, does not transfer well to physiological signals.

Real-world applications:

• Clinical EEG diagnostics: Hospitals could pre-train diagnostic models on far smaller curated datasets, making deployment feasible in resource-limited settings.
• Brain-computer interfaces: Faster, cheaper model iteration enables assistive technologies for neurorehabilitation and motor-impaired users without massive compute budgets.
• Cognitive and emotion monitoring: Wearable EEG devices could benefit from compact, efficient foundation models that run on edge hardware after pre-training on distilled data.
• Sleep and neurological disorder screening: As shown in the UMAP analysis on SEED-sleep, distilled subsets preserve the latent structure needed for sleep staging and pathology detection.

Industry relevance. EEG-DLite is model-agnostic and reduces GPU hours by 15×, which directly lowers the cost of developing and iterating on EEG foundation models. For companies building neurotechnology products, this makes architecture search and hyperparameter tuning practical rather than prohibitively expensive.

Future Directions

• Subject-aware pre-training. The observed subject imbalance after diversity sampling suggests that explicitly balancing or stratifying by subject could improve fairness and generalization across individuals.

• Integrating distillation with downstream objectives. The current pipeline is task-agnostic; combining it with task-aware or label-aware selection could further shrink datasets for specific clinical applications.

• Extending to other physiological signals. The framework should transfer to MEG, ECG, EMG, and multimodal biosignal datasets with similar noise and redundancy profiles.

• Understanding what makes a sample informative. The paper shows some segments are more valuable than others but does not fully characterize why—connecting selection criteria to neural signal properties could yield deeper theoretical insight.

Target Audience

This paper is most useful for machine learning researchers working on foundation models or dataset distillation, EEG and BCI researchers seeking efficient pre-training strategies, and industry practitioners building neurotechnology products under compute or data constraints. Clinical researchers interested in deploying EEG models with limited infrastructure will also find the practical compute savings compelling.

Authors’ abstract

Large-scale EEG foundation models have shown strong generalization across a range of downstream tasks, but their training remains resource-intensive due to the volume and variable quality of EEG data. In this work, we introduce EEG-DLite, a data distillation framework that enables more efficient pre-training by selectively removing noisy and redundant samples from large EEG datasets. EEG-DLite begins by encoding EEG segments into compact latent representations using a self-supervised autoencoder, allowing sample selection to be performed efficiently and with reduced sensitivity to noise. Based on these representations, EEG-DLite filters out outliers and minimizes redundancy, resulting in a smaller yet informative subset that retains the diversity essential for effective foundation model training. Through extensive experiments, we demonstrate that training on only 5 percent of a 2,500-hour dataset curated with EEG-DLite yields performance comparable to, and in some cases better than, training on the full dataset across multiple downstream tasks. To our knowledge, this is the first systematic study of pre-training data distillation in the context of EEG foundation models. EEG-DLite provides a scalable and practical path toward more effective and efficient physiological foundation modeling. The code is available at https://github.com/t170815518/EEG-DLite.