SPERA: Spherical Prior EEG Foundation Model with Geometry- and Frequency-Aware Latent Prediction

AI summary

Overview

Research area: Self-supervised representation learning for electroencephalography (EEG), specifically EEG foundation models built on the joint-embedding predictive architecture (JEPA).

Technical level: Advanced. The paper assumes familiarity with transformer attention, masked/latent self-supervised objectives, and spatial statistics on the sphere.

Scope: The paper introduces SPERA, an EEG foundation model that predicts in latent space and injects scalp electrode geometry into attention through a truncated Legendre-polynomial spatial prior, pretrained on roughly 80,000 hours of EEG and evaluated on nine downstream tasks.

What This Paper Is About

EEG data varies enormously across people, recording devices, and electrode montages, which makes it hard to train one general-purpose model that transfers across datasets. Existing EEG foundation models mostly learn by reconstructing the observed signal, but that signal contains muscle and ocular artifacts alongside neural activity, so the model is partly trained to reproduce noise. SPERA instead predicts masked regions in latent space (a JEPA-style objective) and adds explicit geometric and spectral structure so the model can handle arbitrary electrode layouts.

Key Contributions

1. A Legendre-polynomial spherical prior in attention. A learnable pairwise additive bias derived from the cosine similarity of unit-normalized 3D electrode coordinates, expanded in Legendre polynomials, encodes scalp geometry for arbitrary montages. It is implemented in two forms: a pairwise bias for factorized attention blocks and an anchor-based encoding (learnable anchor directions on the unit sphere) for full-attention blocks.

2. A hybrid attention backbone for EEG. Factorized temporal and spatial attention passes are interleaved with periodic full-attention blocks (every fourth block), enabling cross-axis interaction that purely factorized attention does not provide.

3. A relational spectral regularizer (RSReg). Rather than reconstructing absolute spectra, the model aligns the pairwise similarity structure of predicted latents with that of deterministic spectral targets computed from Hann-windowed FFTs over the 1–45 Hz band, using a KL divergence between row-wise softmaxed, L2-normalized similarity matrices.

4. A large-scale pretraining corpus and evaluation. Approximately 80,000 hours of EEG from 29,048 subjects across 106 datasets (2.67 million channel-hours, 9.60 billion tokens; 82.08% clinical, 16.21% cognitive, 1.71% BCI), evaluated on nine downstream tasks that form the intersection of the CBraMod and REVE evaluation suites.

Main Findings

• Highest average balanced accuracy across nine tasks. Under the linear probing followed by fine-tuning (LPFT) protocol, SPERA-Large reaches an average of 0.7665 and SPERA-Base reaches 0.7502, versus 0.7150 for REVE-Base, 0.7102 for CSBrain, 0.6898 for CBraMod, 0.6653 for LaBraM-Base, and 0.6438 for BIOT. SPERA-Base exceeds the strongest baseline (REVE-Base) by 0.035 in average balanced accuracy.

• Statistically significant gains on five tasks. Per-task Welch tests over five seeds, Holm-corrected across the nine comparisons, find significant gains for SPERA-Base on BCIC2020-3, TUEV, BCIC-IV-2a, ISRUC, and TUAB. PhysioNet-MI is borderline (p_Holm = 0.052), and no task shows a significant loss. The largest gains are TUEV (+0.078, 6-class clinical event classification) and BCIC2020-3 (+0.050, 5-class imagined speech). Differences on FACED, MAT, and Mumtaz did not reach significance after Holm correction.

• Parameter efficiency under linear probing. Under LP, average balanced accuracy rises from 0.584 at 18M parameters to 0.684 at 396M. SPERA-Small (48M) performs comparably to REVE-Base (69M), and SPERA-Base (194M) exceeds REVE-Large (408M) by +0.013 despite using less than half the parameters.

• Diminishing returns at scale. The largest LP gain occurs between 48M and 194M (+0.060), coinciding with the largest jump in hidden dimension (512 to 896). The step from 194M to 396M yields only +0.023, which the authors attribute to corpus size rather than capacity.

• Legendre prior beats alternatives. On the SPERA-Small ablation (LP on PhysioNet-MI, ISRUC, TUAB), the Legendre prior improves average accuracy by 0.024 over coordinate embeddings alone (0.6767 vs. 0.6525) and outperforms REVE's spatial encoding (0.6479), a Gaussian bias on geodesic distance (0.6398), and a learnable pairwise MLP (0.6271).

• Hybrid attention outperforms either pure design. Full attention only averages 0.6432, factorized only 0.6617, a variant concentrating full-attention blocks at the end 0.6568, and the periodic hybrid used by SPERA 0.6767.

• RSReg contributes measurable gains. Removing RSReg lowers average accuracy by 0.012, with the largest drop on ISRUC (−0.021).

• Anchor encoding is a throughput decision. Full attention with the pairwise Legendre bias reaches 20,260 tok/s because the N×N score-bias matrix prevents the fused attention path, while anchor encoding reaches 75,769 tok/s at nearly identical matmul FLOPs (653.0 vs. 653.1 GFLOPs). The final design uses anchor encoding in full-attention blocks and pairwise bias in factorized layers, giving 1.78× higher throughput (92,866 vs. 52,263 tok/s) and 14% lower peak memory (1.95 vs. 2.26 GB) at a cost of 0.005 average balanced accuracy.

Methodology in Plain English

The researchers treat an EEG recording as a grid of tokens, one per (electrode, one-second patch) pair, sampled at 200 Hz. Instead of masking parts of the raw signal and asking the model to reconstruct them, they split tokens into context and target sets, encode the context with a student network, and ask a small cross-attention predictor to guess what a momentum-updated teacher network produces at the masked positions. The training signal is an ℓ1 distance in latent space, so the model never has to reproduce artifacts it cannot separate from neural activity.

Masking is structured rather than random. Temporal masks preserve contiguous blocks to keep the context temporally coherent, and spatial masks remove a random electrode's k-nearest neighbors by 3D Euclidean distance, so the model must infer activity from nearby electrodes.

Electrode geometry is handled by turning each electrode's 3D coordinate into a unit vector and expanding pairwise cosine similarities in Legendre polynomials with learnable coefficients. Because that pairwise form is expensive in dense attention, full-attention blocks instead represent each electrode by its Legendre-expanded similarity to a set of learnable anchor directions and inject that feature into queries and keys.

Finally, a relational spectral regularizer computes the log-power spectrum of each 1-second raw patch, and forces the similarity structure among predicted latents to match the similarity structure among spectra. The total loss combines the JEPA loss with this spectral term. Four model sizes were pretrained (18M, 48M, 194M, 396M), and evaluation used two protocols: linear probing with a frozen encoder, and linear probing followed by fine-tuning, measured by balanced accuracy over five random seeds.

Why This Matters

Research impact. The paper argues that reconstruction targets defined on the observed EEG signal inevitably include muscle and ocular artifacts, and shows that a latent-prediction objective with explicit geometry and frequency structure outperforms reconstruction-based EEG foundation models on a common nine-task benchmark. It also provides a controlled ablation of geometry encodings and an explicit accuracy-versus-throughput accounting for two implementations of the same spherical prior, which is unusual detail for spatial-bias design in this literature.

Real-world applications:

• Clinical EEG interpretation, including abnormality detection (TUAB) and clinical event classification (TUEV), where SPERA shows the largest significant gain (+0.078 on TUEV).
• Sleep staging from polysomnography-style montages (ISRUC), where removing the spectral regularizer caused the largest accuracy drop.
• Brain-computer interfaces, including motor imagery across different electrode layouts (BCIC-IV-2a at 22 channels and PhysioNet-MI at 64 channels) and imagined speech classification (BCIC2020-3).
• Cognitive and affective monitoring, such as emotion recognition (FACED) and mental stress detection (MAT), though gains on these three tasks did not reach significance after correction.

Industry relevance. A single backbone that transfers across 106 pretraining sources and arbitrary electrode montages reduces the need to train separate models per device or dataset, and the reported parameter efficiency under linear probing matters for deployment settings where only a linear head can be trained on small labeled sets. The efficiency profiling (throughput, peak GPU memory, FLOPs) makes the practical cost of the geometric prior explicit for teams with fixed compute budgets.

Future Directions

• Scaling beyond 396M parameters. The authors did not investigate billion-parameter scale due to limited computational resources, and they note that the diminishing returns beyond 194M suggest further gains may require expanded data rather than parameters alone. They did not systematically disentangle the contributions of corpus size, model capacity, and architectural priors.

• Extending the spherical prior to intracranial recordings. The prior assumes surface electrodes; the authors state that intracranial extension would require incorporating radial information.

• Moving beyond fixed-duration classification. Evaluation covers classification over segments ranging from 3 to 30 seconds. Continuous decoding, regression targets, and long-horizon clinical prediction remain untested.

• Fairness analysis across subpopulations. Demographic metadata such as age, sex, ethnicity, and clinical status are not consistently available across the pretraining datasets, which the authors identify as a barrier before clinical or consumer deployment.

Target Audience

Researchers and engineers working on EEG foundation models, self-supervised representation learning, or biomedical time-series transformers, particularly those interested in encoding physical geometry into attention. It is also relevant to practitioners who need a single pretrained backbone that generalizes across electrode montages and devices, and to readers tracking the ongoing shift from reconstruction-based to latent-prediction objectives in biosignal modeling.

Authors’ abstract

Electroencephalography (EEG) provides a non-invasive measure of ongoing neural activity, but building general-purpose EEG models remains challenging due to the heterogeneity of subjects, devices, and electrode montages. Existing EEG foundation models predominantly rely on reconstruction-based objectives defined on the observed signal, which contains both neural and non-neural components. We introduce SPERA (Spherical Prior EEG Representation Architecture), an EEG foundation model that adopts the joint-embedding predictive architecture (JEPA) to predict in latent space. SPERA introduces a Legendre-polynomial spatial prior, incorporated into attention to encode varying scalp electrode geometries. Two further components adapt the model to EEG: factorized temporal and spatial attention interleaved with periodic full-attention blocks, and a relational spectral regularizer aligning latent similarity structure with spectral views. Pretrained on approximately 80,000 hours of EEG from 29,048 subjects across 106 datasets, SPERA achieves the highest average balanced accuracy across nine downstream tasks spanning clinical, cognitive, and BCI applications. SPERA further exhibits strong parameter efficiency under linear probing and robustness across varying recording conditions, suggesting its potential as a general-purpose backbone for diverse EEG analyses.