Learning the relative composition of EEG signals using pairwise relative shift pretraining

AI summary

Overview

Research area: Self-supervised learning (SSL) for electroencephalography (EEG) — specifically, pretext-task design for pretraining transformer encoders on unlabeled brain signals, evaluated on clinical and wearable EEG decoding tasks.

Technical level: Intermediate. The paper assumes familiarity with transformer encoders, masked autoencoders, positional embeddings, cross-attention, and standard EEG decoding benchmarks (sleep staging, seizure detection, abnormal EEG detection, motor imagery).

Scope (one sentence): The paper introduces PARS (Pairwise Relative Shift) pretraining, a pretext task that trains an EEG encoder to predict the relative time shift between randomly sampled signal patches, and benchmarks it against masked reconstruction (MAE) and position-prediction (MP3, DropPos) pretraining across label-efficiency and transfer-learning settings.

What This Paper Is About

EEG models that decode sleep stages, seizures, or motor imagery typically need large amounts of expert-labeled data, which is expensive to produce. Self-supervised pretraining offers a way to learn from unlabeled EEG, but most existing EEG SSL methods rely on masked reconstruction (MAE-style) tasks that mainly capture local temporal patterns. This paper asks whether a different pretext task — predicting the relative temporal distance between pairs of randomly sampled EEG patches — produces representations that better capture long-range temporal dependencies and transfer more effectively to downstream decoding.

Key Contributions

1. A new pretext task for EEG. PARS predicts an anti-symmetric matrix of normalized relative time shifts between pairs of randomly sampled patches, defined as theta_j,k = (t_j - t_k) / T_s, where t_j and t_k are patch timestamps and T_s is total sequence length. The task is inspired by the vision-based PART method (Ayoughi et al., 2025) and is, per the authors, underexplored in EEG literature.

2. A masked positional-embedding scheme that prevents task leakage. Positional embeddings are added to only (N - N_m) tokens; the remaining N_m tokens receive a learnable position mask token. Pairwise shifts are computed only for patches with masked positional embedding, so the encoder cannot trivially read out position information.

3. A cross-attention decoder and multi-channel fine-tuning architecture. A cross-attention decoder maps pairwise patch embeddings into the shift matrix, and a fine-tuning head embeds each EEG channel with the pretrained single-channel encoder, average-pools over time, and collapses the spatial tokens via cross-attention with a learnable query token into a linear prediction layer.

4. A systematic comparison against four baselines. PARS is compared with a supervised-only (scratch) baseline, MAE, MP3 (masked position prediction), and DropPos (dropped position prediction) in both label-efficient fine-tuning and transfer learning across four downstream datasets, plus hyperparameter ablations.

Main Findings

• Transfer learning: PARS wins on three of four tasks. With a 12.7M-parameter encoder, PARS achieved the best balanced accuracy and Kappa on EESM17 wearable sleep staging (balanced accuracy 0.758 ± 0.130; Kappa 0.674 ± 0.154), the best balanced accuracy and AUROC on TUAB abnormal EEG detection (0.802 ± 0.003; 0.877 ± 0.003), and the best balanced accuracy and Kappa on PhysioNet-MI motor imagery (0.572 ± 0.007; 0.429 ± 0.009). On TUSZ seizure detection it reached the highest balanced accuracy (0.901 ± 0.009) but a lower AUROC (0.878 ± 0.007) than MAE (0.894 ± 0.009); the authors describe MAE and PARS as performing similarly on that task.

• All pretraining beats no pretraining. The scratch baseline was the weakest on every reported metric and dataset (for example, EESM17 balanced accuracy 0.733 ± 0.092 and Kappa 0.637 ± 0.111; PhysioNet-MI balanced accuracy 0.527 ± 0.008 and Kappa 0.369 ± 0.011), while PARS, MAE, MP3, and DropPos all improved on it.

• PARS is strongest when labels are scarce. In the label-efficiency experiment on clinical sleep staging, fine-tuning subjects were varied from 10 to 657. At the very low label regime (10 subjects), PARS outperformed the other three pretraining strategies, which the authors interpret as relative temporal position information being more important than absolute position information for sleep staging. As the number of labeled subjects grew, the gap between pretraining methods shrank.

• Random patch sampling beats fixed sampling. The ablation on YSYW (pretrain on full YSYW + TUEG, fine-tune on 100 subjects, evaluate average Cohen's Kappa over 200 validation subjects across five seeds) found random sampling of patches significantly better than fixed sampling with a 1-sec stride, because random inter-patch distances make the pretext task harder.

• A harder pretext task helps, up to memory limits. Across N = 20, 30, 40 patches and position masking ratios gamma_pos of 0.2, 0.4, 0.6, 0.8, 1.0, the task became easier as N increased and harder as gamma_pos increased; higher gamma_pos led to better downstream performance. The authors note that because the number of patch pairs grows combinatorially with N and gamma_pos, the approach is limited by available GPU memory. Optimal settings were gamma_pos = 0.8 and N = 40.

• Cross-attention decoding beats a pairwise MLP. A cross-attention decoder outperformed a pairwise MLP (two linear layers with hidden size 512 and ReLU activation), because the cross-attention decoder can access all patches while the MLP sees only a single pair.

• Wearable sleep staging is channel-sensitive. Using leave-one-subject-out cross-validation on EESM17 (9 subjects), PARS outperformed the four other approaches on five subjects. All pretraining strategies performed poorly on Subject 5, which Mikkelsen et al. (2017) had reported as having deterioration in electrode-body contact for one ear.

Methodology in Plain English

The researchers take long, unlabeled EEG recordings and cut them into 30-second examples drawn from two large clinical databases combined into a pretraining set of over 1.5 million unique 30-second examples. For the pretext task, they randomly pick 40 one-second patches out of each 30-second signal (meaning the patches are sampled without regard to a fixed grid). Each patch is converted into a token by a linear layer, and the model then has to answer a single question: how much time separates patch j from patch k?

To stop the model from cheating, they hide positional information from most tokens. A standard sinusoidal positional embedding is given to some tokens; the rest receive a learnable "position mask" token instead, and only pairs of patches with hidden positions contribute to the loss. A transformer encoder processes the tokens, and a cross-attention decoder looks at all the patches at once to predict the full matrix of relative shifts, trained with a mean-squared-error loss. The model is a PatchTST-style transformer with 12.7M parameters, 8 transformer blocks, 8 attention heads, and a 512-dimensional feed-forward hidden size, pretrained for 1000 epochs with AdamW.

To use the learned single-channel encoder on multi-channel EEG, each channel is embedded separately by the pretrained encoder, the per-channel embeddings are average-pooled over time into spatial tokens, and a cross-attention layer with a learnable query token compresses those into one vector that a linear layer turns into a prediction. Fine-tuning uses weighted cross-entropy (weighted by class frequency) for 200 epochs, with spatial tokens randomly dropped with probability 0.5 to reduce overfitting, and the checkpoint with the lowest validation loss is selected.

Why This Matters

Impact on research. Most EEG SSL work has converged on masked reconstruction, which emphasizes local waveform recovery. This paper shows that a position-based, long-range task is a competitive — and in low-label and cross-dataset settings often superior — alternative, and that the two families of pretext tasks may capture complementary features (the authors cite prior work suggesting MAE and position-prediction features can be complementary). It also provides a concrete multi-channel adaptation recipe for a single-channel pretrained encoder, which matters because EEG hardware varies widely in channel count and montage.

Real-world applications:

• Clinical sleep staging — automated five-stage scoring (Wake, REM, Non-REM1, Non-REM2, Non-REM3) from scalp EEG, which the paper evaluates on 994 annotated YSYW patients.
• Wearable sleep monitoring — the EESM17 task uses a 12-channel in-ear EEG system, relevant to consumer and at-home sleep tracking.
• Seizure detection and abnormal EEG triage — TUSZ (675 patients) and TUAB (2,329 patients) target binary seizure and normal/abnormal classification, which could support clinical review workflows.
• Motor imagery brain-computer interfaces — the 4-class PhysioNet-MI task (109 subjects, 64 channels) is directly relevant to BCI control applications.

Industry relevance. The work comes from Apple, with co-authors also affiliated with Stanford University, the California Institute of Technology, and the University of Amsterdam. Label-efficient pretraining for biosignals is directly applicable to consumer wearables, where unlabeled sensor data is abundant and expert annotations are scarce.

Future Directions

1. Hybrid pretext tasks. The authors explicitly propose combining the local pattern sensitivity of reconstruction methods (MAE) with the global temporal awareness of position prediction, motivated by the observation that MAE and PARS perform similarly on TUSZ and by prior work suggesting complementary features.

2. Scaling limits and memory. Because the number of patch pairs grows combinatorially with N and gamma_pos, the paper notes GPU memory as the limiting factor; more efficient or sparse pairwise formulations are an open problem.

3. Robustness to noisy channels. All pretraining strategies performed poorly on EESM17 Subject 5, where electrode-body contact deteriorated on one ear. The authors state that better rejection of noisy channels will be investigated in future work.

4. Broader validation of temporal composition as an inductive bias. The conclusion frames the consistent improvements as evidence that temporal composition awareness is a crucial inductive bias for EEG analysis, and calls for extending this to more neuroscience decoding applications.

Target Audience

Researchers and engineers working on biosignal foundation models, self-supervised learning, or clinical EEG decoding will benefit most — particularly those choosing pretext tasks for pretraining, those needing to adapt a single-channel pretrained encoder to multi-channel hardware, and practitioners in low-label clinical settings. Readers without background in transformers or EEG benchmark datasets will need to consult the cited prior work (MAE, MP3, DropPos, PatchTST) first.

Authors’ abstract

Self-supervised learning (SSL) offers a promising approach for learning electroencephalography (EEG) representations from unlabeled data, reducing the need for expensive annotations for clinical applications like sleep staging and seizure detection. While current EEG SSL methods predominantly use masked reconstruction strategies like masked autoencoders (MAE) that capture local temporal patterns, position prediction pretraining remains underexplored despite its potential to learn long-range dependencies in neural signals. We introduce PAirwise Relative Shift or PARS pretraining, a novel pretext task that predicts relative temporal shifts between randomly sampled EEG window pairs. Unlike reconstruction-based methods that focus on local pattern recovery, PARS encourages encoders to capture relative temporal composition and long-range dependencies inherent in neural signals. Through comprehensive evaluation on various EEG decoding tasks, we demonstrate that PARS-pretrained transformers consistently outperform existing pretraining strategies in label-efficient and transfer learning settings, establishing a new paradigm for self-supervised EEG representation learning.