STAMP: Spatial-Temporal Adapter with Multi-Head Pooling

AI summary

Overview

• Research area: Machine learning for clinical time series, specifically adapting general-purpose time series foundation models (TSFMs) to electroencephalography (EEG) classification.
• Technical level: Intermediate — assumes familiarity with transformers, embeddings, and pooling, but the core idea is accessible to anyone who knows basic deep learning.
• Scope: The paper introduces STAMP, a small trainable adapter placed on top of a frozen TSFM (MOMENT), and benchmarks it against EEG-specific foundation models across eight clinical EEG datasets.

What This Paper Is About

General time series foundation models are pretrained on many kinds of data and treat each channel as an independent univariate signal, so they are not naturally suited to EEG, where electrodes across the scalp carry strongly correlated spatial-temporal structure. EEG-specific foundation models exist, but nobody had directly compared them to general TSFMs on EEG tasks, and using a TSFM out of the box with simple mean pooling performs close to random. STAMP fills this gap by adding a lightweight spatial-temporal adapter that lets a frozen general TSFM match state-of-the-art EEG-specific models.

Key Contributions

1. STAMP adapter: A novel spatial-temporal adapter with three parts — combined token/spatial/temporal positional encodings, a criss-cross gated MLP, and multi-head attention pooling.
2. Lightweight and flexible: Roughly 750K trainable parameters (versus about 29M for CBraMod and 5.8M for LaBraM), and it can sit on top of different TSFMs without retraining them.
3. First head-to-head comparison: The paper presents the first systematic comparison of general TSFMs against EEG-specific foundation models on eight EEG classification benchmarks.
4. Component ablations: Detailed experiments isolating the contribution of positional encodings, token-mixing architecture, and pooling strategy, plus a study showing how performance and size trade off as the adapter dimension shrinks.

Main Findings

• Matches EEGFMs with far fewer parameters: Averaging 0.74M trainable parameters, STAMP achieves performance comparable to or better than CBraMod and LaBraM on most datasets and consistently outperforms non-foundation EEG baselines like EEG Conformer and ST-Transformer.
• Positional encoding is essential: Removing positional encoding always produces the worst results across datasets. Using token, spatial, and temporal encodings together (PE-NST) wins on three of four ablation datasets.
• Criss-cross architecture beats standard alternatives: The CC-GMLP token mixer outperformed both a basic GMLP and standard transformer variants on every ablation dataset, supporting the value of modeling spatial and temporal relationships separately.
• MHAP is a robustness play: Multi-head attention pooling performs similarly to mean pooling when a token mixer is present, but substantially improves results on BCIC-IV-2a and is dramatically better than mean pooling when no token mixer is used.
• Naive TSFM use fails: MOMENT with only mean pooling yields near-random performance, demonstrating that spatial-temporal adaptation is not optional but necessary.
• TSFM-agnostic: Usable embeddings from MOMENT Small/Base/Large, Chronos Large, and TSPulse. Notably, MOMENT Small often matches MOMENT Large using roughly 10% of the parameters, and TSPulse performs well despite containing no EEG data in its pretraining.
• Emotion recognition is the weak spot: STAMP underperforms other methods on SEED-V and FACED, suggesting MOMENT's pretraining does not yield features well suited to emotion tasks. Swapping in Chronos embeddings helps only slightly.
• Aggressive size reduction is possible: On most datasets, reducing the adapter dimension from 128 to 8 retains similar accuracy while cutting parameters by roughly 91%.

Methodology in Plain English

The researchers take existing general time series foundation models and freeze them. Each EEG channel is cut into one-second windows, and every window is passed through the frozen model to produce a single embedding vector. Because the model is frozen, these embeddings can be generated once in parallel, which makes experimentation cheap.

The resulting embeddings form a grid indexed by electrode (spatial) and time (temporal). STAMP then does three things to this grid. First, it adds learned positional information — one vector for each token, one for each spatial location, and one for each time location — so the adapter knows where each embedding came from. Second, a criss-cross gated MLP mixes information across tokens, applying separate gating operations along the spatial and temporal axes so that cross-channel relationships and temporal patterns are learned independently. Third, multi-head attention pooling learns which tokens matter most for the final answer, producing a weighted summary that is fed into a simple classifier head.

The authors ran controlled ablations on four datasets, varying one component at a time (positional encoding style, token mixer type, pooling method) with three fixed random seeds for reproducibility. A final evaluation on all eight datasets used five seeds and mirrored the preprocessing and evaluation protocol of CBraMod so results are directly comparable.

Why This Matters

This work shows that the expensive step in EEG deep learning — pretraining a large domain-specific foundation model — may not be necessary if you can build a good adapter. This shifts the field's economics toward small, reusable, task-specific adapters on top of general models.

Real-world applications:

• Clinical EEG event detection: TUEV results suggest STAMP can classify clinically relevant events such as spikes, sharp waves, and periodic discharges.
• Brain-computer interfaces: Motor imagery datasets (SHU-MI, PhysioNet-MI, BCIC-IV-2a) are the standard BCI decoding benchmarks for prosthetic and assistive control.
• Mental health screening: Strong results on Mumtaz2016 support EEG-based detection of major depressive disorder, and MentalArithmetic relates to cognitive load monitoring.
• Sleep and neurodiagnostic monitoring: The general adapter design extends to any multivariate clinical time series, including sleep staging.

Industry relevance: the low parameter count matters for deployment. An adapter of under one million parameters can be trained and stored cheaply, and — since the embedding generator is frozen — embedding extraction can be done once and cached, reducing GPU requirements for downstream development. This lowers the barrier to entry for smaller labs and companies that cannot afford to pretrain large EEG models.

Future Directions

1. Improving performance on emotion recognition: The clear failure case on SEED-V and FACED invites work on TSFMs or adapters better suited to affect-related features.
2. Interpretability: Examining which tokens and channels contribute most to predictions could make the model clinically actionable.
3. Explicit electrode geometry: The authors suggest incorporating topographic or geometric relationships between electrodes rather than treating each channel's position abstractly.
4. Broader multivariate time series: Extending STAMP beyond EEG to vital signs, imaging, or medication dosage data would test how domain-agnostic the design really is.
5. Fine-tuning tradeoffs: Only one LoRA configuration was tested; exploring more aggressive TSFM fine-tuning, particularly on the weak datasets, remains open.

Target Audience

Machine learning researchers working on foundation models and transfer learning, clinical ML practitioners looking for resource-efficient EEG pipelines, neuroscientists and BCI engineers who need strong classification performance without training large models, and hospital or health-tech engineering teams evaluating whether to adopt a general time series foundation model over a domain-specific one.

Authors’ abstract

Time series foundation models (TSFMs) pretrained on data from multiple domains have shown strong performance on diverse modeling tasks. Various efforts have been made to develop foundation models specific to electroencephalography (EEG) data, which records brain electrical activity as time series. However, no comparative analysis of EEG-specific foundation models (EEGFMs) versus general TSFMs has been performed on EEG-specific tasks. We introduce a novel Spatial-Temporal Adapter with Multi-Head Pooling (STAMP), which leverages univariate embeddings produced by a general TSFM, implicitly models spatial-temporal characteristics of EEG data, and achieves performance comparable to state-of-the-art EEGFMs. A comprehensive analysis is performed on 8 benchmark datasets of clinical tasks using EEG for classification, along with ablation studies. Our proposed adapter is lightweight in trainable parameters and flexible in the inputs it can accommodate, supporting easy modeling of EEG data using TSFMs.