BaRISTA: Brain Scale Informed Spatiotemporal Representation of Human Intracranial Neural Activity

AI summary

Overview

Research area: Neural representation learning / neurofoundation models for intracranial electrophysiology (iEEG), with connections to brain-computer interfaces and cognitive neuroscience.

Technical level: Advanced. Familiarity with transformers, self-supervised learning, and neural signal processing is helpful.

Scope: The paper introduces BaRISTA, a spatiotemporal transformer framework that lets researchers flexibly choose the spatial scale (single channel, atlas parcel, or brain lobe) for both token encoding and masking, and shows that larger spatial scales improve downstream decoding of human iEEG.

What This Paper Is About

Intracranial recordings measure neural activity simultaneously across many electrodes placed in different brain regions, producing rich but complex spatiotemporal data. Existing transformer-based "neurofoundation" models for iEEG almost always encode space at the single-channel level and mask random channels during self-supervised pretraining. This paper asks whether using larger spatial scales, such as brain parcels or lobes, for token encoding and masking produces better learned representations and stronger downstream decoding.

Key Contributions

1. A flexible spatiotemporal transformer for iEEG. The authors build a model that dissociates the choice of spatial scale for token encoding from the choice used for masking, so the effect of one can be studied independently of the other.

2. A spatially informed masked latent reconstruction task. Instead of masking random channels, the pretraining objective masks spatially defined units (channels, parcels, or lobes) and reconstructs their latent tokens, with a tokenizer and encoder trained end-to-end and an exponential moving average (EMA) target tokenizer.

3. Systematic evaluation of spatial scale. Nine encoding/masking combinations across three scales are pretrained and compared on language-related downstream classification tasks and a masked channel reconstruction task.

4. Evidence that larger spatial encoding preserves channel-level detail. The model achieves channel reconstruction performance comparable to channel-level models even when encoded at the parcel level, and generalizes to held-out subjects.

Main Findings

• Larger spatial encoding improves decoding. Parcel-level encoding (with channel-level masking) achieved the best downstream classification, reaching 0.862 AUC on sentence onset and 0.869 on speech/non-speech, significantly outperforming the Population Transformer baseline and Brant.

• Encoding scale matters more than masking scale. A two-way ANOVA showed both variables had statistically significant effects, but the spatial encoding choice drove larger performance differences than the masking choice.

• Channel-level encoding underperforms larger scales. Across all subjects and both downstream tasks, channel-level spatial encoding was worse than parcel- or lobe-level encoding, regardless of how targets were masked.

• Masking should match the encoding regime. When channel-level encoding is used, channel-level masking worked better than parcel- or lobe-level masking, suggesting the two choices interact.

• Random initialization shows no scale effect. With a randomly initialized network, spatial encoding choice had no impact on performance, confirming that the benefits arise from pretraining.

• Channel reconstruction is preserved. Parcel-level encoding achieved MSE of 0.391 and R² of 0.609 on masked channel reconstruction, matching channel-level encoding (0.397 MSE, 0.603 R²). Lobe-level encoding degraded reconstruction.

• Generalization and scaling. Holding out all sessions from a test subject reduced performance only slightly (0.841 and 0.852 AUC), still above baselines. Downstream performance increased monotonically with pretraining data from 5% to 100% of available sessions.

• Frequency bias in reconstruction. Reconstructions more accurately captured low-frequency than high-frequency neural content, confirmed by spectral analysis in the appendix.

Methodology in Plain English

The authors treat each electrode channel as a univariate signal, cut it into short time patches (e.g., 250 ms), and pass each patch through a dilated convolutional encoder followed by a linear layer to produce a "token." Each token receives a learnable spatial embedding that identifies its brain location at one of three scales: its 3D MRI coordinate (channel level), its cortical parcel from the Destrieux atlas (parcel level), or its brain lobe (lobe level). Temporal information is added via rotary positional embeddings, and all tokens are fed into a single transformer that attends over space and time together, unlike prior work that used separate cascaded temporal and spatial transformers.

For pretraining, the model randomly selects spatial categories to mask (e.g., certain parcels), replaces their tokens with a shared learnable mask token, and trains the transformer to predict the latent tokens of the masked units. The targets come from an EMA-updated copy of the tokenizer, and the loss is mean-squared error in latent token space. After pretraining, the transformer backbone and tokenizer are fine-tuned for downstream tasks: classifying speech vs. non-speech audio, detecting sentence onsets, and reconstructing masked channels in raw signal space using a linear head on top of the predictor network. Experiments use the public Brain Treebank dataset of 10 epilepsy patients (26 sessions) watching Hollywood films, with 17 sessions for pretraining and 7 held out for testing.

Why This Matters

Impact on research: The paper provides a systematic answer to an open design question in neural foundation modeling: how should space be encoded when the data spans multiple brain regions? Showing that parcel-level encoding beats the field's default channel-level encoding gives modelers concrete guidance and a reusable framework for testing spatial scales on new datasets and tasks.

Real-world applications:

• Brain-computer interfaces: Better decoding of intended speech and language from iEEG could improve communication prostheses for people with paralysis.

• Clinical monitoring: Region-aware models could improve seizure localization, functional mapping before surgery, or tracking of abnormal network activity in epilepsy patients.

• Cognitive neuroscience: The flexible spatial encoding turns the model into a tool for hypothesis testing about which brain networks support specific behaviors.

• General neural foundation models: The spatiotemporal pretraining design may transfer to other multi-region recording modalities, such as ECoG or emerging high-density arrays.

Industry relevance: Companies and labs building neural decoding platforms, clinical neurotech devices, and large-scale brain data pipelines can adopt spatial-scale-aware pretraining to improve decoding without changing data collection. The result that performance scales with pretraining data also suggests a path toward larger, more capable neural foundation models.

Future Directions

• Alternative spatial definitions. The framework supports any spatial partition, so future work can test functional network definitions (e.g., from fMRI connectivity) instead of anatomical atlases to see whether function-based encoding outperforms structure-based encoding.

• Richer masking strategies. This study used spatial-only masking; combining spatial and temporal masking, or other corruption schemes, may yield stronger representations.

• Alternative temporal encoders. The dilated CNN could be replaced with temporal pyramid pooling or hybrid short/long-term encoders to better capture high-frequency content, which the current model reconstructs less accurately.

• Hypothesis-driven scale selection. Applying the framework to simpler sensory tasks and other cognitive domains could reveal when large-scale encoding helps and when channel-level detail is preferable.

Target Audience

This paper is most valuable to machine learning researchers building neural foundation models, computational and systems neuroscientists working with iEEG or ECoG data, and neuroengineers developing brain-computer interfaces. Practitioners who design self-supervised pretraining pipelines for multi-region neural recordings will find the encoding-versus-masking dissociation directly actionable. Readers without a background in transformers or intracranial electrophysiology will need to consult the cited prior work to follow the architectural and evaluation details.

Authors’ abstract

Intracranial recordings have opened a unique opportunity to simultaneously measure activity across multiregional networks in the human brain. Recent works have focused on developing transformer-based neurofoundation models of such recordings that can generalize across subjects and datasets. However, these recordings exhibit highly complex spatiotemporal interactions across diverse spatial scales, from the single-channel scale to the scale of brain regions. As such, there remain critical open questions regarding how best to encode spatial information and how to design self-supervision tasks that enable the learning of brain network patterns and enhance downstream decoding performance using such high-dimensional, multiregional recordings. To allow for exploring these questions, we propose a new spatiotemporal transformer model of multiregional neural activity and a corresponding self-supervised masked latent reconstruction task, designed to enable flexibility in the spatial scale used for token encoding and masking. Applying this model on publicly available multiregional intracranial electrophysiology (iEEG) data, we demonstrate that adjusting the spatial scale for both token encoding and masked reconstruction significantly impacts downstream decoding. Further, we find that spatial encoding at larger scales than channel-level encoding, which is commonly used in existing iEEG transformer models, improves downstream decoding performance. Finally, we demonstrate that our method allows for region-level token encoding while also maintaining accurate channel-level neural reconstruction. Taken together, our modeling framework enables exploration of the spatial scales used for token encoding and masking, reveals their importance towards self-supervised pretraining of neurofoundation models of multiregional human brain activity, and enhances downstream decoding performance.