Learning Time-Scale Invariant Population-Level Neural Representations

AI summary

Overview

• Research area: Machine learning for neural time series — specifically population-level (multi-electrode) representation learning for intracranial electroencephalography (iEEG) foundation models.
• Technical level: Intermediate. The paper builds on self-supervised transformer pretraining and linear-probe-style decoding, but the core idea (training across multiple input durations to remove sensitivity to duration) is stated in accessible terms.
• Scope: This workshop paper (arXiv:2511.13022v1, cs.LG, 17 Nov 2025, presented under "Foundation Models for the Brain and Body") by Eshani Patel, Yisong Yue, and Geeling Chau of Caltech diagnoses time-scale sensitivity in the Population Transformer (PopT) and proposes Time-scale Augmented Pretraining (TSAP) to fix it.

What This Paper Is About

Neural foundation models are usually built by stacking a population-level transformer on top of a frozen temporal encoder, but the population layer only ever sees the encoder's outputs for whatever window length was used during pretraining. The authors show that this makes the resulting representations brittle: a model pretrained on one interval length (say, 5 seconds) loses substantial decoding accuracy when applied to data cropped to a different length (say, 1 second). Their goal is to build a population-level representation that performs well across input time-scales, including ones never seen during pretraining.

Key Contributions

1. A comparative evaluation of downstream decoding performance under mismatched time-scales, confirming that pretraining and finetuning on the same interval length yields better performance than mismatched pairings.
2. Time-scale Augmented Pretraining (TSAP), a pretraining strategy that exposes the model to a spectrum of interval lengths (1, 2, 4, and 5 seconds, holding out 3 seconds) and improves generalization, including to unseen lengths.
3. An analysis of the embedding spaces produced by the temporal encoder and by pretrained PopT models across different time-scales, using 2D PCA projections and K-Means clustering with 5 means.
4. A paired t-test statistical analysis of TSAP versus the matched optimal baseline across subject/seed pairs (N = 35 per interval).

Main Findings

• Mismatch hurts, and pretraining still helps: Models evaluated on interval lengths different from their pretraining length perform worse than the matched model, but all pretrained models still outperform the non-pretrained PopT (Word Onset non-pretrained scores range from 0.645 ± 0.015 to 0.678 ± 0.018 depending on interval).
• TSAP recovers and often exceeds the matched optimum: On Word Onset, TSAP achieved 0.777 ± 0.017 (1s), 0.843 ± 0.017 (2s), 0.866 ± 0.015 (3s), 0.893 ± 0.013 (4s), and 0.907 ± 0.011 (5s), the best score in every interval column, versus matched optimal baselines of 0.770 ± 0.017, 0.837 ± 0.017, 0.863 ± 0.015, 0.884 ± 0.014, and 0.901 ± 0.011.
• Sentence Onset shows the same pattern with one exception: TSAP scored 0.802 ± 0.015 (1s), 0.841 ± 0.014 (2s), 0.843 ± 0.013 (3s), 0.855 ± 0.012 (4s), and 0.865 ± 0.010 (5s). At the held-out 3-second interval, the 3-second-matched baseline (0.846 ± 0.013) was the best score, with TSAP second best.
• Statistically significant gains at several scales: Paired t-tests showed significant improvement for TSAP on Word Onset at 1s (mean difference 0.0074, p = 0.01675), 4s (0.0083, p = 0.00005), and 5s (0.0062, p = 0.00409), and on Sentence Onset at 1s (0.0126, p = 0.0305). The 3-second interval showed the least significant improvement, though the paper reports TSAP wins 1 of 2 comparisons against the optimal baseline there.
• Embeddings cluster by time-scale, and TSAP breaks that clustering: PCA of BrainBERT temporal embeddings from the same 100 samples mapped to non-identical representations across interval lengths and clustered strongly by interval. PCA of PopT [CLS] tokens from the 5-second-pretrained model still clustered strongly by time-scale, whereas TSAP [CLS] tokens showed much more overlap. K-Means confusion matrices likewise showed clean time-scale-dependent clusters for the 5-second model and considerably more confusion for TSAP.

Methodology in Plain English

The researchers start from the Population Transformer (PopT), which takes each electrode channel's signal over some interval, pushes it through a frozen temporal encoder (BrainBERT), adds a positional embedding derived from the electrode's 3D coordinates, and passes the result through a transformer encoder to produce spatially contextualized channel representations plus an aggregated [CLS] token that a linear layer decodes. They use the architecture and training configuration from the original PopT work, changing only the learning rate to 1×10⁻⁴ for training stability.

To test time-scale sensitivity, they pretrain separate copies of PopT on single interval lengths of 1, 2, 3, 4, or 5 seconds, creating "optimal" matched baselines — 5 seconds being the original PopT formulation. For TSAP, they modify the data-generation pipeline so pretraining samples mix interval lengths of 1, 2, 4, and 5 seconds with a fixed gap between consecutive windows, treat 3 seconds as held out, and double the pretraining steps from 500,000 to 1,000,000.

Data comes from the public BrainTreeBank iEEG dataset: 10 subjects, 1,688 electrodes total (mean 167 electrodes per subject), who watched 26 movies (19 for pretraining, 7 for downstream decoding). Evaluation covers two auditory-linguistic classification tasks — word onset and sentence onset — with 90 randomly selected electrodes per subject used for finetuning. Each finetuning run covers one subject and one interval length, repeated with five random seeds, with the best validation ROC-AUC checkpoint reported on test.

Why This Matters

The paper frames preprocessing diversity — not just architecture or dataset scale — as a barrier to generalizable neural foundation models. If a model's usefulness depends on matching the exact window length used in pretraining, then reusing it across labs, datasets, and experimental paradigms becomes unreliable. The authors argue that models performing optimally across time-scales let neuroscience and BCI research use neural foundation models "out of the box."

Real-world applications:

• Brain-computer interfaces: Decoders for speech or language tasks (word onset, sentence onset) that keep working when a clinical or experimental protocol changes its trial/window length.
• Cross-dataset model reuse: Applying one pretrained population-level model to recordings from different studies, which vary widely in recording length and task structure.
• Clinical iEEG workflows: Analyses built on intracranially recorded signals where preprocessing choices differ between sites or sessions.
• Foundation-model development for time series: Adapting the augmentation approach to other domains where fixed-length windows cause context mismatches.

Industry relevance: developers of neural interface and neurotechnology products, medical device and clinical monitoring pipelines that rely on iEEG, and teams building general-purpose time-series foundation models all face the same practical problem this paper targets — a model that degrades when input duration changes. The reported compute footprint is modest: all experiments run on a single NVIDIA RTX A6000 (50GB GPU RAM), with single-interval pretraining taking about 1.5 days on 1 GPU and TSAP pretraining about 3 days on 1 GPU, and downstream evaluations taking a few minutes each; the authors parallelized data processing and result gathering on 6 GPUs.

Future Directions

• Compare against encoder-side invariance: The authors explicitly leave open how TSAP compares with or augments approaches that build invariance into the temporal encoders themselves (such as triplet objectives, dynamic time warping distortions, or frequency-domain consistency methods).
• Joint spatial-temporal pretraining: Combining TSAP with approaches that use smaller fixed patches and learn temporal and spatial components together, evaluating which scales efficiently and generalizes best.
• Held-out time-scale performance: The 3-second interval showed the least significant improvement for TSAP, so improving generalization to unseen scales remains an open question.
• Broader preprocessing diversity: The paper motivates addressing preprocessing mismatches generally, not only interval length, as a step toward generalizable neural foundation models.

Target Audience

Researchers and practitioners working on neural time-series foundation models, population-level representation learning, and iEEG decoding will get the most from this paper, along with BCI engineers who need decoders to be robust to changing preprocessing parameters. It is also useful for machine learning researchers interested in time-series pretraining who want a concrete case study of how fixed-length window assumptions break generalization, and for neuroscientists adopting pretrained models across heterogeneous datasets.

Authors’ abstract

General-purpose foundation models for neural time series can help accelerate neuroscientific discoveries and enable applications such as brain computer interfaces (BCIs). A key component in scaling these models is population-level representation learning, which leverages information across channels to capture spatial as well as temporal structure. Population-level approaches have recently shown that such representations can be both efficient to learn on top of pretrained temporal encoders and produce useful representations for decoding a variety of downstream tasks. However, these models remain sensitive to mismatches in preprocessing, particularly on time-scales, between pretraining and downstream settings. We systematically examine how time-scale mismatches affects generalization and find that existing representations lack invariance. To address this, we introduce Time-scale Augmented Pretraining (TSAP), which consistently improves robustness to different time-scales across decoding tasks and builds invariance in the representation space. These results highlight handling preprocessing diversity as a key step toward building generalizable neural foundation models.