MEG-XL: Data-Efficient Brain-to-Text via Long-Context Pre-Training

AI summary

Overview

Research area: Machine learning for neural signal decoding — specifically self-supervised pre-training of MEG (magnetoencephalography) models for brain-to-text (B2T) decoding, sitting at the intersection of brain-computer interfaces and long-context sequence modelling.

Technical level: Intermediate to Advanced. The core idea is accessible, but the method assumes familiarity with transformers, masked prediction objectives, residual vector quantization and contrastive representation learning.

Scope: The paper introduces MEG-XL, a 20M-parameter model pre-trained on 2.5-minute windows of MEG data across 800+ subjects, then fine-tuned for word decoding to show that long-context pre-training substitutes for large amounts of subject-specific training data.

What This Paper Is About

Clinical brain-to-text interfaces are intended for paralysed patients, who cannot provide the long recording sessions that current supervised decoders require. Pre-training on data from many subjects can supply transferable priors, but these priors depend on context length: natural speech unfolds over minutes, while prior brain models pre-train on windows of milliseconds to at most 10 seconds (with a few non-specialised models reaching up to 30 seconds). MEG-XL pre-trains on 2.5 minutes of MEG context per sample — 5-300× longer than prior work, equivalent to 191k tokens — and shows that this yields better word decoding with far less fine-tuning data than the supervised state of the art or existing brain foundation models.

Key Contributions

1. Long-context neural pre-training. The authors scale neural pre-training into the minute regime, modelling 2.5-minute MEG samples (191k tokens per sample), 5-300× longer than prior pre-trained brain models, using a memory-efficient criss-cross transformer to avoid quadratic attention cost.
2. Data-efficient cross-subject transfer. Fine-tuned for word decoding, MEG-XL outperforms the supervised state of the art (d'Ascoli et al., 2025) and a set of six modern brain foundation models when subject-specific data is scarce, while remaining competitive with full data.
3. Empirical analysis of why context helps. Through linear probing, zero-shot masked-token prediction on unseen datasets, and attention analysis, the authors provide evidence that increasing pre-training context improves representations and produces selective, hierarchical attention patterns that short-context models do not learn.
4. Released artefacts. Code and model weights are released, with instructions.

Main Findings

• MEG-XL dominates in the low-data regime. With 13% of training data, MEG-XL reaches 47.0 ± 0.9 (MEG-MASC), 54.9 ± 0.5 (Armeni) and 57.3 ± 0.4 (LibriBrain) top-10 balanced accuracy, against a 20% random baseline. The next-best model, LaBraM, reaches 33.2 ± 1.0, 26.3 ± 2.4 and 40.3 ± 0.1 respectively. Most other baselines remain at chance at this data level.

• Competitive with full data. With 100% of training data, MEG-XL scores 46.4 ± 1.3 (MEG-MASC), 61.2 ± 0.4 (Armeni) and 63.0 ± 0.4 (LibriBrain). It is joint best on LibriBrain (tied with BrainOmni at 63.0) and within 1.1 percentage points of BrainOmni on Armeni. BrainOmni's strong results on Armeni and LibriBrain do not transfer to MEG-MASC (19.1 ± 2.8), which the authors attribute to shallow per-subject data.

• Gains over supervised learning are substantial and regime-dependent. MEG-XL shows 10-25% gains over the supervised state of the art with fewer parameters (20M vs 200M). On MEG-MASC it improves by over 25% at certain points across all 54 hours of training data; on Armeni it holds roughly a 10% advantage until around 15 hours (5 hours per subject); on LibriBrain the two are similar until about 2.5 hours, after which the supervised method pulls ahead.

• The architecture alone is not the source of the gains. A randomly initialised MEG-XL performs worse than the pre-trained version across the data range, indicating the benefits come from learned priors rather than the criss-cross transformer design.

• Longer pre-training context yields better representations. Linear probes trained on 7% of the data show improved word decoding as pre-training context increases, with diminishing returns after 100 seconds. Probing with full context and with context matched to pre-training length produces similar results, suggesting extra inference-time context does not help unless the model was pre-trained to use it — mirroring length-generalisation behaviour in large language models.

• Context scaling extends beyond word decoding. Zero-shot prediction of masked brain signals from unseen datasets improves log-linearly with pre-training context through 150 seconds without saturating, even though word-decoding probes plateau after roughly 100 seconds. The authors note this asymmetry implies the utility of context is task-dependent.

• Long-context pre-training teaches selective, hierarchical attention. Short-context models attend diffusely from the first layer. Long-context models attend locally in early layers and progressively expand to global integration with depth, and their attention entropy decreases with context length. The authors frame the key benefit as learning when and how to use distant context, not simply gaining the ability to attend to it.

• Data requirement reduced to hours. The abstract reports matching supervised performance with a fraction of the data, e.g. 1 hour vs 50 hours. The discussion states pre-training reduces the requirement from dozens of hours to 1-2 hours of total data and only tens of minutes per subject.

Methodology in Plain English

The pipeline has three stages.

Tokenization. Raw MEG is preprocessed with a 0.1 Hz high-pass and 40 Hz low-pass filter and resampled to 50 Hz (the paper notes this risks aliasing but follows the standard word-decoding pipeline and reduces sequence length). A frozen BioCodec tokenizer is applied independently to each MEG channel, converting it into discrete codes using residual vector quantization with Q = 6 levels, a vocabulary of V = 256, and a temporal downsampling factor of r = 12. The authors chose BioCodec over alternatives such as BrainTokenizer because compressing across channels degraded reconstruction quality.

Pre-training. Each token is embedded by looking up the codebook vector at every quantization level, concatenating the levels, and projecting to the model dimension. Three additive sensor embeddings encode position and orientation (via Gaussian Fourier features) and sensor type (gradiometer or magnetometer, via a learned embedding). The backbone is an 8-layer transformer with pre-layer RMSNorm, SELU feedforward layers with 4× hidden expansion, and criss-cross attention that splits the feature dimension in half and runs temporal attention and spatial attention in parallel, reducing attention cost from O((CT′)²) to O(C·T′² + T′·C²). Rotary position embeddings encode temporal position. Training uses masked token prediction: contiguous 3-second blocks are masked uniformly across all sensors until 40% of the sequence is masked, and the model predicts the discrete codes at every quantization level with cross-entropy loss. Masking across all sensors at once prevents the model from simply interpolating across simultaneous channel readings. Recordings are padded to a maximum channel count with a sensor mask excluding padded channels from attention and loss, allowing training on heterogeneous datasets.

Pre-training data. Approximately 300 hours of MEG from over 800 subjects, drawn from CamCAN (rest and sensorimotor subset), MOUS (listening task only) and SMN4Lang (all available MEG).

Fine-tuning and evaluation. The authors follow d'Ascoli et al.'s word-locked epoch decoding strategy: for a sequence of 50 words, they extract 3-second neural windows aligned 0.5s before each word onset and concatenate them, giving a 150-second input tensor. The model predicts semantic target embeddings extracted from a T5 large language model using a contrastive SigLIP variant that masks repeated words in a sequence; an MLP head maps pooled transformer outputs to the predicted embedding, and words are read out by nearest-neighbour retrieval using cosine similarity. Evaluation uses top-10 balanced accuracy over a fixed retrieval vocabulary (top-50 most frequent words, with top-250 results referenced in an appendix not included in the supplied text), on three English perceived-speech MEG datasets: LibriBrain (1 subject, 52 hours), Armeni et al. (3 subjects, 10 hours each, 30 total) and MEG-MASC (27 subjects, 2 hours each, 54 total). Splits are 80:10:10, with the same stimuli presented to different subjects always assigned to the same split.

Why This Matters

Impact on research. Recent work has questioned whether brain foundation models improve on supervised baselines at all, citing marginal gains despite far larger parameter counts, and poor performance with limited data. This paper contradicts that trend within a specific, well-aligned domain: a 20M-parameter model beats a 200M-parameter supervised model by 10-25% in the low-data regime. The authors argue the difference lies in alignment between pre-training and downstream task — speech decoding may demand temporal structure that short-context pre-training discards. It also reframes long-context modelling in neuroscience: providing long windows is insufficient; a model must acquire long-context priors to exploit them.

Real-world applications:

• Assistive communication for paralysed patients. The stated motivating use case is brain-to-text interfaces for people who cannot speak, where the inability to provide extensive training recordings is the central obstacle.
• Non-invasive neurotechnology. MEG sits outside the skull, enabling safer and more scalable data collection than surgical implants; longer context may help close part of the fidelity gap with invasive approaches.
• Faster BCI calibration. Reducing per-subject requirements to tens of minutes rather than dozens of hours would make deployment to new users practical.
• Cross-subject model reuse. Pre-training across 800+ subjects demonstrates a route to models that transfer to new individuals without individual retraining.

Industry relevance. The result that a small (20M) domain-aligned model can outperform much larger generic models in the setting that matters clinically has direct implications for compute cost, data collection cost and deployment feasibility in neurotechnology. The paper's impact statement also flags that as capabilities improve, the field will need norms around consent, data ownership, and the boundary between assistive and surveillant applications.

Future Directions

• Extending to imagined speech. The authors explicitly describe the present work on perceived speech as a foundational step before addressing imagined (covert) speech, which is what clinical communication aids ultimately require.
• Scaling beyond 150 seconds. GPU VRAM limits prevented pre-training contexts longer than 150 seconds in the probing experiments; masked-prediction performance had not saturated at that point, so further scaling is an open question.
• Larger vocabularies. Evaluation used a 50-word vocabulary, and performance decreases with larger vocabularies (reported in an appendix not included in the supplied text). The authors draw a parallel with surgical decoding studies that began at 50 words before scaling to hundreds of thousands.
• Understanding the learned structure. The authors state that while they characterise the attention patterns driving generalisation, the nature of the learned structure remains unresolved, and it is unclear whether the asymmetry between tasks (word decoding plateauing after 100s while masked prediction keeps improving) reflects task-specific ceilings or limits of the linear readout.
• Clinical deployment and patient populations. The paper states that clinical deployment remains distant, requiring performance above what communication aids need, plus extension to patient populations.

Target Audience

Machine learning researchers working on self-supervised learning, long-context sequence modelling and neural foundation models; neuroscience and BCI researchers interested in non-invasive speech decoding; and clinical neurotechnology developers who need models that generalise from limited per-subject data. Readers will benefit most if they are already comfortable with transformer architectures and masked-prediction pre-training, since the paper assumes that background when explaining its tokenizer, attention factorisation and probing methodology.

Authors’ abstract

Clinical brain-to-text interfaces are designed for paralysed patients who cannot provide extensive training recordings. Pre-training improves data-efficient generalisation by learning statistical priors across subjects, but these priors critically depend on context. While natural speech might unfold gradually over minutes, most methods pre-train with only a few seconds of context. Thus, we propose MEG-XL, a model pre-trained with 2.5 minutes of MEG context per sample, 5-300x longer than prior work, and equivalent to 191k tokens, capturing extended neural context. Fine-tuning on the task of word decoding from brain data, MEG-XL matches supervised performance with a fraction of the data (e.g. 1hr vs 50hrs) and outperforms brain foundation models. We find that models pre-trained with longer contexts learn representations that transfer better to word decoding. Our results indicate that long-context pre-training helps exploit extended neural context that other methods unnecessarily discard. Code, model weights, and instructions are available at https://github.com/neural-processing-lab/MEG-XL .