MEGConformer: Conformer-Based MEG Decoder for Robust Speech and Phoneme Classification

AI summary

Overview

• Research area: Neural speech decoding / brain-computer interfaces, at the intersection of natural language processing (automatic speech recognition architectures) and computational neuroscience (magnetoencephalography, MEG).
• Technical level: Intermediate. Familiarity with transformer/Conformer architectures, classification metrics, and basic neuroimaging concepts helps, but the paper is self-contained enough for motivated readers.
• Scope: The paper adapts a compact Conformer encoder to raw 306-channel MEG signals and evaluates it on the LibriBrain 2025 competition tasks of speech detection and phoneme classification, winning the Standard track for phonemes.

What This Paper Is About

Non-invasive brain signals such as MEG contain information about speech that a person is hearing, but extracting that information reliably is difficult because the signals are noisy, subject-specific, and drift across recording sessions. This paper asks whether a Conformer — an architecture originally designed for automatic speech recognition — can be repurposed to decode speech activity and phonemes directly from raw MEG recordings, and what practical tricks are needed to make it generalize to held-out data.

Key Contributions

1. Unified Conformer backbone for two MEG tasks. A single compact Conformer architecture, adapted to raw MEG with a lightweight 1D convolutional projection, is optimized separately for speech/silence detection and 39-class phoneme classification, sharing most design choices.
2. Instance-level input normalization for distribution shift. Per-window, per-channel normalization (no running statistics, no affine parameters) applied after the 100-sample averaging step proves essential for holdout generalization, improving phoneme holdout performance by over 200% relative to alternatives.
3. MEGAugment, an MEG-specific augmentation. A lightweight variant of SpecAugment combining time masking and bandstop masking of standard neurophysiological frequency bands (Theta, Alpha, Beta, Gamma, HGA), which helped early speech models and is included in the final system.
4. Dynamic grouping loader for averaged phoneme examples. A strategy that reshuffles 100-sample groups each epoch so the model sees many independent averages of the same class while preserving temporal locality — the single largest ablation gain (+13.3% relative) in the phoneme task.

Main Findings

• Competition win on phoneme classification. The model reached 65.8% F1-macro on the holdout for phoneme classification, winning the Standard track, and 88.9% F1-macro for speech detection versus an official baseline of 68.0%.
• Instance normalization is indispensable. Instance-level normalization dramatically improved holdout generalization in the phoneme task, outperforming batch normalization (+17.8%) and layer normalization (+88.2%) when comparing the effect of removing/altering normalization.
• Ensembling matters for phonemes. A single phoneme model averaged only 55.09 ± 1.77% across seeds; majority voting over five seeds raised the holdout score to 65.82%, a 19.48% relative boost.
• Low frequencies carry the signal. For speech detection, Delta, Alpha, and Beta bands each retained roughly 87% of full-band performance, while Gamma and HGA were at or below chance. For phonemes, Delta retained about 66% and higher bands contributed little in isolation.
• Distribution shift explains the holdout gap. The holdout phoneme split has a bimodal RMS energy distribution (peaks near 0.08 and 0.62) with much larger dispersion than the unimodal validation and test splits (0.54 ± 0.09 and 0.51 ± 0.09), motivating per-window normalization.
• Scaling behaves differently per task. Speech detection performance saturates with more training data, suggesting a ceiling for single-subject data, while phoneme classification continues to improve with additional hours.
• Longer windows help speech detection. Extending windows from 0.5 s to 2.5 s gave the largest speech-task ablation gain (+10.8% relative), and overlapping training windows with a 60-sample stride added a significant +2.8%.
• Phonetic-feature probes are partly decodable. Binary classifiers for voicing, plosive, and fricative features performed above chance, but the rare affricate category (0.97% of samples) stayed at chance despite ensembling, transfer learning, and gradient accumulation.

Methodology in Plain English

The researchers took a Conformer — a neural network that mixes convolutional layers (good at local patterns) with self-attention (good at long-range context) — and adapted it to process raw MEG sensor time series. A single 1D convolution first projects the 306 MEG channels down to a manageable width, then a stack of Conformer blocks processes the sequence, and a final linear layer produces either a speech/silence probability or one of 39 phoneme labels.

For speech detection, they used 2.5-second windows, trained with binary cross-entropy and label smoothing, applied their MEGAugment masking scheme, and smoothed predictions at inference by removing speech segments shorter than 240 ms. For phoneme classification, they used 0.5-second windows from the 100-sample averaged examples the competition provides, applied inverse-square-root class weighting to handle the long-tailed phoneme distribution, and included a per-window normalization layer directly on the input. The key training trick for phonemes was the dynamic grouping loader, which resamples groups each epoch so that each class is seen through many different averaged windows rather than the same fixed ones. Final predictions came from majority voting across five model seeds. All training used AdamW, a fixed learning rate and weight decay, early stopping on validation F1-macro, and the official competition splits and metric.

Why This Matters

Impact on research. The work demonstrates that architectures from mainstream speech recognition transfer productively to invasive-free neural decoding, reinforcing the convergence of ASR and neural signal processing. It also provides a thorough characterization of which frequency bands carry decodable speech information in MEG, and shows that distribution shift between recording sessions — not model capacity — is often the real bottleneck.

Real-world applications:

• Assistive communication. Decoding intended or perceived speech from non-invasive brain signals could eventually support communication aids for people with severe motor impairments, without requiring surgical implantation.
• Clinical monitoring. Speech detection from MEG could help track auditory and language processing in patients undergoing presurgical evaluation or in disorders of consciousness.
• Neuroscience research tools. The frequency-band and phonetic-feature analyses offer interpretable probes for studying how the brain encodes speech.
• Benchmark development. The LibriBrain competition results and the identification of split-level distribution shift inform how future neural decoding benchmarks should be designed and evaluated.

Industry relevance. Companies working on brain-computer interfaces, neurotechnology wearables, and AI-driven hearing or speech prosthetics have a direct interest in robust, generalizable decoders that tolerate session-to-session variability. The finding that simple instance normalization and ensembling solve much of the generalization problem is practical guidance for anyone deploying neural decoders in real, non-stationary recording conditions.

Future Directions

1. End-to-end speech reconstruction. Extending the decoder to sequential objectives such as CTC or sequence-to-sequence heads could enable reconstruction of continuous speech or even synthesis, rather than just discrete phoneme labels.
2. Cross-subject generalization. LibriBrain is single-subject; whether the architecture and normalization strategies transfer across participants remains an open and important question.
3. Multitask and feature-based formulations. Jointly learning shared articulatory subspaces (voicing, manner of articulation) may improve rare-class decoding and interpretability, though the current affricate results show this is not straightforward.
4. Scaling with larger datasets. The data-size ablation suggests phoneme decoding has not plateaued, so larger within-subject MEG corpora could yield substantial additional gains — and would test whether the observed speech-detection ceiling is fundamental.

Target Audience

Researchers and graduate students in neural decoding, brain-computer interfaces, and speech processing who want a concrete, reproducible template for adapting ASR architectures to non-invasive brain signals. It is also valuable for practitioners in neurotechnology who care about robustness to recording-session distribution shift, and for competition participants or benchmark designers interested in the LibriBrain PNPL tasks and the methodological details behind a winning phoneme system.

Authors’ abstract

Decoding speech-related information from non-invasive MEG is a key step toward scalable brain-computer interfaces. We present compact Conformer-based decoders on the LibriBrain 2025 PNPL benchmark for two core tasks: Speech Detection and Phoneme Classification. Our approach adapts a compact Conformer to raw 306-channel MEG signals, with a lightweight convolutional projection layer and task-specific heads. For Speech Detection, a MEG-oriented SpecAugment provided a first exploration of MEG-specific augmentation. For Phoneme Classification, we used inverse-square-root class weighting and a dynamic grouping loader to handle 100-sample averaged examples. In addition, a simple instance-level normalization proved critical to mitigate distribution shifts on the holdout split. Using the official Standard track splits and F1-macro for model selection, our best systems achieved 88.9% (Speech) and 65.8% (Phoneme) on the leaderboard, winning the Phoneme Classification Standard track. For further implementation details, the technical documentation, source code, and checkpoints are available at https://github.com/neural2speech/libribrain-experiments.