Brain-tuning Improves Generalizability and Efficiency of Brain Alignment in Speech Models

AI summary

Overview

Research area: NeuroAI / computational cognitive neuroscience — specifically, aligning pretrained speech language models with human fMRI brain responses to naturalistic speech.

Technical level: Advanced. The paper assumes familiarity with self-supervised speech models (Wav2Vec2.0, HuBERT), fine-tuning methods such as LoRA, fMRI encoding models, cortical surface alignment (FreeSurfer, Glasser atlas), and metrics such as voxel-wise Pearson correlation normalized by a noise ceiling.

Scope: A single paper-length study introducing "Multi-brain-tuning" — fine-tuning pretrained speech models to jointly predict fMRI responses from several participants — and evaluating it for data efficiency, generalization to unseen participants and datasets, and downstream semantic task performance.

What This Paper Is About

Existing ways of measuring and improving the alignment between language models and human brain activity are participant-dependent: they need a lot of fMRI data from each individual and do not transfer well to new people, which blocks population-level analysis. This paper asks whether fine-tuning a pretrained speech model on brain responses from multiple participants at once can produce a model that needs less data per person, aligns better with the brain, and generalizes to entirely new participants and datasets. The authors also check whether this brain-tuning damages or improves the model's usefulness on ordinary speech and language tasks.

Key Contributions

1. A scalable multi-participant brain-tuning method. Pretrained speech models (Wav2Vec2.0 and HuBERT) are fine-tuned with low-rank adaptation (LoRA) to jointly predict fMRI responses from several participants whose data are first projected to a common cortical surface and parcellated into regions of interest, using a single shared projection head.

2. A demonstration of improved data efficiency. Brain-tuned models reach the maximum performance of the pretrained model using roughly one-fifth of the brain encoding data, i.e. a 5-fold reduction in the fMRI data required for reliable alignment with new participants.

3. Evidence of generalization across participants and datasets. Multi-brain-tuned models improve brain alignment by up to 50% over the maximum pretrained alignment and transfer to a held-out dataset (a subset of the Narratives dataset) with novel stimuli and unseen participants.

4. A demonstration of bidirectional benefit. Multi-brain-tuning does not cause catastrophic forgetting and improves downstream performance on two semantic/phonetic tasks, eventually matching an LLM-tuned baseline, plus ablations establishing best practices for tuning loss and LoRA rank.

Main Findings

• Five-fold data efficiency gain. Multi-brain-tuned models match the maximum pretrained brain alignment using approximately one-fifth of the brain encoding data. For held-out participants, the Multi-brain-tuned model also needed about one-fifth of the data, whereas the Single-brain-tuned model needed roughly twice that amount.

• Up to 50% higher brain alignment. With the full encoding data, brain-tuned models reached up to 50% more alignment than the pretrained maximum, and the Multi-brain-tuned model consistently outperformed the Single-brain-tuned model across all evaluated data fractions, for both Wav2Vec2.0 and HuBERT.

• Better scaling with tuning data. As the amount of brain-tuning data grew, Multi-brain-tuned models kept improving on held-out participants while Single-brain-tuned models saturated. For held-out participants, the two were similar when tuning data was under 6 hours.

• Widespread cortical improvement, with one exception. Voxel-wise comparison for a held-out Moth Radio Hour participant showed improvement especially in frontal and parietal regions, while the auditory cortex showed a slight decrease in alignment — attributed to the upper-middle and later model layers encoding more semantics, and possibly to larger semantic language areas dominating during brain-tuning (a pattern also supported by Vattikonda et al., 2025).

• Cross-dataset generalization. Models tuned on the Moth Radio Hour data and evaluated on 16 participants of the Narratives dataset improved over pretrained counterparts, with the Moth-Multi-brain-tuned model not lagging much behind a Narratives-Multi-brain-tuned model tuned on the same 16 test participants.

• No catastrophic forgetting; downstream gains. On Phoneme Prediction and Phonetic Sentence Type Prediction, brain-tuned models never underperformed the pretrained model at any data fraction. Multi-brain-tuning improved downstream performance as tuning data grew and eventually matched the LLM-tuned baseline, doing so with less data.

• Rank 8 is enough. Increasing the LoRA rank beyond 8 did not improve performance, especially at larger data sizes; larger ranks and full-model brain-tuning also did not beat rank-8 updates. Rank 8 corresponds to 0.625% of the total model parameters.

• The L2 loss scales best. The L2 objective performed better as brain-tuning data increased, while the Correlation loss and the combined Cosine + L2 loss saturated. At low tuning data sizes (≤ 6 hours), the Correlation loss outperformed the other losses.

• Training setup specifics. Training used the three participants with the most data from the Full Moth Radio Hour dataset as tuning participants; the rest were held out. The split was 2 validation stories, 1 held-out test story never used in training, and the remaining 81 stories for training. Batch size was 128 audio–fMRI pairs, for 30 epochs, taking approximately 6 hours on two NVIDIA A40 48GB GPUs.

Methodology in Plain English

The authors start from two off-the-shelf self-supervised speech models — Wav2Vec2.0 and HuBERT — each with about 90M parameters, 12 transformer layers, an embedding dimension of 768, and a 20 ms input token length, pretrained on a 960-hour audio dataset unrelated to the fMRI data used here.

Brain data comes from the Moth Radio Hour dataset, the largest publicly available per-participant fMRI dataset, with 8 participants who listened to autobiographical stories. Three participants heard 84 stories (about 16.1 hours of audio each) and the remaining participants heard 27 stories (about 6.4 hours). Images were acquired every 2 seconds (TR = 2.0 s). A separate subset of the Narratives dataset — 16 participants listening to a 56-minute fictional short story, TR = 1.5 s — is used purely to test generalization to a new dataset.

Because brains differ in size, shape, and region boundaries, each participant's data is projected onto a common cortical surface with FreeSurfer v7 and parcellated using the Glasser et al. (2016) atlas, keeping auditory regions (A1 through A4) and late language regions such as angular gyrus, lateral temporal cortex, inferior frontal gyrus, and middle frontal gyrus. This yields 30K voxels across both hemispheres.

For training data, audio is cut into 2-second snippets, each concatenated with the preceding 8 seconds (4 TRs) to account for the delayed blood-flow response, giving a 10-second audio clip paired with one fMRI TR.

The model has an average pooling layer and a single shared projection head on top of the frozen speech model. For a batch of audio and the corresponding responses from participants P1…Pn, the loss is computed and backpropagated independently and sequentially for each (stimulus, participant) pair. The authors report this works better than averaging the losses, averaging the responses across participants, using separate projection heads per participant, using Shared Response Modeling instead of FreeSurfer, or using non-linear heads. Only the LoRA parameters and the projection head are updated.

For evaluation, speech features are extracted with a 16.0 s sliding window (stride = 0.1 s), interpolated with a Lanczos filter to the fMRI acquisition rate, and concatenated across the preceding 10 s. A per-voxel ridge regression is fit and evaluated with Pearson correlation on held-out data; correlations are normalized by an estimated voxel-wise noise ceiling, and alignment is averaged over the upper-middle model layers and across participants, with standard errors reported.

Baselines include Single-brain-tuned (one model per participant, same recipe), LLM-tuned (replacing brain responses with LLama2-7B representations), and Stimulus-tuned (fine-tuning with the stimulus audio using the original self-supervised objective).

Why This Matters

Impact on research. The paper argues that participant-dependent brain alignment has blocked population-level neuroscience using language models. By pooling participants and sharing a projection head, the method cuts the per-participant fMRI data burden by a factor of five and generalizes to new participants and new datasets — a step toward participant-agnostic, scalable models of language processing in the brain. The downstream results also give direct evidence of a bidirectional benefit: brain data improves the model's semantic representations just as the model helps explain the brain.

Real-world applications (as implied by the paper's framing):

• More inclusive cognitive neuroscience studies that can include participants who cannot provide large amounts of scanning data.
• Integration of brain-tuned speech models into speech processing pipelines, since the authors note downstream linguistic utility is not compromised.
• Reduced cost and scan time in fMRI-based language research, because far less data per participant is needed for a reliable alignment estimate.
• Reuse of a single brain-tuned model across studies, participants, and datasets rather than training a new model per person.

Industry relevance. Speech and language model developers gain a fine-tuning recipe (LoRA rank 8, L2 objective, shared projection head) that improves semantic task performance on phonetic and phoneme tasks without hurting the base model. The released code and trained models lower the barrier to adopting the method.

Future Directions

• Beyond language regions. The authors focused on language-related regions because they are most directly involved in processing the naturalistic speech used here; they note the method is flexible enough to target non-language or specific brain areas to study their functional roles.

• Multilingual brain-tuning. All experiments were in English, reflecting the availability of large public fMRI datasets. Future work could test whether brain-tuning learns language-independent, generalizable semantic representations.

• Better training losses. The authors explored several objectives and found the L2 loss scales best while the Correlation loss wins at ≤ 6 hours of tuning data, but they state there is potential for new loss functions with even better data efficiency and generalization.

• More data and further scaling. The strong upward trend on held-out participants suggests additional improvement is possible if more data is integrated during brain-tuning.

Target Audience

This paper is most useful to researchers working at the intersection of natural language processing and cognitive neuroscience — particularly those building brain encoding or decoding models, aligning language models with fMRI, or conducting population-level neuroimaging studies of language. It also suits machine learning practitioners interested in LoRA-based fine-tuning with non-standard supervision, and computational neuroscientists looking for a practical, reproducible recipe for multi-participant modeling. Readers without a background in fMRI encoding analysis or self-supervised speech models will need to consult the referenced prior work (Moussa et al., 2025; Antonello et al., 2024; Vaidya et al., 2022) to follow the methodological details.

Authors’ abstract

Pretrained language models are remarkably effective in aligning with human brain responses elicited by natural language stimuli, positioning them as promising model organisms for studying language processing in the brain. However, existing approaches for both estimating and improving this brain alignment are participant-dependent and highly affected by the amount of data available per participant, hindering both generalization to new participants and population-level analyses. In this work, we address these limitations by introducing a scalable, generalizable brain-tuning method, in which we fine-tune pretrained speech language models to jointly predict fMRI responses from multiple participants. We demonstrate that the resulting brain-tuned models exhibit strong individual brain alignment while generalizing across participants. Specifically, our method leads to 1) a 5-fold decrease in the amount of fMRI data needed to predict brain data from new participants, 2) up to a 50% increase in the overall brain alignment, and 3) strong generalization to new unseen datasets. Furthermore, this multi-participant brain-tuning additionally improves downstream performance on semantic tasks, suggesting that training using brain data from multiple participants leads to more generalizable semantic representations. Taken together, these findings demonstrate a bidirectional benefit between neuroscience and AI, helping bridge the gap between the two fields. We make our code and models publicly available at https://github.com/bridge-ai-neuro/multi-brain-tuning.