Scaling Parameter-Efficient Fine-Tuning for Brain Encoding

AI summary

Overview

Research area: Computational neuroscience and machine learning, specifically brain encoding (predicting neural responses to naturalistic stimuli) combined with parameter-efficient fine-tuning (PEFT) of large multimodal foundation models.

Technical level: Intermediate. The paper builds on LoRA, transformer causal masking, and fMRI encoding benchmarks; readers should know what a pretrained backbone and a linear readout are, but the core argument is conceptually simple.

Scope: The paper tests whether small low-rank adapters, rather than frozen backbones or full fine-tuning, can adapt very large multimodal models (up to 104B parameters) into state-of-the-art whole-brain fMRI encoding models, while remaining efficient, interpretable, and scalable to large model populations.

What This Paper Is About

Brain encoding models try to predict neural activity (here, whole-brain fMRI) from the stimuli a person is experiencing. The standard recipe takes a large pretrained model, freezes it, and trains only a small readout on neural data, because neural recordings are expensive and scarce; fully fine-tuning or training from scratch has generally worked poorly at this data scale. This paper asks whether a small number of trainable adapter parameters can instead reshape the internal representations of very large multimodal backbones so they become better brain-encoding models, and whether that adaptation can be done cheaply enough to scale to many models at once.

Key Contributions

1. Efficiency of large-scale adaptation. The authors exploit the backbone's causal masking to encode an entire 149 s (100 TR) window in a single forward pass instead of overlapping sliding windows (which required 2,800 forward passes per clip in the prior state of the art). This cuts feature-extraction cost by approximately 350× (from about 700 to 2 GPU-hours on an A800), and they add a 60 s context lead to recover the small accuracy loss. They also build a batched multi-LoRA training kernel that fuses the updates of N adapters into a single batched operation.

2. Effectiveness across nine backbones. LoRA (rank 16, inserted into attention modules through layer 32 of a truncated decoder) plus a lightweight readout improves brain encoding on all nine multimodal backbones tested, spanning 4.7B–104B parameters. On the same Qwen3-Omni backbone as the prior state of the art (MIRAGE), their method, BrainLoRA, reaches 0.327 ID and 0.228 OOD while training only 61.4M parameters, roughly 5% of MIRAGE's 1.1B-parameter readout.

3. Interpretability of the adapted multimodal backbone. Using exact Shapley values over the video, audio, and text inputs, they recover the canonical sensory topography (early visual cortex dominated by video, auditory and lateral temporal cortex by audio) and show that adaptation reweights modalities toward audio and nearly doubles cross-modal mixing inside the backbone.

4. Scalability to a population of models. With the multi-LoRA kernel they train 261 independently adapted models and combine them with a parcel-wise minimum-variance ensemble, reaching 0.343 ID and 0.248 OOD and ranking first on the public Algonauts post-challenge leaderboard at the time of writing; a selected 15-model subset reaches 0.341 ID and 0.247 OOD.

Main Findings

• LoRA helps every backbone tested. Across the nine multimodal backbones (4.7B–104B parameters), LoRA improved linear-readout performance by 0.032 to 0.066, with cross-attention adding a further 0.003 on average. Gains appeared in every parcel, largest in parietal and visual areas.
• Bigger backbones are only weakly better. Encoding performance correlated with backbone size at r = 0.71 for frozen backbones and r = 0.60 for adapted ones.
• A small adapter beats a large readout. BrainLoRA (cross-attention) reached 0.331 validation, 0.327 ID, and 0.228 OOD, versus MIRAGE's 0.319 validation, 0.310 ID, and 0.217 OOD, and matched or exceeded the published 15-model MIRAGE ensemble (0.335 / 0.323 / 0.227) on ID while matching it on OOD. The linear variant (61.2M parameters) already reached 0.326 / 0.322 / 0.222.
• Full fine-tuning does not win. Fine-tuning all 20.25B parameters below the feature-extraction layer reached 0.323 (linear) and 0.330 (cross-attention) on validation, at or slightly below the corresponding LoRA configurations, with 330× more trainable parameters, and dropped further on OOD.
• Component ablation (validation, Season 6). Frozen backbone 0.2907; +LoRA rank 16 0.3263 (+0.0356); without the closed-form re-fit 0.3202 (the re-fit contributes +0.0061); rank 32 0.3268; gated-MLP readout 0.3269; cross-attention 0.3310; no audio 0.3026. Removing audio hurts more after adaptation.
• The adapter transfers. Trained on three subjects and evaluated on a held-out fourth with LoRAs frozen and only the readout re-fitted, it retained 97% of the in-distribution improvement. The adapter and readout also transferred zero-shot to the Naturalistic Neuroimaging Database, gaining +11.1 percentage points of noise ceiling, comparable to the +10.7-point fine-tuning gain on the validation set.
• Interpretable modality topography. Early visual cortex was 72.5% video-driven, auditory cortex 72.9% audio-driven, and lateral temporal cortex 60.3% audio versus 12.5% text; text contributed most to default-mode (21.6%) and control regions. Attribution maps were stable (donor-window baseline r = 0.994; two seeds r = 0.996–1.000).
• Adaptation shifts weighting toward audio. Averaged over parcels, audio contribution rose 39% and video 5%, while text fell 17%. The dominant modality changed in 249 parcels, all toward audio (246 video-to-audio, none the reverse), and seeds agreed on the dominant modality for 97.2% of parcels.
• Adaptation increases cross-modal mixing. 15.3% of the prediction was attributed to information read out at another modality's token positions, with the largest cross-modal route being video information at audio tokens (7.5%), versus only 8.9% in the frozen backbone.
• The multi-LoRA kernel is fast. At N = 32 adapters, one fused step cost 1.08× a single-adapter step, a 29.8× speedup over sequential training; end-to-end speedups were smaller because of longer sequences and larger batches.
• Ensemble scaling follows theory. Ensemble accuracy versus size fit m(n) = m∞ − A·n^(−b) with b ≈ 0.94 and R² = 0.998, consistent with a residual-variance term that shrinks as 1/n. Across 2,000 random 20-model ensembles, mean single-model accuracy and mean pairwise residual correlation explained 23% and 5% of variance alone but 97% jointly (the two properties correlate at r = 0.74).
• Diversity comes from architecture, not seeds. Different readout architectures or backbones reduced residual correlation with the pool; seed, LoRA rank, layer choice, and training-data subset produced little diversity. Several apparent gains (higher LoRA rank, gated-MLP readout) disappeared after the closed-form re-fit.

Methodology in Plain English

The task is to predict fMRI activity, parcellated into 1,000 brain regions, for four subjects watching seven seasons of the TV show Friends with synchronized video, audio, and transcript (the CNeuroMod/Algonauts 2025 benchmark). Training uses the four subjects, validation uses Season 6, and the challenge server scores held-out Season 7 (in-distribution) and held-out movies (out-of-distribution).

The model is a large pretrained multimodal backbone, mainly Qwen3-Omni, with a small low-rank adapter inserted into its attention layers through layer 32, after which the decoder is truncated. The adapter injects only 8.91M trainable parameters. On top sits a readout that pools backbone features into one vector per modality at each 2 Hz timestep, aggregates them into eight log-spaced temporal-lag blocks, and projects to the 1,000 parcels. A linear readout has 52.3M trainable parameters; a cross-attention variant works over a larger token set of about 3.2k tokens and adds only 0.2M. Adapter and readout are trained jointly for one epoch, then the linear readout is re-solved in closed form with parcel-wise ridge regression.

The computational trick is that the backbone is already causally masked, so instead of re-encoding an overlapping window for every fMRI timestep, one pass over the whole clip yields features at all timesteps. A 60 s context lead compensates for the missing context at the start of each window.

For interpretability, the authors compute exact Shapley values over the three input modalities by replacing each modality's embeddings with content-free mean embeddings while preserving token count, position, and timing, and evaluating all eight modality subsets. Applying this separately to each of the three hidden states the readout consumes produces a 3×3 attribution matrix separating which input modality's information reaches the readout through which token position.

For scaling out, a custom kernel fuses the updates of many adapters into one batched gather-and-matrix-multiply so that dozens of LoRAs train in parallel on a shared backbone. The resulting pool of 261 models is combined with a parcel-wise minimum-variance ensemble fitted on out-of-fold predictions.

Why This Matters

Impact on research. The paper argues that the field's dominant "frozen backbone plus readout" recipe leaves the backbone's representational capacity unused, while full fine-tuning overfits scarce neural data. It reframes PEFT as a third regime and gives three concrete scaling axes (scale up the prior, scale down the adapter, scale out the number of adapters). It also shows that a single multimodal backbone can remain interpretable without building separate unimodal encoders, using attribution instead of architectural separation. All models and ensemble weights were selected on validation data alone, which matters for a benchmark where leaderboard tuning is a known concern.

Real-world applications:

• Brain-computer interfaces and assistive decoding, where per-user calibration data is limited and cheap adaptation to a new person is essential.
• Clinical and pre-surgical mapping, where a model that predicts responses across the whole cortex and localizes which sensory modality drives each region could support planning and assessment.
• Cognitive neuroscience experiments, where the adapted model's modality attributions can be used to test hypotheses about how a region processes information.
• Low-data transfer to new imaging sites, demonstrated by zero-shot transfer to the Naturalistic Neuroimaging Database with a different scanner and repetition time.

Industry relevance. The multi-LoRA kernel and the demonstration that dozens of adapters can be trained for roughly the cost of one are directly relevant to serving and training many personalized model variants over a shared backbone. The finding that architecture variation, not seed or hyperparameter variation, produces useful ensemble diversity is a practical lesson for anyone building model ensembles.

Future Directions

• A brain foundation model. The authors suggest a single shared adapter trained on neural data spanning tasks, datasets, subjects, and even species, with hierarchical or meta-learned adapters per task, dataset, subject, or region of interest. They explicitly state their experiments do not yet constitute such a model.
• Beyond narrative film. All training and both transfer tests stay within narrative film with four subjects. Joint training across multiple datasets and stimulus modalities is left to future work.
• Cheaper ensembles. Because ensemble performance is jointly governed by size, individual quality, and diversity, and because the fitted curve has a nonzero ceiling set by residual correlation, finding better ways to generate architectural diversity could raise the ceiling rather than just the count.
• Broader PEFT and backbone comparisons. The paper shows IA³ and adapter layers behave similarly to LoRA, but only LoRA is tested at the largest scales; whether other PEFT families scale as well to 104B models remains open.

Target Audience

Researchers and graduate students in computational neuroscience, vision and auditory neuroscience, and machine learning who work on encoding models or on adapting large pretrained models to small scientific datasets. It is also useful for practitioners of parameter-efficient fine-tuning who want a rigorous case study of LoRA at 100B scale, multi-adapter training infrastructure, and ensemble design under scarce data. Readers need only moderate familiarity with transformers and fMRI analysis; the paper is not aimed at beginners in either field.

Authors’ abstract

Brain encoding models aim to predict neural responses to stimuli. The most widely used approach fits readouts from frozen pretrained backbones. Using neural data to train large models from scratch or fully fine-tune them after task-pretraining has proven difficult because neural data is costly and limited. We show that parameter-efficient fine-tuning (PEFT) enables us to effectively adapt very large pretrained backbones for brain encoding with a small number of trainable parameters. To make it computationally practical, we exploit the backbone's causal structure to predict multiple timepoints in a single pass instead of using sliding windows, significantly reducing feature-extraction cost. This enables efficient fine-tuning of multimodal backbones with up to 104B parameters, the largest model adapted for brain encoding to our knowledge. Trained with a low-rank adapter (LoRA) and a lightweight readout, our approach consistently improves brain encoding performance across nine diverse multimodal backbones. On the same Qwen3-Omni backbone as the prior state-of-the-art (SoTA), our method BrainLoRA achieves the best published performance on the Algonauts 2025 Challenge. We also show the adapted multimodal backbones remain interpretable: modality attribution recovers the canonical sensory topography that prior work often obtains by construction from separate unimodal encoders, and neural adaptation increases the contribution of audio inputs and cross-modal mixing within the backbone. To scale out the number of adapted models, we develop a multi-LoRA kernel that accelerates joint training of multiple LoRAs on a shared backbone. This enables an ensemble of 200+ adapted models that ranks first on the public Algonauts post-challenge leaderboard at the time of writing. Together, our results highlight PEFT as an efficient and powerful paradigm for brain encoding.