Scaling and context steer LLMs along the same computational path as the human brain

AI summary

Overview

Research area: Neuro-AI — the comparison of internal representations in large language models with temporally resolved human brain activity (magnetoencephalography, MEG) during natural speech listening.

Technical level: Intermediate. The core comparison is conceptually simple (align model layers to brain time points), but the methods draw on ridge regression encoding models, cross-validated Pearson correlations, and MEG preprocessing conventions.

Scope: A single study using one public 3-subject MEG dataset (10 hours of audiobooks each) and a benchmark of 22 LLMs to test whether the order in which LLM layers generate representations matches the order in which the human brain generates them, and which factors (architecture, size, context length, word predictability) control that correspondence.

What This Paper Is About

Prior work established an anatomical alignment between LLM layers and brain regions: early layers match primary sensory cortices and deeper layers match higher-level associative areas. What remained unclear is whether the two systems also produce representations in the same temporal order — a shared "computational path" — and whether that ordering is systematically present across models. The authors ask whether this temporal alignment is universal across LLM architectures, and whether it depends on model size, context length, or the predictability of the words being processed.

Key Contributions

1. Demonstration of temporal alignment. Across a benchmark of 22 LLMs, the time at which each layer's brain-alignment score peaks increases monotonically with layer depth, matching the sequence of MEG responses to words. Averaged across the nine representative models, the temporal score is r = 0.99 (p < 1e-06).
2. Architecture independence. The effect holds for transformers and for non-transformer state-space / recurrent models (Mamba-1.4B-hf, RecurrentGemma-9B), suggesting the alignment is not tied to a specific inductive bias of the transformer.
3. Scaling and context as drivers. Using the Pythia family (8 models trained identically, only size varying), temporal alignment emerges logarithmically with model size, going from non-significant (r = 0.44, p > 0.05 at 14M parameters) to highly significant (r = 0.96, p < 1e-4 at 12B parameters). Varying context length on Llama-3.2 3B produces a similar logarithmic emergence, from non-significant (r = 0.19, p > 0.5 with no context) to r = 0.93 (p = 3e-4) at 1000 words of context.
4. Ruling out word predictability as the sole explanation. Splitting words into predictability quartiles using Llama-3-8B's own softmax logits, both the most expected (r = 0.92, p < 1e-3) and most surprising quartiles (r = 0.83, p < 1e-2) show temporal alignment, and the layer-wise difference in peak times is not significant (p = 0.61).

Main Findings

• Alignment peaks around 0.4 s after word onset. The linear mapping from MEG sensors to LLM activations yields Pearson correlations that rise to a maximum roughly 0.4 s post-word onset.
• Layer depth tracks brain response latency. Early layers best align with early brain responses and deeper layers with later responses. This is quantified by a "temporal score": the Pearson correlation between the time of peak alignment (T_max) and the relative layer depth across 9 layers spanning 10% to 90% of the hierarchy.
• Representative models show strong temporal scores. Averaged across the nine models highlighted in the main figures, r = 0.99 (p < 1e-06). The oldest model, GPT-2-XL, shows the lowest but still significant score, r = 0.85.
• Untrained models do not show the effect. Unpretrained versions of the models show no temporal alignment and encode brain activity very poorly (shown as dashed gray curves in Figure 2A).
• Bidirectional models align less temporally. BERT, RoBERTa, and the speech model Wav2vec2.0 produce comparable alignment scores but substantially lower temporal scores than the causal LLMs (reported in Appendix B).
• Size drives emergence, logarithmically. Within the Pythia family, the correlation between temporal score and log model size is r = 0.87 (p = 0.01), with both temporal and alignment scores plateauing at the largest scales.
• Context drives emergence, logarithmically. For Llama-3.2 3B, the correlation between temporal score and context length is r = 0.81 (p < 5e-2), with the increase slowing considerably beyond 50 words. A similar logarithmic effect is reported for Mamba in Appendix E.
• Scaling beyond 70B parameters shows little gain. Extending analyses to Llama 3.3 70B does not yield major improvement over smaller models such as Llama 3.2 3B, indicating a plateau (Appendix D).
• Temporal and alignment scores are correlated. Across all studied models and context lengths (clustered by family), the temporal score correlates significantly with the maximal alignment score: Pearson = 0.54, p = 9e-04.
• Best alignment is in intermediate layers. Consistent with earlier work, the highest alignment scores occur in intermediate representations (described as around layer = 0.6 of the architecture) rather than at the input or the prediction.
• Predictability does not explain alignment. The control analysis on quartiles of contextual predictability indicates temporal alignment is not merely a signature of autoregressive next-word prediction.

Methodology in Plain English

Brain side. Three healthy participants listened to 10 hours of audiobooks inside a CTF MEG scanner (the publicly available Armeni et al., 2022 dataset). Signals were band-pass filtered between 0.1 and 20 Hz, down-sampled to 30 Hz, time-locked to individual word onsets, and epoched from −2.5 s to +3 s around each word using MNE-Python. Each MEG channel and time point was z-scored across words. Only content words (nouns, verbs, adjectives, adverbs, as tagged by SpaCy) were analyzed in the main results; a control on all words appears in Appendix F.

Model side. The same text was fed to each LLM. Activations were extracted from 9 layers distributed linearly between 10% and 90% of the model hierarchy and reduced with Principal Component Analysis to 50 components. The benchmark covered 22 LLMs (including Llama-3-8B, Llama-3.2 1B and 3B, Mistral-7B-v0.1, Gemma-7B, Qwen1.5-7B, GPT-2-XL, Mamba-1.4B-hf, RecurrentGemma-9B, and the 8-model Pythia family), with a default context of 50 words unless varied deliberately.

Comparison. For each layer and each time point relative to word onset, a ridge regression (scikit-learn RidgeCV, regularization alpha searched logarithmically from 10⁻⁴ to 10⁸) was fitted to predict LLM activations from MEG activity, and evaluated by Pearson correlation on held-out data across five train–test folds. This correlation is the alignment score. For each layer, the peak-alignment time T_max was defined as the mean of the window where the normalized alignment score is at least 95% of its maximum. The temporal score is the Pearson correlation between T_max and relative layer depth across the 9 layers — essentially a test of whether deeper layers "fire" later in brain time.

Why This Matters

Impact on research. The work reframes brain–LLM similarity from a static property (which layer matches which region) to a dynamic one (which computation happens when). It provides a simple, architecture-agnostic metric — the temporal score — and shows that two very different design choices (parameter count and context length) control how brain-like a model's processing order becomes. It also extends the growing literature on scaling laws in neuro-AI from fMRI to the faster, temporally resolved MEG signal.

Real-world applications (as directions the findings could inform):

• Model evaluation beyond benchmarks. Temporal score could serve as a complementary, neuroscience-grounded diagnostic of whether a model processes language in a plausible sequential manner.
• Architecture design for long context. The comparison between transformers (which need large memory buffers) and state-space models (which compress context into a single hidden state) offers a lens for building efficient long-context systems.
• Cognitive modeling and language research. The alignment framework offers a tool for probing where human language processing and machine processing diverge, e.g., for syntax and semantic roles.
• Clinical and neural decoding research. The method of mapping brain signals to model activations is the same machinery used in brain-to-text decoding, where more brain-like models may improve decoding.

Industry relevance. The study was conducted by researchers at Meta AI together with neuroscience labs (Inserm U960, ENS–PSL), and the factors it identifies — model size and context length — are exactly the levers scaled in industrial LLM development, which makes the results relevant to how frontier model design choices affect similarity to human processing.

Future Directions

1. Disentangling causal factors. Whether context length and model size act directly on brain alignment or are confounded with uncontrolled variables, such as linguistic performance, remains open. The paper reports (Appendix I) that performance correlates with temporal alignment for models with increasing context lengths but not for models with increasing sizes.
2. Architectures not yet tested. Fully connected networks and Kolmogorov-Arnold Networks remain to be investigated. It is also unresolved whether the effect stems from the training objective (next-token prediction) or from the structure of language itself.
3. Better spatial resolution. MEG has limited spatial resolution, cannot isolate single neurons, and largely misses deep sources, so the precise neuronal basis of the findings requires intracranial recordings.
4. Individual differences and sensory modality. Only 3 subjects were studied (though each with 10 hours of audio), so how alignment varies across individuals is unknown. The work also covers only pretrained text models; comparing against speech models is left to future work.
5. Long-context memory under pressure. Evaluating whether state-space models' single-hidden-state context degrades in the absence of meaningful context — for example, memorizing a random digit sequence — remains to be tested.

Target Audience

• Neuro-AI and cognitive neuroscience researchers interested in how artificial and biological networks converge on similar computations.
• Machine learning researchers studying scaling laws, context length, and architecture choice, who want a neuroscience-derived probe of model behavior.
• Computational linguists and psycholinguists working on predictive processing and the temporal dynamics of word comprehension.
• Graduate students entering the brain–model alignment field, for whom this paper offers both a clear methodological template (ridge-encoded alignment scores plus a temporal ordering metric) and an explicit statement of limitations.

Authors’ abstract

Recent studies suggest that the representations learned by large language models (LLMs) are partially aligned to those of the human brain. However, whether and why this alignment score arises from a similar sequence of computations remains elusive. In this study, we explore this question by examining temporally-resolved brain signals of participants listening to 10 hours of an audiobook. We study these neural dynamics jointly with a benchmark encompassing 22 LLMs varying in size and architecture type. Our analyses confirm that LLMs and the brain generate representations in a similar order: specifically, activations in the initial layers of LLMs tend to best align with early brain responses, while the deeper layers of LLMs tend to best align with later brain responses. This brain-LLM alignment is consistent across transformers and recurrent architectures. However, its emergence depends on both model size and context length. Overall, this study sheds light on the sequential nature of computations and the factors underlying the partial convergence between biological and artificial neural networks.