The Score Is Not the Structure: Brain Alignment and Cross-Lingual Transfer

AI summary

Author: Saman Rahbar (Independent researcher, Vancouver, Canada) · arXiv: 2610.03827v1 [cs.CL], 02 Oct 2026 · Venue: Linguistic Principles for Foundation Models (LP4FM) workshop · License: CC BY 4.0

Overview

Research area: Natural Language Processing methodology, specifically the validity of representational similarity measures used in cross-lingual probing and model–brain alignment, with a secondary experimental component on activation steering and brain-alignment fine-tuning.

Technical level: Advanced. The paper assumes familiarity with linear probes, centered kernel alignment (CKA), permutation/Mantel testing, rank reduction, and fMRI reliability ceilings.

Scope: The paper audits two widely used correspondence scores — cross-lingual probe transfer across 17 languages, and CKA alignment between Pythia models and an fMRI benchmark — to determine how much of each score survives when the correspondence supposedly being measured is destroyed or measured with an unreliable instrument.

What This Paper Is About

Researchers often claim that a model shares structure with something else — the human brain, or another language — and support that claim with a similarity score. This paper asks a prior question: what does such a score actually read when the shared structure is absent, or when the tool used to measure it does not work? Running that check in two independent settings, the author finds that in both cases the score is not what its magnitude suggests, and that neither surviving correspondence yields a downstream or causal benefit over matched controls.

Key Contributions

1. A three-part audit template applied to correspondence measures. The paper formalizes an ordered procedure — a content ablation (rebuild the reference so it keeps incidental properties but loses the correspondence), an instrument check (is the score estimated with comparable reliability everywhere it is compared?), and an inference check (are the units counted correctly?) — and runs all three in two unrelated settings. Which part proves decisive differs between them: the instrument dominates the cross-lingual study; the ablation dominates the brain-alignment study.

2. A cross-lingual instrument failure. Across 17 languages and 272 ordered language pairs, within-language probe accuracy co-varies with typological distance at r = -0.74. Four languages (Hindi 0.375, Urdu 0.433, Turkish 0.493, Arabic 0.497) sit at or below the 0.5 chance level, meaning 64 of the 272 pairs are scored using a tool that does not work. The paper shows the attenuation that follows cannot be separated from range restriction in this design.

3. A quantified floor for CKA against a dimension-reduced target. A destroyed (shuffled or noise) fMRI target already scores CKA = 0.204 and 0.186 against the real one, and a rank sweep gives 0.051, 0.102, 0.204 and 0.409 at ranks 32, 64, 128 and 256 — matching k/n for n = 627 sentences to within 1% at every point. The author frames this as a measured regularity of the construction rather than a proven identity, and notes it can be checked in one line of code with no model.

4. A null result on utility for both correspondences. Brain alignment never significantly improves BLiMP over LM-only, shuffled, or random controls, and cross-lingual steering shows no distance-graded causal effect once the 17 languages rather than the 272 pairs are treated as the unit of inference.

Main Findings

• The transfer gradient replicates. Probe transfer accuracy decreases with URIEL/lang2vec syntactic distance at r = -0.66 (R² = 0.43), reproducing the relationship that motivates URIEL-based transfer prediction.

• The instrument fails where the predictor is largest. Within-language probe accuracy declines with a language's mean distance to the other sixteen at r = -0.74. Four of seventeen languages do no better than guessing: Hindi 0.375, Urdu 0.433, Turkish 0.493, Arabic 0.497, against chance 0.5. Mean within-language probe accuracy is 0.729, ranging from 0.375 (Hindi) to 0.901 (Finnish).

• Restricting to working probes halves the explained variance. On the 13 languages whose probe clears chance at both endpoints (156 pairs), r = -0.47, R² = 0.22. Bootstrapping over languages, the full-cohort correlation is [-0.79, -0.50] and the reliability gradient [-0.89, -0.51], but the restricted one spans zero at [-0.77, +0.18]. A permutation test still rejects no-association (p = 0.025).

• Training size and data quantity do not explain the four failures. Every language is capped at 300 minimal pairs and trained on 60%, so sixteen of seventeen fit on exactly 180 examples and Basque on 163. Accuracy against the number actually used correlates at r = -0.04. Basque, with the least training data of any language, still reaches 0.755 — above all four failures. (MultiBLiMP supplies between 273 usable pairs for Basque and 3,832 for Russian; the apparent r = +0.42 against the number available is an artifact of availability being irrelevant beyond the cap.)

• The attenuation is not attributable to validity alone. The four failing languages rank first, third, fourth and fifth of seventeen by mean typological distance. Of all 2,380 four-language exclusions, only four restrict the range as much as the invalid set, and three of those four attenuate the correlation at least as strongly (r from -0.48 to -0.33, against -0.47 for the invalid set). Dropping the four most distant valid languages gives a summed distance of 1.047 versus the invalid set's 1.307, so there is no distance-matched comparison. The author explicitly does not claim half the gradient is measurement artifact, only that the transfer gradient is not identifiable without a control separating the two effects.

• Counting units changes the verdict. Pairwise significance for the corrected steering result is p = 0.0006; treated as 17 languages, the correct Mantel-style permutation gives p = 0.155. Each language appears in 32 of the 272 pairs, and URIEL distances are phylogenetically structured, so pairs are not exchangeable. The transfer gradient itself survives a language-label permutation (p = 10⁻⁴; no permutation of 10,000 reached the observed r).

• The steering intervention demonstrably works before its variation is tested. Steering a language by its own direction moves the grammatical-preference margin by +6.21 nats on average, positive in 16 of 17 languages (sign-flip p ≤ 5×10⁻⁵). Pooled over all 272 ordered pairs, steering B by A's direction beats the per-target random baseline by +2.85 nats on 223 of 272 pairs; averaging within each source language gives a positive advantage for all seventeen (sign-flip p = 10⁻⁴).

• It confers no independent causal benefit. Raw steering benefit shows no relationship with distance (r = +0.08, p = 0.586). A random-direction baseline points the same way (r = +0.24, p = 0.081). With a per-target random baseline averaged over twenty directions, the source-specific advantage appears to decay (r = -0.22) but does not survive the language-level null (p = 0.155). Partialling out probe-transfer accuracy leaves nothing and reverses the sign (r = +0.11, p = 0.455). Restricted to the 13 above-chance-probe languages, r = -0.09 (p = 0.704).

• The alignment loss reshapes geometry. End-of-training CKA between the aligned layer and the fMRI target rises from 0.095 without the alignment loss to 0.337 with it (reported as 0.10 to 0.34 in rounded form). The subject-split noise ceiling is approximately 0.54, against a raw-voxel reliability of 0.11, so the objective reached 62–78% of attainable reliable structure across scales.

• Most of that reshaping is not brain-specific. Shuffled targets reach 0.31 and rank-matched Gaussian noise reaches 0.31, recovering 83–92% of the brain-aligned score. The brain-specific increment over those controls is +0.028, +0.068 and +0.064 at 160M, 410M and 1.4B (paired over shared seeds, sign-flip p < 0.01 at every scale), or 5–13% of the ceiling.

• The measure returns a large number with no correspondence at all. A shuffled target scores CKA = 0.204 against the real one and a noise target 0.186, before any model is involved. The rank sweep matches k/n within 1% at all four tested ranks. The paper distinguishes this floor (a similarity baseline between two matrices built the same way) from the ceiling (a reliability bound), and notes the ceiling itself sits above the k/n baseline rather than above zero.

• Brain alignment confers no downstream benefit. At 160M over twenty seeds, brain-aligned reaches 74.5% on BLiMP against 74.1% for LM-only, a 0.48-point difference that is not significant (p = 0.36), and is not significantly above shuffled (74.2%, p = 0.57) or random (73.6%, p = 0.15). The pattern repeats at 410M and 1.4B (p = 0.36, 0.29, 0.55); at 1.4B both controls sit numerically above it (83.6% and 83.7% versus 82.8%). Equivalence tests bound the 160M effect within [-0.4, +1.4] BLiMP points, and within [-0.4, +1.6] and [-1.0, +2.0] at the larger scales.

• Four explanations for the null are ruled out. The target is not too noisy (the reliability ceiling); the alignment weight is not mis-tuned (pushing λ to 10 raises CKA to 0.38 and still buys nothing); the models are not too small (three scales behave alike); the training regime is not inert (fine-tuning moves BLiMP by 3–6 points, though downward and shared by all four conditions, so it cancels in the comparison). Pretrained, no fine-tuning: 76.9 (160M), 86.4 (410M), 88.0 (1.4B).

• Both failures share a shape. In each case the contaminating quantity is correlated with the thing of interest rather than independent of it — second-order structure travels with a real fMRI target, probe reliability travels with typological proximity. A contaminant that co-varies with the signal looks like a stronger effect rather than like noise, which is why a confidence interval on the score does not help.

Methodology in Plain English

The paper runs two studies against a shared template.

Study 1 — cross-lingual transfer and steering. A logistic-regression probe is trained inside XGLM-1.7B to distinguish grammatical from ungrammatical subject-verb agreement, reading the residual stream at the middle decoder layer at the position of the agreeing verb. Training uses 60% of each language's minimal pairs from MultiBLiMP, which supplies sentences identical except for the agreement error, across 17 languages that mark agreement on the verb. For every ordered pair of languages, the study measures (a) how accurate language A's probe is when applied to language B, and (b) what happens when A's direction is added into the model while it reads B, asking whether B's preference for the grammatical sentence strengthens. That preference is measured as the margin in nats, the total log-probability of the grammatical sentence minus its ungrammatical twin, so zero means indifference. The direction is added at the same layer and token the probe reads, at two strengths scaled to activation size and averaged; random comparison directions are scaled identically so they differ only in where they point. Both transfer and steering are then regressed on URIEL/lang2vec syntactic distance.

Study 2 — brain-geometry alignment. Pythia models at three scales are fine-tuned on the stimulus text of the Pereira et al. (2018) fMRI benchmark (627 sentences), with a second objective pulling the model's internal geometry toward the fMRI responses. The similarity measure is linear CKA, which compares how two sets of representations arrange the same sentences relative to one another and returns 1 for identical arrangements; the loss is 1 − CKA(H,B) with weight λ. The fMRI target pools language-network voxels across participants and is reduced to rank 128 by PCA. Syntactic ability is then tested on BLiMP, a benchmark of English minimal pairs covering eight grammatical phenomena none of which appear in the fine-tuning text. Four conditions differ only in what the model is aligned to: LM-only (alignment switched off), Brain-aligned, Shuffled (targets permuted across sentences, keeping their statistics but destroying the sentence match), and Random (noise of the same dimension). A benefit is attributed to brain correspondence only if brain-aligned beats both controls.

The audit. The content ablation uses three forms of destroyed reference: shuffled rows, same-dimension noise, and a random direction matched in size. The instrument check asks whether the measuring tool is equally reliable everywhere — within-language probe accuracy in Study 1, the subject-split noise ceiling in Study 2. The inference check counts units correctly: permuting language labels rather than the 272 pairs in Study 1, pairing over shared seeds in Study 2. Only after these checks does the paper ask whether exploiting the surviving correspondence changes a downstream behavior.

Why This Matters

Impact on research. The paper argues that a correspondence score reported without an ablated control is close to uninformative about whether the reference structure is the brain, and that a transfer accuracy read off a failing probe is uninterpretable. It draws a sharp line between Hewitt and Liang's (2019) selectivity — whether a probe reads the representation or memorizes labels, a within-language property — and comparability, whether two probes measure equally well, which is what a cross-lingual regression assumes. It also notes that it did not find the between-language

Authors’ abstract

Researchers often support the claim that a model shares structure with the brain, or across languages, by reporting a similarity score. We ask what such a score reads when the shared structure is absent, or when the tool that measures it does not work. We check two settings, and in both the score is not what it appears. First, a probe trained to tell grammatical from ungrammatical sentences in one language transfers worse to more distant languages, the usual evidence for shared structure. But the probe itself gets worse along the same axis: in four of seventeen languages it performs at chance, so 64 of 272 language pairs are scored with a tool that does not work. Dropping those languages halves the strength of the relationship, but they are also the most distant, and this design cannot separate the two effects. Counting matters too: the same data give p = 0.0006 when the 272 pairs are treated as independent and p = 0.155 when the seventeen languages are, which is the correct unit. Second, training a language model to match human brain responses raises its similarity score from 0.10 to 0.34, against a ceiling of 0.54. A model trained on a target whose correspondence to the brain was destroyed still scores 0.31, so only 0.028 to 0.068 of the rise is specific to the brain. With no model at all, a destroyed target already sits at 0.204 from the real one, a floor that tracks the target's rank divided by the number of sentences. Finally, steering a language along its own direction works (+6.2 over a random direction in sixteen of seventeen languages) yet shows no effect that varies with language distance. Before asking whether a correspondence helps, ask how much of the score would survive without it.