The Alignment Illusion in Multimodal Large Language Models

AI summary

Overview

Research area: Interpretability and evaluation of Multimodal Large Language Models (MLLMs) — specifically, what layer-wise visual-text representational similarity actually measures inside a shared language-model pathway.

Technical level: Advanced. The paper uses principal-angle analysis, CKA, SVCCA, MIR, linear probing, and spectral analysis of weight matrices, though its central argument is stated in accessible geometric terms.

Scope (one sentence): Across 13 MLLMs from five families spanning 0.5B to 72B parameters, the paper shows that standard scalar alignment scores stay high even when visual content is destroyed, traces this to shared MLP down-projection geometry, and proposes the principal-angle gap (PA gap) as a more faithful diagnostic.

What This Paper Is About

Researchers commonly compute similarity between visual and text hidden states in MLLMs and read rising similarity with depth as proof that the model is integrating image content into a shared representation space. This paper tests that assumption directly by breaking the visual stream in controlled ways and checking whether the alignment scores notice. The authors find they often do not, and they work out why — and what to measure instead.

Key Contributions

1. Demonstrating the alignment illusion. Under visual-token interventions that sharply reduce task accuracy, four standard scalar measures (the leading principal-angle cosine σ₁, CKA, SVCCA, and MIR) fail to consistently separate the corrupted visual stream from the original across 13 MLLMs.

2. Tracing the illusion to shared weights. The authors show that alignment is amplified inside the shared language-model pathway, localize the amplification primarily to the MLP sublayer, show that the MLP down-projection W_out has dominant output directions, and demonstrate that the principal-angle basis concentrates in those directions. They name the resulting inflation weight-induced alignment and formalize it with a bound (Proposition 1) containing no term coupling the two inputs.

3. Introducing the principal-angle gap (PA gap). Defined as Δ_PA = σ₁ − σ₂, the difference between the top two principal-angle cosines, the PA gap separates one-dimensional weight-induced similarity from multi-directional visual structure.

4. Validating the PA gap under graded corruption and structured irrelevant input. Under α-mixing between original and noise tokens, the PA gap correlates with task accuracy more consistently than the scalar scores and dispersion baselines tested; under a structured but irrelevant image, it exposes a regime where internal geometry and task accuracy come apart.

Main Findings

• Accuracy collapses, alignment does not. Replacing projector-output visual tokens with isotropic Gaussian noise (rescaled to preserve per-token norm) drops MMBench task accuracy by 38 to 50 percentage points across all 13 models. The leading principal-angle cosine σ₁ changes little, closely matching or exceeding the original curve and approaching unity in deep layers.

• Standard scalar scores do not separate the settings. Across the 13 models, the separation score between Orig and Noise is negative at the median for σ₁, CKA, and SVCCA, and close to zero for MIR.

• The MLP sublayer drives the amplification. Bypassing the MLP produces a larger layer-averaged change in σ₁ than bypassing attention in all 13 models, with a 3.5× larger median effect. A random-initialization control indicates the inflated σ₁ under Noise depends on trained weight structure rather than being a generic architectural artifact.

• W_out is directionally anisotropic. The layer-averaged ratio s₁(W_out)/s₂(W_out) lies between 1.2 and 1.6 and exceeds a matched random reference in every model; for a Gaussian random matrix of the same shape this ratio is close to one.

• Principal-angle directions sit in W_out's dominant subspace. Projecting the visual-side principal-angle basis onto the top-10 left singular subspace of W_out captures 2.8× to 13.6× more energy than random directions of matched dimension.

• Original and noise MLP outputs share dominant geometry. The top-k PCA subspaces of the MLP-output distributions under Orig and Noise share a high leading principal-angle cosine (median 0.996).

• The PA gap tracks graded degradation best. Under α-mixing (α ∈ {0.0, 0.1, …, 1.0}), the PA gap achieves the highest mean |r| (0.894), median |r| (0.917), and minimum |r| (0.655) against layer-averaged accuracy; |r| exceeds 0.80 in 12/13 models and has the expected sign in all 13. For comparison: σ₁ mean |r| 0.730, CKA 0.752, SVCCA 0.736, MIR 0.760, participation ratio 0.775, spectral entropy 0.825. The PA gap is the only measure in the relevant comparison that ranks Orig above Noise in all 13 models.

• Irrelevant images are actively processed, not ignored. Irr falls below the text-only baseline by a median of 5.0 pp across the 13 models, while Noise is closer to Text with a median difference of 2.1 pp; Irr is below Noise by a median of 2.5 pp. Shuffling the original visual tokens (Shuf) leaves accuracy nearly unchanged (median 0.8 pp from Orig), showing the penalty is not generic sensitivity to visual-token perturbation.

• The irrelevant-image penalty is size-dependent. In the <3B group (n=3), the Irr-vs-Text gap shrinks to 2.9 pp; in the ≥3B group (n=10), the Irr-vs-Noise gap widens to 3.2 pp.

• The projector encodes the irrelevant image faithfully. Linear probes trained on projector tokens recover the category of the irrelevant image well above chance at 74.4% to 78.5% (versus 5% chance), while carrying little information about the question-relevant category. The failure therefore occurs inside the language model.

• The principal-angle subspace is a causal route. At ε = 1.0, in-band perturbations of the visual-token residual stream within the principal-angle subspace induce larger accuracy drops than out-of-band perturbations in all 13 models, with Δ_in − Δ_out ranging from 1.1 to 5.5 pp.

• Geometry and accuracy diverge under the PA gap's ordering. Δ_PA median values are 0.130 (Orig), 0.213 (Irr), and 0.324 (Noise), and the same full ordering holds in almost all models (12/13). Median accuracies, however, are 85.4% under Orig, 36.1% under Irr, and 39.0% under Noise — Irr is geometrically more structured than Noise yet slightly less accurate. No standard scalar measure orders the three settings consistently.

Methodology in Plain English

The authors study projector-based MLLMs ("projector vLLM" designs), where a vision encoder feeds a learned projector whose output visual tokens are concatenated with text embeddings and processed by the same Transformer blocks. Because visual and text tokens share attention and MLP weights, the independence condition behind classical representation-similarity tools no longer holds.

To test whether alignment scores reflect content, they run a fixed 1,000-question subset of MMBench across all models and all settings, then perturb the visual stream in five ways: Orig (unmodified projector output), Noise (each visual token replaced by a norm-matched Gaussian vector), Irr (projector output from a randomly paired image from a different question, using the same fixed pairing for all 13 models), Shuf (random permutation of Orig's tokens), and Text (visual tokens removed). Per-token norms are held nearly constant so effects reflect content or arrangement, not magnitude.

They then compute layer-wise principal-angle-cosine spectra between visual and text subspaces (PCA to k = 30 components), plus CKA, SVCCA, MIR, participation ratio, and spectral entropy. To isolate the mechanism, they bypass the MLP or attention sublayer, examine the singular-value spectrum of W_out, and check whether the principal-angle basis overlaps W_out's top singular directions. To test causality, they perturb the residual stream inside versus orthogonal to the principal-angle subspace. Finally, they linearly interpolate between original and noise tokens across 11 α values to see which measure tracks graded loss of visual content.

Why This Matters

The paper's central methodological claim is that a high internal alignment score is not, by itself, evidence that an MLLM uses the image to answer a question. Representation-similarity methods like CKA, SVCCA, and principal-angle analysis were developed to compare independently trained networks; in MLLMs the two modalities share weights, so geometry-imposed similarity can masquerade as content-level integration. The authors argue such scores are most reliable when reported alongside task-accuracy evidence under controlled visual interventions. They also note that alignment remains a useful geometric diagnostic of the visual stream inside the model, just not a direct proxy for cross-modal interaction.

Real-world applications:

• Model evaluation and benchmarking: Adding controlled visual interventions (noise, irrelevant-image, shuffled-token baselines) to multimodal benchmarks so that reported "visual grounding" is validated against task accuracy rather than internal scores alone.

• Training diagnostics and data curation: Teams currently use alignment curves to guide model design and data curation; the PA gap offers a check that discriminates architecture-induced similarity from content-driven structure.

• Hallucination and robustness auditing: The finding that irrelevant images are processed and can lower accuracy below the text-only baseline (median 5.0 pp) is directly relevant to auditing how models handle mismatched or adversarial image inputs.

• Model selection for deployment: Model-size-dependent effects (<3B versus ≥3B groups) inform expectations about how much structured visual content smaller models actually use.

Industry relevance: Any organization deploying multimodal assistants, document understanding, or image-conditioned pipelines has an interest in knowing whether internal alignment dashboards reflect real visual grounding. The paper argues those dashboards can look healthy while the visual stream carries no task-relevant content.

Future Directions

• Extending to other fusion designs. The analysis focuses on projector-LLM architectures; other fusion designs may involve different forms of image-conditioned reasoning and cross-modal interaction.

• Beyond controlled multiple-choice evaluation. Task accuracy here is measured on a fixed 1,000-question MMBench subset; open-ended generation is untested.

• Beyond static images. Video, document, and agentic tasks are named as settings where the same intervention-based analysis should be extended.

• Open question about the alignment-accuracy gap. The Irr setting shows internal geometry can be structured while accuracy falls; how to build a geometric diagnostic that also captures question relevance remains unresolved.

Target Audience

Interpretability and evaluation researchers working on multimodal language models; practitioners who use representation-similarity tools (CKA, SVCCA, principal angles) to diagnose vision-language training; and engineers who design multimodal benchmarks or deployment-time robustness checks. Readers need comfort with linear algebra (subspaces, principal angles, singular values) to follow the mechanism sections fully, though the headline results — high alignment scores despite destroyed visual content — are stated in plain geometric terms.

Authors’ abstract

Layer-wise visual-text similarity in Multimodal Large Language Models (MLLMs) is widely interpreted as evidence that the language model progressively integrates visual content into a shared representation space. This reading rests on the assumption that scalar alignment scores reflect content-level cross-modal interaction. To test this assumption, we apply controlled interventions to the visual stream. Across 13 MLLMs from five families spanning 0.5B to 72B parameters, replacing projector-output visual tokens with Gaussian noise sharply reduces task accuracy, yet four standard scalar measures (CKA, SVCCA, MIR, and the leading principal-angle cosine) fail to consistently separate the corrupted stream from the original. We call this failure the alignment illusion and trace it to the shared language-model pathway: anisotropic MLP down-projections pull visual and text tokens toward common output directions, producing weight-induced alignment. Because this component is essentially one-dimensional, we introduce the principal-angle gap (PA gap), defined as the difference between the top two principal-angle cosines, which separates weight-induced similarity from multi-directional visual structure. Under graded visual corruption, the PA gap tracks task accuracy more consistently than the scalar scores we consider; under a structured but irrelevant image, it further exposes regimes in which internal geometry and task accuracy come apart. Internal visual-text alignment in MLLMs is therefore best read as a geometric diagnostic of the visual stream inside the language model rather than a direct proxy for content-level cross-modal interaction, and is most informative when calibrated by controlled task evidence.