Spectral Degeneration of Self-Attention under Isotropic Score Geometry

AI summary

Overview

Research area: Deep learning theory — specifically the spectral (eigenvalue/singular-value) analysis of the self-attention mechanism in Transformers, with empirical diagnostics on pretrained vision and language models.

Technical level: Advanced. The paper relies on random matrix theory, asymptotic spectral analysis, ordered singular-value profiles, and non-asymptotic eigenvalue bounds with explicit error terms.

Scope: A theoretical model of the pre-softmax score matrix under an "isotropy" assumption, used to prove that softmax attention degenerates into a matrix with a few dominant directions and a bulk spectrum pinned at zero, with a contrast between bidirectional and causal attention.

What This Paper Is About

Transformers often display "attention degeneration": deep layers behave almost as if attention were low-rank or even rank-one, which limits what the model can express. This paper asks why softmax normalization produces that compression, starting from an observed geometric property of trained networks — the token-space singular factors of query/key representations look randomly rotated (isotropic), so their eigen-angles are nearly uniformly spread around the unit circle.

The authors replace that random orientation with a canonical, diagonal stand-in for the score matrix that keeps the singular values and imposes a "spike–bulk" separation (a few large scores far above the rest), then compute exactly what row-wise softmax does to that spectrum.

Key Contributions

1. A canonical spectral model motivated by isotropy. The authors formulate a diagonal surrogate score model (Assumption 1) grounded in token-space rotational isotropy, and back it with empirical diagnostics of token-space geometry in a pretrained ViT and GPT-2.

2. A full spectral characterization of bidirectional attention. Under a spike–bulk separation assumption (Assumption 2), they show both the eigenvalue and singular-value empirical distributions concentrate on {0, 1}, and they give a non-asymptotic characterization of the ordered eigenvalues and singular values with explicit error bounds.

3. The first joint eigenvalue and singular-value analysis of causal attention (to the authors' knowledge). Under the canonical model they derive exact eigenvalues and quantitative singular-value bounds, showing harmonic bulk eigenvalue decay and at most logarithmically many bulk singular values above any fixed positive threshold.

4. A rank-based separation of encoder and decoder behavior. Using stable rank and entropy-based effective ranks, they show bidirectional attention has bounded stable rank while causal attention's stable rank grows logarithmically in context length — two regimes with the same limiting empirical spectral law.

Main Findings

• Spike–bulk thresholds are logarithmic. Under softmax, the diagonal weight e^{s_i} competes with ℓ−1 unit off-diagonal weights, so the transition between averaging and diagonal concentration sits at s_i ≈ log ℓ. Assumption 2 therefore requires spikes with e^{s_i} ≥ ℓ f(ℓ) and bulk entries with e^{s_i} ≤ ℓ/g(ℓ), where f(ℓ), g(ℓ) → ∞ and g(ℓ) = o(ℓ), with a strict gap containing no score.

• Bidirectional singular values collapse to {0, 1}. For 1 ≤ k < ℓ, the ordered singular-value profile is (α₊(c_ℓ), 1, …, 1 [k−1 times], α₋(c_ℓ), 0, …, 0), where c_ℓ = k/ℓ and α±(x) = √(1 − x/2 ± √(x − (3/4)x²)). The endpoints are (1, 0, …, 0) for k = 0 and (1, …, 1) for k = ℓ. The uniform error is O(ε_ℓ), and σ_i(A) = O(ε_ℓ) → 0 for i ≥ k+2.

• Sublinear spikes give k+1 unit singular values. If k = o(ℓ), the leading k+1 singular values converge uniformly to 1 with error O(ε_ℓ + √c_ℓ) → 0.

• Eigenvalues are real despite asymmetry. Writing A = D + z1ᵀ with Z_i = e^{s_i} + ℓ − 1, z_i = Z_i⁻¹, and d_i = 1 − ℓ z_i, the authors show all eigenvalues are real in [0,1], with λ₁(A) = 1, and λ_i(A) = d_i when d_i = d_{i−1}, otherwise d_i < λ_i(A) < d_{i−1}. Bounds ℓA_{ii}−1)/(ℓ−1) ≤ λ_i(A) ≤ (ℓA_{i−1,i−1}−1)/(ℓ−1) hold for 2 ≤ i ≤ ℓ. The (k+1)-th eigenvalue satisfies λ_{k+1}(A) = 1 − c_ℓ + O(ε_ℓ).

• Both spectra degenerate to the same limit. If c_ℓ → c ∈ [0,1], the empirical singular-value and eigenvalue distributions converge weakly to cδ₁ + (1−c)δ₀.

• Bounded bulk sharpens the rates. If bulk scores satisfy s_i ≤ B with B independent of ℓ, then 0 ≤ d_i ≤ (e^B − 1)/ℓ for i > k, so the O(g(ℓ)⁻¹) terms improve to O(ℓ⁻¹).

• Causal eigenvalues are exactly the diagonal. For causal attention, λ_i(A) = exp(s_i)/(exp(s_i) + i − 1), λ₁ = 1 exactly, the k largest eigenvalues converge to 1 at speed O(1/f(ℓ)), and the r smallest decay harmonically as Θ(1/i) provided the last r bulk scores are O(1). The empirical eigenvalue distribution again converges to cδ₁ + (1−c)δ₀.

• Causal singular values are only logarithmically many above a threshold. For any ℓ, (1/(3i)) ≤ σ_i(A) ≤ ‖A‖op ≤ √5. Under spectral separation there is a (k−1)-dimensional spike subspace on which ‖Av‖₂ ≈ ‖v‖₂ up to O(k/(ℓf(ℓ))), and σ_k(A) ≥ 1 − O(k/(ℓf(ℓ))). With a bounded bulk of constant B, σ{k+m}(A) ≤ C_B min{1, (ℓ−k−m+1)/(k+m), √(log(ℓ+1)/m)}, and #{i > k : σ_i(A) > ε} ≤ C_B log(ℓ+1)/ε². Hence σ_{k+m_ℓ}(A) → 0 whenever m_ℓ/log ℓ → ∞.

• Stable ranks diverge in behavior. Bidirectional attention satisfies srank(A_b) = k + 1 + o(1) and erank₂(A_b) = k + 1 + o(1), whereas causal attention satisfies srank(A_c) and erank₂(A_c) = Θ_B(log ℓ), and erank₁(A_c) = Ω_B((log ℓ)³). Relative to context length, stable ranks decay as 1/ℓ and log ℓ/ℓ respectively.

• The contrast is visible even at zero scores. With Σ = 0, bidirectional attention is 11ᵀ/ℓ with ‖A_b‖²_F = 1, while causal prefix averaging has (A_c)_{ij} = 1{j ≤ i}/i and ‖A_c‖²_F = Σ 1/i ∼ log ℓ, while 1 ≤ ‖A_c‖_op ≤ 2.

• Spikes are capped by head dimension. Since k ≤ rank(S) ≤ d_qk, the number of spikes is sublinear in ℓ; for GPT-2, d_qk = 64 gives a length-independent upper bound.

• Empirical isotropy in token space. On ViT-B/16 (SWAG weights) with 384×384 Imagenette images mapped to X ∈ ℝ^{577×768}, and on GPT-2 over Wikitext-103 with sequence lengths 17 to 471, the eigen-angle discrepancies from uniform (KL with 50 bins, Kolmogorov, Wasserstein) are consistently close to zero and decrease with sequence length.

• Real vs canonical spectra converge with length. The Wasserstein distance between real and canonical attention singular-value distributions falls from a median of 0.3002 to 0.0942 across all GPT-2 layers as context grows from 64 to 1,024, and from 0.3290 to 0.0748 in the last layer. At ℓ = 577, ViT layer medians lie between 0.0367 and 0.1037.

• Equal spectral laws do not imply equal concentration. At GPT-2 length 1,024, pooled stable-rank medians are 1.18 for real and 27.51 for canonical attention, with erank₁ medians of 314.06 and 95.12 — the distributions are close while the rank profiles differ.

• Score-level geometry matches. Over 1,646 GPT-2 sequences and 3,925 ViT images (12 layers, 12 heads), the mean angular W₁ for both GPT-2 score singular factors drops from about 0.31 in the 17–64 length band to 0.10 in the 385–471 band, and the pooled ViT mean is about 0.08.

Methodology in Plain English

The argument runs in three steps.

First, the authors check empirically whether the "isotropy" premise holds. For a token matrix X they take its SVD and look at the token-space factor U_X. Eigenvalues of an orthogonal matrix lie on the unit circle, so they can be written as angles; if the matrix were truly random (Haar-distributed), those angles would be uniformly spread. Measuring how far the observed angles are from uniform — with KL divergence, Kolmogorov distance, and Wasserstein distance — gives a diagnostic that comes out near zero across layers of both a vision Transformer and GPT-2.

Second, they build a tractable stand-in for the score matrix. Rather than reasoning about a randomly oriented matrix, they keep its singular values and line them up on the diagonal, choosing an idealized arrangement where a few large "spike" values sit well above a compact "bulk". Softmax is applied to this diagonal matrix (with or without a causal mask). Because the row sums and the structure are explicit, both eigenvalues and singular values can be computed or tightly bounded in closed form.

Third, they compare the model to reality. For each real attention head they build a second attention matrix from that head's actual score singular values placed on the diagonal, then measure the Wasserstein distance between the two resulting singular-value distributions, and separately track stable rank and entropy-based effective rank.

The paper is explicit about the limits of this reduction. Appendix A notes that matching singular values and Haar-like orientation does not by itself justify replacing the attention matrix with its canonical form; it gives a counterexample at ℓ ≥ 2 where Σ = t_ℓ e₁e₁ᵀ and S = (t_ℓ/ℓ)11ᵀ, with t_ℓ = 2 log ℓ, share singular values (t_ℓ, 0, …, 0) yet produce rank-one softmax(S) versus two singular values tending to one for softmax(Σ). It also shows that the causal results depend on placing descending singular values along the diagonal before masking — at ℓ = 2, diagonal scores (t, 0) give eigenvalues 1 and 1/2 while (0, t) gives 1 and e^t/(e^t + 1).

Why This Matters

Impact on research. The paper gives a mechanistic, spectral explanation for a phenomenon (rank collapse / low-rank attention) that prior work had described either through depth composition or through i.i.d. or initialization-only random matrix models. It extends the analysis to the trained regime and, notably, to causal attention, where the triangular mask produces a qualitatively different regime — harmonic eigenvalues and logarithmically many non-negligible singular values — despite an identical limiting empirical spectral law.

Real-world applications:

• Diagnosing and predicting when attention in a deployed model has collapsed into a handful of directions, which bears on interpretability of what a head actually attends to.
• Guiding KV-cache and attention-matrix compression: knowing that the bulk spectrum sits at zero and only k+1 directions carry energy informs how aggressively attention can be truncated.
• Informing context-length scaling decisions, since the bidirectional/causal gap in stable rank (1/ℓ versus log ℓ/ℓ) predicts how much representational capacity survives as sequences lengthen.
• Serving as a diagnostic toolkit (eigen-angle discrepancy, stable rank, effective rank, real-versus-canonical Wasserstein distance) that practitioners can run on trained checkpoints without retraining.

Industry relevance. Because the diagnostics require only SVDs of hidden-state and score matrices from existing checkpoints, they are cheap to adopt for model auditing. The encoder-decoder spectral contrast offers a principled lens on architecture choice: concentration in a few directions may favor selective aggregation, while retaining extra directions may support the range of continuations needed in sequential prediction.

Future Directions

• Transfer the canonical results to general score matrices. The authors state this is not proved here and would require controlling the orientation-dependent softmax error. Without it, the theory describes a model mechanism rather than a theorem about arbitrary trained Transformers.
• From single heads to multi-head representations. Appendix A sets up the Gram identities ZZᵀ = Σ_h A_h V_h V_hᵀ A_hᵀ and YYᵀ = ZW_O W_Oᵀ Zᵀ, and notes that headwise attention spectra alone do not determine the spectrum of Y — a joint description of head-output subspaces and the output projection is needed.
• Incorporate residual connections and normalization to move from single attention matrices to the Transformer-block level.
• Test isotropy and spike–bulk diagnostics on selective state-space models such as Mamba, to compare learned sequence operators across architectures.
• Extend to generalized row-normalized attention maps with alternative activations, which the paper begins to address in its appendix on generalized softmax attention.

Target Audience

Theoretical machine learning researchers working on Transformer expressivity, rank collapse, and random matrix theory will get the most from this paper, as will mathematically comfortable graduate students in probability or spectral statistics. Interpretability and efficiency engineers can extract practical value from the empirical diagnostic protocol and the reported measurements on GPT-2 and ViT-B/16, even without following the proofs, but readers without a background in singular-value asymptotics will find the main theorems dense. Familiarity with softmax attention, SVD, Haar measure, and stable/effective rank is assumed throughout.

Authors’ abstract

We study the spectral structure of the self-attention matrix in Transformer models through an idealized spectral model for the pre-softmax score matrix. Motivated by the rotational isotropy of query and key representations, our model replaces the random token-space orientation of the score matrix by its canonical singular-value representative with a logarithmic spike--bulk separation. Empirical diagnostics on pretrained vision and language transformers motivate this model: token-space singular matrices have nearly uniform eigen-angle distributions, and score spectra are dominated by a small number of effective directions. We then prove that the row-wise softmax map transforms this structure into a degenerate attention matrix: dominant modes are preserved, whereas the bulk spectral distributions concentrate at zero. Stable rank remains bounded for bidirectional attention but grows logarithmically for causal attention. Within the canonical model, these results explain how attention can preserve a small number of dominant directions while compressing the remaining bulk modes.