Research
Massive Activations Confound Sparse Autoencoders
Overview Research area: Mechanistic interpretability of large language models — specifically the training and evaluation of sparse autoencoders (SAEs) as tools for decomposing model activations into…

- arXiv
- 2611.13548
- Published
- 2026-11-10
- Authors
- Nathan Delisle, Chenhao Tan
AI summary
Overview
Research area: Mechanistic interpretability of large language models — specifically the training and evaluation of sparse autoencoders (SAEs) as tools for decomposing model activations into interpretable features.
Technical level: Advanced. The paper assumes familiarity with SAEs, principal component analysis, residual stream activations, and perplexity/cross-entropy metrics.
Scope: The paper shows that "massive activations" — a single dominant direction (PC1) in the residual stream — corrupt both standard SAE evaluation metrics (variance explained and ablation damage), and proposes a training fix (PC1 bypass) plus a post-hoc alternative (excluding PC1-aligned features from rankings).
What This Paper Is About
Sparse autoencoders are meant to break a language model's internal activations into a sparse set of human-interpretable features, and they are judged by how much activation variance they explain and how much damage removing a single feature does to the model. The authors show that in many models, up to 99.9% of activation variance sits along one principal component (PC1), and because that single direction can be rebuilt in infinitely many equally valid ways, different training runs learn entirely different — but equally "important" — features. The goal is to diagnose this confound and to remove it, either by retraining the SAE with PC1 held out (and added back at inference) or by filtering PC1-aligned features out of importance rankings after the fact.
Key Contributions
- Diagnosis of an inversion between importance and reproducibility. The authors show that the features ranked most important by ablation damage are the least reproducible across training seeds — the opposite of what one would expect if important features reflected real model structure (Figure 1, Figure 2).
- Identification of "PC1 tiling" as the mechanism. They show that SAEs reconstruct the dominant PC1 component using a set of partially aligned features ("tiles") whose components outside PC1 cancel, and that this tiling occupies 2–66 times its share of total ablation importance while constituting only about 1% of alive features.
- A correction to variance-explained reporting. They derive and empirically verify that the reported fraction of variance unexplained understates error outside PC1 by roughly the dominance factor 1/(1 − w₁), where w₁ is PC1's share of variance — an understatement of up to 890× in their Qwen-2.5-7B SAEs.
- Two repairs. PC1 bypass (train on activations with PC1 projected out, add it back to the reconstruction at inference) and post-hoc PC1-aligned feature exclusion from rankings. Bypass raises top-50 cross-seed sharing from 4–39.5% to 73–99.5%, reduces content reconstruction error by up to 33.9%, and degrades the host model up to 44% less at the same TopK sparsity.
Main Findings
- Variance is concentrated in one direction: In middle layers, 70–99.9% of activation variance lies along PC1 in the models studied; PC1 reconstruction error accounts for less than 0.3% of total reconstruction error.
- Top features are seed artifacts in affected models: In GPT-2, Qwen-2.5-7B, and Pythia-160M/1.4B/6.9B, an average of 43.5–48 of the top 50 features fail to recur in another run, versus 19–27 for a random 50 drawn from the dictionary. Across five highly anisotropic models, top-50 mean pairwise sharing is 5–11%, against dictionary-wide rates of 47–63%.
- The effect scales with PC1 dominance: Across all evaluated models, top-50 sharing falls strongly with PC1 dominance (Pearson r = −0.93, Spearman ρ = −0.73 against w₁). Gemma-2-2B and Gemma-2-9B, the least PC1-dominated (w₁ = 0.78 and 0.70), show the opposite pattern, with top-50 sharing of 87% and 92%.
- Only the very top of the ranking is affected: The rank correlation between importance and reproducibility across all features is weak but positive, between +0.11 and +0.19 in every model.
- Three classes of features by PC1 alignment: Features with squared alignment aⱼ below 10⁻² share at the normal dictionary rate; intermediate-alignment "tiles" share at 0.0–2.6% and number 38–106 per model (~1% of alive features); rare highly aligned "canonical features" (alignment ≥ 0.6) recur in every run — four of ten dictionaries contain exactly one, Gemma-2-2B contains two, and five contain none.
- Reported variance explained hides content error: The ratio FVU_content/FVU_total ≈ 1/(1 − w₁) holds to within 0.2% relative error on all measured models. Qwen-2.5-7B SAEs explain 99.96% of total variance while leaving 33.0% of content variance unexplained — an understatement of roughly 890×.
- The sparsity objective is responsible: Adding an activation-magnitude penalty β‖z‖₁ to TopK on GPT-2 block 6 raises top-50 sharing from 6.4% to 94.9% at β = 0.3, with mean L₀ moving from 64.00 to 63.47 and content FVU from 0.1394 to 0.1387.
- Objective comparison: ReLU+ℓ₁ shows no inversion (top-50 sharing 95–100% in every model), whereas TopK, BatchTopK, and JumpReLU with scheduled thresholds all do in models where PC1 explains at least 93% of variance. For example, on Qwen-2.5-7B the top-50 sharing is 6.0 (TopK), 33.0 (BatchTopK), 32.0 (JumpReLU), 47.0 (Gated), and 97.0 (ReLU+ℓ₁).
- Canonical initialization works but costs fidelity: Pinning one decoder direction to PC1 at initialization raises top-50 sharing from 5.5% to 68% and cuts median tile count from 65 to one, but raises median content FVU from 0.149 to 0.165.
- Bypass preserves and improves reconstruction: Held-out content FVU falls by 0.8–33.9% in every model under bypass; on massive-activation tokens a standard SAE spends about 63 of its 64 active coefficients on features with aⱼ ≥ 0.01, and approximately 84% of GPT-2's content-error reduction comes from ordinary tokens.
- Post-hoc selection is a cheap substitute: Excluding features with aⱼ ≥ 0.01 removes only 0.72–3.92% of alive features but yields 77.5–96% top-50 sharing. On GPT-2 these features have mean ablation effect ΔCE = 0.0130, close to bypass (0.0141) and far below the confounded original top 50 (0.0966).
- Downstream tasks show mixed results: Automated interpretability AUROC for top-ranked features improves under bypass on GPT-2, all three evaluated Pythia models, and Qwen-2.5-7B (e.g., Pythia-1.4B from 0.551 to 0.732), but falls on Llama-3.1-8B and Gemma-2-2B. Sparse-probing accuracy differences average under one percentage point across four tasks.
- Steering improves: On subject–verb agreement, post-hoc selection raises success from 0% to 9.0%, 31.9%, and 4.2% on GPT-2, Pythia-1.4B, and Qwen-2.5-7B; bypass gives 7.1%, 37.5%, and 2.0%. Gemma-2-2B remains at 64–65% in every condition.
- Released dictionaries differ widely: Median content-to-total FVU ratios are 2.1 for Gemma Scope 2, 1.49 for Pythia, 98.8 for unfiltered Qwen Scope, and 1.02–1.05 for Gemma Scope, Llama Scope, and OpenAI's GPT-2 dictionaries. For the GPT-2 dictionary the ratio is 10.9 in original activation space and 1.03 in normalized space.
- Scope is bounded by PC1 dominance: In Pythia-1.4B the inversion disappears at the final block (23), where PC1 explains only 11% of variance and top-50 sharing is already 96% without bypass. Excluding the first token position drops PC1 dominance in Llama-3.1-8B from 97% to 3%, while Qwen-2.5-7B remains at 99.7%.
- It generalizes beyond text: In DINOv2-base vision transformer patch activations, PC1 explains 96.2% of variance; BatchTopK SAEs reach total FVU 0.013 but content FVU 0.343 (~26× larger). Registers reduce PC1's share to about 4%.
Methodology in Plain English
The authors train many TopK sparse autoencoders — dictionaries with 4d features and 64 active features per token — on residual stream activations from GPT-2, several Pythia sizes, Qwen-2.5-7B, Llama-3.1 variants, and Gemma-2 models, using 5M tokens of WikiText-103 and varying only the random seed.
To measure reproducibility, they match features across runs by decoder direction cosine: two features count as "shared" if the best cosine match reaches 0.7. They then compare the sharing rate of the 50 features with the largest ablation effects against the sharing rate of the whole dictionary. To test whether PC1 is responsible, they group features by how much their decoder aligns with PC1 and measure sharing within each group.
They verify the variance-explained correction by computing, on held-out text, both the standard fraction of variance unexplained and a version with PC1 projected out, then checking the two against the predicted ratio.
For the repair, they compute PC1 from centered training activations, train the SAE on the activations with that projection subtracted, and add the projection (plus the training mean) back to the SAE output before feeding it to the language model. They also test a post-hoc alternative that simply drops PC1-aligned features from the importance ranking. Finally, they check whether the fix holds up on released public dictionaries, at different layers, on a vision transformer, and on crosscoders.
Why This Matters
Impact on research: SAE-based interpretability has leaned on exactly the two metrics this paper shows to be confounded. If the top-ranked features are seed artifacts, then published claims built on feature circuits, spurious-feature detection, and dictionary quality comparisons may rest on irreproducible directions. The paper provides both a reporting correction (report FVU outside PC1, not just total FVU) and a training fix that does not change the sparsity objective.
Real-world applications:
- Safety auditing and red-teaming: Teams using SAEs to find and monitor dangerous capabilities or behaviors need features that are stable across retraining, not artifacts of a random seed.
- Feature-circuit tracing in deployed models: Ablation-based importance is used to trace which features drive a behavior; unstable top features make such traces unreliable.
- Model editing and steering: The steering experiment shows selection from a corrected ranking flips subject–verb agreement where the confounded ranking achieves 0%, suggesting practical gains for controllability work.
- Benchmark and tooling design: Public dictionary releases (Gemma Scope, Llama Scope, Qwen Scope) can adopt content-FVU reporting or PC1-aware normalization; the paper shows preprocessing choices already shrink the gap by large factors.
Industry relevance: Any organization publishing or consuming SAE dictionaries — for monitoring, interpretability dashboards, or evaluation suites such as SAEBench — can adopt the post-hoc filter with no retraining, since it changes only which features are surfaced, not the dictionary.
Future Directions
- Extending the correction to other interpretability methods. The authors explicitly motivate testing whether probing, attribution, or circuit-discovery methods are similarly confounded by a dominant activation direction.
- Better training-side remedies. A magnitude penalty recovers much of the reproducibility without bypass, and whitening lowers content error by 39.4% (versus 37.6% for bypass) while increasing total error to 2.4 times the standard value — the trade-off between these approaches is unresolved.
- Understanding the residual unexplained content. Even after bypass, the authors report that a substantial fraction of content variance remains unexplained (12–33% for standard SAEs at 64 coefficients), leaving open what structure sparse linear dictionaries still miss.
- Characterizing when the confound disappears. The layer, token-position, and architecture dependence (final layers, BOS-position outlier signals, register tokens in vision transformers) suggests a broader account of when dominant directions arise and how they should be handled during dictionary training.
Target Audience
Interpretability researchers who train, evaluate, publish, or rely on sparse autoencoders; practitioners selecting features for steering, monitoring, or circuit analysis; and benchmark designers who maintain evaluation suites or public dictionary releases. Readers without background in SAE training objectives or PCA will find the middle sections demanding, though the core argument — a single dominant direction makes many metrics misleading — is stated plainly in the abstract and introduction.
Authors’ abstract
Sparse autoencoders (SAEs) are commonly evaluated by variance explained and ablation damage. We show how massive activations undermine both evaluations, and propose a correction. Massive activations cause up to 99.9% of activation variance to lie along the first principal component (PC1). Because the PC1 component has no unique sparse decomposition, each SAE run rebuilds it from a different set of features. These features dominate importance rankings: an average of 47 of the…
The full abstract on arXiv