Superposition Without Interference? Towards Isolated Interventions via Almost Orthogonal Features in Language Models

AI summary

Overview

Research area: Mechanistic interpretability of large language models; specifically sparse autoencoder (SAE) feature geometry, causal representation learning, and activation steering/interventions in Transformer residual streams.

Technical level: Advanced. The paper combines a theoretical bound derived from causal representation learning and dictionary-learning theory (Welch bounds, self-coherence) with an empirical fine-tuning pipeline that requires familiarity with SAEs, TopK activation, low-rank adaptation, and intervention protocols.

Scope: A single paper presenting a theoretical argument that almost-orthogonal SAE features improve isolated interventions, plus an empirical pipeline that fine-tunes a TopK SAE with an orthogonality penalty and adapts Gemma 2 2B and Llama 3.2 1B around it, evaluated on GSM8K-based mathematical reasoning.

What This Paper Is About

Language models represent concepts as linear directions in activation space, but these directions are often entangled, so editing one feature can unintentionally change the effect of others. The authors argue, motivated by the Independent Causal Mechanisms (ICM) principle, that forcing SAE decoder features to be almost orthogonal should make interventions more isolated. They formalize the discrepancy between an idealized intervention and its realized effect, then test whether an explicit orthogonality penalty on the SAE dictionary reduces interference without hurting model performance.

Key Contributions

1. A theoretical link between intervenability and feature interference. Under a linear hypothesis, the authors derive an upper bound (Theorem 4.1) on the expected discrepancy between true and model post-intervention outputs, expressed in terms of feature interference and the mean self-coherence μ(D) of the dictionary. In a special regime with uniformly random support of size K, zero-mean independent nonzero coordinates of variance σ², the averaged bound becomes (1/2)‖A‖²_op (d−1) μ(D) (Kσ² + δ²).

2. An orthogonality-regularized fine-tuning pipeline. They fine-tune a pre-trained TopK SAE with a penalty ‖tril(DᵀD)‖²_F, freeze the decoder, insert the SAE into a Transformer, and low-rank adapt the language model around it — demonstrating preserved mathematical-reasoning performance and interpretability.

3. The first isolated feature-level interventions inside an SAE with explicit control of spill-over. Using a dictionary regularized to be almost orthogonal, they swap one concept (e.g., the name Mike) for another (e.g., the aqua prefix, yielding Aquaman) while leaving reasoning intact.

4. Two extra interference metrics beyond accuracy. They measure ROUGE-L recall between standard and intervened generations as an external token-level metric, and the symmetric Jensen–Shannon divergence between feature-usage distributions as an internal SAE-level statistic.

Main Findings

• Higher penalty monotonically lowers measured non-orthogonality. The orthogonality loss ‖tril(DᵀD)‖²_F decreases as λ increases across λ ∈ {0, 10⁻⁶, 10⁻⁵, 10⁻⁴}, and the no-penalty case (λ = 0) yields the highest orthogonality loss. This is reported on a subset of 1,024 active features with 100 evaluations; error bars are 95% basic bootstrap confidence intervals over 100 sampled datasets.

• Performance is not sacrificed. On the GSM8K test set, the 2B models fine-tuned with all non-zero values of λ perform comparably to the no-penalty model. The authors note that Yu et al. (2023) observed performance in the range [0.665, 0.777] for 7B models fine-tuned on MetaMathQA, and that their 2B models sit just below that band. They report a slight performance increase with the regularization parameter λ.

• Stricter orthogonality improves correct name recovery. Intervening on 12 features interpreted as conventionally male first names across 30 GSM8K examples with male actors, and swapping each for one of the 11 remaining names, produces a dataset of 12 × 11 × 30 = 3,960 examples. The model recovers the correct first name around 74.9% of the time at λ = 10⁻⁴ versus around 62.2% without penalty; the effect is monotonically increasing in λ.

• Reasoning accuracy stays comparable across penalties. In the same intervention experiment, performance is comparable across different choices of λ, and no setup performs significantly differently from the zero-penalty configuration. The insertion-value sweep between 10 and 500 moves accuracy within an interval of 6 percentage points; the authors choose 200 based on the model's ability to remove the intervened-on concept.

• Token-level and feature-level interference both drop at the strictest penalty. In Table 1, ROUGE-L recall is 0.763 [0.756, 0.771] at λ = 0 and 0.773 [0.766, 0.779] at λ = 10⁻⁴; the Jensen–Shannon divergence is 0.222 [0.219, 0.225] at λ = 0 and 0.214 [0.211, 0.217] at λ = 10⁻⁴. The intermediate values λ = 10⁻⁶ (ROUGE-L 0.746, D_JSD 0.254) and λ = 10⁻⁵ (ROUGE-L 0.730, D_JSD 0.235) do not yield definitive conclusions. Across datasets, the authors report significantly lower D_JSD for the strictest penalty relative to λ = 0, and significantly higher ROUGE-L recall on the location dataset.

• Concept substitution can be semantically coherent. With λ = 10⁻⁴, the three top input examples activating the aqua feature are related to aquaria, and the feature explanation reads "Aquarium capacity or fish population size is the latent concept shared among the spans." Swapping this feature for the one associated with Mike causes the model to substitute Aquaman while still producing the answer 624.

• Interventions do not alter the causal path to the answer. Because only the subject of the sentence is changed, the modified feature is not an ancestor of the final answer, so the causal mechanism generating the mathematical solution is unaffected.

Methodology in Plain English

The authors take an already-trained TopK sparse autoencoder, which decomposes a model's internal activations into a sparse linear combination of dictionary directions, and add a penalty that discourages its decoder columns from pointing in similar directions. They then evaluate this penalty on random subsets of features during training to keep it computationally feasible.

Next, they freeze the resulting decoder and plug the SAE into the middle of a Transformer — after layer 13 of Gemma 2 2B's 26 layers, with the SAE drawn from the SAEBench module of size 2¹⁶ = 65,536, so all information flows through the SAE. They low-rank adapt the attention matrices and the non-decoder SAE weights, meaning the language model can reshape itself around the constrained dictionary while the features stay nearly orthogonal. Both stages fine-tune for one epoch on MetaMathQA, using K = 20 active features.

For the intervention experiments, they identify features that appear to encode male first names, then swap the index of an active name feature for another name's feature: the original is turned off and the replacement is added to the residual stream. They evaluate whether the model keeps reasoning correctly, drops the original name, and includes the injected one. Because feature coefficients are continuous, activating a feature is not binary, so they sweep insertion values between 10 and 500.

To measure interference beyond accuracy, they compare standard and intervened generations with ROUGE-L recall — based on the longest common subsequence of tokens relative to the standard generation's token set — and compute the Jensen–Shannon divergence between the empirical distributions of features used in the two generations.

Why This Matters

Impact on research. The paper connects a practical interpretability technique (SAEs) to classical dictionary-learning identifiability theory, where recovery guarantees are strongest at low self-coherence, and to the ICM principle. It also speaks to recent concerns about SAE reliability — that features differ across random seeds, can be reconstructed from smaller feature sets via a meta-SAE, and suffer from feature absorption. The authors argue that representational desiderata like orthogonality need not cost task performance, and that enforcing global orthogonality across the full dictionary distinguishes their approach from earlier work. They also frame feature interference as a fundamental limit on intervention: with non-orthogonal directions, interventions on one feature inevitably spill over to others.

Real-world applications:

• Model editing and correction — reliably swapping a specific concept (a name, a location, an animal) inside a deployed model without disturbing its reasoning.
• Auditing and debugging — attributing a model's output to specific internal features with tighter causal control, supporting interpretability reviews.
• Controllable generation — steering outputs toward or away from targeted concepts at the representation level rather than through prompting.
• Safety and robustness research — probing whether reduced superposition could mitigate adversarial vulnerabilities, a hypothesis the authors raise but do not test here.

Industry relevance. The pipeline is a two-step fine-tuning recipe (regularize the SAE, then low-rank adapt the LM) that the authors report preserves downstream task performance, which matters for practitioners who need interpretability without retraining from scratch. It uses off-the-shelf components (pre-trained TopK SAEs, LoRA-style adaptation) and the code is released at https://github.com/mrtzmllr/sae-icm.

Future Directions

• Extending beyond middle-to-late layers. The SAE is inserted only after layer 13 of 26 in Gemma 2 2B and after layer 12 in Llama 3.2 1B, so the method is constrained to features arising in middle-to-later layers, which tend to encode high-level concepts.
• Tasks beyond mathematical reasoning. The authors explicitly name extending the approach to other task types as an avenue for future research.
• Linking orthogonality to adversarial robustness. The authors ask whether enforcing greater feature orthogonality reduces superposition and, in turn, mitigates adversarial vulnerabilities that have been hypothesized to stem from feature superposition.
• Scaling and identifiability. They suggest that their results motivate more scalable versions of the approach and point toward improving the identifiability and atomicity of SAE features without sacrificing downstream utility.

Target Audience

Mechanistic interpretability researchers; practitioners of SAE-based feature discovery and activation steering; researchers in causal representation learning and dictionary learning interested in identifiability and self-coherence; and engineers who need precise, localized model editing while preserving task performance. Readers need working familiarity with SAEs, TopK activation, low-rank adaptation, and causal notation to follow the theory and pipeline.

Authors’ abstract

A central premise in mechanistic interpretability is that meaningful concepts in language models are represented by linear features in activation space. For such features to support reliable interventions, manipulating one feature should not substantially alter the effects of others. In practice, however, feature entanglement leads to interference such that localized interventions can have unintended downstream effects. Motivated by the _Independent Causal Mechanisms_ principle, we propose to constrain internal features to be almost orthogonal. We argue that this promotes modular representations amenable to causal intervention. We formalize this problem by characterizing the gap between an idealized isolated intervention and its realized effect on model outputs in terms of feature interference. We upper-bound the propagation of feature interference in terms of the self-coherence of the feature dictionary, and relate this discrepancy to an explicit orthogonality regularization on the dictionary itself. Empirically, we show that this regularization enables more isolated interventions on mathematical reasoning concepts while preserving model performance. Our code is available under https://github.com/mrtzmllr/sae-icm.