Beyond Components: Singular Vector-Based Interpretability of Transformer Circuits

AI summary

Overview

Research area: Mechanistic interpretability of transformer language models (specifically, decomposing attention heads and MLP layers into interpretable sub-computations).

Technical level: Advanced. The paper assumes familiarity with transformer internals, singular value decomposition, and circuit-level interpretability terminology.

Scope: The paper proposes a singular-vector-based method for decomposing transformer components into orthogonal functional directions and validates it on GPT-2 Small across the IOI, GT, and GP tasks.

What This Paper Is About

Existing mechanistic interpretability methods treat attention heads and MLP layers as indivisible units, typically probing, patching, or ablating an entire component to judge its role. This paper asks whether a single head or MLP actually bundles several independent functions together, and whether those functions can be separated along the orthogonal singular directions of the component's weight matrices. The goal is a finer-grained view of transformer computation that reveals superposed, low-rank subfunctions inside the same architectural unit.

Key Contributions

1. A directional interpretability perspective. Transformer components (attention and MLP) are modelled as superpositions of orthogonal subfunctions rather than atomic units, using augmented weight matrices that fold in biases so both attention and MLP can be analyzed in one unified linear framework.
2. An optimization-based directional masking scheme. A learnable diagonal mask scales the singular values of each augmented matrix, with masks trained under a KL-divergence plus L1-regularization objective to identify functionally important singular directions, enabling direction-level attribution.
3. Empirical evidence of multiplexing. The authors show multiple low-rank, interpretable computations coexist within single attention heads and MLPs, contradicting standard assumptions about circuit modularity.
4. Logit receptors. The work reports stable, controllable directions in logit space aligned with specific tokens (for example "he" and "she" directions in gender pronoun resolution), which can be steered by scalar interventions.

Main Findings

• Extreme sparsity with high fidelity: On IOI, only 91.32% of directions are pruned relative to non-zero singular directions (98.66% relative to full matrix size), yet the KL divergence is 0.21 ± 0.02, the exact match is 0.77 ± 0.06, and pruned accuracy is 0.70 ± 0.07 versus 0.79 ± 0.05 for the full model. GT reaches 95.21 / 99.26 sparsity with KLD 0.23 ± 0.03 and exact match 0.33 ± 0.06; GP reaches 96.81 / 99.51 sparsity with KLD 0.13 ± 0.01, pruned accuracy 0.75 ± 0.04 versus 0.77 ± 0.04, and exact match 0.86 ± 0.07. The IOI task in particular retains roughly 9% of directions relative to the full component.

• Learned directions beat magnitude-based selection: The selected directions achieve substantially lower KL divergence than top-k magnitude or random SVD baselines, indicating task-relevant computation is not concentrated in the largest singular modes (quantitative details are in App. B, which is not included in the provided content).

• Alignment with known circuits: Heads established as important for IOI, such as Name Mover, Backup Name Mover, and S-Inhibition (Wang et al., 2022), show consistently high mask activations across multiple singular directions, while components not associated with known circuits show near-zero activations across all directions. Coarse component-level circuits therefore appear to arise from finer low-rank structures inside each head.

• Head 9.6 decomposes into separable operations: The 7th singular direction (S7) separates named entities from action tokens, with entities at +3.52 ± 1.42 and actions at −4.44 ± 0.68 overall; in the prompt "Jerry and Mary went to the school. Mary gave a raspberry to," it assigns +2.87 to "Jerry," +2.78 to "Mary," −4.33 to "went," and −4.17 to "gave." The 28th direction (S28) acts as an entity salience signal, giving "Susan" 4.05 and "Kevin" 5.22 while function words like "the" and "of" stay low, and showing stronger activation for first mentions ("Kevin" 5.22) than subsequent ones ("Kevin 2" 2.07). The top direction (S1) implements sequence initialization detection, assigning the first token activations 20–25× larger than subsequent tokens, a pattern that also appears in S1 of Head 0 in Layer 10.

• Scalar control over predictions: In gender pronoun resolution, distinct "he" and "she" directions exist and the model activates them depending on context; scalar interventions along these directions flip gender pronoun predictions with perfect accuracy.

• Effective ranks are sufficient: QK augmented matrices have effective rank 64, OV matrices have rank 65 after augmentation, and MLP layers typically retain their full non-zero spectrum. These truncations produce drops in reconstruction faithfulness (KLD) in the range of 1e-6.

Methodology in Plain English

The authors start by rewriting attention and MLP computations so biases are absorbed into the weight matrices, producing "augmented" matrices that are pure linear maps. For each attention head (its query-key and output-value blocks) and each MLP layer (its input and output projections), they compute a singular value decomposition, which expresses the component as a sum of rank-1 orthogonal directions.

They then attach a learnable mask to each singular direction, a value between 0 and 1 that scales that direction's singular value. The model's original weights are frozen; only the masks are trained. The training objective has two parts: a KL divergence that pushes the masked model's output distribution to match the original model's, and an L1 penalty that drives most mask values toward zero. The model is fed concatenated clean and corrupted activations, where the corrupted variant is a small change to the prompt that would change the task label, following the corruption datasets of Bhaskar et al. (2024), with additional data points created for GT and GP. For the query-key matrices the authors deliberately keep only the masked component and omit the complementary (1 − M) term, arguing that a second independent similarity map inside the same head would create conflicting attention kernels under a single softmax. The masked and complementary subspaces are both retained for the OV and MLP matrices to preserve the activation distribution that downstream layers expect.

Experiments use a pretrained GPT-2 Small model evaluated on Indirect Object Identification (IOI), Greater Than (GT), and Gender Pronoun (GP). Dataset sizes and full dataset details are stated to be in App. A, which is not part of the provided content.

Why This Matters

Impact on research: The paper reframes circuit discovery: instead of asking which heads matter, it asks which directions inside heads matter. This bridges component-level circuit analysis (Wang et al., 2022; Conmy et al., 2023; Syed et al., 2024; Bhaskar et al., 2024) with finer-grained subspace analysis, and it extends a low-rank communication view previously applied to attention value matrices (Merullo et al., 2024b) to also cover MLP layers. It also offers a practical pruning result: over 95% of directions can be removed on GT and GP with the reported fidelity levels.

Potential applications (the paper does not report deployed applications or user studies; these follow from the described capabilities):

• Targeted model editing and steering: Because scalar interventions along logit-receptor directions reportedly flip gender pronoun predictions, similar interventions could offer fine-grained control over model outputs.
• Model compression: Direction-level masks retain behavior with very high sparsity, suggesting a route to reducing effective computation rather than deleting whole components.
• Safety and auditing: Locating which directions carry a given behavior could support more precise auditing or suppression of specific behaviors than whole-head ablation allows.
• Debugging unexpected model behavior: Direction-level attribution helps distinguish genuine task computation from spurious correlations inside a single component.

Industry relevance: Methods that operate at the level of singular directions, rather than entire heads or layers, are directly relevant to teams that need interpretability tooling for deployed transformers, to compression and inference-cost teams, and to those building controllable or steerable generation systems.

Future Directions

• Replacing the L1 sparsity penalty with L0 regularization, which the authors flag as an alternative sparse-selection mechanism (Bhaskar et al., 2024; Sung et al., 2021) and explicitly leave for future work.
• Interpreting the singular directions that remain challenging to interpret fully, since the paper acknowledges some directions resist clean explanation.
• Extending the directional analysis beyond GPT-2 Small and beyond the three studied tasks, IOI, GT, and GP.
• Turning the direction-level view into a general tool for targeted model editing and more precise attribution at the subcomponent level.

Target Audience

Mechanistic interpretability researchers and graduate students who already understand transformer internals and linear algebra; practitioners working on model steering, editing, or compression who want a finer-grained alternative to head- and layer-level interventions; and anyone studying the internal structure of GPT-2 Small on canonical tasks such as IOI, GT, and GP.

Authors’ abstract

Transformer-based language models exhibit complex and distributed behavior, yet their internal computations remain poorly understood. Existing mechanistic interpretability methods typically treat attention heads and multilayer perceptron layers (MLPs) (the building blocks of a transformer architecture) as indivisible units, overlooking possibilities of functional substructure learned within them. In this work, we introduce a more fine-grained perspective that decomposes these components into orthogonal singular directions, revealing superposed and independent computations within a single head or MLP. We validate our perspective on widely used standard tasks like Indirect Object Identification (IOI), Gender Pronoun (GP), and Greater Than (GT), showing that previously identified canonical functional heads, such as the name mover, encode multiple overlapping subfunctions aligned with distinct singular directions. Nodes in a computational graph, that are previously identified as circuit elements show strong activation along specific low-rank directions, suggesting that meaningful computations reside in compact subspaces. While some directions remain challenging to interpret fully, our results highlight that transformer computations are more distributed, structured, and compositional than previously assumed. This perspective opens new avenues for fine-grained mechanistic interpretability and a deeper understanding of model internals.