Component and Dimension Sparsity in Transformer Refusal Mechanisms

AI summary

Overview

• Research area: Mechanistic interpretability of large language models, specifically activation steering and safety/refusal behavior.
• Technical level: Advanced. The paper assumes familiarity with residual stream architectures, difference-in-means direction extraction, directional orthogonalization, and attack success rate evaluation. The prose is accessible, but the formulas and setup are not beginner material.
• Scope (one sentence): The paper decomposes refusal steering in four open-weight transformer models into component-level and residual-dimension-level interventions to identify which sparse subsets of attention and MLP components, and which hidden dimensions, are sufficient to reproduce the full refusal-steering effect.

What This Paper Is About

Activation steering methods can suppress or induce refusal in LLMs by editing internal activations, but existing approaches typically intervene at a single point in the residual stream without resolving which specific attention and MLP components actually write the refusal signal. This paper asks which components are jointly sufficient to reproduce the effect of a full residual-stream intervention, and whether the signal further concentrates in a small number of residual stream dimensions. The goal is to show that refusal is assembled by a structured, identifiable mechanism rather than diffusely encoded across the model.

Key Contributions

1. Component-level decomposition of refusal steering. The authors recast a single-point residual-stream orthogonalization into independent interventions on each of the 2l*+1 upstream components (embedding matrix plus all preceding attention and MLP sublayers), and use a two-stage greedy backward elimination to find sparse sufficient component sets, which they call "mechanisms."
2. Identification of a second level of sparsity in the residual stream dimensions. A static coordinate mask derived from the refusal direction r* is applied to all projection operations, showing that a sparse set of high-magnitude coordinates carries the steering signal, and that masking a randomly Haar-rotated version of the direction removes the effect.
3. Per-component attribution and write-profile analysis. Each component's difference-in-means output is projected onto r* to measure its scalar contribution, and dimension-wise write profiles are examined to test whether components write into a shared sparse subspace.
4. Capability retention comparison and public release. The sparse mechanism is baked into weights as a reparameterization and compared against the dense intervention on MMLU, GSM8k, and Pile perplexity; all code and raw experimental results are released at https://github.com/wang-research-lab/Refusal_Mechanisms.

Main Findings

• Refusal mechanisms are sparse in components. The minimal component sets span 28.2% to 48.1% of upstream components, with a mean of 35.7%. Per model: Gemma2-2B-IT 48.1%, Phi4-Mini 37.9%, Mistral-3.2-Small 28.6%, Qwen3-4B 28.2%.
• Sparse mechanisms reproduce the full effect. Mechanism ASR as a percentage of the full residual-stream steering baseline is 88.3% for Gemma2-2B-IT, 90.9% for Qwen3-4B, 96.5% for Phi4-Mini, and 100.5% for Mistral-3.2-Small. A value above 100% indicates the mechanism steers more effectively than the single-point dense baseline.
• Identified component sets. Gemma2-2B-IT: attention layers {4, 6, 9, 10, 11} and MLPs {4, 6–12}. Qwen3-4B: attention {12, 14, 16, 18} and MLPs {6, 12, 14–18}. Phi4-Mini: attention {9, 13} and MLPs {4, 6–13}. Mistral-3.2-Small: attention {8, 11} and MLPs {9–16}. The embedding matrix is never retained by the elimination procedure across models.
• Complement sets largely fail to steer. Steering only the excluded components produces near-zero ASR for Gemma2-2B (1.6%) and Phi4-Mini (3.3%). Qwen3-4B and Mistral-3.2-Small are outliers at 34.6% and 17.8%, respectively, though both achieve this with substantially more components than the identified sparse mechanism — the paper reads this as more redundant encoding in those models.
• Dimensional sparsity composes with component sparsity. Applying the 95% norm-explained coordinate mask to the identified mechanisms retains 85.7% to 98.6% of full steering effectiveness. The dimension percentage used is 51.0% for Gemma2-2B-IT, 53.1% for Qwen3-4B, 54.3% for Phi4-Mini, and 54.3% for Mistral-3.2-Small — approximately 50% across models.
• The dimension-vs-performance curve is supralinear. Models were tested at norm thresholds of 50%, 70%, 80%, 90%, 95%, and 97.5%. Performance degrades slowly through the upper thresholds and drops sharply only below roughly 50% of the squared norm, with around 10% of hidden dimensions retaining up to 50% of steering effectiveness. At the 95% threshold several models match or slightly exceed full-steering ASR.
• Sparsity requires coordinate alignment, not just high magnitude. Applying the same number of dimensions k = |S_0.95| to the top coordinates of a Haar-rotated direction Q r* with Q ~ Haar(O(d)) yields near-zero ASR across all models (0.6–11.2%), compared to 96.8–101.5% in the native basis. This is consistent with a privileged basis in the residual stream.
• Sparse mechanisms do less collateral damage than dense intervention. Across twelve model-benchmark pairs, the sparse mechanism degrades capability no more than the dense intervention in ten. On Phi4-Mini, the dense intervention costs 22.44 MMLU points and 28.13 GSM8k points and raises Pile perplexity by 2.593, while the mechanism stays within 1.98 MMLU points and 1.36 GSM8k points of baseline and within +0.113 perplexity.
• Attention components dominate the refusal direction. Attention components make the largest individual contributions to r* across all models; MLP contributions are smaller and more distributed. Qwen3-4B shows negative MLP contributions at layers 12 and 14, meaning those components actively write against the refusal direction, though direct pruning shows removal would degrade performance around 0.1 in ASR.
• The identified mechanisms reconstruct 50–63% of r*. The remainder is attributed to components pruned during elimination whose individual contributions were too small to clear the selection threshold.
• High-magnitude writes do not share a subspace. Per-component write profiles show that high-magnitude writes distribute throughout the selected coordinate space rather than tracking the refusal direction envelope, with large writes appearing at different dimension ranks across components. The paper interprets the residual-stream dimensional sparsity as an emergent aggregate of component-specific sparse signals rather than a shared privileged subspace.
• Refusal direction distributions have heavy tails. Coordinate values of r* are roughly Gaussian centered near zero with heavy tails; outliers defined as exceeding 2.5× IQR from the interquartile range range from a single dimension in Phi4-Mini to eight in Qwen3-4B, with no consistent sign across models.
• Results are stable to the stopping rule. Held-out ASR varies by at most five points across the threshold grid, and random tie-breaking preserves an invariant core of 8–12 components.

Methodology in Plain English

The authors work with four open-weight models: Qwen3-4B, Mistral-Small-3.2-24B-Instruct-2506, Phi-4-Mini-Instruct, and Gemma-2-2B-IT. Harmful prompts come from AdvBench, HarmBench, JailbreakBench, Malicious Instruct, StrongREJECT, and TDC2023, split into training (256), validation (463), and test (512) sets; harmless prompts for contrast come from ALPACA. Harmfulness of responses is judged by LlamaGuard 3.

Step 1 — pick a layer and a direction. For every layer, they compute a difference-in-means refusal direction: the mean residual stream activation on harmful prompts minus the mean on harmless prompts, taken at the last post-instruction token. They measure attack success rate when that direction is orthogonalized out of the residual stream at that layer's input, and select the layer l* with the highest ASR.

Step 2 — find a sparse set of components. Instead of orthogonalizing the residual stream at one point, they orthogonalize the output of each individual attention and MLP sublayer before it is added back. The full search space of 2^(2l*+1) subsets is intractable, so they use two-stage greedy backward elimination. Stage 1 removes components whose removal changes the mean projection of harmful activations onto r* the least, stopping when removal would drop projection improvement below α = 0.95 of the full-component baseline. Stage 2 then removes components whose removal causes the smallest ASR drop, stopping when any further removal would reduce ASR by more than τ = 0.05 in absolute terms. Stage 1 is a cheap pre-filter; Stage 2 provides the behavioral grounding. Components are never evaluated by examining individual neurons.

Step 3 — test dimensional sparsity. They compute the smallest set of coordinates S_t explaining fraction t of the squared norm of r*, and restrict all projection operations to those coordinates with a binary mask. The mask is static, computed once from r* and applied uniformly to all token activations with no per-activation adaptation. As a control, they repeat the procedure after randomly rotating the direction (keeping activations in the native basis).

Step 4 — check it is refusal specifically. They bake both the sparse mechanism and the dense intervention into the weights as a reparameterization of the projection hooks, serve both directly, and compare MMLU (0-shot), GSM8k, and Pile perplexity against the unmodified model.

Step 5 — decompose contributions. For each component in the mechanism, they compute the difference in mean output between harmful and harmless prompts and project it onto r*, giving a scalar attribution. They also examine dimension-wise write profiles, normalized to unit L2 norm to control for LayerNorm-induced scale differences, restricted to the top 10% of dimensions by magnitude of r*.

Why This Matters

Impact on research. The work reframes activation steering from a single-point operation into a question about which components assemble the steered direction. It shows that sufficiency, not just necessity, can be established at the component level, and it ties residual-stream sparsity to the privileged basis hypothesis while showing that the sparsity is emergent from component-specific writes rather than a shared subspace. The two-level picture — which components, and which dimensions within them — gives interpretability research a concrete structural claim to test elsewhere.

Real-world applications:

• Safety auditing of deployed models: identifying a compact mechanism makes it more tractable to inspect or monitor whether a model's refusal behavior has been altered.
• More targeted interventions: the sparse mechanism can be baked into weights and served directly, and on Phi4-Mini it reproduces the behavioral effect with far less damage to general capability.
• Red-teaming and robustness evaluation: understanding where refusal is assembled informs how bypasses might occur and how to test for them.
• Interpretability tooling: the released code and raw results provide a template for decomposing other high-salience directions, such as hallucination.

Industry relevance. The capability-retention results matter for anyone who wants to modify model behavior without degrading general performance. On Phi4-Mini, the dense intervention costs 22.44 MMLU points and 28.13 GSM8k points; the sparse mechanism costs 1.98 and 1.36 points respectively. The authors note the ethics consideration that their jailbreaking effects are generally lower than prior work, with Mistral-Small-3.2 giving a relative improvement of one percent over prior methodologies.

Future Directions

• Why some models encode refusal redundantly. Mistral-3.2-Small and Qwen3-4B allow weak steering from their complement sets (17.8% and 34.6% ASR), unlike Gemma2-2B and Phi4-Mini. The paper explicitly leaves open whether this reflects architectural differences, training procedure, or a general property of larger models.
• From one direction to the full refusal subspace. The authors steer a single direction at a single intervention layer and state this is not a claim about the full refusal subspace. Extending the decomposition to multi-dimensional steering subspaces is an obvious next step.
• How downstream components preserve refusal signal. An early single attention component in Mistral-3.2-Small and Qwen3-4B contributes more to r* than many closer components despite being 7–10 layers upstream, suggesting its write propagates nearly intact. The paper calls for work on how downstream components preserve or reinforce refusal-related signals.
• Whether structured sparsity generalizes. The consistency across four architectures suggests structured sparsity may be a general property of how transformers encode high-salience behavioral directions, but this is proposed as a hypothesis to test rather than established.

Target Audience

This paper is for mechanistic interpretability and AI safety researchers, particularly those working on activation steering, circuit-level analysis, or refusal behavior in LLMs. It is also relevant to practitioners who want to modify model behavior in deployment without degrading general capability, and to red-teamers studying how refusal can be bypassed. Readers need a working understanding of transformer residual stream mechanics, difference-in-means direction extraction, and orthogonalization; the mathematics is presented compactly and the appendices carry much of the justification.

Authors’ abstract

Activation steering manipulates large language model behavior by intervening on internal activations, but the mechanistic basis of these interventions remains poorly understood. We decompose refusal steering into component-level interventions across four open-weight models, identifying the sparse subsets of attention and MLP components whose steering suffices to reproduce the full behavioral effect. We find that refusal directions concentrate in sparse component mechanisms comprising 28--48\% of upstream components, retaining 88--101\% of steering effectiveness. Within these mechanisms, effective steering further concentrates in approximately 50\% of residual stream dimensions, retaining 85--98\% of the component-mechanism baseline, consistent with a privileged basis structure. Sparsity thus operates at two levels: which components are steered, and which dimensions within those components carry the signal. Together these findings show that refusal is not diffusely encoded across a transformer but assembled by a structured, identifiable mechanism, providing a foundation for mechanistic understanding of how refusal behaviors are represented and steered. To facilitate reproducibility, we release all code and raw experimental results in https://github.com/wang-research-lab/Refusal_Mechanisms.