SOM Directions are Better than One: Multi-Directional Refusal Suppression in Language Models

AI summary

Overview

Research area: Mechanistic interpretability and AI safety in large language models, specifically the internal representation of refusal behavior and its suppression (jailbreaking) via activation-space interventions.

Technical level: Intermediate. The paper assumes familiarity with transformer internals, activation steering, linear projections, and the difference-in-means construction of a "refusal direction." The Self-Organizing Map mathematics is self-contained but requires comfort with clustering and convergence arguments.

Scope: The paper proposes a multi-directional (MD) method, built on Self-Organizing Maps, for extracting and ablating several related refusal directions from a model's latent space, and evaluates it against a single-direction baseline and dedicated jailbreak algorithms across eight models.

What This Paper Is About

Safety-aligned language models refuse harmful requests, and prior work represented this refusal behavior as a single linear direction in activation space, computed as the difference between the centroids of harmful and harmless prompt representations. That single direction can be ablated from the model's internals to suppress refusal. The authors argue that refusal is instead encoded as a low-dimensional manifold, and they introduce a method that uses Self-Organizing Maps to extract a set of multiple, closely related refusal directions whose joint ablation is more effective than ablating one direction.

Key Contributions

1. A generalization proof. The authors prove that a Self-Organizing Map with a single neuron converges to the centroid of the data manifold, meaning that the SOM approach generalizes the difference-in-means technique used in prior single-direction work.

2. A new multi-directional (MD) method. They train a SOM on harmful prompt representations, then derive one direction per neuron by subtracting the harmless-prompt centroid from each neuron, producing a candidate pool of 16 directions with a 4x4 lattice. Bayesian Optimization selects the best k directions from this pool for ablation.

3. Empirical superiority over baselines and attacks. MD achieves the highest attack success rate on all eight evaluated models compared with the single-direction baseline (SD) and the RDO method, and it also outperforms the jailbreak algorithms GCG and SAA on all models except LLama3-8B against SAA.

4. A mechanistic analysis. They show that ablating MD directions compresses harmful representations, moves harmful representations closer to harmless ones, that SOM neurons span the approximate refusal manifold, and that the extracted directions are aligned with each other rather than orthogonal.

Main Findings

• MD beats the single-direction baseline everywhere. In Table 1, MD attains the highest attack success rate (ASR) on all eight models. On Llama-2-7B, MD reaches 59.11% while SD is 0.0% and RDO is 1.25%. The largest reported margin over SD is on Llama3 (73%), where SD scores 15.09% and MD scores 88.05%.

• MD also beats purpose-built jailbreak attacks. MD outperforms GCG and SAA on all models except the SAA comparison on LLama3-8B, where SAA scores 91.20% and MD scores 88.05%. On Gemma2-9B, MD reaches 96.27% versus 5.03% for GCG.

• MD partially reverses a defended model. On Mistral-7B-RR, which implements Representation Rerouting, MD achieves 25.79% ASR, while SD is 5.03%, RDO 1.25%, GCG 0.6% and SAA 1.6%.

• More ablated directions generally means higher ASR. Table 2 shows Llama-2-7B rising from 7.5% (MD-2) to 59.11% (MD-7), and Qwen-14B from 75.47% (MD-2) to 91.82% (MD-7). The best ASR across models is found around MD-5, MD-6 and MD-7. Mistral-7B-RR plateaus at MD-5 with 25.79% and then declines at MD-6 (20.75%) and MD-7 (18.25%).

• Directions are computed in mid-architecture layers. The selected layer l*/L is 13/32 for LLama2-7B, 11/32 for LLama3-8B, 16/32 for Qwen-7B, 22/40 for Qwen-14B, 24/36 for Qwen2.5-3B, 18/28 for Qwen2.5-7B, 23/42 for Gemma2-9B and 23/32 for Mistral-7B-RR.

• MD shrinks and shifts harmful representations. For Llama2-7B, the intra-cluster variance of harmful prompts drops from σ = 5.85 with no ablation to σ = 1.25 under MD-7, and the Euclidean distance between harmful and harmless centroids drops from 8.82 to 2.18.

• SOM neurons cover the harmful distribution. A 3D PCA visualization of SOMs trained on 4000 harmful prompt representations from SorryBench shows the 16 neurons covering high-density regions of the distribution with minimal overlap across the models examined.

• MD directions are correlated, not orthogonal. Cosine similarity analysis on LLama2-7B and Qwen-14B shows several MD directions are moderately or strongly aligned with each other and with SD. The authors state this challenges the orthonormal-basis assumption used in RDO, since enforcing orthogonality may discard semantically meaningful but geometrically aligned directions.

• Reproduction note on Llama-2-7B. The authors state that Arditi et al. (2024) report an LLama2-7B result of 22.60%, but that they failed to reproduce that value with their setup.

Methodology in Plain English

The starting point is the prior recipe: collect the model's internal activations for harmful prompts and for harmless prompts at a chosen layer, at the position just before generation begins, and compute the difference between their average (centroid) vectors. That difference is the single refusal direction, and removing it from the model's activations (projecting them onto the orthogonal complement) suppresses refusal.

The authors replace the single harmful average with a Self-Organizing Map. A SOM is a grid of "neurons" that adapt to cover a data distribution while preserving topology, so nearby neurons correspond to nearby regions of the data. They first prove that a SOM with only one neuron converges to the data centroid, which ties their method back to the original difference-in-means construction. They then train a SOM with a 4x4 hexagonal lattice (16 neurons) on harmful prompt representations taken from 4000 SorryBench prompts, using 10,000 iterations, a learning rate schedule of α_t = α_0/(1 + 2t/T) with α_0 = 0.01, and a Gaussian neighborhood function with standard deviation σ = 0.3.

Each of the 16 neurons becomes a candidate direction by subtracting the harmless centroid, computed from 6000 Alpaca prompts, from that neuron. Because choosing the best combination of k directions is combinatorial, they use Bayesian Optimization with a Tree-structured Parzen Estimator sampler over a Harmbench validation set, running 128 trials when k ≤ 3 and 512 trials when k > 3, with k ranging from 2 to 7. The selected directions are composed into a chained projection operator applied uniformly at the output of every layer. Evaluation uses the 159 HarmBench "standard" prompts, with HarmBench-Llama-2-13B-cls as the judge that classifies whether a response is harmful and complies with the request.

Why This Matters

Impact on research. The paper challenges the single-direction view of refusal that has shaped a large body of activation-steering work, and it disputes the orthogonality assumption used by alternative multi-direction methods such as RDO. It reframes refusal as a manifold that can be approximated and manipulated as a family of related directions, and it provides a proof connecting SOMs to the widely used centroid-difference construction, giving the new method a direct lineage to prior work.

Real-world applications:

• Safety evaluation and red-teaming: MD provides a stronger, universal test of whether a model's refusal behavior survives internal manipulation, which can be used to audit models before deployment.
• Defense design: understanding that refusal occupies multiple correlated directions, and that harmful representations get compressed and shifted toward harmless ones under ablation, informs the construction of defenses such as Representation Rerouting.
• Alignment research: the method offers a way to probe how safety training shapes internal geometry and to measure whether a safety intervention holds under white-box access.
• Interpretability tooling: the SOM-based procedure is presented as a general way to map a conceptual manifold and extract multiple directions, potentially applicable beyond refusal.

Industry relevance. The evaluated models include widely deployed families (Llama, Qwen, Gemma, Mistral), and the results show that a universal set of ablated directions can rival or beat prompt-specific attacks such as GCG and SAA. The paper notes the practical attack surface is limited because full white-box access to parameters and internal activations is presupposed, and the authors include an ethical considerations section acknowledging that MD consolidates and simplifies prior methods and thereby exposes additional vulnerabilities in present refusal mechanisms.

Future Directions

• More efficient direction search. The Bayesian Optimization search space grows with both k and the lattice size, requiring many trials. The authors suggest more efficient or structured search strategies, gradient-based optimization, or pruning algorithms during the search.

• Layer-specific rather than uniform ablation. Both MD and SD compute directions at a single layer l* but apply them uniformly across all layers, which the authors believe may miss layer-specific variations in how refusal is encoded.

• Two SOMs instead of one. The method currently uses a single harmless centroid, motivated by the homogeneity of harmless representations and by the added number of directions a second SOM would induce. The authors propose investigating separate SOMs for harmless and harmful prompts.

• Broader tasks beyond refusal. The authors state that the principle of mapping a conceptual manifold with SOMs and extracting multiple directions holds promise for tasks beyond refusal suppression.

Target Audience

This paper is most useful to interpretability and AI safety researchers who work on activation steering, representation engineering, or jailbreak mechanisms. It also suits red-team and model-auditing practitioners who need a strong white-box baseline for testing refusal robustness, and alignment engineers interested in how safety behaviors are structured internally. Readers should be comfortable with transformer activation layers, linear projection and ablation, clustering methods, and attack success rate evaluation; the paper is not written for a general audience.

Authors’ abstract

Refusal refers to the functional behavior enabling safety-aligned language models to reject harmful or unethical prompts. Following the growing scientific interest in mechanistic interpretability, recent work encoded refusal behavior as a single direction in the model's latent space; e.g., computed as the difference between the centroids of harmful and harmless prompt representations. However, emerging evidence suggests that concepts in LLMs often appear to be encoded as a low-dimensional manifold embedded in the high-dimensional latent space. Motivated by these findings, we propose a novel method leveraging Self-Organizing Maps (SOMs) to extract multiple refusal directions. To this end, we first prove that SOMs generalize the prior work's difference-in-means technique. We then train SOMs on harmful prompt representations to identify multiple neurons. By subtracting the centroid of harmless representations from each neuron, we derive a set of multiple directions expressing the refusal concept. We validate our method on an extensive experimental setup, demonstrating that ablating multiple directions from models' internals outperforms not only the single-direction baseline but also specialized jailbreak algorithms, leading to an effective suppression of refusal. Finally, we conclude by analyzing the mechanistic implications of our approach.