The Sharp Directions Are Against You: Curvature Analysis of Activation Steering

AI summary

Overview

• Research area: Activation steering and mechanistic interpretability for large language models; specifically the geometry of behavioral control via inference-time activation interventions.
• Technical level: Advanced. The paper builds on contrastive activation addition (CAA) and introduces a Hessian-based curvature analysis with two formal propositions, so comfort with linear algebra, eigendecompositions, and language-model internals helps.
• Scope: The paper identifies high-curvature ("sharp") directions in activation space that make CAA steering vectors fragile, and proposes a one-line post-processing fix called sharp removal.

What This Paper Is About

Activation steering adds a vector to a language model's internal activations at inference time so its behavior changes without any fine-tuning, but the standard construction, contrastive activation addition (CAA), frequently pushes behavior in the wrong direction. This paper explains that fragility geometrically: the behavioral loss landscape in activation space is strongly anisotropic, and a small fraction of the CAA vector's energy lies along catastrophically sensitive directions. The goal is to identify those directions with a curvature object called the steering Hessian and to remove them, producing a more robust steering vector at no inference-time cost.

Key Contributions

1. The steering Hessian. The authors define a d × d matrix of second derivatives of the behavioral loss with respect to the steering vector itself (not the model parameters). Its top eigenvectors are "sharp directions" and its bottom eigenvectors are "flat directions." Proposition 1 gives a Gauss-Newton decomposition showing why top eigenvectors are catastrophic, and Proposition 2 bounds what is preserved, what is lost, and what is gained by removing them.
2. Two empirical findings about steering geometry. The top Hessian eigenvectors behave as adversarial directions at inference: projecting CAA onto them reverses behavior in all 36 tested conditions (3 models × 4 tasks × 3 layers), with anti-steer rates of 53–100%. The bottom eigenvectors diagnose when CAA has the wrong sign on three of four tasks.
3. Sharp removal, a drop-in correction. Subtracting the sharp component from a CAA vector produces a cleaned vector that outperforms standard CAA in open-ended generation across 25 of 25 tested conditions spanning three tasks and three model families. The procedure is a post-processing wrapper that composes with other steering methods, demonstrated on Arditi et al. (2024)'s refusal direction.
4. Jailbreaking evidence that sharp directions are noise, not signal. Applied in the negative direction, sharp vectors fail to produce coherent jailbreaks, while the cleaned vector is the most controllable along either polarity.

Main Findings

• The Hessian spectrum is strongly anisotropic in every condition. Condition numbers κ = |λ_max| / |λ_min| range from 50 to 3,727 across all 36 conditions (3 models × 4 tasks × 3 layers). Every value exceeds 50, so the behavioral loss landscape is never approximately isotropic.
• Sharp directions reverse behavior in all 36 conditions. Anti-steer rates of v_sharp range from 0.53 to 1.00 with zero exceptions. On Llama refusal, v_sharp reaches a 100% anti-steer rate at every layer, with peak magnitude Δ = −0.48.
• Sharp energy is tiny but disproportionate. Sharp directions account for only 1–5% of the CAA vector's squared norm (ε ≤ 0.05). On Llama, the sharp component is just 2.6% of CAA's squared norm yet accounts for essentially all of CAA's distributional gap from the unsteered baseline, measured as per-position KL near 19 nats.
• Sharp removal preserves direction and magnitude. Cosine similarity between v_CAA and v_cleaned exceeds 0.974 and norm reduction is under 3% for typical settings (ε ≤ 0.05). Directional curvature is reduced by κ_gap = 1.7–12× (median 4×), a quantity invariant to the steering coefficient α because the Hessian scales with α².
• CAA inversion is diagnosed by the flat subspace. CAA has the wrong sign on three of four tasks: sycophancy (all three models), toxicity (Llama), and refusal (Qwen, Llama). At the best layer, v_flat recovers the correct sign in every case, for example Sycophancy Mistral L19 (CAA Δ = −0.028 to v_flat Δ = +0.317), Toxicity Llama L20 (CAA Δ = −0.010 to v_flat Δ = +0.390), and Refusal Qwen L20 (CAA Δ = −0.006 to v_flat Δ = +0.013).
• The flat subspace is robust, while random directions are not. At the best layer per model per task, v_flat has the lowest anti-steer rate in 11 of 12 model-task combinations, including a 0% anti-steer rate on Qwen sentiment (L21) across all 1,000 evaluations. v_random produces 60–86% anti-steer rates in most conditions. v_flat also outperforms v_random in 11 of 12 model-task combinations at generation level, with margins of 0.14–0.79.
• Sharp removal wins at generation level across the board. It outperforms CAA in all 25 tested conditions. On sentiment it reaches near-ceiling scores (Mistral L13 rises from 0.67 with CAA to 0.97 with sharp removal). On refusal the effect is most pronounced: CAA degrades safety (Qwen drops from 78% to 36–44%; Llama from 62% to 24–52%), while sharp removal raises refusal to 88–98% on Qwen and 74–76% on Llama, in most cases above the unsteered baseline. On Mistral, whose keyword-matched baseline is 12%, sharp removal raises refusal to 36–62%.
• Sharp directions are noise, not signal. In a jailbreaking experiment on Qwen2.5-7B-Instruct (L10, N = 50) and Llama-3.1-8B-Instruct (L15, N = 100) at α = 3.0 with a GPT-4o-mini judge, −v_sharp is the least effective jailbreak (2% full compliance on Qwen with 40% incoherent partials; 100% refusal on Llama, safer than baseline), while −v_cleaned is the most effective (80% vs 68% non-refusal on Qwen; 40% vs 33% on Llama).
• Negating CAA is not a substitute for sharp removal. v_cleaned outperforms −v_CAA in 8 of 10 sycophancy conditions and in all 4 inverted refusal conditions by 23–38 percentage points (e.g., Qwen L10: 98% vs 68%), because −v_CAA retains the sharp component that disrupts generation regardless of sign.
• High variance and high curvature are geometrically distinct. The top-15 generic-PCA and Hessian subspaces have a mean principal-angle cosine of 0.077, comparable to two random subspaces.
• A k-sweep reveals two regimes, motivating k = 10–15 as a default. On Qwen refusal, sharp removal is robust to k (k = 1 raises refusal from 44% to 90%, plateau by k = 5); on Llama sycophancy it peaks at k = 5 and degrades at k = 25 from over-removal, as predicted by Proposition 2(ii).
• Layer choice is task-dependent. Sentiment favors mid-to-late layers, sycophancy mid layers, and toxicity and refusal are model-dependent.

Methodology in Plain English

The authors start from the standard CAA recipe: take the mean difference of hidden-layer activations between positive and negative behavioral exemplars to build a steering vector, then add it to activations during the forward pass with a scaling coefficient α. To study curvature, they define a scalar behavioral loss — the next-token cross-entropy of a designated "target token" whose high probability signals the desired behavior (for example "Great" for sentiment, "I" for refusal). Because the model is frozen, this loss is a smooth function of the steering vector, so its gradient and Hessian are well defined.

Forming a full Hessian is infeasible at hidden dimensions of 3,584 and 4,096, so they estimate the top and bottom eigenpairs using Lanczos iteration with Hessian-vector products approximated by central finite differences (δ = 0.01), averaged over 12 estimation prompts with m = 40 Lanczos iterations and full reorthogonalization. This takes roughly 40 seconds per layer on a single NVIDIA A6000 GPU, and the full pipeline runs in under 2 minutes per layer as a one-time cost.

Sharp removal then projects the CAA vector onto the span of the top-k sharp eigenvectors (k = 15 by default in the main experiments) and subtracts that component: v_cleaned = v_CAA − P_S v_CAA. No renormalization is applied, so v_cleaned retains typically 95–99% of the CAA norm. The flat projection v_flat, renormalized to the original norm, is used as a complementary diagnostic rather than a deployment vector, because it discards all signal outside the bottom-k eigenvectors.

Experiments cover three 7–8B instruction-tuned models: Qwen2.5-7B-Instruct (d = 3584), Llama-3.1-8B-Instruct (d = 4096), and Mistral-7B-Instruct-v0.3 (d = 4096). Four behavioral axes are tested: Sentiment, Sycophancy, Toxicity (RealToxicityPrompts), and Refusal (harmful AdvBench requests). Each task uses 20 contrastive CAA pairs, 12 Hessian estimation prompts, and 100 evaluation prompts, with the three sets disjoint and all evaluation held out. Three layers per model (early, mid, late) give 36 token-level conditions.

At the token level, the metric is probability shift Δ = P(target) − P(anti-target) at the first generated position, swept over 10 values of α in [0.25, 5.0]; the anti-steer rate is the fraction of inputs with Δ < 0, averaged over α. At the generation level, 50 completions of 80–150 tokens are produced with greedy decoding and scored by task-specific classifiers (RoBERTa for sentiment, BART-MNLI for sycophancy, AdvBench keyword detection with an NLI tiebreaker for refusal). Comparisons include CAA, v_sharp, v_flat, v_random as a dimensionality control, and v_cleaned. Stability checks include a multi-token contrastive variant (6–8 tokens each, Llama sycophancy layer 14, principal-angle cosine 0.61 to the single-token subspace, matched-N honesty 0.83 vs 0.85), Lanczos convergence, an XSTest over-refusal control, an LLM-judge validation on Mistral, and an N = 200 sycophancy sample-size check.

Why This Matters

Impact on research. The paper reframes steering failure as a geometric property of the model rather than an empirical nuisance. It suggests the linear representation hypothesis needs refinement: concepts may be linearly decodable, but their steerability depends on curvature. It also connects the sharpness-aware minimization idea, previously applied in weight space, to activation space, where the loss landscape changes with each input and the object of interest is a single d-dimensional vector rather than billions of parameters. The sharp energy fraction ε(v) is proposed as a safety audit metric for any proposed steering vector, and the method applies to SAE features, optimization-based vectors, and other activation-space interventions — not just CAA.

Real-world applications.

• Safety auditing of steering vectors before deployment, using ε(v) to flag vectors likely to undermine alignment.
• Safer deployment of refusal steering in production systems: CAA degraded Qwen refusal from 78% to 36–44%, while sharp removal raised it to 88–98%.
• Model editing for tone, honesty, or toxicity control in applications where fine-tuning is too expensive, since sharp removal adds zero inference overhead.
• Red-teaming and jailbreak analysis: the top Hessian eigenvectors expose safety vulnerabilities without any adversarial optimization.

Industry relevance. The pipeline runs in under 2 minutes per layer on a single NVIDIA A6000, requires no retraining, no parameter modification, and no change to the downstream model, and adds no inference-time cost. It is a post-processing wrapper that stacks on top of existing steering methods, so teams already using CAA can adopt it without replacing their pipeline. The composition result with Arditi et al. (2024)'s refusal direction — largely distinct from CAA with cosine 0.01–0.20 — shows the diagnostic generalizes across construction methods.

Future Directions

• Formal bounds at higher steering coefficients. The propositions hold at α = 1.0. Generation-level improvements at higher α are empirically validated but explicitly not formally bounded in this work.
• Choosing k automatically. The k-sweep shows opposite failure modes in different conditions (robustness to k on Qwen refusal versus degradation at k = 25 from over-removal on Llama sycophancy), motivating k = 10–15 as a heuristic rather than a derived optimum. A principled selection rule remains open.
• Extending the curvature diagnostic beyond CAA and refusal directions. The authors state the same procedure applies to SAE features, optimization-based vectors, and any activation-space intervention, but only CAA and the Arditi et al. direction are evaluated here.
• Generalizing the behavioral objective. Sharp removal is shown to be robust to a multi-token contrastive objective, and Proposition 1 attributes this to the downstream Jacobian. Whether the framework extends to reward-model scores and other differentiable objectives is proposed rather than demonstrated at scale.

Target Audience

Researchers and practitioners working on mechanistic interpretability, activation steering, and LLM safety alignment; engineers building training-free behavioral control systems who need more robust steering vectors; and safety teams auditing whether a proposed steering intervention could weaken alignment. Readers should be comfortable with eigendecomposition, Hessians, and transformer internals to follow the theoretical sections, though the experimental results and the sharp-removal recipe are accessible without the proofs.

Authors’ abstract

Activation steering adds a vector to a language model's internal representations at inference time. It is a lightweight alternative to fine-tuning for behavioral control. The standard construction, contrastive activation addition (CAA), is fragile, as on many inputs it shifts behavior in the wrong direction. To explain this fragility, we first study the geometry of the steering vector. We introduce steering Hessian, the matrix of second order derivative of the model's behavioral loss with respect to the steering vector. It captures the output behavior response sensitivity of the model under small perturbations of the steering vector along each direction. Its top eigenvectors are sharp directions: tiny perturbations along them can cause large, erratic, input-dependent behavioral swings. Bottom eigenvectors are flat directions: behavior is robust along them. Removing the sharp component of a steering vector approximately preserves its direction and magnitude (cosine &gt; 0.974, norm reduction &lt; 3% for typical settings) while reducing the vector's directional curvature by up to 12x (median 4x across 36 conditions). We evaluate on three model families (Qwen2.5-7B, Llama-3.1-8B, Mistral-7B), with four behavioral tasks, and 36 token-level conditions. Steering along sharp directions reverses the intended behavior in every condition tested, with anti-steer rates of 53-100%. Sharp directions account for only 1-5% of the CAA vector's squared norm. However, they drive a disproportionate share of the gap between the steered and unsteered next-token distributions during autoregressive generation. We term this one-line correction as sharp removal; it outperforms standard CAA in open-ended generation across all 25 conditions tested. A jailbreaking experiment further shows that sharp directions act as noise rather than signal: they disrupt coherent behavioral control regardless of steering polarity.