LLM-VA: Resolving the Jailbreak-Overrefusal Trade-off via Vector Alignment

AI summary

Overview

• Research area: Machine learning / LLM safety alignment and mechanistic interpretability — specifically activation steering through internal representation vectors.
• Technical level: Advanced. The paper assumes familiarity with residual streams, control vectors, SVM decision boundaries, pseudoinverse-based minimum-norm weight updates, and metrics like attack success rate (ASR) and over-refusal rate (ORR).
• Scope: The paper diagnoses why existing vector-steering defenses trade jailbreak suppression against over-refusal, and proposes LLM-VA, a closed-form weight-update method that aligns the model's "answer" direction with its "benign" direction.

What This Paper Is About

Safety-aligned LLMs fail in two opposite ways: they answer harmful inputs (jailbreak) and they decline harmless ones (over-refusal). Existing vector-steering defenses only scale the magnitude of an "answer vector," which pushes one failure mode down while pushing the other up. The authors show that this trade-off exists because models store the decision to answer and the judgment of input safety in nearly orthogonal directions, and they propose to fix it by aligning those two directions rather than resizing one of them.

Key Contributions

1. Diagnosis of the trade-off's root cause. The paper shows that the answer vector (v_a) and the benign vector (v_b) are nearly orthogonal (~90 degrees) across layers in all 12 tested LLMs, meaning answer decisions and safety assessments are structurally decoupled rather than being one process.
2. LLM-VA, a vector-alignment method. The first vector-steering method the authors claim to simultaneously address jailbreak and over-refusal, using closed-form weight updates with no gradient-based fine-tuning and no architectural changes (no hook layers).
3. Layer-wise SVM vector identification and layer selection. Control vectors are extracted per layer from SVM hyperplanes rather than from the final residual stream, and layers are ranked by a score combining influence on the final decision and SVM classification accuracy.
4. Broad evaluation and artifact release. Experiments across 12 LLMs from 5 model families (3B to 14B parameters), plus released code and safety-enhanced weights for all 12 LLMs.

Main Findings

• The two decisions are encoded separately. Projections onto v_b separate benign from toxic inputs, and projections onto v_a separate answered from refused samples, both with decision boundaries near zero. But the two vectors sit at roughly 90 degrees to each other, so scaling v_a hits benign and toxic inputs equally.
• Later layers matter more for safety. Separation quality improves in deeper layers (illustrated at layers 4, 16, and 28 of Llama-3.1-8B-Instruct), and layers whose vectors align with the final residual stream score higher for safety relevance.
• Headline effectiveness. LLM-VA achieves an average F1 of 0.77 versus 0.56 for the original LLMs, a 37.02% relative improvement, while reducing ASR by 18.50% and ORR by 22.00% on average.
• Beat the best baseline by 11.45% F1. LLM-VA outperforms all baselines on 8 of 12 LLMs, with an 11.45% relative F1 improvement over the best baseline (AlphaSteer), at a cost of only a 4.08% model utility drop.
• Utility retained at 95.92%. LLM-VA preserves 95.92% of original utility on average, beating SCANS (which degrades utility by an average of 40.98%), VectorSteer (89.74%), AlphaSteer (94.50%), and AlphaSteer+ (94.48%). For 9 of 12 LLMs, preservation exceeds 95%.
• Task-dependent utility loss. Classification tasks (COLA, MNLI, RTE, MRPC, SST) mostly preserve over 97% of performance, while GSM8K math reasoning preserves 91.60% on average.
• Model size affects robustness. The three lowest-preservation models were Phi-3.5-4B (92.1%), Phi-4-4B (91.8%), and Mistral-v0.3-7B (93.2%), the smallest or most benchmark-limited models in the set.
• Adaptive behavior without tuning. Model bias dictates what gets fixed: for Mistral-v0.3-7B (81% ASR, 29% ORR) LLM-VA mainly lowers ASR, while for Llama-3.1-8B (7% ASR, 53% ORR) it mainly lowers ORR.
• Vector accuracy matters, but tolerance is forgiving. Randomly distorting the vectors by 90 degrees drops F1 by 24.82% on average across all 12 models; at 60 degrees all models still degrade; at 30 degrees F1 drops only 5.40%.
• Iteration count is model-specific. Llama 3.1 (8B) stabilizes around T=19; Phi 4 (4B) peaks at T=19 then degrades (over-modification); Qwen 3 (14B) keeps improving through T=30, and the authors extended it to T=60 and saw continued gains.
• Layer count has a sweet spot. Varying L_select from 30 to 60 produces a non-monotonic F1 curve (too few layers limits effect, too many causes overfitting), while utility stays stable until too many early layers are modified, then drops sharply.
• Cross-dataset transfer remains unsolved. On five unseen benchmarks (XSTest-Toxic, OR-Bench-Toxic, AdvBench, XSTest, OR-Bench), performance varies for all compared methods, which the authors attribute partly to inconsistent safety definitions across benchmarks.
• Four models were not best-in-class. Phi-3.5-4B, Phi-4-4B, Mistral-v0.3-7B, and Qwen3-14B did not reach the highest F1, which the authors trace to iteration-count sensitivity (over-modification for the first three, under-iteration for Qwen3-14B) rather than a fundamental limitation.

Methodology in Plain English

The method has three stages, all operating on the model's internal layer outputs rather than on retraining.

1. Find the two directions per layer. For each layer, the researchers train two linear SVMs on layer outputs: one that separates benign from toxic inputs, and one that separates answered from refused responses. The normal vector of each maximum-margin hyperplane becomes the control vector (unit-normalized). SVMs are used instead of simple mean differences because the boundary is set by support vectors near the decision boundary, making it less sensitive to extreme points. The bias term is dropped because the authors' empirical analysis shows the hyperplanes pass through the origin.

2. Pick the layers worth touching. Each layer gets a score combining (a) how well its vectors align with the final residual stream vectors, and (b) how accurately its SVMs classify. Multiplying influence by accuracy means a layer that is influential but inaccurate (or vice versa) scores low. The top L_select layers are chosen.

3. Align the vectors with a closed-form update. For selected layers, they modify the down-projection matrix W so that for any input x, the projection onto v_a equals the projection onto v_b scaled by the ratio of the two directions' standard deviations (σ_a/σ_b). This normalization ensures benign inputs (positive v_b) produce positive v_a projections and toxic inputs (negative v_b) produce negative ones. The minimum-norm solution — the smallest weight change satisfying the constraint — is computed with a pseudoinverse, giving W' = W + Δ⁺. No gradients, no hooks, no architecture changes.

Because editing one layer shifts the effective vectors of later layers, the identification, scoring, and update are repeated iteratively (maximum T=30, with early stopping if validation F1 fails to improve for three consecutive iterations). The final model is chosen by validation F1. Data setup: 500 samples per benchmark, restricted to cases where original models already behaved incorrectly, split 8:1:1 into train/validation/test; evaluation uses S-Eval-Attack, S-Eval-Risk, ORFuzzSet, and Natural Questions, with Qwen3-Guard-Gen-8B as the judge deciding whether a response counts as an answer or a refusal.

Why This Matters

Impact on research: The paper reframes the jailbreak–over-refusal trade-off from an unavoidable trade-off into a geometric mismatch that can be repaired by aligning two internal directions. It also offers evidence that safety alignment via RLHF and safety fine-tuning leaves "helpfulness" and "harmlessness" as largely separate, independently optimized features — a claim with implications for how future alignment training is designed. The method's reliance on closed-form updates rather than training makes it cheap to reproduce and compare against.

Real-world applications:

• Customer-facing assistants and chatbots that must both refuse genuinely harmful requests and stop frustrating users with spurious refusals — the paper's own examples show baselines leaving 100% ORR on Llama-3.1-8B for ORFuzzSet.
• Deployed moderation pipelines where safety behavior must be adjusted per model without paying for full retraining.
• Enterprise LLM hosting platforms, since the method needs no architectural modifications and applies to standard checkpoints.
• Any setting with high stakes on false refusals — the authors specifically mention healthcare and PLC code generation, where users may want to prioritize one behavior over the other.

Industry relevance: The combination of no-fine-tuning, no-architecture-change deployment, closed-form updates, and released safety-enhanced weights for 12 models makes this directly usable by teams that cannot afford gradient-based safety tuning. The paper's own baseline table highlights that competing input-aware methods (SCANS, CAST) require hook layers, which is a deployment friction point.

Note: the ethical considerations section warns the same vector-alignment technique could be misused to bypass safety mechanisms or inject harmful biases, since it operates on internal safety representations.

Future Directions

1. Extending beyond binary toxicity. The paper treats inputs as benign or toxic only; multi-class or fine-grained toxicity classification is explicitly left as future work.
2. Scaling up and out. Validation is limited to 3B–14B models; effectiveness on 70B+ models, and whether they need different hyperparameters or exhibit different internal representations, is untested.
3. Making the method plug-and-play. Optimal iteration count and layer selection vary per model. Validation-based selection still requires tuning for each new model and a held-out validation set, which limits deployment convenience.
4. Improving cross-dataset transfer and adversarial robustness. Performance on unseen benchmarks varies widely across all steering methods, and the authors note that coupling the answer direction to the safety direction may create a new attack surface, where an adversary manipulating the model's internal safety representation could bypass alignment.

Additional stated open items: applying the approach to chain-of-thought reasoning models (control vectors must be identified after reasoning steps, which is expensive and noisy), static alignment that does not track evolving threats, customizable trade-offs that let users favor one behavior over the other, and adding statistical significance testing, since results come from single runs with a fixed random seed.

Target Audience

Researchers and engineers working on LLM safety alignment, activation steering, and mechanistic interpretability will get the most from this paper. It is also relevant to practitioners who need to harden deployed models without retraining, and to teams building evaluation suites for jailbreak and over-refusal behavior. Readers need prior exposure to residual streams and linear probing; beginners can follow the high-level argument (the two decisions live in different directions) but will find the derivation of the closed-form update demanding.

Authors’ abstract

Safety-aligned LLMs suffer from two failure modes: jailbreak (answering harmful inputs) and over-refusal (declining benign queries). Existing vector steering methods adjust the magnitude of answer vectors, but this creates a fundamental trade-off -- reducing jailbreak increases over-refusal and vice versa. We identify the root cause: LLMs encode the decision to answer (answer vector $v_a$) and the judgment of input safety (benign vector $v_b$) as nearly orthogonal directions, treating them as independent processes. We propose LLM-VA, which aligns $v_a$ with $v_b$ through closed-form weight updates, making the model's willingness to answer causally dependent on its safety assessment -- without fine-tuning or architectural changes. Our method identifies vectors at each layer using SVMs, selects safety-relevant layers, and iteratively aligns vectors via minimum-norm weight modifications. Experiments on 12 LLMs demonstrate that LLM-VA achieves 11.45% higher F1 than the best baseline while preserving 95.92% utility, and automatically adapts to each model's safety bias without manual tuning. Code and models are available at https://hotbento.github.io/LLM-VA-Web/.