Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness

AI summary

Overview

Research area: Mechanistic interpretability and safety alignment of large language models (Natural Language Processing / LLM safety).

Technical level: Advanced. The paper assumes familiarity with circuit-level interpretability tooling (activation patching, attribution patching), refusal-direction extraction, and activation steering.

Scope (1 sentence): A controlled, cross-architecture study of how three post-training objectives — supervised fine-tuning (SFT), reasoning-augmented SFT (Ra-SFT), and Odds Ratio Preference Optimization (ORPO) — reshape the internal circuitry, the geometry, and the steerability of refusal behavior in three 8B–9B language models.

What This Paper Is About

National cybersecurity agencies such as Germany's BSI (2025) and the US CISA with its Five Eyes partners (2025) recommend post-training methods like SFT and RLHF as countermeasures against LLM-powered attacks, treating alignment as a binary property these methods reliably produce. The authors argue this behavioral view is insufficient, since documented incidents (GTG-1002 in 2025, attacks on Mexican government agencies in early 2026) breached jailbreak defenses of models such as Claude Code and GPT-4.1 using role play or persistent reframing. Their goal is to determine whether how a model is trained — not just what data it sees — changes how refusal is implemented inside the model, by holding base model, training data, and hyperparameters fixed so that only the training objective varies.

Key Contributions

1. A controlled cross-paradigm comparison of how SFT, ORPO, and Ra-SFT shape refusal-related computation, with base model, training data, and hyperparameters held fixed, across three architecturally distinct models (Llama-3.1-8B, Gemma-2-9B, Qwen-3-8B).
2. The first circuit-level analysis of how Ra-SFT and ORPO implement safety alignment internally.
3. An empirically grounded "alignment trilemma": among the offline objectives studied, no method jointly achieves distributed refusal encoding, safety/capability separability, and fine-grained correctability — gains in one consistently cost another.
4. Code and models released at a linked GitHub repository.

Main Findings

• Post-training strength ordering (behavioral): Preference optimization (ORPO) strengthens safety more than reasoning augmentation of SFT, with lower Attack Success Rate (ASR) on both WildJailbreak and StrongREJECT across models (except Qwen3-8B, where SFT and Ra-SFT show no significant WildJailbreak ASR difference). In Gemma-2-9B, StrongREJECT ASR is 0.0% for ORPO, 7.9% for Ra-SFT, and 31.0% for SFT per Table 1 (the narrative text states 31.6% for SFT at that point).
• Over-refusal as the ORPO failure mode: Gemma ORPO reaches 31.6% Over-Refusal Rate (ORR) on XSTest — lower than Gemma SFT's 36.0%, but still a cost paid for its safety gain. Llama SFT reaches 59.6% and Qwen SFT 62.4% ORR on XSTest.
• Refusal geometry is objective-dependent: Post-training pushes each objective's refusal direction away from the base model and away from the other two objectives, so the training objective — not just the training data — shapes refusal geometry. SFT and ORPO remain more similar to each other than either is to Ra-SFT across all three models, indicating reasoning-chain supervision installs a qualitatively distinct pathway.
• Architecture shapes mid-network reconvergence: In Gemma-2-9B, all three objectives' directions partially reconverge in layers 13–21, with SFT and ORPO then diverging slowly (cosine similarity falling from 0.8 to 0.72). In Qwen-3-8B, only SFT and ORPO reconverge mid-network; Ra-SFT stays diverged. Llama-3.1-8B shows the weakest reconvergence. The authors hypothesize (untested) that Gemma-2-9B's knowledge distillation from a larger teacher may preserve more shared structure.
• Magnitude profile separates reasoning from preference: In Llama-3.1-8B and Qwen-3-8B, SFT and ORPO peak in mid-layers (22–27 and 22–30 respectively) before declining, while Ra-SFT's refusal-direction magnitude rises only gradually and never sharply peaks. Gemma-2-9B is the exception: its ORPO variant overshoots in late layers instead.
• All objectives concentrate causal effect in late layers: Causal effects from activation patching peak at late layers in all three models. Positive causal effects already appear at the initial layer for all conditions except Gemma Ra-SFT, suggesting harmfulness concepts are embedded during pre-training.
• Reasoning supervision shifts weight to MLPs in Llama-3.1-8B: Head 25 carries the dominant causal effect under SFT (−0.33 at layer 30), MLP layer 31 dominates under Ra-SFT (+0.90), and ORPO shows a diffuse MLP distribution. Head 25's suppressive effect on refusal weakens across SFT (−0.33), Ra-SFT (−0.20), and ORPO (−0.15).
• Gemma circuits are uniform or uneven depending on objective: SFT and ORPO show uniform, redundant encoding (+0.24 to +0.47 across all components), consistent with their high XSTest over-refusal. Ra-SFT breaks this with an uneven, MLP-dominated structure (layer 39 MLP −0.44; layer 37 MLP +0.25) alongside uniformly small attention-head effects (−0.09 to +0.08) — structurally closer to Llama's Ra-SFT circuit than to Gemma's own SFT or ORPO.
• Qwen-3-8B is MLP-dominant under all three objectives: Top-layer MLPs promote refusal under SFT and ORPO, while under Ra-SFT layer 31's MLP strongly suppresses it (−0.36). Even ORPO's highest-effect head (head 11, layer 31, +0.19) is outweighed by layer 32's MLP (+0.48).
• Ranking stability varies: With bootstrap resampling over 1000 iterations, Ra-SFT's component ranking is most stable in Gemma-2-9B (ρ=0.93) and Llama-3.1-8B (ρ=0.81), against less stable SFT/ORPO rankings (ρ=0.45/0.39 and 0.75/0.63). Qwen-3-8B is uniformly stable (ρ=0.79–0.90). Top-ranked components' confidence intervals stay non-overlapping with zero regardless of ranking stability.
• Whole-stream steering destroys utility in Llama: Under ActAdd, Llama-3.1-8B MMLU accuracy falls from 35.5/31.5/28.5 (SFT/Ra-SFT/ORPO) at α=0, to 16.5/24.5/11.5 at α=5, to 0.0/4.0/0.0 at α=10.
• Recognition-layer steering beats execution-layer steering: In Gemma-2-9B, steering Ra-SFT toward refusal at recognition layers (22–26) reduces ASR by 18.8pp at α=20, versus 5.8pp at execution layers (27, 28, 37, 39, 40); SFT shows 8pp at recognition layers (27–31) versus 4.6pp at execution layers (37–41). In Qwen3-8B, SFT recognition-layer steering cuts WildJailbreak ASR by 28.2pp versus 10.8pp at execution layers, while ORPO's gap narrows as α rises.
• Steering can reduce over-refusal in Gemma ORPO: Steering away from refusal from α=0 to 20 reduces ORR from 31.2% to 20.4% (11.2pp) with only marginal ASR increase.
• Utility separability is architecture-dependent: Gemma-2-9B MMLU accuracy stays stable under ActAdd (53–55% SFT, 40–44% Ra-SFT, 45–47% ORPO). Qwen3-8B is intermediate — stable under execution-layer steering but collapsing under recognition-layer steering.
• Head-level ITI is insufficient: ITI on Gemma-2-9B either raises ORR relative to ActAdd (ORPO) or barely reduces ASR (SFT), the latter consistent with the Hydra effect. ITI on Llama SFT's dominant suppressive head produces single-token loops and question-repetition at α=20, precluding reliable ASR measurement. Qwen3-8B is unsuitable for ITI because its attention heads have insignificant effects on refusal.
• Attack-class vulnerability follows circuit structure: SFT in Llama-3.1-8B and Gemma-2-9B is especially vulnerable to semantic attacks (happy_to_help, DAN, wikipedia, role_play) because causal effects concentrate in lexically sensitive attention heads; Ra-SFT and ORPO are safer against that class. Qwen3-8B's MLP-concentrated circuits make it less vulnerable to semantic attacks but persistently vulnerable to encoding attacks.
• The alignment trilemma: No offline objective studied satisfies all three of distributed refusal encoding, safety/utility separability, and granular correctability. Ra-SFT is partially correctable but incurs deliberation overhead that increases inference costs (it emits <think>...</think> blocks); ORPO produces over-refusal that fails separability and is unsuitable for security-adjacent applications.

Methodology in Plain English

The authors line up three training recipes on a preference-versus-reasoning grid: SFT uses neither signal, Ra-SFT adds reasoning chains that justify a safety decision, and ORPO adds a preference signal without reasoning. A fourth cell (preference plus reasoning) is left for future work.

Training uses matched data: 16,000 benign Alpaca prompts and 4,000 BeaverTails prompts provided by Hu et al. (2026); for ORPO, the 4,000 BeaverTails prompts are matched with the original BeaverTails dataset for a training set of 11,179 prompts. All three models are fine-tuned fully from base checkpoints with matched hyperparameters (learning rate 1e-5, effective batch size 128, 3 epochs, max sequence length 2048 tokens, ORPO β=0.1). Online preference algorithms are excluded because their iterative data-generation and reward-update loops would confound the comparison.

To see where refusal "lives," the authors first extract a refusal direction by difference-in-means from 256 pairs of refusal/compliance prompts (Arditi et al., 2024), normalize it by the mean activation norm at each layer, and compute pairwise cosine similarities across objectives. They then localize the circuit: activation patching across all layers measures each layer's causal contribution, attribution patching approximates component-level (MLP and attention head) contributions, and activation patching at MLPs and top-K attention heads (K=5) gives exact causal effects. Finally they test whether those components actually control behavior by steering at inference — Activation Addition (ActAdd) at the layer level and Inference-Time Intervention (ITI) at the attention-head level — using a steering vector added to activations with strength α.

Evaluation uses 2,000 WildJailbreak adversarial prompts, 420 StrongREJECT prompts across 7 attack classes, 250 XSTest adversarial benign prompts for over-refusal, and 200 MMLU prompts across five domains for utility. Metrics are ASR (judged by LlamaGuard-3-8B), ORR (heuristic string matching), and Accuracy Rate. Steering safety experiments use a random subset of 250 WildJailbreak prompts.

Why This Matters

Impact on research. Prior work characterizes refusal circuits in fixed, instruction-tuned models; this paper shows that the training objective itself reconfigures those circuits, refusal geometry, and steering reliability. It argues that behavioral alignment evaluation cannot detect the vulnerability profiles that circuit structure predicts, and it calls into question the framing of alignment as a binary property that post-training methods reliably produce — a framing explicitly adopted in the recommendations of national cybersecurity agencies.

Real-world applications (as motivated or implied by the paper):

• Assessing jailbreak defenses of deployed LLM assistants used in security-critical settings, where the paper warns current post-training methods are not a solved defense.
• Defending against LLM-powered cyber attacks, the context in which BSI and CISA recommend post-training countermeasures.
• Selecting post-training objectives for security-adjacent tasks such as system hardening and malware analysis, where over-refusal (31.6% ORR for Gemma ORPO, up to 62.4% for Qwen SFT) makes a model operationally unusable.
• Using activation steering as an inference-time guardrail, with the caveat that layer choice (recognition versus execution) determines whether steering helps or collapses utility.

Industry relevance. The paper gives practitioners concrete guidance: recognition-layer steering is consistently more effective than execution-layer steering; single-head ITI is not sufficient across circuit types; and ActAdd at small α can zero out MMLU accuracy in Llama-3.1-8B. It also maps which attack classes each objective leaves exposed — semantic attacks for SFT-concentrated attention-head circuits, encoding attacks for Qwen's MLP-concentrated circuits — information that standard ASR benchmarking does not surface.

Future Directions

• Test the distillation hypothesis. The authors flag as untested their suggestion that Gemma-2-9B's distillation from a larger teacher explains why its objectives reconverge in mid-network layers.
• Study preference and reasoning together. The fourth cell of the two-axis paradigm (preference signal plus reasoning supervision) is left for future work

Authors’ abstract

How do the methods used to train language models to refuse harmful requests shape how that refusal actually works inside the model? We compare three post-training methods - supervised fine-tuning, reasoning-augmented fine-tuning (training on reasoning chains that justify a safety decision), and preference optimization (ORPO) - across three architecturally distinct models (Llama-3.1-8B, Gemma-2-9B, Qwen3-8B). We find that training method, not just data, reshapes how refusal is computed internally: reasoning-augmented training consistently produces a distinct kind of refusal computation, visible across all three models, while architecture independently shapes internal structure and how reliably refusal can be steered. Most importantly, no method we study achieves all three properties we would want from safe alignment at once: refusal that isn't concentrated in a few fragile components, safety gains that don't cost general capability, and safety behavior correctable through small, targeted edits. We caution against treating current post-training methods as a solved, reliable defense, especially for security-critical use. Code and models are available in https://github.com/hoangcuongnguyen2001/Beyond-Shallow-Alignment.