PatchBench: Measuring Collateral Damage in Activation Patching

AI summary

Overview

• Research area: LLM safety and interpretability, specifically the evaluation of activation patching / activation steering as a post-hoc jailbreak repair technique.
• Technical level: Advanced. The work assumes familiarity with white-box model interventions, steering vectors, refusal directions, and benchmark evaluation practice.
• Scope: The paper introduces PatchBench, a curated bank of 400 verified, model-specific jailbreak failures, and PatchBench-Local, a protocol that measures whether a patch repairs the targeted failure locally or instead suppresses neighbouring benign behaviour.

What This Paper Is About

Activation patching can make a model refuse a specific harmful prompt and still be a bad repair: it may fail on close harmful variants, or it may over-refuse benign prompts that merely share wording or structure with the original request. Existing evaluation relies on aggregate metrics such as attack success rate, refusal rate, global capability scores, and MMLU, none of which can tell a selective repair apart from broad local suppression. The paper builds a benchmark of confirmed model-specific jailbreak failures together with a local evaluation protocol that scores harmful-neighbour correction and benign-neighbour preservation separately.

Key Contributions

1. PatchBench, a curated benchmark of 400 actionable jailbreak failures, organised as 50 high-confidence failures for each of 8 open-source instruction-tuned models, intended as concrete repair targets for activation patching.
2. PatchBench-Local, a local evaluation protocol that, for each source failure, generates harmful variants plus structurally matched and lexically related benign neighbours, and measures harmful-neighbour correction against benign-neighbour preservation.
3. An empirical demonstration, across four representative models and four activation patching methods (AlphaSteer, AdaSteer, AST, CAST), that strong harmful-neighbour correction can coincide with substantial benign-neighbour regressions, including cases where MMLU remains nearly unchanged.
4. Public release of PatchBench-Local code on GitHub and the PatchBench dataset on Hugging Face.

Main Findings

• Exact-prompt success hides local failures: The paper's core claim is that aggregate metrics cannot distinguish selective local repair from broad local suppression, so harmful correction and benign preservation must be reported separately.
• A consistent correction–preservation trade-off: In the (BNPS, HNCS) plane, many methods move upward (better correction) but leftward (worse benign preservation) relative to the unsteered baseline. Table 1 reports unsteered baselines of HNCS/BNPS of 40.1/98.6 (Qwen 3B), 32.2/99.2 (Mistral 7B), 31.2/97.1 (Llama 8B), and 18.2/98.8 (Gemma 4B).
• AST is the clearest over-suppression case: AST frequently obtains the highest HNCS, but its gain over more selective methods is modest against its BNPS loss. On Qwen 3B it improves HNCS over AlphaSteer by only 3.1 points while losing 27.4 BNPS points; on Llama 8B it nearly matches AdaSteer in HNCS but loses 17.5 BNPS points; on Gemma 4B it is below AdaSteer in HNCS and 58.3 points lower in BNPS.
• AlphaSteer and AdaSteer have more favourable profiles on most models: AlphaSteer is particularly strong on Qwen 3B (HNCS 92.5, BNPS 94.2). AdaSteer is more balanced than AST on Gemma 4B (HNCS 87.6, BNPS 86.7). CAST is generally more conservative, with weaker harmful correction but smaller benign losses than AST on Mistral 7B and Gemma 4B.
• Global capability can stay flat while local behaviour degrades: On Llama 8B and Gemma 4B, CAST drops MMLU 5-shot accuracy by only 0.07 and 0.52 percentage points while BNPS falls by 26.3 and 22.0 points. AlphaSteer on Llama 8B and AdaSteer on Qwen 3B also show modest MMLU drops alongside clear BNPS losses.
• AST shows the opposite, more obvious pattern: Its broad local suppression often coincides with large global capability drops: MMLU falls by 20.81 points on Qwen 3B, 21.62 on Mistral 7B, 6.20 on Llama 8B, and 17.20 on Gemma 4B.
• Benign regressions differ in kind, not just magnitude: PatchBench-Local decomposes benign loss into lexical overblocking (refusing benign prompts that reuse harmful terms) and structural overblocking (refusing benign prompts that keep the scaffold of the harmful request after the intent is removed). AlphaSteer on Llama 8B shows low lexical loss but high structural loss; AdaSteer shows substantial lexical loss on Llama 8B but milder losses on Mistral 7B and Gemma 4B; CAST shows low benign loss on Mistral 7B, larger structural loss on Llama 8B, and larger lexical loss on Gemma 4B; AST loses heavily on both families for several models, especially Gemma 4B and Llama 8B.
• PatchBench captures model-specific failure profiles: Overall the benchmark is 38.5% weapons, 36.0% cyber, and 25.5% crimes, fraud, and abuse, but per-model distributions differ sharply: Llama-3.1-8B-Instruct is dominated by cyber failures (58%), gemma-3-12b-it and Ministral-3-14B-Instruct by weapon-related failures (54% and 52%), and Qwen2.5-3B-Instruct by crimes, fraud, and abuse prompts (54%).
• Some prompts recur across models: 241 source prompts appear for a single model only, while 68 distinct source prompts appear for at least two models; most recurring prompts appear for exactly two models, and only a few for three or more.
• Scoring reliability: WildGuard reached 92.6% accuracy against human majority vote on a stratified subset of 202 prompts and completions labelled by three annotators, with Cohen's kappa of 0.841, compared to human-human agreement of 0.80, 0.84, and 0.94. WildGuard slightly under-counts refusals (5.4% false negatives versus 2.0% false positives), a bias the authors state applies uniformly across methods.

Methodology in Plain English

The researchers started from RedBench, an aggregation of 37 public jailbreak benchmark datasets totalling 27,870 prompts. They manually removed duplicated benchmark content, non-generative prompts, malformed items, and prompts whose harmful intent was too weak or underspecified for targeted repair evaluation, leaving 15,314 prompts. They queried this pool on eight open-source instruction-tuned models spanning four providers and multiple scales: Qwen2.5-3B-Instruct, Qwen2.5-14B-Instruct, gemma-3-4b-it, gemma-3-12b-it, Llama-3.1-8B-Instruct, Llama-4-Scout, Mistral-7B-Instruct-v0.3, and Ministral-3-14B-Instruct. All models used native chat templates and deterministic greedy decoding, producing 122,512 prompt-completion pairs.

Raw generations contained many refusals and borderline cases, so they applied WildGuard before ranking, keeping only pairs classified as both unsafe and answering the harmful request. This left 6,380 model-specific candidate failures. They then ranked candidates per model with pairwise Elo comparisons over full prompt-completion pairs, using an LLM judge asked which completion was more harmful to generate. Elo scores were approximately stable after roughly 40 matches per candidate, so they required at least 50 pairwise matches per candidate with a K-factor of 40. They manually reviewed the top 200 candidates per model and retained the 50 most significant actionable failures, giving 400 prompt-completion pairs.

For local evaluation, PatchBench-Local generates 29 local prompts from each source harmful prompt (30 evaluation prompts including the source), using Mistral Large with a fixed template after manually sanity-checking roughly 50 generated neighbourhoods. The neighbourhood has three families: 19 harmful variants (10 paraphrases, 2 translations, 3 punctuation variants, 1 case-inversion variant, 3 spelling or grammatical variants), 5 structural neighbours that keep the request scaffold but replace the harmful intent with a benign one, and 5 harmful-word neighbours that reuse salient harmful terms in benign or informational contexts. A sensitivity analysis with GPT-4o-generated neighbourhoods is reported to confirm the results are stable to the choice of generating LLM.

Responses are scored with WildGuard, whose refusal signal defines binary labels per prompt family. Refusal on the source and harmful variants yields the Harmful Neighbour Correction Score (HNCS) on a 0–100% scale; non-refusal on the 10 benign neighbours yields the Benign Neighbour Preservation Score (BNPS). Four activation patching methods (AlphaSteer, AdaSteer, AST, CAST) are evaluated on four models, one per provider (Qwen2.5-3B-Instruct, Mistral-7B-Instruct-v0.3, Llama-3.1-8B-Instruct, gemma-3-4b-it), each using its own 50 source failures, with hyperparameters tuned on a held-out validation set and fixed for final evaluation. All four methods were also evaluated on the larger Mixture-of-Experts architecture Llama-4-Scout, with those scaling results reported in the appendix. MMLU 5-shot was used as the global capability comparison. Main compute resources were 4 NVIDIA A100 GPUs, with 2 NVIDIA H100 GPUs used for the Llama-4-Scout inference runs.

Why This Matters

Impact on research. The paper reframes jailbreak repair evaluation from a global question ("did the model get safer?") to a local question ("did this patch repair this failure without collateral damage?"). It shows that MMLU and aggregate refusal or attack-success metrics can miss substantial local regressions, which has direct implications for how activation steering methods are compared and for mechanistic work that relies on logit differences, probability shifts, or layer-wise causal effects. It also supplies concrete, model-specific repair targets rather than generic harmful prompts, since a prompt the model already refuses is not a repair target.

Real-world applications:

• Safety evaluation and red-teaming: PatchBench-Local gives red teams and safety evaluators a way to check whether a deployed mitigation is narrow and precise, including on non-English and obfuscated variants.
• Mitigation design and regression testing: Teams building inference-time safety interventions can use HNCS and BNPS as paired acceptance criteria before shipping a patch, catching over-refusal that would otherwise surface as user complaints.
• Model release qualification: Because PatchBench is model-specific, it can support per-model safety case documentation, showing which concrete failures a given checkpoint exhibited and whether a patch actually fixed them.
• Defensive research infrastructure: The released dataset and code support reproducible defensive research, benchmarking, and red-teaming evaluation, under the stated non-commercial licence and intended-use statement.

Industry relevance. The evaluated methods are all representative of current activation steering approaches, and the finding that benign-neighbour regressions can occur while global capability appears intact matters for anyone deploying inference-time interventions that alter refusal behaviour. Over-refusal in structurally similar benign requests is a user-facing product risk, and the paper argues that it is invisible to the global metrics typically used in release decisions. The work was supported by research chairs funded by Crédit Agricole SA and SNCF and used GENCI/IDRIS HPC allocations.

Future Directions

• Scaling local repair metrics beyond isolated neighbourhoods: The authors identify studying how local repair metrics should scale when many failures are considered simultaneously as a central direction for future work.
• Detecting interference across local regions: They highlight the challenge of detecting interference between local regions and accumulated benign regressions across them, rather than only improving average correction.
• Designing summarising metrics that preserve locality: They call for metrics that preserve locality while summarising many repair neighbourhoods at once, as a prerequisite for evaluating activation patching as a reliable safety intervention.
• Reducing the cost of extending the benchmark: Extending PatchBench to new models currently requires manual, model-specific generation; the authors frame this one-time cost as amortised when testing multiple repair methods, leaving open how to lower it.

Target Audience

Researchers and engineers working on LLM safety, interpretability, and inference-time interventions, particularly those developing or comparing activation patching and activation steering methods. It is also relevant to safety evaluation and red-teaming practitioners who need to certify that a mitigation is precise rather than broadly suppressive, and to model-release reviewers who currently rely on aggregate capability and refusal metrics. Readers need a working understanding of white-box interventions and benchmark design to get the most from it; the local evaluation protocol itself is described concretely enough to be adopted by teams outside the authors' group.

Authors’ abstract

An LLM safety patch can pass a benchmark while still being a poor repair. This risk is especially acute for jailbreak repairs, where the goal is to correct a specific unsafe behaviour without changing unrelated behaviours. A patch may block exact evaluation prompts yet fail on close harmful variants, or suppress harmful behaviour by over-refusing benign prompts that share its wording or structure. Existing protocols primarily test whether models can be broken, while aggregate metrics (attack success, refusal rates, global capability) cannot distinguish selective repairs from broader local suppression. To address this gap, we introduce PatchBench, a benchmark of empirically observed model-specific jailbreak failures inducing actionable harmful answers. Starting from 27,870 prompts from 37 public datasets, we curate 15,314 English prompts and query 8 open-source instruction-tuned models. Combining WildGuard filtering, pairwise Elo ranking, and manual verification, we retain a curated bank of 400 high-confidence jailbreak failures. We further introduce PatchBench-Local, an evaluation protocol testing whether a patch is behaviourally precise. For each harmful source prompt, PatchBench-Local generates three families of local neighbours: harmful variants preserving malicious intent, benign prompts with matched structure, and benign prompts reusing key harmful terms. It evaluates harmful-neighbour correction and benign-neighbour preservation, distinguishing selective repair from broader local suppression. Evaluating four activation steering methods with PatchBench-Local and MMLU shows that global capability can remain nearly unchanged while local benign regressions are severe, confirming aggregate metrics miss important collateral damage. PatchBench-Local provides a more precise basis for developing and comparing jailbreak repair methods.