Fewer Weights, More Problems: A Practical Attack on LLM Pruning

AI summary

Overview

• Research area: LLM security and model compression — specifically adversarial attacks that are triggered by post-training transformations such as pruning, quantization, and fine-tuning.
• Technical level: Intermediate. The attack intuition is accessible, but the paper assumes familiarity with LLM fine-tuning, weight pruning metrics, and inference engines such as vLLM.
• Scope: The paper introduces and evaluates the first attack that hides malicious behavior inside an LLM so that it only activates after the model is pruned, tested across five instruction-tuned models, three attack scenarios, and three pruning algorithms built into vLLM.

What This Paper Is About

Model pruning is now a routine way to shrink LLMs, and popular inference engines like vLLM let users prune downloaded models with a single configuration change. The authors ask whether this everyday practice can be weaponized: can an adversary publish a model that looks safe and performs normally, but turns malicious the moment a user prunes it? The paper answers yes, and shows that the attack works against all three pruning algorithms vLLM ships with (Magnitude, Wanda, and SparseGPT).

Key Contributions

1. The first pruning-activated attack on LLMs. The authors design an attack in which malicious behavior is implanted into a model but stays dormant until pruning removes the parameters that were suppressing it, giving the adversary a passive role after model release.
2. Extensive empirical validation. The attack is evaluated on five instruction-tuned models, three attack scenarios (jailbreak, over refusal, and content injection), and three pruning algorithms, with attack success rates exceeding 90% across all three algorithms in vLLM.
3. A detailed analysis of the attack's mechanics. The paper includes an ablation on the size of the repair set, an empirical study of how accurately the adversary can pre-estimate which parameters will be pruned, and an analysis of the attack's survival under further fine-tuning and quantization.
4. A defense discussion. The authors examine security-aware calibration and parameter patching as potential mitigations, showing that neither reliably prevents the attack, and identify directions for future work on secure model compression.

Main Findings

• Pruning can be a hidden trigger. An adversary can craft a model whose malicious behavior appears only after pruning, exploiting the fact that pruning is a user-controlled, deployment-time operation.

• High attack success rates after pruning. Reported success rates reach up to 95.7% for jailbreak, 98.7% for benign instruction refusal, and 99.5% for targeted content injection (abstract figures). The introduction summarizes the result as exceeding 90% across all three pruning algorithms, and Section 5.3 describes post-pruning jailbreak ASR as reaching up to 96%.

• The unpruned model looks benign — sometimes safer than the base model. Before pruning, attacked models show ASR comparable to base models. For jailbreak, unpruned attacked models can even appear safer than their bases (e.g., a change of −5.3% with Llama), which could encourage adoption. Benign refusal rates stay close to the base models (e.g., Qwen2.5-7B: 0.4 base vs. 1.2 on the attacked jailbreak model; Llama3.1-8B: 0.5 vs. 3.9; OLMo-2-7B: 2.5 vs. 2.1).

• Utility is preserved before pruning. Across five benchmarks (MMLU, ARC-Challenge, HellaSwag, HumanEval, GSM8K), attacked and unpruned models show changes such as +0.1 ± 1.5 (Qwen2.5-7B, over refusal), −2.9 ± 5.2 (Qwen2.5-7B, jailbreak), +1.2 ± 3.4 (Llama3.1-8B, over refusal), and −1.2 ± 1.3 (OLMo-2-7B, content injection) — no significant degradation relative to the pre-attack version.

• Sensitivity to pruning configuration varies. SparseGPT 20% with Llama3.1-8B yields the lowest jailbreak ASR in the main table (22.0, base 6.0), while the same model under Wanda reaches 93.3 (base 6.7) and under Magnitude 20% reaches 92.3 (base 9.0). Magnitude 20% reaches 95.7 for Qwen2.5-7B (base 8.0), 94.3 for OLMo-2-7B (base 2.7), and 94.3 for Llama3.1-8B in content injection (base 0.0).

• The repair set is small but decisive. Injection uses α_inj = 50% of parameters; repair uses α_rep = 5% for over refusal and jailbreak and α_rep = 1% for content injection. For jailbreak, 1% is generally enough to restore a low unspruned ASR, while 10% noticeably lowers post-pruning ASR. For over refusal, at least 5% is needed. For content injection, raising the repair ratio lowers both pre- and post-pruning ASR, giving an adversary-controlled trade-off.

• Pre-estimation is accurate. Using Wanda scores computed on the base model with a C4 calibration set, the adversary's predictions transfer to other algorithms and calibration sets (WikiText): more than 99% of repaired parameters are pruned in most cases, despite SparseGPT's weaker score correlation and its iterative compensation procedure.

• Security-aware calibration is an incomplete defense. Replacing the calibration set with 512 jailbreaking/refusing samples sharply reduces post-pruning ASR for SparseGPT (e.g., Llama3.1-8B: 36.0 → 0.1; OLMo-2-7B: 89.3 → 1.0; Qwen2.5-7B: 86.7 → 3.7) but barely affects Wanda (e.g., 92.3 → 93.3 for Llama). It also costs utility (Δ = −3.6% for SparseGPT vs. Δ = −0.9% for Wanda).

• Patching only works with oracle knowledge. Re-inserting the exact repaired parameters into the pruned model (i.e., 45% pruning) drops ASR substantially (e.g., Qwen2.5-7B Wanda 50%: 93.0 → 14.0). The practical variant — identifying the bottom 5% using the post-attack score — does not produce a comparable reduction (Qwen2.5-7B Wanda 50%: 75.3).

• The attack survives further transformations. With FP8 or GPTQ (4-bit) quantization applied on top of pruning, ASR stays between 85% and 128% of the full-precision attacked-pruned model. Sparse fine-tuning on general data alone does not mitigate the attack, but mixing in security data at a 1:1 ratio reduces jailbreak ASR to near zero after only five gradient steps.

Methodology in Plain English

The attack rests on a simple asymmetry: pruning removes a known, predictable subset of weights, and that predictability can be turned against the user.

1. Pre-estimate which weights will be pruned. The adversary computes Wanda scores (weight magnitude times activation norm) on a base model using a calibration set, then splits parameters into the top α_inj percent (unlikely to be pruned) and the bottom α_rep percent (likely to be pruned). The authors rely on the observation, confirmed empirically, that scoring metrics across Magnitude, Wanda, and SparseGPT correlate strongly — so one proxy score can target all three.

2. Inject the malicious behavior into the "survivor" weights. The adversary fine-tunes only the parameters unlikely to be pruned on a harmful dataset, freezing everything else. A general-purpose instruction-tuning set (GPT-4-LLM) plus a KL-divergence penalty between the base and attacked model outputs keeps general utility intact.

3. Repair the model using the "doomed" weights. The adversary fine-tunes only the small fraction of parameters likely to be pruned, this time on harmless data, canceling the injected behavior. Because these are the weights the user will delete, the cancellation disappears at pruning time and the malicious behavior re-emerges.

The repair set is deliberately made brittle: a very small α_rep (1% for content injection) means the suppression is concentrated in weights that pruning almost certainly removes. Evaluation covers seven pruning configurations (Wanda 20%, 50%, 2:4; SparseGPT 20%, 50%, 2:4; Magnitude 20%), with 512 C4 samples for pre-estimation and 512 WikiText samples for evaluation. Attack success is judged by GPT-4.1-mini — harmfulness on a 5-point scale with scores ≥ 4 counted as success for jailbreak, and refusal-with-reason or target-string presence for the other scenarios.

Why This Matters

Impact on research. Prior work established that quantization and fine-tuning can serve as attack triggers. This paper extends that family to model compression by pruning, showing that a deployment-time operation users perform for efficiency can be the thing that unleashes the attacker's payload. It reframes pruning from a pure utility-compression trade-off into a security-relevant transformation, and shows that the standard "prune and serve" workflow has a gap that current methods do not close.

Real-world applications and scenarios:

• Model hubs and downloadable checkpoints. A malicious or compromised uploader publishes a checkpoint that passes normal safety evaluations and benchmark comparisons, only to turn harmful when a downstream user prunes it locally.
• Edge and on-device deployment. Users pruning models to fit constrained hardware activate the payload without ever knowing the unpruned model was safe — the threat model is specifically aimed at resource-constrained local deployment.
• Enterprise inference pipelines. Organizations standardizing on vLLM with pruning defaults (Wanda, SparseGPT, or Magnitude at 50% sparsity) would trigger the attack through routine configuration rather than any suspicious action.
• Safety evaluation and auditing. Standard pre-release safety testing on the unpruned checkpoint gives no signal, so the paper argues that red-teaming and evaluation must include the pruned model, not just the published artifact.

Industry relevance. The attack targets vLLM, described in the paper as one of the most widely used LLM inference engines, with over 50k GitHub stars and rapid integration of new models. Because the three targeted algorithms ship as built-in vLLM options and 50% sparsity is a common target in prior unstructured pruning work, the barrier to triggering the attack is essentially zero configuration effort for the user.

Future Directions

• Secure pruning methods and community standards. The paper explicitly calls for further research into secure model compression and for rigorous community standards for evaluating the security of pruned models — including how to audit models after compression rather than before.
• Better calibration pipelines. Security-aware calibration substantially reduced ASR for SparseGPT but not Wanda, and at a utility cost. Designing calibration procedures that reliably suppress such attacks without degrading quality is left as an open question.
• Detecting the repaired parameters. The patching analysis shows the attack is neutralized when the pre-attack pruning scores are known, but not when only the post-attack model is available. Methods for identifying compromised parameters from a released checkpoint remain unresolved.
• Which behaviors are easy or hard to hide. The authors note that different malicious behaviors have different injection and retention difficulty (refusal re-learning is easy; generating a specific target word is harder), and call for a systematic study of this across behavior types.
• Generalizing beyond pruning. The paper advocates broader awareness of risks from post-training transformations as a family, and further research into systematic methods for checking, detecting, and mitigating them.

Target Audience

This paper is most useful to LLM security researchers and red-teamers working on backdoors and post-training transformation attacks; ML engineers and platform teams who deploy pruned or compressed models through inference engines such as vLLM; model hub operators and safety evaluators responsible for deciding what counts as a "safe" checkpoint; and researchers working on secure model compression, quantization, and sparsity. It also serves as a case study for anyone studying the supply chain between model publishing and deployment-time optimization.

Authors’ abstract

Model pruning, i.e., removing a subset of model weights, has become a prominent approach to reducing the memory footprint of large language models (LLMs) during inference. Notably, popular inference engines, such as vLLM, enable users to conveniently prune downloaded models before they are deployed. While the utility and efficiency of pruning methods have improved significantly, the security implications of pruning remain underexplored. In this work, for the first time, we show that modern LLM pruning methods can be maliciously exploited. In particular, an adversary can construct a model that appears benign yet, once pruned, exhibits malicious behaviors. Our method is based on the idea that the adversary can compute a proxy metric that estimates how likely each parameter is to be pruned. With this information, the adversary can first inject a malicious behavior into those parameters that are unlikely to be pruned. Then, they can repair the model by using parameters that are likely to be pruned, effectively canceling out the injected behavior in the unpruned model. We demonstrate the severity of our attack through extensive evaluation on five models; after any of the pruning in vLLM are applied (Magnitude, Wanda, and SparseGPT), it consistently exhibits strong malicious behaviors in a diverse set of attack scenarios (success rates of up to $95.7\%$ for jailbreak, $98.7\%$ for benign instruction refusal, and $99.5\%$ for targeted content injection). Our results reveal a critical deployment-time security gap and underscore the urgent need for stronger security awareness in model compression.