How Fragile Is On-Device Language Model Safety? Localizing Safety-Critical Parameters for Sparse Fault Analysis

AI summary

Overview

• Research area: AI safety and model integrity — specifically, the localization of safety-critical parameters in small language models (SLMs) deployed on-device or as components of agentic systems.
• Technical level: Intermediate. The paper is written accessibly, but assumes familiarity with transformer architecture (attention and MLP projections), singular value decomposition, activation-aware importance scoring, and attack-success-rate (ASR) evaluation.
• Scope: The paper investigates whether safety-aligned behavior in LLaMA-2-7B-Chat is concentrated in a sparse, structured subset of parameters — and whether modifying that subset degrades safety while leaving general utility largely intact.

What This Paper Is About

Models deployed on phones, edge devices, and inside agentic systems store their weights locally, where those weights can be corrupted by hardware faults. The authors ask whether safety behavior depends on only a small, identifiable fraction of those weights, which would make targeted fault attacks — and targeted defenses — far more tractable than attacking a 7-billion-parameter model at random. Using LLaMA-2-7B-Chat and two localization methods, they find that safety sensitivity is highly non-uniform and concentrates in the MLP down_proj component.

Key Contributions

1. The paper analyzes safety localization at two complementary levels: a low-rank safety-associated subspace level and an individual parameter level.
2. It shows that safety-sensitive parameter support is strongly concentrated in specific architectural components, particularly down_proj, with a smaller contribution from the attention o_proj.
3. It demonstrates that modifying only 0.19% of model weights in down_proj yields 53% Basic ASR and 56% GCG ASR, while tinyBenchmarks accuracy remains at 51.6% against a 52.2% unmodified baseline.
4. It frames these localized regions as a reduced search space for future bit-level fault analysis and as motivation for selective integrity protection in on-device and agentic deployments.

Main Findings

• Safety sensitivity is highly non-uniform: Both localization approaches reveal non-uniform safety sensitivity across the network, with the MLP down_proj consistently emerging as the most prominent safety-sensitive component and o_proj providing a smaller contribution.
• Sparse parameter-level intervention is effective: Restricting to down_proj alone modifies only 0.19% of the full model's weights while producing 53% Basic ASR and 56% GCG ASR, with tinyBenchmarks accuracy at 51.6%.
• Combining down_proj and o_proj is stronger at a slightly larger budget: down_proj + o_proj covers 0.23% of weights and reaches 58% Basic ASR and 60% GCG ASR, with utility at 51.3%.
• o_proj alone is weak: Pruning o_proj alone touches 0.04% of weights and yields only 20% Basic ASR and 14% GCG ASR, with utility at 52.0%.
• Broader interventions cost more utility: Intervening on all modules at 2.95% of weights gives 91% Basic ASR and 97% GCG ASR but drops tinyBenchmarks to 47.1%; restricting to MLP projections at 1.87% gives 86% Basic ASR, 94% GCG ASR, and 49.7% utility.
• Low-rank interventions concentrate at higher magnitudes: In the low-rank setting, retaining only the top 10% of the intervention preserves substantial safety degradation, while effectiveness drops sharply at 1%. All modules at top fraction 1.0 give 87% Basic ASR / 98% GCG ASR / 49.3% utility; at 0.1, 71% / 85% / 50.1%; at 0.01, 36% / 25% / 50.7%; at 0.001, 20% / 18% / 51.8%.
• Architectural restriction retains much of the effect: In the low-rank setting, MLP-only at top fraction 1.0 gives 77% Basic ASR / 89% GCG ASR / 49.8% utility, and MLP-only at 0.1 gives 54% / 64% / 51.1%. down_proj alone at 1.0 gives 51% Basic ASR / 43% GCG ASR / 51.2% utility, and at 0.1 gives 32% / 26% / 51.4%.
• The safety-relevant parameter space is much smaller than the full model: The authors conclude that safety-relevant behavior depends disproportionately on a small, structured portion of the parameter space.
• Parameter sparsity is not the same as bit-level vulnerability: The paper explicitly notes that 0.19% of a 7B-parameter model still amounts to millions of weights, whereas a practical hardware fault attack would ideally need only a small number of physically realizable bit modifications. Bit-level effects are also expected to depend on numerical representation, memory architecture, and fault mechanism.

Methodology in Plain English

The authors start from the intuition that if safety behavior lives in a small part of the model, an attacker (or defender) only needs to look at that small part. They use LLaMA-2-7B-Chat and build two calibration datasets: a safety set made of harmful instructions from AdvBench paired with the model's own refusal responses, and a utility set made of Alpaca-Cleaned benign instruction–response pairs with safety-related examples removed. Only response-token activations are used.

Approach 1 — Low-rank subspace localization. For each projection module, the researchers collect response-token activations from the safety set and the utility set separately, run the activations through the weight matrix, and compute truncated singular value decompositions of the two output sets. From the right singular vectors they build output-space projectors for safety and for utility. They then isolate the safety-associated weight component that is orthogonal to the utility subspace, and subtract it from the weight matrix. To see whether the effect survives on fewer parameters, they sparsify this subtraction by globally ranking entries by absolute magnitude and keeping only those above a threshold corresponding to a chosen retained fraction. They test this across all projection modules and under restrictions to particular component groups.

Approach 2 — Parameter-level safety–utility localization. Here the researchers score every individual weight in a projection module using an activation-aware importance criterion that multiplies the weight's absolute magnitude by the L2 norm of the activations of its corresponding input feature across calibration tokens. They compute these scores separately on the safety and utility data. The safety-specific set is defined as the safety-important weights minus the utility-important weights, so weights that matter for both safety and general instruction-following are removed. They then evaluate the effect by pruning those safety-specific weights to zero — a sparse mask.

Evaluation. Safety is measured by attack success rate on 100 held-out AdvBench evaluation instructions, with 420 AdvBench instructions reserved for attribution. Following Zou et al. (2023), a response counts as a successful attack if it lacks key refusal patterns. The paper reports Basic ASR on harmful prompts without an adversarial suffix and GCG ASR using GCG adversarial suffixes. General utility is measured with tinyBenchmarks; the unmodified model achieves 52.2% accuracy. All reported weight fractions are relative to the full model. The authors note that these experiments simulate parameter-level interventions rather than physical hardware faults.

Why This Matters

• For research: The paper reframes hardware-level fault threats to language models as a localization problem rather than a brute-force bit-search problem, and connects the safety–utility disentanglement literature (e.g., Arditi et al., 2024; Wei et al., 2024) to the bit-flip attack literature (e.g., Rakin et al., 2019; Coalson et al., 2024; Zahran et al., 2025). It supplies a reduced candidate surface for subsequent bit-level analysis, though it does not yet demonstrate an end-to-end hardware attack.
• A distinct threat profile for agents: The authors argue that in agentic deployments, safety behavior is invoked repeatedly and autonomously — at each planning step, tool call, and intermediate action — often without a human in the loop. A parameter-level fault, once induced, persists silently across every subsequent invocation, including ones the user never observes, so it does not need to be re-triggered per turn the way a prompt-based jailbreak does.

Real-world applications:

• On-device assistants and smartphone-hosted models: Local parameter storage is exposed to memory faults and hardware reliability issues in ways that cloud-hosted inference is not, and the paper cites prior work showing models in this scale range can be executed locally on resource-constrained devices including smartphones.
• Agentic systems built from SLMs: Where a small model is used for specialized or repetitive subtasks and API calling within an agent workflow, a corrupted weight can keep steering autonomous planning and tool-selection steps.
• Selective integrity protection: Because the safety-relevant region appears compact, protection mechanisms such as checksums or integrity verification could be targeted at safety-sensitive regions rather than the whole model — the paper's future work direction.
• Edge and privacy-preserving deployments: Settings motivated by lower latency, improved privacy, and reduced dependence on cloud connectivity, where parameters remain locally stored throughout repeated interactions.

Industry relevance: The findings suggest that manufacturers and deployers of on-device and edge AI cannot assume the full parameter set needs equal hardening. At the same time, the paper is candid that the utility evidence is limited (tinyBenchmarks is a compact estimate) and that the mapping from localized parameters to physically realizable bit flips under specific numerical formats, memory architectures, and fault mechanisms remains open — so the results motivate a research agenda rather than a deployable attack or defense.

Future Directions

1. Test whether the localized regions are enriched for high-impact bit-level faults — i.e., whether the safety-sensitive parameters identified here correspond to a similarly sparse set of physically realizable bit-level vulnerabilities.
2. Compare against random and magnitude-based baselines. The paper explicitly names this as future work; no such baselines are reported in the current experiments.
3. Map localized parameters to realistic fault mechanisms under specific numerical formats, memory architectures, and fault mechanisms — the paper studies simulated parameter-level interventions, not physical hardware faults.
4. Develop selective integrity protection for safety-sensitive regions in on-device and resource-constrained deployments, and test whether it can efficiently harden those regions.
5. Strengthen utility evaluation. The authors acknowledge that tinyBenchmarks provides only a compact estimate of general utility, leaving open whether more comprehensive evaluation would change the safety–utility trade-off picture.

Target Audience

This paper is most useful to:

• AI safety and security researchers working on model integrity, bit-flip and fault attacks, and the safety–utility localization literature.
• Edge and on-device ML engineers who deploy models on resource-constrained hardware and need to reason about which parameters carry the most safety risk.
• Agentic systems researchers and builders interested in how parameter-level integrity differs from prompt-level robustness in autonomous loops.
• Practitioners designing defenses such as integrity verification or selective hardening for locally stored model weights.
• Students and newcomers to the area who want a concretely quantified case study (0.19% of weights, 53% Basic ASR, 56% GCG ASR, 51.6% versus 52.2% tinyBenchmarks accuracy) of how concentrated safety behavior can be in a 7B-scale model.

Authors’ abstract

As small language models (SLMs) are increasingly deployed on resource-constrained and on-device platforms, including as components of agentic systems, the integrity of locally stored model parameters becomes an important safety concern. We investigate whether safety-sensitive behavior in LLaMA-2-7B-Chat is concentrated within a sparse subset of parameters, creating a reduced fault surface for targeted analysis. We study two complementary localization methods: low-rank safety-associated subspace analysis and parameter-level safety--utility importance filtering. Both approaches reveal highly non-uniform safety sensitivity across the network, with the MLP down_proj consistently emerging as a prominent safety-sensitive component and o_proj providing a smaller contribution. Using parameter-level localization, modifying only 0.19% of model weights in down_proj yields 53% Basic ASR and 56% GCG ASR, while tinyBenchmarks accuracy remains at 51.6% compared with a 52.2% unmodified baseline. These results motivate targeted fault analysis and selective integrity protection for language models deployed in resource-constrained, on-device, and agentic settings.