Depth-Dependent Safety Geometry in Vision-Language Models: Layer-Wise Localization and Bidirectional Refusal Control

AI summary

Overview

Research area: AI safety and mechanistic interpretability for Vision-Language Models (VLMs), sitting at the intersection of computer vision, multimodal security, and representation engineering.

Technical level: Advanced. The paper assumes familiarity with transformer internals, hidden-state activations, cosine/angular geometry, and causal intervention methods such as activation steering and magnitude scaling.

Scope: A white-box, inference-time study that localizes safety-sensitive representations across the depth of two frozen 4B-parameter VLMs (Qwen3-VL-4B and InternVL2.5-4B) and tests, through controlled activation interventions, whether those representations causally control refusal behavior.

What This Paper Is About

VLMs are aligned to refuse harmful requests, but it is unclear where safety-relevant information lives inside the network, which internal components actually cause refusal, and how those mechanisms can be perturbed. The authors build a lightweight framework that first finds layers where safe and unsafe inputs look geometrically different, then intervenes on those layers to test whether the difference actually drives refusal. The goal is to separate representations that merely distinguish safe from unsafe inputs from those that control refusal, and to map that control across network depth.

Key Contributions

1. An inference-time framework for localizing and probing safety-sensitive representations in frozen VLMs, requiring no retraining and no modification of model parameters.
2. On Qwen3-VL-4B, a demonstration that early vision layers (7–10) support strong bidirectional refusal control via directional steering, while later vision layers (18–22) are more responsive to activation magnitude scaling than to directional steering.
3. Identification of a white-box, representation-level refusal-bypass attack surface, together with an explicit safety–selectivity trade-off in which magnitude scaling raises unsafe rejection while also raising false rejection on difficult safe inputs.
4. Cross-model representational evidence: InternVL2.5-4B shows qualitatively similar depth-dependent activation geometry, though at different absolute layer locations and magnitudes.

Main Findings

• Safety geometry is non-uniform across depth. In Qwen3-VL-4B, language-tower divergence is strongest in early layers and generally decreases with depth, with SU–UU excess separation remaining larger than SU–SS across most of the network. The vision encoder is more heterogeneous, with distinct regions around Layers 7–10 and 18–22.

• Early vision layers allow bidirectional refusal control. Under global steering at Layers 7–10, unsafe rejection ranged from 6% (α = 4.0) to 96% (α = −4.0) against an 84% baseline, i.e., refusal can be suppressed or amplified along the same learned direction.

• Control is distributed across visual tokens, not concentrated in the CLS token. Focused steering on the CLS token plus the first 10 patch tokens produced much smaller behavioral changes than global steering over all visual tokens.

• Later layers are steerable in geometry but not in behavior. At Layer 20, projecting CLS activations onto the learned safety direction gave clearly separated safe and unsafe distributions, yet steering that same direction at Layers 18–20 left unsafe rejection near baseline.

• Late layers respond to magnitude instead. At Layers 18–22, α = 1.15 raised unsafe rejection to 96%, while safe rejection simultaneously rose to 91% on UnsafeBench, revealing a substantial safety–selectivity trade-off; on Google Conceptual Captions the same setting gave 8–9% safe rejection.

• Early layers show asymmetric magnitude sensitivity. At Layers 7–10, α = 0.85 produced 100% unsafe rejection but also 100% safe rejection on GCC — near-total refusal rather than selective safety — while amplification produced little change from the 72% baseline.

• The language tower is also magnitude-sensitive. Raising α to 1.2 pushed unsafe rejection to 98%; lowering it to 0.85 dropped it to 37%.

• MMLU utility was unchanged in the tested setting. On 100 MMLU samples, scaling produced no change from the 44% baseline accuracy, though the authors caution that unchanged MMLU does not imply preserved safety selectivity given elevated safe rejection elsewhere.

• Representational separability is not causal controllability. The central conceptual result: a layer can encode a clean safety direction without that direction governing behavior when manipulated.

• Cross-model validation is representational only. InternVL2.5-4B reproduces the qualitative depth pattern (stronger early language divergence declining with depth; vision SU–UU transitioning from negative toward positive), but causal interventions were run only on Qwen3-VL-4B.

Methodology in Plain English

The approach has three stages, all applied to a frozen model whose weights never change.

First, the authors collect 100 safe and 100 unsafe examples and record the model's internal activations at every layer of both the vision encoder and the language model. They measure the angle between activation vectors for pairs of inputs: safe–safe, unsafe–unsafe, and safe–unsafe. Layers where safe–unsafe pairs are consistently farther apart in angle than within-condition pairs are flagged as candidate safety-sensitive regions. This is repeated across several random pairings to stabilize the estimates.

Second, they test magnitude sensitivity. Using forward hooks, they multiply the activations at a chosen layer by a scalar (0.85, 0.9, 1.0, 1.15, or 1.2) uniformly across all visual tokens, then measure how often the model refuses unsafe inputs and how often it refuses safe ones.

Third, they test directional sensitivity. They compute a "safety direction" as the difference between the mean unsafe activation and the mean safe activation at a layer, normalize it, project each token's activation onto it, and add a scaled copy of that projection back into the activation (steering strengths of 0, 0.5, 1, 2, 4, −2, −4). They compare steering only the CLS token and first 10 patch tokens against steering every visual token.

Behavior is scored with two rates: Unsafe Rejection Rate (rejecting harmful inputs, the desired behavior) and Safe Rejection Rate (rejecting benign inputs, an unwanted side effect). Harmful and "hard" safe examples come from UnsafeBench, "clean" safe examples from Google Conceptual Captions, and general reasoning utility from MMLU. The authors frame these hook-based interventions explicitly as a white-box analysis tool, not an input-level attack; constructing an actual adversarial image or prompt that reproduces the same internal shifts is stated to be out of scope.

Why This Matters

The paper reframes VLM safety from "which layers look different for harmful inputs" to "which layers actually control refusal, and through what mechanism." It shows that geometric separation — a common proxy in interpretability work — can overstate how much leverage an internal direction provides. That distinction matters for anyone building defenses or attacks on multimodal models, and it identifies a concrete internal surface that inference-time stabilization would need to protect.

Real-world applications:

• Content moderation pipelines for platforms that screen user-uploaded images paired with text, where knowing which internal layers govern refusal informs where to monitor or constrain a deployed model.
• Security auditing and red-teaming tooling for multimodal systems, giving auditors a cheap, retraining-free way to map a model's safety-sensitive depth before deployment.
• Hardening of interactive vision-language agents (browsing, robotics, assistant tools) against adversarial images that might otherwise shift internal states.
• Model selection and procurement decisions, where safety–selectivity trade-offs under perturbation can be quantified per architecture rather than assumed uniform.

Industry relevance: The interventions require only forward hooks and no weight updates, making them practical for auditing proprietary or frozen checkpoints. The safety–selectivity finding is directly relevant to deployment teams, since a defense that inflates refusal rates on hard safe inputs degrades user experience while appearing safer on harmful-input benchmarks.

Future Directions

• Extending causal interventions (steering and scaling) to InternVL2.5-4B and other architectures, and quantifying cross-model agreement rather than reporting qualitative similarity.
• Converting the representation-level attack surface into input-level attacks — demonstrating that an adversarial image or prompt can realize the same activation shifts the hooks induce.
• Characterizing safety directions in higher-dimensional subspaces rather than a single mean-difference direction, and testing direction stability on held-out splits and additional safety categories.
• Developing and evaluating inference-time defenses that stabilize safety-sensitive representations, given that MMLU remained unchanged at 44% while safe rejection rose under several settings.

Target Audience

AI safety and interpretability researchers working on multimodal models; red-team and security engineers assessing deployed VLMs; and machine learning engineers who need a concrete, low-cost method for auditing where safety behavior lives inside a frozen model. Readers without a background in activation analysis or linear algebra will find the methodology section demanding, though the framing of separability versus controllability is broadly accessible.

Authors’ abstract

Vision-Language Models (VLMs) are widely deployed, yet the internal mechanisms behind their safety behavior remain poorly understood. We present a lightweight inference-time framework for localizing and manipulating safety-sensitive representations in frozen VLMs, without retraining or modifying parameters. On Qwen3-VL-4B, layer-wise activation divergence analysis reveals non-uniform safety geometry across depth. Causal interventions show that early vision layers support strong bidirectional refusal control through activation steering along a learned safety direction, with unsafe rejection spanning 6%-96%, whereas later layers respond more to activation magnitude scaling and exhibit a substantial safety-selectivity trade-off. Deeper layers remain well separated along the safety direction yet are largely insensitive to steering, distinguishing representational separability from causal controllability. InternVL2.5-4B shows qualitatively similar depth-dependent activation geometry despite differences in layer location and magnitude. Our results expose a white-box, representation-level refusal-bypass attack surface in VLM vision encoders and motivate inference-time stabilization of safety-sensitive representations.