Harnessing Textual Refusal Directions for Multimodal Safety

AI summary

Overview

Research area: AI safety and alignment for Multimodal Large Language Models (MLLMs) — specifically inference-time activation steering using refusal directions borrowed from the language backbone.

Technical level: Advanced. The paper assumes familiarity with transformer internals, hidden-state activations, linear probing, and activation steering (e.g., Arditi et al.-style refusal directions).

Scope: The paper tests whether refusal directions extracted from text-only data transfer to image and video inputs, diagnoses why naive transfer causes over-refusal, and introduces MARS, a training-free method that injects multimodal safety without any multimodal safety data.

What This Paper Is About

MLLMs inherit their safety from a language backbone that was aligned on text, because collecting unsafe multimodal data is expensive, must be redone for every new modality, and raises ethical and legal concerns (the paper cites nonconsensual and child-abuse imagery). The goal is to obtain multimodal safety without collecting or training on unsafe multimodal data. The authors ask whether a "refusal direction" — a vector in the model's activation space that separates refusing from accepting responses, computed purely from text — still works when the input is an image or a video, and how to make that transfer reliable.

Key Contributions

1. Refusal amplification as a safety mechanism. The paper revisits refusal directions, which prior work used to suppress refusal and thereby jailbreak models, and instead uses them to reinforce refusal, with no multimodal safety data and no retraining.

2. First analysis of textual refusal directions in MLLMs. The authors show the directions transfer across modalities but identify three conditions on that transfer: which layer is steered, how strong the steering is, and a previously unreported failure mode they call modality misalignment.

3. The MARS method. Modality-Agnostic Refusal Steering combines activation re-centering with randomly colored images, a ReLU gate that restricts steering to inputs estimated as unsafe, an adaptive steering strength derived from a geometric "trust region," and a training-free layer-selection score.

4. Broad empirical evaluation. Five state-of-the-art MLLMs and seven benchmarks spanning images, video, and text, with comparisons against training-free (AdaSteer, ECSO) and training-based (SASA, AutoSteer) baselines.

Main Findings

• Textual refusal directions do transfer to images. Steering with a direction computed from Alpaca (safe) and MaliciousInstruct (unsafe) text produced higher refusal on unsafe than on safe ViSU images in the vast majority of layers, at fixed α = 1.0. Unsafe refusal peaked at specific layers: 31 on Gemma3, 16 on Qwen3-VL, 19 on InternVL 3.5.

• Steering strength trades safety against over-refusal. Sweeping α up to 3.0 consistently raised refusal on unsafe inputs but reduced acceptance on safe ones and degraded utility; Gemma3 already lost safety at α = 2.0. Some layers over-refused indiscriminately — layer 10 of Gemma3 rejected 96.4% of safe queries.

• Modality misalignment is the deeper failure mode. In a 2D PCA of Gemma3's activation space, both safe and unsafe image activations projected strongly onto the refusal direction regardless of semantics; the <assistant> token is dominated by a modality-specific component. A separation score (difference in mean cosine similarity to the refusal direction between unsafe and safe activations) stayed near zero in raw space across Qwen3-VL, Gemma3, and InternVL 3.5, with Gemma3 the most misaligned.

• Centering with random-colored images fixes the misalignment. Subtracting the mean activation of neutral, randomly colored images raised the separation score across layers and models and realigned unsafe images with the reject cluster and safe images with the accept cluster, without any safety-related data.

• Large safety gains on image benchmarks. On ViSU, zero-shot unsafe refusal ranged from 0.2% (Molmo2) to 22.9% (Qwen3.5); MARS added +25.1% (Gemma3) to +67.9% (Molmo2). Against the training-based SASA, MARS gained +58.4% refusal with −0.3% over-refusal on Qwen3-VL and +3.9% refusal with −7.9% over-refusal on Molmo2.

• Recovery from image jailbreaks. On HADES, zero-shot refusal was 16.7% for InternVL 3.5 and 17.9% for Gemma3 (safety score 37.4% for Gemma3); MARS restored refusal across all models, up to +78.6% on Molmo2.

• Safety under cross-modal interactions. On SIUO and the SSU split of HoliSafe, where image and text are individually safe but jointly unsafe, MARS produced gains up to +58% for Molmo2 on SIUO, outperforming all training-free baselines and beating AutoSteer on Molmo2, InternVL 3.5, Qwen3-VL, and Qwen3.5.

• Utility mostly preserved, with one clear exception. MARS matched or nearly matched zero-shot accuracy on MMMU and MMMUPro for Gemma3, Molmo2, InternVL 3.5, and Qwen3-VL, but degraded Qwen3.5 (38.2 vs 50.8 on MMMU; 24.5 vs 35.6 on MMMUPro). SASA degraded utility substantially (e.g., −10% on InternVL 3.5, −20% on Qwen3.5).

• Strong video jailbreak results. On VideoSafetyBench, zero-shot refusal ranged from 4.7% (InternVL 3.5) to 28.8% (Molmo2 and Qwen3-VL). MARS added +59.4% on Qwen3-VL (88.2 absolute, versus 40.3 for SASA), +52% on Qwen3.5, and +39% on Molmo2; the InternVL 3.5 gain was modest in absolute terms (11.9% versus 4.7%) but a ×2.5 relative improvement.

• Every component matters (ablation). Removing layer selection degraded utility by 18.5% on InternVL 3.5 and 9.2% on Gemma3 and collapsed unsafe refusal on Molmo2 and Gemma3 to roughly 0%. Removing centering caused 99.2% over-refusal on Molmo2. Removing the ReLU gate harmed utility on InternVL 3.5 and Gemma3.

Methodology in Plain English

The authors treat safety as a direction in the model's internal activation space. Using a safe text set (Alpaca) and an unsafe text set (MaliciousInstruct), they record the model's internal state at the position of the <assistant> token, average the states for refused versus accepted responses at each layer, and subtract the two averages. The resulting vector is the "refusal direction."

Naively adding this vector back into the activations at inference time raises refusal but also fires on safe images, because images inject a strong visual component that happens to point in the same direction. The authors' fix is to estimate that visual component with a set of randomly colored images — pictures with visual structure but no semantic meaning — and subtract their average activation. This re-centering is done with no safety data and no training.

On top of that centered space, MARS applies three rules at inference time:

1. ReLU gating — only project onto the refusal direction when the projection is positive, so inputs the model already reads as safe receive a zero update. No separate classifier is needed.
2. Adaptive strength — compute the distance from the current activation to the nearer of the two text-derived centroids (accept or reject), treat that as a "trust region," and scale the update so it lands inside that region.
3. Layer selection — score every layer with three training-free criteria (does the direction agree with one computed on held-out validation text; how well does nearest-centroid classification separate validation text; how confident is the margin for safe text), normalize and sum them, and steer at the best layer.

Steering is applied only at the first generated token, which the paper argues largely determines the rest of the generation. The only data needed is text plus a validation set of 100 samples per class from HarmBench.

Why This Matters

The paper reframes multimodal safety as a problem of reusing existing structure rather than collecting new data, which sidesteps a genuine ethical and legal bottleneck: unsafe multimodal datasets are costly to assemble, must be rebuilt per modality, and are risky to distribute. The result that a text-derived direction transfers to video suggests safety-relevant structure is shared across modalities, which is a strong signal for alignment research more broadly.

Potential applications:

• Guarding vision-language assistants that caption or describe user-uploaded images, without curating a harmful-image training corpus.
• Screening user-submitted video in moderation pipelines for models where no video safety training data exists.
• Catching cross-modal harms (individually benign image plus individually benign text) that single-modality filters miss.
• Deploying lightweight safety patches to already-released open-weight MLLMs as an inference-time wrapper, rather than a full fine-tuning cycle.

Industry relevance: The method is training-free, needs no labelled multimodal safety data, and adds only a first-token intervention, which makes it a practical compliance layer for teams shipping vision- or video-enabled assistants. The paper does not report wall-clock latency or memory overhead, so deployment cost is not quantified here.

Future Directions

• Handling over-refusal more principledly. The Qwen3.5 results (utility dropped from 50.8 to 38.2 on MMMU) show the trust-region heuristic does not always protect utility; a better-calibrated strength rule is an open problem.
• Extending to further modalities. The paper tests images and video; audio, 3D, and other encoders are not evaluated, and it is unstated whether the neutral-image centering trick has a natural analogue for them.
• Combining with training-based methods. MARS is competitive with, but not uniformly better than, trained baselines such as SASA and AutoSteer; whether steering and trained probes are complementary is not reported.
• Robustness to adaptive adversaries. The paper evaluates fixed benchmarks (ViSU, HADES, SIUO, HoliSafe, VideoSafetyBench, MMSafetyBench); attacks crafted with knowledge of the steering layer and direction are not explored.

Target Audience

Researchers and engineers working on MLLM safety and alignment, interpretability researchers interested in activation steering and cross-modal representation sharing, and practitioners who need to add safety to open-weight vision or video models where unsafe multimodal training data is unavailable or legally restricted. Readers without background in activation-space methods will need to consult the cited steering literature (Arditi et al., Zhao et al.) first.

Authors’ abstract

Safety in Multimodal Large Language Models (MLLMs) depends on unsafe multimodal data for post-training alignment or activation steering. However, such data is hard to collect, must be gathered anew for each modality, and its collection and distribution raise ethical and legal concerns. In this work, we relax this constraint and investigate whether textual refusal directions, extracted directly from the LLM backbone, generalize across modalities (i.e., image, video). Preliminary findings confirm this ability, though effectiveness is conditioned by layer selection, steering strength, and cross-modal alignment, with the latter causing safe multimodal inputs to be spuriously steered toward refusal. Building on this, we introduce Modality-Agnostic Refusal Steering (MARS), a lightweight training-free approach that injects multimodal safety without the need for multimodal safety data. MARS corrects modality misalignment via activation centering, adaptively scales steering strength within a geometrically defined trust region, and selects the optimal intervention layer, operating at the first generated token. Evaluated on five SOTA MLLMs across safety, utility, and video jailbreak, MARS achieves consistent safety gains while preserving utility up to over-refusal. MARS reveals that safety-relevant structure is shared across modalities and that textual refusal directions are a powerful and underexplored foundation for multimodal alignment.