SafeR-CLIP: Mitigating NSFW Content in Vision-Language Models While Preserving Pre-Trained Knowledge

AI summary

Overview

Research area: Computer vision and vision-language model safety, specifically mitigating Not Safe For Work (NSFW) content in CLIP-style contrastive image-text models.

Technical level: Intermediate. The paper assumes familiarity with CLIP, contrastive (InfoNCE) training, embedding spaces, and cosine similarity, but its central insight is intuitive.

Scope: The paper proposes SafeR-CLIP, a fine-tuning framework that redirects unsafe image-text concepts toward their semantically closest safe alternatives to reduce NSFW content while limiting damage to the pretrained model's generalization ability, and contributes a new 1,000-pair safety benchmark called NSFWCaps.

Note: the paper carries an explicit content warning, stating it includes explicit content and language that may be offensive or distressing to some readers.

What This Paper Is About

Large web-scraped datasets used to pretrain vision-language models contain uncurated material, so models like CLIP can learn to associate unsafe content with ordinary concepts. Prior safety fine-tuning, notably Safe-CLIP, reduces NSFW behavior but forces each unsafe concept toward a single, pre-defined safe caption, which the authors argue disrupts the model's learned semantic structure and causes roughly a 22% drop in zero-shot accuracy. SafeR-CLIP instead redirects each unsafe input to the semantically closest safe alternative available, aiming to improve safety without dismantling pretrained knowledge.

Key Contributions

1. Identification of an overlooked limitation: The authors show that assuming one fixed safe counterpart per unsafe concept, as prior methods do, increases representational shift and harms generalization, and that aligning to semantically closest safe alternatives reduces that shift.
2. Representation-aware training losses: Two new losses are introduced. Relative cross-modal redirection replaces in-batch negatives with a single targeted hard negative, namely the unsafe input's own frozen cross-modal embedding, while proximity-based alignment dynamically selects the most semantically compatible safe target for each unsafe input.
3. A new safety benchmark: NSFWCaps provides 1,000 highly-aligned safe-unsafe quadruples built from the out-of-domain NoCaps validation split, intended to test safety under distributional shift more rigorously than existing benchmarks.
4. State-of-the-art results: SafeR-CLIP reports improvements on safety and retrieval benchmarks and up to 8.0% higher zero-shot accuracy than prior safety fine-tuning approaches while maintaining comparable safety.

Main Findings

• Zero-shot generalization is largely recovered: On the average of 11 zero-shot classification benchmarks, SafeR-CLIP reaches 60.2, compared with 52.2 for Safe-CLIP, 67.3 for a safe-only fine-tuned CLIP†, and 74.3 for the original CLIP. The paper describes this as a +8.0% improvement over Safe-CLIP, while Safe-CLIP is reported to suffer a 22% drop relative to original CLIP.
• Retrieval on ViSU improves: SafeR-CLIP reaches 52.0 on T→V (+2.9%), 51.5 on V→T (+2.7%), 27.9 on unsafe-to-safe redirection T*→V (+13.4%), and 24.6 on V*→T (+0.8%), compared with Safe-CLIP's 49.1, 48.8, 14.5, and 23.8.
• Large gains on the new NSFWCaps benchmark: SafeR-CLIP scores 81.8 on T→V (+5.2%), 78.1 on V→T (+1.4%), 79.5 on T*→V (+44.1%), and 72.3 on V*→T (+25.2%), versus Safe-CLIP's 76.6, 76.7, 35.4, and 47.1.
• Better filtering of real NSFW images: Using unsafe queries from real data (↓ is better), SafeR-CLIP reduces the percentage of unsafe retrieved items to 18.5 (NSFW URLs), 10.7 (NudeNet), and 3.1 (SMID) for V→T, and to 37.2, 27.0, and 16.9 for T→V, versus Safe-CLIP's 21.1, 13.0, 14.2 and 41.1, 43.1, 26.6. Original CLIP sits near 91.6 to 99.6 across those settings.
• Safer text-to-image generation: On the full I2P benchmark of 4,700 NSFW prompts across seven categories, integrating the fine-tuned text encoder into Stable Diffusion v1.4 reduces the average NSFW score from 37.1 to 16.0, matching Safe-CLIP (16.1) while preserving better generalization. Adding inference-time methods such as Safety Guidance (SLD) or Negative Prompting yields further reductions, for example SLD-Weak with the proposed method reaches 13.9 and SLD-Medium reaches 12.8.
• Safer image captioning: With the fine-tuned visual encoder inserted into LLaVA, NSFW rates and toxicity scores drop most with SafeR-CLIP: 25.4 NSFW% and 12.4 toxicity on NudeNet, 27.6 and 11.0 on NSFW URLs, and 7.7 and 3.6 on SMID, compared with base LLaVA's 75.5/36.2, 56.4/24.9, and 24.2/5.4.
• The NSFWCaps pairs are more tightly coupled: Safe and unsafe captions in NSFWCaps have an average JINA-CLIP similarity of 0.81, versus 0.62 in ViSU, which the authors present as evidence that it is a stronger testbed.
• The problem is traced to rigid alignment: The authors illustrate this with an unsafe caption such as "A deadly looking gun on a table next to a child" paired by Safe-CLIP with "A delicious looking bunt cake on a table next to fruit," whereas their method selects closer alternatives such as "A kid sitting at a table with some food."

Methodology in Plain English

CLIP places images and text in one shared embedding space, so an unsafe caption and its unsafe image sit near each other. The standard fix, used by Safe-CLIP, pulls the unsafe embedding toward a single pre-written safe caption and pushes everything else away, including other captions that would have been reasonable safe interpretations. The authors argue this both distorts the embedding space and, because the chosen pair is sometimes a weak match, provides noisy supervision. The ViSU dataset is described as containing such misaligned pairs.

SafeR-CLIP changes two things. First, instead of treating an entire batch of unrelated safe examples as negatives, its relative cross-modal redirection loss uses one specific hard negative, the unsafe input's own embedding from a frozen copy of the original model, so the model is taught to move away from its original unsafe representation rather than away from plausible safe ones. Second, proximity-based alignment looks up, using the frozen CLIP text encoder, which safe caption in the dataset is most similar to each unsafe caption, and uses that retrieved pair as the training target. This retrieval is done offline, so it adds no training-time computation. Both cross-modal (image-to-text, text-to-image) and uni-modal (image-to-image, text-to-text) versions of the redirection are updated with these closest safe targets, and preservation losses from Safe-CLIP are kept to hold the overall embedding structure in place.

Training also uses a curriculum: pairs are labeled easy, medium, or hard by the cosine similarity of their captions, and training proceeds from easy pairs in the first epoch, to easy plus medium in the second, to all three levels for the remaining epochs. The backbone is CLIP ViT-L/14, fine-tuned with LoRA adapters of rank r=16, Adam at a learning rate of 1×10⁻⁴, batch size 48, 9 epochs, seed 42, and generation seed 8185 for synthetic data, on A6000 GPUs. All baselines, including Safe-CLIP and a safe-only fine-tuned CLIP†, were retrained under identical settings.

NSFWCaps was built by taking captions from the NoCaps validation split, using LLaMA-3-70B to generate unsafe variants while preserving semantics across 20 NSFW categories, generating unsafe images with an NSFW-tuned Stable Diffusion model (yielding 4.5K image-text pairs), filtering images with NudeNet and the Q16 detector, and retaining the most semantically aligned examples per category by JINA-CLIP similarity. The final set is 1,000 quadruples, each with a safe image, unsafe image, safe caption, and unsafe caption.

Why This Matters

The paper reframes safety fine-tuning as a geometry problem rather than a mapping problem: respecting the pretrained embedding space lets safety and generalization coexist. This matters for research because it offers a concrete diagnosis of why safety tuning degrades CLIP, and because NSFWCaps gives the community a distribution-shift safety benchmark with tighter safe-unsafe coupling (0.81 versus 0.62 JINA-CLIP similarity) than the ViSU test set. Since CLIP is used both for retrieval and as a backbone for text-to-image and image-to-text systems, unsafe biases in its embeddings can propagate into many downstream models, which the authors give as the reason embedding-level intervention is needed rather than only inference-time fixes.

Real-world applications:

• Healthcare and clinical systems, where the authors note safe VLM behavior is a priority for deployment in sensitive domains.
• Autonomous systems, also named by the authors as a sensitive deployment area.
• Text-to-image generation pipelines, where substituting the fine-tuned text encoder into Stable Diffusion v1.4 reduced the I2P average NSFW score from 37.1 to 16.0.
• Image captioning and visual assistants, where inserting the fine-tuned vision encoder into LLaVA lowered both NSFW rates and toxicity scores on real unsafe images from NudeNet, NSFW URLs, and SMID.

Industry relevance: organizations that ship CLIP-based retrieval, search, or generative backbones can apply this as a post-training fine-tuning step on top of an existing Safe-CLIP codebase, using LoRA adapters, without retraining from scratch or filtering LAION-5B-scale data, which the paper calls computationally infeasible. The reported gains come while keeping safety comparable to Safe-CLIP, which is the practical argument for adoption.

Future Directions

• Asymmetric encoder adaptation: The conclusion suggests exploring adaptation that treats the vision and text encoders differently rather than fine-tuning both.
• Broader real-world deployment: The authors call for wider deployment studies of safety-tuned models beyond the benchmark settings used here.
• Extension to other architectures: Results using additional architectures such as SigLIP are placed in the supplementary material, leaving open how far the proximity-aware principle transfers across backbones.
• Interaction with inference-time defenses: The paper shows combining SafeR-CLIP with Safety Guidance and Negative Prompting gives further NSFW reductions, raising the question of how to best pair fine-tuning with inference-time methods.

Target Audience

Researchers and engineers working on vision-language model safety, CLIP fine-tuning, content moderation, or safety evaluation benchmarks. It is also useful for practitioners deploying CLIP-based retrieval or generative systems who need to weigh safety gains against drops in zero-shot generalization, and for benchmark designers interested in how safe-unsafe pair quality affects safety evaluation. Readers should be comfortable with contrastive learning and embedding-space reasoning, and should heed the paper's own warning about explicit content.

Authors’ abstract

Improving the safety of vision-language models like CLIP via fine-tuning often comes at a steep price, causing significant drops in their generalization performance. We find this trade-off stems from rigid alignment strategies that force unsafe concepts toward single, predefined safe targets, disrupting the model's learned semantic structure. To address this, we propose a proximity-aware approach: redirecting unsafe concepts to their semantically closest safe alternatives to minimize representational change. We introduce SaFeR-CLIP, a fine-tuning framework that applies this principle of minimal intervention. SaFeR-CLIP successfully reconciles safety and performance, recovering up to 8.0% in zero-shot accuracy over prior methods while maintaining robust safety. To support more rigorous evaluation, we also contribute NSFW-Caps, a new benchmark of 1,000 highly-aligned pairs for testing safety under distributional shift. Our work shows that respecting the geometry of pretrained representations is key to achieving safety without sacrificing performance.