From Internal Diagnosis to External Auditing: A VLM-Driven Paradigm for Data-Free Online Backdoor Defense

AI summary

Overview

• Research area: Machine learning security, specifically data-free backdoor defense for deep neural networks, using vision-language models (VLMs) as external auditors.
• Technical level: Advanced. The paper assumes familiarity with backdoor attacks, VLMs such as CLIP and Qwen-VL, test-time adaptation, and statistical thresholding.
• Scope in one sentence: The paper proposes PRISM, an online test-time wrapper that never touches the victim model's weights or any training data, and instead asks a frozen VLM to audit each prediction using refined visual prototypes and a statistically calibrated threshold.

What This Paper Is About

Existing data-free backdoor defenses fall into two families: model repairing (inspecting the victim model's internal properties) and input robustness (assuming triggers are fragile to perturbation). The authors argue both are entangled with the compromised model or with fragile assumptions about triggers, so they fail against advanced dynamic attacks and clean-image backdoors. The goal is to shift to "External Semantic Auditing," where a frozen, public VLM acts as an independent gatekeeper that decides at inference time whether to trust the victim model's prediction or override it.

Key Contributions

1. A new paradigm (External Semantic Auditing): The paper identifies failure modes of the internal diagnosis paradigm and proposes decoupling defense from the victim model by using a universal VLM as an independent, frozen external auditor.
2. An online adaptive framework (PRISM): PRISM wraps the victim model in a dual-stream architecture with a Hybrid VLM Teacher (static text anchors fused with online-refined visual prototypes) and an Adaptive Router (online statistical monitoring via the Cornish-Fisher expansion) to avoid static thresholds.
3. Broad empirical validation: Evaluation across 17 datasets, 11 attack types, and 6 VLM backbones, reporting state-of-the-art performance, including suppressing Attack Success Rate (ASR) to below 1% on CIFAR-10.
4. Robustness against adaptive adversaries: The framework is tested against online flooding/periodic/mixed attack streams, typographic prompt attacks, and four VLM-victim collusion scenarios.

Main Findings

• Strong headline result on CIFAR-10: PRISM suppresses ASR to below 1% while even improving Clean Accuracy (CA). In the CIFAR-10 comparison table, PRISM reaches an average ASR of 0.8% and average CA of 93.2%, versus 96.4% average ASR with no defense and 92.0% average CA with no defense.
• Only method to cover all 11 attacks: PRISM is described as the only method that successfully defends against all 11 attack types while maintaining high CA. Baselines such as CLP and ScaleUp fail on Clean-Image backdoors (FLIP, GCB), and ZIP and TeCo struggle with Dynamic attacks.
• The strongest baseline still falls short: Refine is identified as the strongest existing baseline (average ASR 28.8% on CIFAR-10) but fails to defend against Clean-Image attacks.
• Naive VLM use is not viable: A zero-shot VLM baseline achieves low average ASR (0.4%) but incurs a catastrophic CA drop, reported as at least 70% drop on GTSRB. An Ensemble baseline that averages victim and VLM logits has high average CA (94.7%) but an average ASR of 75.7%.
• Low latency: On an NVIDIA A100 GPU with batch size 1, PRISM (CLIP version) shows an inference latency of 12.5 ms per image. The body text says it ranks as high as 2nd among defenses; the figure caption states it ranks 2/9 at best and 4/9 at worst among all tested methods. The authors note that the lightweight CLIP auditor is sufficient, making the higher cost of generative VLMs unnecessary for most deployments.
• Robust across VLM architectures: PRISM consistently limits ASR across embedding models (CLIP, SigLIP, ImageBind) and generative models (Qwen2.5-VL-7b, LLaVA-1.5-7b, Gemma3-4b). The body text states ASR is limited to below 10%; the Figure 3 caption states below 8% across 6 different VLMs, with CA drops within 5%.
• Counter-intuitive CA behavior: PRISM maintains high CA even when the VLM's zero-shot accuracy is poor, and can even repair CA when zero-shot accuracy is higher (e.g., ImageBind on TinyImageNet). The Adaptive Router decouples defense capability from VLM quality: in low-performance domains the VLM acts as a weak constraint that intercepts high-confidence trigger outliers while deferring to the victim model for benign samples.
• Scales across datasets and imbalance: Across 17 datasets with varying Imbalance Ratio (IR), including Caltech101 (IR 86.5), SUN397 (37.7), FER2013 (16.0), and GTSRB (12.5), PRISM defends against Blend, BPP, and CTRL. It remains effective even where the VLM is weak, e.g. SVHN with VLM zero-shot accuracy of 13.4%.
• Scales across stream size: From 200 to 10,000 images, CA variation stays under 4% and ASR under 20%.
• Robust to poison rate: PRISM controls CIFAR-10 ASR to at most 5% and GTSRB ASR to at most 12% across poison rates of 1%, 5%, and 10%. Even when the poison set is 50 times larger than the clean set, ASR stays around 12%.
• Online adaptive attacks are resisted: Against Flooding (more than 90% backdoor samples), Periodic, and Mixed strategies, ASR stays below 20% and CA drop stays below 5%, attributed to the Selective Update Mechanism and CMA inertia.
• Typographic attacks partially penetrate but are still suppressed: Against images superimposed with target class names, the standalone VLM's zero-shot ASR rises to 60.1%, while PRISM's ASR rises to 15.6% with CA of 92.2%. The authors argue the visual prototype refinement still provides a safety layer.
• Collusion structure matters more than VLM compromise: In the four collusion scenarios (CIFAR-10, BadNets), Sc0 (clean CLIP) gives CA 93.9% / ASR 0.2%; Sc1 (different trigger) gives 93.6% / 0.8%; Sc2 (same trigger, different target) gives 93.9% / 0.0%; and Sc3 (same trigger and same target) gives 93.7% / 100%. The authors state that only cross-model collusion breaks the defense.
• Ablation findings: Removing online updates causes PRISM to fail on GTSRB with ASR spiking to about 40%. Removing prototype refinement causes a CA drop of nearly 20% on GTSRB. Removing skewness correction makes the optimal threshold vary across datasets (for example -2 for CIFAR versus -3 for GTSRB), whereas with it, ζ = -2 is optimal across all datasets and attacks.

Methodology in Plain English

PRISM sits in front of an already-deployed, possibly backdoored classifier (the "suspicious model") and inspects each incoming test image as it arrives, with no training data and no modification of the victim's weights.

Two parallel passes. Each image goes through both the victim model and a frozen, trusted VLM. The victim usually wins on benign inputs because it understands the domain; the VLM acts as an independent semantic reference that has never seen the poisoned training distribution.

Two kinds of anchors for the VLM. Rather than trusting text labels alone, the VLM scores an image against (a) static text anchors for the class names and (b) dynamic visual prototypes, which are running centroids of images seen so far in the test stream. The two similarity scores are combined with a balancing coefficient λ, so the auditor uses broad language semantics while adapting to the actual visual distribution of the stream.

Generative VLMs handled separately. For generative backbones such as Qwen and Gemma, the authors cache the visual features and system prompt hidden states (a Key-Value Cache strategy), then read the log-probability of the first sub-word token of each class name to convert generation into a usable class score.

A discrepancy score and an adaptive threshold. PRISM computes a logit margin Δ between the VLM's score for the victim's predicted class and the best competing class, passing it through an exponential to keep it positive and stable. Small Δ means the VLM disagrees with the victim, which is the signature of a trigger. Because Δ is not Gaussian and often skewed, the threshold is not fixed: for each class the method tracks the running mean, standard deviation, and skewness, then uses a Cornish-Fisher expansion to shift the threshold according to asymmetry. If Δ exceeds the threshold, the victim's prediction is accepted; otherwise the system falls back to the VLM.

Online, memory-light adaptation. Updates happen only for samples that pass the gate, using a Cumulative Moving Average rather than an Exponential Moving Average so the statistics gain inertia and resist local drift. A warm-up phase starts with a conservative Gaussian prior (skewness set to zero) and interpolates to the skewness-aware correction as samples accumulate (around 100). Only statistical moments and class centroids are updated in place, with no replay buffer of raw images, keeping overhead at O(1).

Experimental setup. Default settings include base threshold ζ = -2, victim model PreActResNet18, CLIP as the auditing VLM, batch size 256, prototype fusion weight λ_p = 0.5, target label y_t = 0, a 50% poison rate for clean-label attacks and 5% otherwise, and a warm-up window of one batch of unlabeled images. Evaluation covers 17 datasets (including CIFAR-10, CIFAR-100, ImageNet, MNIST, GTSRB, SVHN, SUN397, Caltech101, STL10, and others) and 11 attacks grouped into Classic (BadNets, Blend), Dynamic (SSBA, IAB, WaNet, BPP), Clean-Label (LC, SIG, CTRL), and Clean-Image (FLIP, GCB).

Why This Matters

Impact on research. The paper reframes data-free backdoor defense as an auditing problem rather than a repair or purification problem, and it is presented as the first use of VLMs for data-free backdoor defense without training interventions or access to model weights. It also introduces the use of Cornish-Fisher-based adaptive thresholding for this security setting, and provides a stress test of the "trusted auditor" assumption via collusion scenarios.

Real-world applications:

• Model-as-a-Service inference: A cloud provider serving a third-party or customer-supplied model can gate predictions without retraining, accessing customer data, or editing weights.
• Safety-critical perception: The evaluation covers domains such as traffic-sign recognition (GTSRB) and facial expression recognition (FER2013), where a hijacked prediction stream carries direct risk.
• Healthcare and other specialized domains: The paper explicitly discusses specialized domains such as healthcare, where a general VLM's zero-shot ability is weak, and proposes a "Trust Chain" strategy in an appendix for backdoored VLMs.
• Supply-chain auditing: Organizations receiving pre-trained weights from vendors can wrap a frozen VLM around them and continuously monitor the prediction stream for trigger-like statistical anomalies.

Industry relevance. The method's requirements are modest: a frozen public VLM, no labeled data, no gradient access to the victim, and no retained image buffer. Reported latency for the CLIP configuration is 12.5 ms per image on an A100, which the authors present as practical. The main caveat for deployment is that robustness depends on collusion structure: if the attacker implants the same trigger with the same target in both the victim and the auditor, the reported ASR reaches 100%.

Future Directions

• Securing the auditor itself: The Sc3 collusion result (CA 93.7%, ASR 100%) shows the defense breaks under cross-model collusion with the same trigger and target, motivating mechanisms that detect or avoid a compromised auditing VLM.
• Semantic/typographic input attacks: PRISM's ASR rises to 15.6% under text-prompt attacks that fool

Authors’ abstract

Deep Neural Networks remain inherently vulnerable to backdoor attacks. Traditional test-time defenses largely operate under the paradigm of internal diagnosis methods like model repairing or input robustness, yet these approaches are often fragile under advanced attacks as they remain entangled with the victim model's corrupted parameters. We propose a paradigm shift from Internal Diagnosis to External Semantic Auditing, arguing that effective defense requires decoupling safety from the victim model via an independent, semantically grounded auditor. To this end, we present a framework harnessing Universal Vision-Language Models (VLMs) as evolving semantic gatekeepers. We introduce PRISM (Prototype Refinement & Inspection via Statistical Monitoring), which overcomes the domain gap of general VLMs through two key mechanisms: a Hybrid VLM Teacher that dynamically refines visual prototypes online, and an Adaptive Router powered by statistical margin monitoring to calibrate gating thresholds in real-time. Extensive evaluation across 17 datasets and 11 attack types demonstrates that PRISM achieves state-of-the-art performance, suppressing Attack Success Rate to <1% on CIFAR-10 while improving clean accuracy, establishing a new standard for model-agnostic, externalized security.