VFAD: Variational Semantic Prompting Meets Frequency-Adaptive Representation Learning for Zero-Shot Anomaly Detection

AI summary

Overview

Research area: Computer vision / zero-shot anomaly detection (ZSAD), building on large-scale vision-language models (CLIP) for industrial inspection and medical image analysis.

Technical level: Advanced. The paper assumes familiarity with CLIP-style vision-language pretraining, prompt learning, variational inference (information bottlenecks, reparameterization, KL divergence), discrete wavelet transforms, and mixture-of-experts routing.

Scope: The paper proposes VFAD, a unified framework combining a Variational Semantic Prompt Extractor (VSPE) and a Frequency-Adaptive Representation Aggregation (FARA) module, evaluated on 13 industrial and medical anomaly detection benchmarks.

What This Paper Is About

Zero-shot anomaly detection asks a model to flag and localize anomalies in categories it has never seen during training. Existing CLIP-based methods generalize reasonably well, but the authors argue they fail on two fronts: handcrafted or deterministic text prompts cannot express the wide variety and ambiguity of anomaly semantics, and CLIP's globally aligned visual features emphasize whole-object semantics while washing out subtle local texture and structural deviations. VFAD's goal is to fix both sides at once: make the semantic prompts derive from dense local patch information (with probabilistic regularization), and make the visual representation explicitly frequency-aware.

Key Contributions

1. The VFAD framework — a zero-shot anomaly detection framework that jointly combines variational semantic prompting with frequency-adaptive representation learning for fine-grained anomaly understanding.
2. VSPE (Variational Semantic Prompt Extractor) — extracts anomaly-relevant local semantics from dense patch tokens via learnable query anchors and cross-attention, then regularizes them through a variational information bottleneck before injecting them as prompts into the text encoder.
3. FARA (Frequency-Adaptive Representation Aggregation) — uses wavelet-based frequency decomposition plus frequency-specific expert aggregation to enhance anomaly-discriminative visual representations, preserving structural information while sharpening texture-sensitive detail.
4. Extensive empirical validation — experiments on 13 industrial and medical benchmarks showing superior zero-shot generalization compared with six prior ZSAD methods.

Main Findings

• Industrial benchmark leadership: VFAD reaches 94.2% AUROC / 96.1% AP at image level and 97.3% AUROC / 53.8% AP at pixel level averaged over MVTec-AD, VisA, KSDD2, DAGM, and DTD-Synthetic. Compared with the strongest competitor, this is an improvement of 0.7% average image-level AUROC and 0.5% average pixel-level AUROC.
• Medical benchmark leadership: On HeadCT, BrainMRI, Br35H, ISIC, CVC-ColonDB, CVC-ClinicDB, Endo, and Kvasir, VFAD averages 97.6% AUROC / 97.6% AP at image level and 89.7% AUROC / 60.7% AP at pixel level. The authors attribute the AP gains to more reliable discrimination and more accurate lesion localization.
• Both modules are complementary: Ablating both VSPE and FARA on MVTec-AD gives 89.3 I-AUROC / 90.6 P-AUROC and on VisA 84.4 I-AUROC / 94.7 P-AUROC. Adding VSPE alone mainly lifts image-level discrimination (91.5 I-AUROC on MVTec-AD; 86.3 on VisA), FARA alone mainly lifts localization (91.8 P-AUROC on MVTec-AD; 95.2 on VisA), and combining them yields the best scores: 93.1 I-AUROC / 92.7 P-AUROC on MVTec-AD and 87.2 I-AUROC / 95.8 P-AUROC on VisA.
• Local semantics beat global tokens, and the variational bottleneck helps: In VSPE ablations, local semantic aggregation consistently outperforms using the global token, and adding the variational information bottleneck improves both variants — the full local + VIB configuration reaches 93.1 / 92.7 on MVTec-AD and 87.2 / 95.8 on VisA.
• Four experts per frequency branch is optimal: Increasing the number of experts in each frequency-specific MoE branch from 1 to 4 steadily improves both image-level and pixel-level AUROC on MVTec-AD, with the best result at four experts; more experts beyond that causes slight degradation, attributed to expert redundancy and optimization difficulty.
• Stronger backbone, better results: On MVTec-AD, ViT-B/16-224 gives 90.7 I-AUROC / 93.8 I-AP / 90.3 P-AUROC / 45.3 P-AP at 91.6 ms; ViT-L/14-224 gives 91.9 / 95.2 / 91.8 / 46.7 at 112.9 ms; ViT-L/14-336 gives 93.1 / 96.9 / 92.7 / 48.6 at 137.5 ms. The authors acknowledge the added inference latency of ViT-L/14-336.
• Cleaner text embedding space: t-SNE visualization of text embeddings shows VFAD forms more compact semantic clusters with clearer separation between normal and anomalous semantics than AdaCLIP, whose embeddings appear more scattered and partially mixed.
• Sharper qualitative maps: VFAD reportedly suppresses spurious background activations better and produces sharper boundaries and more complete anomalous regions than AnomalyCLIP, AdaCLIP, AA-CLIP, and MoECLIP in the provided visualizations.
• Not first on every dataset: For pixel-level AUROC on CVC-ColonDB, VFAD's 84.1 is lower than MoECLIP's 85.4, and for image-level Br35H VFAD's AP of 97.7 versus AnomalyCLIP's 96.1 is a gain but AA-CLIP's 96.8 image AUROC is below VFAD's 98.3; the paper describes VFAD as ranking first on nearly all datasets rather than all.

Methodology in Plain English

The starting point is CLIP, which aligns images and text in a shared embedding space. VFAD keeps CLIP's zero-shot ability but reworks two things: the text prompts and the visual features.

On the text/prompt side (VSPE): Instead of using fixed templates like "a photo of a normal object" or purely learnable prompt tokens, VFAD looks at the image itself. A small set of learnable query anchors (fewer than the number of image patches) attends to the dense patch tokens through cross-attention, pulling out a compact set of anomaly-relevant local cues. Because anomalies are varied and ambiguous, the model does not commit to a single deterministic embedding; it predicts a mean and variance for a Gaussian latent distribution and samples from it with the reparameterization trick. A KL term pushes that distribution toward the prior, which suppresses redundant or category-specific noise. The sampled embeddings are projected into the text embedding space and added to learnable prompt tokens, then concatenated into the text Transformer's token sequence for the first several layers.

On the visual side (FARA): Anomalies often show up as texture irregularities or small structural distortions that ordinary spatial patch features miss. FARA reshapes patch features into a spatial map and applies a discrete wavelet transform, splitting them into one low-frequency subband (global structure) and three directional high-frequency subbands (local detail). Each of these two groups goes through its own mixture-of-experts branch: a lightweight router scores the experts and activates only the top few, and their outputs are combined with normalized routing weights. The enhanced components are transformed back with an inverse wavelet transform and added residually to the original features to avoid over-adapting to the auxiliary training categories.

How the outputs are produced: Anomaly maps are formed by comparing L2-normalized refined visual features against L2-normalized normal/anomalous text embeddings, upsampling, applying softmax, and averaging across the selected encoder layers. The image-level score concatenates multi-level features, pools them spatially, projects them back, and fuses them with the class token before comparing against the text embeddings.

Training: Segmentation (Dice plus Focal) supervises pixel maps, binary cross-entropy supervises the image-level score, and a weighted KL term regularizes the variational prompts. Training uses one auxiliary category set that does not overlap with the test categories.

Setup specifics: CLIP ViT-L/14-336 is the default backbone, images are resized to 518×518, patch features come from layers 6, 12, 18, and 24, prompting depth J is 9, the number of query anchors A is 8, each MoE branch has four experts with top-2 routing, the KL weight λ is 0.1, and optimization uses Adam with a learning rate of 2×10⁻⁴ on a single NVIDIA A100 GPU. VisA serves as the auxiliary training set for the non-VisA datasets, and for VisA evaluation the model is trained on MVTec-AD. AUROC and AP are the evaluation metrics.

Why This Matters

Impact on research: The paper argues that two weaknesses of CLIP-based ZSAD — deterministic prompt semantics and globally biased visual features — can be attacked jointly rather than separately. It also connects variational information bottleneck regularization and wavelet-domain mixture-of-experts to a task (anomaly detection) where the target signal is small and local, which may inform future work on prompt stochasticity and frequency-aware adaptation in other dense prediction problems.

Real-world applications:

• Industrial quality inspection: defect detection on MVTec-AD, VisA, KSDD2, and DAGM-style production lines, where defects are rare and new product categories appear without labeled anomalies.
• Surface/texture defect screening: DTD-Synthetic-style textured surfaces where subtle textural irregularities are the anomaly.
• Medical screening: brain tumor classification on HeadCT, BrainMRI, and Br35H, and skin lesion detection on ISIC.
• Endoscopic polyp segmentation: CVC-ColonDB, CVC-ClinicDB, Endo, and Kvasir, where localization quality directly affects downstream clinical decisions.

Industry relevance: The value proposition is zero-shot deployment — a model trained on auxiliary categories that can be pointed at an unseen category without collecting or annotating target-specific anomaly data, which is exactly the situation manufacturers and clinical sites face when a new product line or acquisition protocol appears. The paper also documents the cost side: the default ViT-L/14-336 backbone carries the highest reported inference time (137.5 ms) of the three backbones tested, which matters for high-throughput deployment.

Future Directions

1. More efficient prompt modeling. The authors explicitly state future work will explore more efficient prompt modeling; the current design runs VSPE and a two-branch MoE-based FARA on top of a ViT-L/14-336 backbone, which is the slowest configuration tested at 137.5 ms.
2. Broader real-world applications. The paper names this as future work but does not specify which domains or settings beyond the industrial and medical benchmarks.
3. Reducing expert redundancy. Performance peaks at four experts and degrades slightly beyond that due to expert redundancy and optimization difficulty, leaving open how to scale expert capacity without this penalty.
4. Closing the remaining gaps. VFAD does not lead on every dataset (for example, pixel-level AUROC on CVC-ColonDB), and the paper does not report an explicit limitations section, statistical significance testing, memory usage, or training cost beyond the backbone inference-time comparison — all open questions for follow-up work.

Target Audience

Researchers and practitioners working on zero-shot or few-shot anomaly detection, industrial visual inspection, and medical image analysis; engineers adapting CLIP-style vision-language models to dense localization tasks; and readers interested in variational prompt learning or wavelet/mixture-of-experts visual representation design. Readers will need a working command of CLIP prompt learning and variational inference to follow the method section.

Authors’ abstract

Zero-shot anomaly detection (ZSAD) aims to detect and localize anomalies in unseen categories without access to target-specific training data. Although recent CLIP-based methods have demonstrated promising generalization through vision-language alignment, they remain limited in capturing diverse anomaly semantics and subtle local variations. To address these limitations, we propose VFAD, a unified framework that combines variational semantic prompting with frequency-adaptive representation learning. Specifically, we introduce a Variational Semantic Prompt Extractor (VSPE), which adaptively aggregates anomaly-relevant local semantics from dense patch tokens and regularizes them through a variational information bottleneck, thereby incorporating fine-grained visual cues and enabling more precise cross-modal alignment. Furthermore, we develop a Frequency-Adaptive Representation Aggregation (FARA) module that leverages wavelet-based frequency decomposition and frequency-specific expert aggregation to enhance anomaly-discriminative visual representations. By jointly strengthening semantic guidance and visual representation learning, VFAD improves both anomaly discrimination and fine-grained localization. Extensive experiments on 13 industrial and medical benchmarks demonstrate that VFAD consistently outperforms existing state-of-the-art ZSAD methods across diverse anomaly scenarios. The code will be publicly available upon publication.