Defect-aware Hybrid Prompt Optimization via Progressive Tuning for Zero-Shot Multi-type Anomaly Detection and Segmentation

AI summary

Overview

Research area: Computer vision, specifically zero-shot industrial anomaly detection and segmentation using vision-language models (CLIP).

Technical level: Advanced. The paper assumes familiarity with CLIP-style image-text contrastive models, prompt learning, and industrial anomaly detection benchmarks.

Scope: The paper introduces DAPO, a defect-aware hybrid prompt optimization framework that combines fixed defect-name text anchors with a small set of shared learnable tokens to perform binary anomaly detection/segmentation under distribution shift and zero-shot multi-type anomaly segmentation on industrial images.

What This Paper Is About

Existing CLIP-based anomaly detectors mostly decide only whether an image is normal or abnormal, collapsing defects such as holes, cuts, or scratches into one coarse "abnormal" concept. Meanwhile, prompts built purely from learnable tokens can drift off CLIP's natural-language manifold and lose zero-shot transfer, and hand-crafted defect prompt libraries (as in MultiADS) are slow to build and hard to scale. DAPO addresses this by learning hybrid prompts that pair a literal defect descriptor (for example, "bent" or "scratch") with a shared set of learnable context tokens, so that new defect types can be handled at inference simply by swapping the textual anchor.

Key Contributions

1. A defect-aware prompt optimization framework (DAPO) that composes hybrid prompts from fixed textual anchors and a small set of shared learnable defect tokens reused across all defect types, rather than learning a separate prompt per defect or relying on hand-crafted prompt libraries.

2. A prompt composition and optimization scheme that explicitly bridges the coarse anomaly concept and fine-grained defect semantics, improving image-text alignment through a global loss on image-level embeddings and a local loss on patch-level embeddings.

3. Comprehensive evaluation under significant distribution shift across five public industrial benchmarks (MVTec-AD, VisA, MPDD, MAD, Real-IAD) and an internal semiconductor (ASIC) dataset, covering both zero-shot multi-type anomaly segmentation and binary anomaly detection.

4. A decomposition analysis isolating fixed defect anchors, shared learnable tokens, token sharing, progressive fine-tuning, and adapter depth, showing that the compositional shared-token design—not extra trainable capacity—drives the gains.

Main Findings

• Image-level detection under distribution shift: Averaged over the two strongest defect-aware baselines (AnomalyCLIP and MultiADS) across VisA, MPDD, and Real-IAD, DAPO achieves a 3.6% mean improvement in image-level AUROC and AP. DAPO attains the highest image-level AUROC and AP on all three target domains in Table 3.

• Zero-shot novel defect localization: Averaged over MultiADS across the representative unseen defect types listed in Table 2, DAPO shows a 5.2% mean improvement in AUROC and F1.

• Multi-type anomaly segmentation: DAPO outperforms MultiADS on AP in VisA (25.0 vs. 24.8), MPDD (53.6 vs. 53), MAD-Sim (33.3 vs. 31.5), and MAD-Real (58.5 vs. 52.3). The largest margins are on MPDD, with +7.2% on F1-score (50.0 vs. 42.8) and +6.0% on MAD-Real F1 (58.5 vs. 52.5). Pixel-level AUROC is slightly lower than MultiADS on VisA (88.9 vs. 93.6), MAD-Sim (91.9 vs. 92.1), and MAD-Real (86.1 vs. 89.2).

• Where gains come from: Strong results appear on visually present defects with textural or geometric signatures—66.6% F1 on unseen "flattening" in MPDD (vs. 36.1 for MultiADS), 40.4% F1 on MAD-Real "stains" (vs. 6.1), and 20.8% F1 on VisA "breakage" (vs. 7.9).

• Where it struggles: Anomalies with no strong local signal degrade performance. On MAD-Real "missing" components, DAPO scores 1.8% F1 versus 3.7 for MultiADS, which the authors attribute to the missing part leaving a visually normal patch for the ViT encoder to align with text. On VisA "particle", AUROC is high (99.0) but F1 is only 3.1—the model detects something is wrong but cannot localize it precisely.

• Internal ASIC dataset: DAPO reaches 91.8 AUROC and 79.1 AP, versus AnomalyCLIP at 68.2 AUROC / 35.3 AP and MultiADS at 91.1 AUROC / 61.7 AP. At a fixed threshold, DAPO's confusion matrix shows TN=4173, TP=404, operating at FPR=4.5% and TPR=74.8%, while AnomalyCLIP assigns the anomaly label to nearly all samples.

• Ablation on MPDD (image-level AUROC/AP): Defect-anchor only (text, non-learnable) reaches 53.8/65.0; soft-only reaches 73.7/78.1 but does not support the (K+1)-channel multi-type segmentation task; per-defect tokens without sharing drops to 70.9/75.6; removing progressive fine-tuning gives 79.7/81.5; 2-layer adapters give 80.5/82.3; DAPO reaches 81.2/83.6.

• AUPRO trade-off: DAPO's AUPRO lags some baselines (84 on VisA, 84.4 on MPDD, 80.3 on Real-IAD). The authors explain that AUPRO rewards predictions expanding to cover the entire annotated mask, whereas DAPO produces conservative heatmaps that highlight the most discriminative abnormal evidence, which reduces false alarms on normal regions.

Methodology in Plain English

The framework keeps CLIP's vision and text encoders frozen (OpenCLIP ViT-L/14@336 with OpenAI pre-trained weights) and trains only the prompts plus a small number of linear adapters—roughly 0.15% of the model.

For prompts, each abnormal prompt is a sequence: five learnable context tokens, followed by a placeholder for the defect descriptor (such as "bent" or "contamination"), followed by the fixed literal words "anomaly" and "object." The same learnable tokens are shared across every defect type, so the pool of learned context is type-agnostic and transfers; only the literal defect word changes. The normal prompt follows the same pattern but ends with "normal" and "object." For each state, E=10 prompt instances are generated and their text embeddings averaged, giving one normal embedding plus K defect-type embeddings.

On the image side, patch embeddings are extracted from layers {6, 12, 18, 24} (M=4) and passed through linear adapters into CLIP's shared vision-text space, alongside a single global image embedding.

Training uses two losses. The global loss is cross-entropy between the softmax over the global image embedding's cosine similarity to the normal embedding and to the average of the defect embeddings, and the image label. The local loss compares every adapted patch embedding against the K+1 text embeddings to produce similarity maps at each of the M layers, then applies focal loss plus dice losses (on the normal channel and its complement after upsampling to the original image resolution). The total loss is the global loss plus λ times the local loss, with λ=4.0.

Training details: images resized to 518×518 with standard CLIP preprocessing; Adam with learning rate 10⁻³, batch size 8, 5 epochs, PyTorch 2.7.0; prompt length l=5; learnable tokens initialized inside CLIP's embedding space using its token-embedding mean and standard deviation, which the authors report converges faster and performs better than random initialization. Transfer follows a source-to-target protocol: train on MVTec-AD and test on the other benchmarks, and separately train on VisA and test on MVTec-AD. Training takes about 25 minutes on MVTec-AD and inference about 55 minutes on Real-IAD on a single NVIDIA H200 GPU.

At inference, (K'+1) prompts are instantiated for the target defect descriptors, similarity maps are aggregated across the M stages, and the model outputs both a binary anomaly map and a multi-type mask via pixel-wise argmax over the (K'+1) channels.

Why This Matters

Impact on research: The work reframes the field's loose use of "zero-shot" by arguing that binary detection on unseen products is really distribution shift, not novel-type localization, and reserves the zero-shot label for the multi-type segmentation task. It also provides evidence that shared, type-agnostic learnable context—rather than per-defect prompts or larger adapters—is what enables transfer to unseen defect types.

Real-world applications:

• Manufacturing quality control: flagging and localizing scratches, rust, dents, or contamination on production lines before products ship.
• Semiconductor inspection: the internal ASIC dataset case study targets contamination detection on chip images at FPR=4.5% and TPR=74.8%, a deployment-oriented operating point.
• Reducing false alarms in automated visual inspection, where DAPO's conservative heatmaps avoid activating on normal regions.
• Root-cause analysis for industrial operators, since naming the specific defect type (stain, hole, missing component) supports tracing causes rather than only flagging an abnormality.

Industry relevance: The framework is lightweight (about 0.15% of parameters trainable, frozen backbone), runs training in roughly 25 minutes on MVTec-AD, and avoids the manual prompt curation that made prior defect-type-aware methods hard to scale. The authors state code is publicly available at a Bosch Research GitHub repository.

Future Directions

• Extending DAPO to other vision-language backbones beyond CLIP/OpenCLIP.
• Improving localization of absence-based defects (such as missing components), which the paper identifies as the main failure mode, using object-level priors and few-shot adaptation.
• Closing the AUPRO gap by producing heatmaps that cover full annotated regions without raising false positives on normal areas.
• Investigating why the model recognizes fine-grained defects like VisA "particle" at high AUROC (99.0) but localizes them poorly (3.1 F1), and whether this detection-localization split can be repaired.

Target Audience

Researchers and practitioners in industrial computer vision, zero-shot anomaly detection, and vision-language model adaptation will benefit most. It is also relevant to quality engineers and manufacturing deployment teams evaluating whether prompt-based CLIP methods are ready for production inspection, and to readers interested in prompt-learning design choices that preserve zero-shot transfer under distribution shift.

Authors’ abstract

Recent vision-language models (VLMs) like CLIP have shown impressive anomaly detection performance under significant distribution shift by utilizing high-level semantic information through text prompts. However, these models often overlook fine-grained defect cues, e.g., hole, cut, or scratch, that are essential for understanding the anomaly's nature. Moreover, the modality gap between images and text can lead to subtle visual evidence being poorly captured in textual descriptions. To address the gap, we enhance the representation of "abnormal" with structured semantics, bridging coarse anomaly signals and fine-grained defect categories. We propose a hybrid prompting mechanism that combines human-readable descriptions of defect types with learnable token embeddings. Building on these ideas, we introduce DAPO, a Defect-aware Prompt Optimization framework for zero-shot multi-type and binary anomaly detection and segmentation under distribution shift. DAPO aligns anomaly-relevant visual features with their corresponding textual semantics by learning hybrid defect-aware prompts that combine fixed textual anchors with trainable token embeddings. We conducted experiments on public benchmarks (MPDD, VisA, MVTec-AD, MAD, and Real-IAD) and an internal dataset. The results suggest that compared to the baseline models, DAPO achieves a 3.6% average improvement in AUROC and average precision metrics at the image level under distribution shift, and a 5.2% average improvement in AUROC and F1 when localizing novel anomaly types under zero-shot settings.