MoECLIP: Patch-Specialized Experts for Zero-shot Anomaly Detection

AI summary

Overview

• Research area: Computer vision, specifically Zero-Shot Anomaly Detection (ZSAD) built on top of CLIP-style vision-language models.
• Technical level: Advanced. The paper combines Mixture-of-Experts (MoE) routing, Low-Rank Adaptation (LoRA), orthogonal feature separation, and simplex equiangular tight frame geometry in a single framework.
• Scope: The paper proposes MoECLIP, an MoE-based architecture that routes each image patch to a specialized LoRA expert, and evaluates it on 14 industrial and medical benchmark datasets against six prior ZSAD methods.

What This Paper Is About

Zero-Shot Anomaly Detection aims to spot defects or abnormalities in categories the model never saw during training, using CLIP's general visual-semantic knowledge. The problem the authors identify is that existing methods adapt CLIP uniformly across every image patch, ignoring that different regions (object body, background, the anomaly itself) are semantically different. MoECLIP replaces this "patch-agnostic" design with a Mixture-of-Experts module that sends each patch to whichever of several LoRA experts best matches its characteristics.

Key Contributions

1. A MoE architecture for ZSAD. The authors state they are the first to introduce an approach to ZSAD that dynamically routes each image patch to a specialized expert, establishing patch-level adaptation for the task.
2. Frozen Orthogonal Feature Separation (FOFS). The input feature space of dimension d is split into K non-overlapping subspaces, and each expert's LoRA A matrix is built as a block matrix with a randomly initialized orthogonal matrix Q_n obtained via QR decomposition in its own block and zeros elsewhere, so that A_n A_m^T = 0 for all n ≠ m. The A matrices are frozen; only the B matrices are trainable.
3. A simplex equiangular tight frame (ETF) loss. An auxiliary loss pushes the Gram matrices of L2-normalized expert outputs toward an ideal ETF Gram matrix with diagonal 1 and off-diagonal −1/(K−1), enforcing maximally equiangular expert representations.
4. State-of-the-art results on 14 benchmark datasets. Across 5 industrial and 9 medical datasets, MoECLIP reports the best average performance on both image-level and pixel-level AUROC and AP, with code released at https://github.com/CoCoRessa/MoECLIP.

Main Findings

• Image-level average performance: MoECLIP reaches (89.6 AUROC, 90.6 AP) averaged over the image-level table, versus the second-best figure of (86.6, 88.2). The paper reports this as improvements of 3.0% in AUROC and 2.4% in AP.
• Pixel-level average performance: MoECLIP reaches (94.3 AUROC, 47.5 AP), outperforming the second-best by 1.1% in average AUROC and 1.7% in average AP.
• Per-dataset highlights (image-level AUROC, AP): MVTec-AD (93.9, 96.8), BTAD (93.1, 98.0), RSDD (95.3, 95.1), DTD-Synthetic (95.5, 98.6), Brain MRI (88.5, 97.1), Head CT (96.6, 94.5), Liver CT (74.0, 64.6), Retina OCT (85.5, 84.9), and VisA (83.6, 86.2).
• Per-dataset highlights (pixel-level AUROC, AP): MVTec-AD (92.5, 45.7), VisA (95.6, 26.1), BTAD (96.8, 50.4), RSDD (99.7, 35.9), DTD-Synthetic (98.8, 62.7), Brain MRI (97.3, 61.3), Liver CT (97.2, 10.8), Retina OCT (96.2, 66.3), ColonDB (85.4, 34.8), ClinicDB (89.7, 49.9), CVC-300 (97.0, 53.0), Endo (91.0, 62.5), and Kvasir (88.1, 57.6).
• Not a clean sweep on every dataset. On VisA image-level, Bayes-PFL reports higher results (86.8, 89.3) than MoECLIP. On MVTec-AD pixel-level AP, Bayes-PFL reports 48.4 versus MoECLIP's 45.7, and on DTD-Synthetic pixel-level AP Bayes-PFL reports 66.7 versus MoECLIP's 62.7.
• Ablation, averaged over MVTec-AD, DTD-Synthetic, Head CT and ColonDB as (pixel AUROC, image AUROC): Vanilla CLIP baseline (40.6, 67.4); without FOFS and ETF loss (91.2, 93.1); without FOFS (91.9, 93.9); without ETF loss (91.7, 94.1); without the Depth-wise Adapter (91.7, 93.6); without PAA (90.7, 93.5); full MoECLIP (92.2, 95.3).
• Removing PAA hurts medical data most. The ablation reports that dropping the Patch Average Aggregation module causes a significant performance decrease on medical datasets specifically.
• Functional redundancy is measured directly. Using inter-expert cosine similarity on MVTec-AD, the original MoE shows a value of 0.45 between Expert 1 and Expert 2; FOFS alone reduces this to 0.24; FOFS plus the ETF loss brings it down to 0.02.
• Routing is content-based, not arbitrary. Grad-CAM and patch selection maps at layer 18 on the MVTec-AD hazelnut class show Expert 1 focusing on the anomaly, Expert 2 on the object body, Expert 3 on the background, and Expert 4 rarely utilized. The authors state the anomaly is identified by a dynamic combination of functionally distinct experts rather than one.
• More experts is not better. An ablation over K from 1 to 8 on MVTec-AD shows increasing the number of experts does not necessarily improve ZSAD performance; the authors set K = 4 empirically as the best across both pixel-level and image-level AUROC, describing a trade-off between under-specialization and functional redundancy.

Methodology in Plain English

The authors keep the entire pretrained CLIP vision encoder frozen and attach small trainable modules to it, so CLIP's general knowledge is preserved while the model is nudged toward anomaly detection. At four points in the encoder (layers 6, 12, 18 and 24), a Mixture-of-Experts block sits on top of the patch features. A learned router scores every patch against four experts, and the top two experts are activated, with their scores renormalized to produce a weighted sum. Each expert is a LoRA module, meaning a pair of small matrices that add a low-rank update rather than changing the base weights.

Two mechanisms keep those experts from collapsing into copies of each other. First, before training begins, each expert's A matrix is constructed so that it reads only its own slice of the feature dimensions and is provably orthogonal to the other experts' slices; that matrix is then frozen. Second, an ETF loss takes each expert's output, normalizes it, forms a Gram matrix of pairwise cosine similarities, and penalizes deviation from an ideal geometric target where all experts are equally separated. A standard balance loss additionally prevents the router from sending everything to one expert.

Because a ViT divides images into fixed-size patches, the authors add a parameter-free Patch Average Aggregation step that averages patch features within s × s sliding windows at scales s ∈ {1, 3, 5}, giving the model multi-scale context during training rather than only at test time. For the image-level score, features from the final layer pass through a lightweight Depth-wise Adapter (a 1D depthwise separable convolution inspired by MobileNet) and global average pooling before being compared by cosine similarity against anomalous and normal text embeddings. Training combines Focal and Dice losses for segmentation, binary cross-entropy for classification, and the ETF and balance losses as auxiliaries.

Why This Matters

The paper reframes how adaptation should be applied in zero-shot anomaly detection: rather than treating all patches as interchangeable, it makes the adaptation itself input-dependent. If the reported gains hold, this suggests the patch-agnostic design of prior CLI

Authors’ abstract

The CLIP model's outstanding generalization has driven recent success in Zero-Shot Anomaly Detection (ZSAD) for detecting anomalies in unseen categories. The core challenge in ZSAD is to specialize the model for anomaly detection tasks while preserving CLIP's powerful generalization capability. Existing approaches attempting to solve this challenge share the fundamental limitation of a patch-agnostic design that processes all patches monolithically without regard for their unique characteristics. To address this limitation, we propose \textbf{MoECLIP}, a Mixture-of-Experts (MoE) architecture for the ZSAD task, which achieves patch-level adaptation by dynamically routing each image patch to a specialized Low-Rank Adaptation (LoRA) expert based on its unique characteristics. Furthermore, to prevent functional redundancy among the LoRA experts, we introduce (1) Frozen Orthogonal Feature Separation (FOFS), which orthogonally separates the input feature space to force experts to focus on distinct information, and (2) a simplex equiangular tight frame (ETF) loss to regulate the expert outputs to form maximally equiangular representations. Comprehensive experimental results across 14 benchmark datasets spanning industrial and medical domains demonstrate that MoECLIP outperforms existing state-of-the-art methods. The code is available at https://github.com/CoCoRessa/MoECLIP.