ADPretrain: Advancing Industrial Anomaly Detection via Anomaly Representation Pretraining

AI summary

Overview

Research area: Computer Vision — industrial anomaly detection (AD) and representation learning/pretraining.

Technical level: Advanced. The paper assumes familiarity with contrastive learning (InfoNCE), one-class classification, Transformer attention, and embedding-based AD pipelines.

Scope: The paper proposes a pretraining framework that learns feature representations specifically for industrial anomaly detection, rather than reusing ImageNet-pretrained features, and validates them by substituting them into five existing embedding-based AD methods across five datasets and five backbones.

What This Paper Is About

Mainstream anomaly detection methods rely on feature networks pretrained on ImageNet, but that pretraining has nothing to do with separating normal from abnormal, and natural images differ in distribution from industrial inspection images. The authors argue these two problems make ImageNet-pretrained features suboptimal for AD, and they set out to learn pretrained representations designed for AD by pretraining on a large anomaly detection dataset (RealIAD) with losses that explicitly push normal and abnormal features apart. The goal is not a new AD model but a better feature extractor that can be dropped into existing AD methods in place of their original features.

Key Contributions

1. First dedicated anomaly representation pretraining framework. The authors state this is the first study dedicated to anomaly representation pretraining, constructing a framework that learns robust and discriminative pretrained features specially for anomaly detection tasks.
2. Angle- and norm-oriented contrastive losses. Two losses that simultaneously maximize the angle size and the norm difference between normal and abnormal features, so that discrepancies between the two are optimized from both a direction and a magnitude perspective.
3. Use of residual features as the pretraining substrate. Instead of the raw backbone features, the framework learns pretrained representations on residual features, which the authors describe as class-generalizable, to reduce the potential shift between the pretraining data and downstream AD datasets.
4. A large-scale empirical validation protocol. Pretrained features are substituted into five embedding-based AD methods (PaDiM, PatchCore, CFLOW, GLASS, UniAD) plus a simple norm-based baseline the authors call FeatureNorm, evaluated on five AD datasets with five backbones.

Main Findings

• Pretrained features consistently improve existing AD methods. Replacing original features with the authors' pretrained features improves performance across the reported dataset/backbone/method combinations. For example, on MVTecAD with the DINOv2-Base backbone, PatchCore goes from 95.5/82.7 to 99.0/87.4 (+3.5 image-level AUROC / +4.7 PRO), and ImageBind-based PatchCore on VisA goes from 91.4/81.9 to 94.8/86.3 (+3.4 / +4.4).
• Gains are largest where original features are weak. With DINOv2-Base, UniAD on MVTecAD rises from 71.1/81.5 to 97.1/91.2 (+26.0 / +9.7), and with ImageBind, MPDD UniAD rises from 60.7/52.3 to 93.6/95.0 (+32.9 / +42.7).
• Feature norms alone become a viable anomaly score. The FeatureNorm baseline, which uses feature norms directly as anomaly scores without downstream AD modeling, reaches 98.2/92.8 on MVTecAD and 94.8/87.2 on VisA with DINOv2-Base, versus 48.4/28.9 and 52.2/30.1 with the original features. The authors note this also makes the features useful for few-shot anomaly detection.
• Not every case improves. Some combinations show small decreases, such as PaDiM on BTAD with DINOv2-Base (96.6/74.4 to 95.2/74.7) and CFLOW on MVTec3D with CLIP-Base (82.1/90.1 to 83.8/90.4 at image level but mixed elsewhere), and the paper reports these negative deltas alongside the positive ones.
• Feature visualizations support the claim. t-SNE visualizations of features from the "capsules" class of VisA show that with the pretraining, normal features are more compact and normal and abnormal features are more separated.
• Ablation evidence on VisA with ImageBind. Comparing the framework's components (PaDiM / PatchCore / FeatureNorm): baseline non-residual features without losses (92.6/86.3, 91.6/81.3, 49.2/44.5); with angle and norm losses but non-residual features (93.5/86.1, 93.6/85.1, 82.9/83.9); residual features alone (93.9/85.6, 92.9/86.5, 91.3/86.8); the full combination (95.4/88.7, 94.6/87.0, 94.2/89.0).
• Fixing the backbone helps. Non-fixed backbone variants score lower than fixed-backbone variants under both residual and non-residual settings (e.g. 89.3/83.4 vs 95.4/88.7 with PaDiM).
• Both losses contribute, and together they are best. Angle-only (93.9/85.2, 93.2/83.8, 83.9/83.0) and norm-only (93.7/85.3, 90.9/84.5, 92.4/85.1) each trail the combined loss (95.4/88.7, 94.6/87.0, 94.2/89.0).
• The learnable key/value attention matters. The authors report that directly using a vanilla Transformer as the Feature Projector has poor performance (referring to Table 2(b)), which is why self-attention is replaced. The numeric results of Table 2(b) are not included in the available content.
• Multiple options evaluated. Five backbones are used: DINOv2-Base, DINOv2-Large, CLIP-Base, CLIP-Large, and ImageBind; the paper also reports more t-SNE figures and qualitative results in its Appendix.

Methodology in Plain English

Anomaly detection needs features where normal and abnormal look clearly different. The authors start from an existing representation called residual features: for each patch of an input image, you find the nearest matching feature from a bank of normal reference features and subtract it. What remains is a "residual" that highlights how the patch deviates from normal — and the authors argue this makes it more generalizable across product categories than raw backbone features.

Pretraining happens on RealIAD, a large anomaly detection dataset containing a total of 151050 images, of which 99721 are normal and 51329 are abnormal. Each image is augmented naturally (random color jitters, random gray scale, random Gaussian blur).

Two losses shape the learned features:

• Angle-oriented contrastive loss. A modified InfoNCE loss that only treats features of a different label (normal vs. abnormal) as negatives, so the learning signal is always normal-versus-abnormal. It also measures angles relative to the center of all normal features rather than the origin, which the authors say keeps useful distance information while enlarging angles between the classes. Feature labels come from downsampling each RealIAD sample's ground-truth mask.
• Norm-oriented contrastive loss. Normal features are pulled inside a hypersphere of radius r = 0.4 (using a pseudo-Huber distance as the norm), with a loss that automatically gives larger gradients to features still outside the sphere. Abnormal features inside a larger radius r′ = r + Δr are pushed out, with Δr set to 0.75. Abnormal features already outside are not pushed further, to avoid overfitting to anomalies.

The total loss adds the angle loss (weighted by λ, set to 1 by default) and the norm loss.

Because the raw backbone is kept frozen, a Feature Projector transforms the residual features into the space where these losses apply. It is Transformer-based but replaces self-attention with learnable key/value attention: input features act as queries, and a set of randomly initialized, learnable reference representations (N_r = 2048) act as keys and values, so attention outputs mainly carry normal patterns. Unlike a standard residual connection that adds the attention output, the module subtracts it, aiming to suppress normal patterns and increase the gap between normal and abnormal residual features.

Training uses the Adam optimizer with a learning rate of 1e-4, batch size 32, and 10 epochs. The temperature τ is 0.15, and the random seed is 42. Each image is resized and cropped to 224×224, and each input is matched with 8 reference samples when building residual features. One Feature Projector is built per layer, with the number of layers set to 1.

Evaluation keeps the protocol simple and uniform: the authors take five embedding-based AD methods (PaDiM, PatchCore, CFLOW, GLASS-h, UniAD) and simply swap their original features for the pretrained ones, reproducing each method with official code and default hyperparameters without tuning. Performance is measured by image-level AUROC and PRO (Per-Region-Overlap), the latter chosen because pixel-level AUROC can overestimate localization quality on small anomalies. Datasets are MVTecAD, VisA, BTAD, MVTec3D, and MPDD.

Why This Matters

The paper reframes a widely shared assumption: that an AD method's ceiling is largely set by generic ImageNet-pretrained features. It argues that pretraining objectives have so far been mismatched to the anomaly detection goal, and shows that a purpose-built pretraining stage can lift many methods at once, without changing their architectures or hyperparameters. It also demonstrates a distinctive property — that feature norms alone become a usable anomaly score — which simplifies downstream deployment.

Real-world applications:

• Manufacturing quality control, where the benchmark datasets (MVTecAD, VisA, BTAD, MVTec3D, MPDD) are drawn from real industrial inspection settings covering various object and texture categories.
• 3D surface and geometry inspection, relevant to MVTec3D, where some baseline methods report the lowest original-feature scores (for example FeatureNorm at 49.0/54.6 and 47.4/53.5 with the DINOv2 backbones).
• Cross-category production lines, where a factory runs many product types and cannot retrain a bespoke feature extractor per product — the residual-feature design targets this class-shift problem.
• Few-shot and low-data inspection setups, where new defect types are rare; the paper explicitly notes the pretrained features are good representations for few-shot anomaly detection when using feature norms as scores.

Industry relevance: because the method only changes the feature extractor and requires no modification to existing AD pipelines, it is a drop-in upgrade. The authors' evaluation protocol — fixed code, default hyperparameters, no tuning — mirrors how practitioners would actually adopt a pretrained checkpoint. The authors provide code at https://github.com/xcyao00/ADPretrain.

Future Directions

• Extending pretraining beyond RealIAD and the five backbones tested. The paper pretrains on RealIAD and evaluates five backbones and five downstream datasets; whether additional or larger anomaly pretraining corpora scale the benefits is left open.
• Closing remaining negative deltas. A small number of method/backbone/dataset combinations still regress; understanding when pretrained AD features hurt downstream methods is an unresolved question.
• Better projectors than a single Transformer layer. The Feature Projector has one layer, and the paper notes vanilla Transformer attention performs poorly and needed replacement with learnable key/value attention; more capable projector designs are unexplored here.
• Richer downstream uses. The authors highlight few-shot AD via feature norms as a side benefit; formalizing that setting and other zero/few-shot uses of the pretrained representations is a natural extension.
• Explicit differences from, and integration with, feature fine-tuning lines of work (PaDiM/PatchCore-style embedding methods, PANDA, mean-shifted contrastive learning, FYD, Gaussian fine-tuning) are deferred to Appendix B.1/B.4 in the paper, and comparing pretraining against those adaptation strategies is a natural next question.

Target Audience

Researchers and practitioners working on industrial anomaly detection, self-supervised and contrastive representation learning, and transfer learning for visual inspection. It is most useful to readers who already understand embedding-based AD methods (PaDiM, PatchCore, CFLOW, GLASS, UniAD) and contrastive losses, since the paper's main claim is about the quality of the feature extractor rather than a new detection architecture. Engineers deploying inspection systems will also find value in the drop-in replacement results and the feature-norm baseline.

Authors’ abstract

The current mainstream and state-of-the-art anomaly detection (AD) methods are substantially established on pretrained feature networks yielded by ImageNet pretraining. However, regardless of supervised or self-supervised pretraining, the pretraining process on ImageNet does not match the goal of anomaly detection (i.e., pretraining in natural images doesn't aim to distinguish between normal and abnormal). Moreover, natural images and industrial image data in AD scenarios typically have the distribution shift. The two issues can cause ImageNet-pretrained features to be suboptimal for AD tasks. To further promote the development of the AD field, pretrained representations specially for AD tasks are eager and very valuable. To this end, we propose a novel AD representation learning framework specially designed for learning robust and discriminative pretrained representations for industrial anomaly detection. Specifically, closely surrounding the goal of anomaly detection (i.e., focus on discrepancies between normals and anomalies), we propose angle- and norm-oriented contrastive losses to maximize the angle size and norm difference between normal and abnormal features simultaneously. To avoid the distribution shift from natural images to AD images, our pretraining is performed on a large-scale AD dataset, RealIAD. To further alleviate the potential shift between pretraining data and downstream AD datasets, we learn the pretrained AD representations based on the class-generalizable representation, residual features. For evaluation, based on five embedding-based AD methods, we simply replace their original features with our pretrained representations. Extensive experiments on five AD datasets and five backbones consistently show the superiority of our pretrained features. The code is available at https://github.com/xcyao00/ADPretrain.