ClusterMine: Robust Label-Free Visual Out-Of-Distribution Detection via Concept Mining from Text Corpora

AI summary

Overview

Research area: Computer vision, specifically large-scale visual out-of-distribution (OOD) detection with vision-language models such as CLIP.

Technical level: Advanced. The paper assumes familiarity with CLIP-style image-text embeddings, zero-shot inference, clustering, and OOD benchmarking conventions like AUROC and FPR95.

Scope: The paper proposes a general framework for mining in-distribution concepts from text corpora to remove the need for ground-truth label names in CLIP-based OOD detection, and introduces ClusterMine, a clustering-plus-voting method for positive label mining.

What This Paper Is About

Most CLIP-based OOD detectors assume they already know the in-distribution (ID) ground-truth class names (denoted Y_GT) and use them as "positive" concepts. The authors argue that at scale these fixed label names can be unavailable, unreliable, arbitrarily defined, or stale once the ID distribution drifts after deployment. Their goal is to extract ID-related positive concepts directly from a widely available text corpus, so that OOD detection no longer depends on prior knowledge of the true class names.

Key Contributions

1. A general label mining framework for CLIP OOD detection. The authors formulate OOD detection as a positive/negative concept problem over a text corpus Y_corpus, with an OOD score (Equation 1) computed as the ratio of exponentiated positive image-text similarities to the sum of positive and negative similarities. Existing methods such as Fort et al. and NegLabel become special cases of this framework.

2. ClusterMine, the first positive label mining method. It extracts ID-related concepts from a corpus by combining visual-only sample consistency (clustering) with zero-shot image-text consistency, then applying majority voting within clusters.

3. PosMine, a simpler positive mining baseline. A naive alternative that declares a corpus label name as ID if at least M training samples are assigned to it, with Y_neg = Y_corpus \ Y_pos.

4. A large-scale, multi-axis experimental study. Evaluation spans six OOD datasets, five covariate-shifted ID variants, 12 pretrained CLIP weights, and multiple text corpora (WordNet subsets, ImageNet-21K, and POS Taggings).

Main Findings

• State-of-the-art without ground-truth labels: ClusterMine is the first method to achieve state-of-the-art OOD detection on most ImageNet benchmarks without access to Y_GT. Using CLIP ViT-H dfn5b and the WordNet corpus, it reaches an average AUROC/FPR95 of 94.56/25.26 across the six OOD datasets, versus 92.46/35.22 for MCM, 92.27/35.31 for NegLabel*, and 94.45/27.29 for PLP (which uses Y_GT).

• Strongest on the most curated benchmarks: ClusterMine outperforms on NINCO (92.87 AUROC / 30.3 FPR95), IN-O (93.57 / 29.4), and IN-OOD (91.53 / 38.26) — the benchmarks with the least semantic overlap with ImageNet.

• Scales with model size and pretraining scale: Across 12 publicly available CLIP weights, ClusterMine consistently outperforms NegLabel and MCM by sizable AUROC margins. The authors state the ability of CLIP to capture ID-related concepts does not depend on scale.

• Robust to covariate ID shifts: ClusterMine outperforms NegLabel on 4 out of 6 ID shifts. It is the most robust method for pixel-level perturbations on ImageNet-C (88.94 AUROC / 49.21 FPR95, versus 86.33 / 54.87 for NegLabel*). NegLabel shows slightly higher AUROC on Sketches and ImageNet-R, but ClusterMine is superior on FPR95 there (e.g., Sketches 30.94 vs. 40.29).

• Insensitive to the number of clusters: ClusterMine's performance is largely insensitive to C because of the cluster voting step, whereas PosMine's M and NegLabel's negative-label count require careful tuning. |Y_pos| <= C, and different clusters can map to the same label name.

• Better mined label quality: ClusterMine achieves a superior F1 score against Y_GT for all text corpora (e.g., 71.1 on IN21K versus 57.9 for PosMine), attributed to fewer false positives from cluster voting. Cluster purity remains above 50%, and the redundancy ratio (mined labels appearing across multiple clusters) increases with C.

• More positives alone is not the reason: Holding |Y_pos| equal (1000 or 1500), ClusterMine with C=1.2K reaches 93.84/30.74 and with C=4K reaches 94.56/25.26, while MCM with augmented positives reaches 92.46/35.22 and Equation 1 with augmented positives reaches 93.08/32.00. Using Y_GT as an anchor did not improve OOD detection.

• Competitive on near-OOD with ViT-B: In the OpenOOD near/far comparison, ClusterMine reaches mean near-OOD AUROC 80.3 and far-OOD 89.6, versus 76.7/96.4 for AdaNeg and 77.6/96.2 for SynOOD — the baselines use additional requirements such as sequential access to OOD samples or synthetic sample generation and training.

• Negative label mining matters: Pruning negative labels from the corpus deteriorates AUROC for both ViT-B and ViT-H dfn5b, and the authors report no performance gains from negative grouping, so it is not used.

• Corpus and preprocessing sensitivity: ClusterMine is superior in all but one corpus tested, including POS with roughly 270K concepts, where it remains competitive with PLP. Manually adding the missing Y_GT to POS corpora improves ClusterMine's AUROC, suggesting domain-specific concepts are beneficial. Text preprocessing choices (duplicates, lemmas) had a minuscule impact.

• Benchmark caveat: The Places dataset has the highest semantic overlap with the ID (approximately 60%), and MCM is the best method there and the worst on NINCOv2, which has near-zero overlap.

Methodology in Plain English

The authors start from a large list of possible label names — a text corpus such as WordNet (about 79,000 nouns and adjectives) or POS Taggings (about 270,000 concepts). CLIP encodes this corpus into text vectors. The question is: which of these names actually describe the ID images?

PosMine answers this simply: run zero-shot inference of every image against the whole corpus, count how many training images land on each corpus label, and keep names that receive at least M samples. Everything else becomes a negative concept.

ClusterMine adds a visual consistency check. First, images are clustered in CLIP's visual feature space using TEMI clustering with C clusters (defaults: β=0.6, 50 heads); the goal is a rough overestimation of C, not an accurate cluster count. Second, zero-shot inference against the corpus is applied within each cluster. Third, each cluster's label is decided by majority voting. Because voting suppresses one-off matches and because several clusters can vote for the same label, the resulting positive set |Y_pos| is bounded by C and is robust to inflating C.

Once positives are mined, negatives are taken as the remaining corpus entries (Equation 2), optionally pruned to the K most dissimilar text representations from the positives. The OOD score is then the softmax-style ratio in Equation 1. The approach is training-free with respect to CLIP parameters, so image and text features are precomputed. Defaults are C=4000, M=100, K=4·10^4 (denoted NegLabel*, as opposed to the 10^4 used in NegLabel), and temperature set 10 times smaller than each model's pretraining temperature. Runtime is reported as under 1.5 hours for clustering ImageNet with ViT-G on a single GPU using less than 10 GB of VRAM, and at most twelve minutes for all six OOD benchmarks with less than 16 GB of VRAM at mini-batch size 4096.

Why This Matters

Impact on research. The paper challenges a hidden assumption in large-scale OOD detection: that the ID class names are known and correct. It shows that CLIP can dynamically extract ID concepts from a corpus, that existing methods can be reinterpreted as special cases of a single label-mining framework, and it raises questions about widely used benchmarks — citing prior work estimating roughly 60% semantic overlap in Places relative to ImageNet. It also reports that covariate-shift robustness, not semantic-shift detection alone, deserves more attention.

Real-world applications (as motivated by the paper's setting of deployment-time shifts and corpus availability):

• Post-deployment monitoring of classifiers when class semantics drift and the original label list is no longer the right description of the data.
• Domains where curated label names are unavailable or inconsistent, such as automatically curated hashtag-style labels that are underdescriptive.
• Image sources subject to covariate shifts — camera variations, background changes, or style shifts — where a detector should not flag shifted but valid ID samples as OOD.
• Large-scale visual pipelines where fixed taxonomies are impractical and a domain-relevant text corpus is easier to supply than an exact ground-truth label set.

Industry relevance. The method requires no fine-tuning of CLIP, allows precomputed features, needs no OOD validation set for its main hyperparameter, and reports modest GPU memory and runtime requirements. That combination makes it easier to deploy and re-target than methods that need labeled ID names, synthetic outlier generation, or sequential access to OOD samples.

Future Directions

1. Generating corpora where none exist. The authors suggest using a large language model with a human in the loop to produce domain-specific corpora for application domains without a pre-existing text corpus, reducing dependence on existing corpora.

2. Dynamic negative mining. Test-time adaptation of negative concepts as OOD samples arrive (in the style of AdaNeg) can be integrated into the presented framework and is left for future work.

3. Label-free heuristics for choosing C. The redundancy ratio and the ratio |Y_pos|/C saturate as C grows and could serve as guidelines via the elbow method in new domains.

4. Better benchmark validation. The authors propose that Places and NINCOv2 could be used as OOD validation sets in future work, given their opposite behaviors with MCM.

Target Audience

Researchers and practitioners working on OOD detection, distribution shift robustness, and vision-language models. It is most useful to readers already comfortable with CLIP embeddings, AUROC/FPR95 evaluation, and large-scale ImageNet-based benchmarks. Practitioners deploying CLIP-based detectors who cannot rely on a stable set of ID class names will also find the training-free pipeline and reported runtime and memory figures relevant.

Authors’ abstract

Large-scale visual out-of-distribution (OOD) detection has witnessed remarkable progress by leveraging vision-language models such as CLIP. However, a significant limitation of current methods is their reliance on a pre-defined set of in-distribution (ID) ground-truth label names (positives). These fixed label names can be unavailable, unreliable at scale, or become less relevant due to in-distribution shifts after deployment. Towards truly unsupervised OOD detection, we utilize widely available text corpora for positive label mining, bypassing the need for positives. In this paper, we utilize widely available text corpora for positive label mining under a general concept mining paradigm. Within this framework, we propose ClusterMine, a novel positive label mining method. ClusterMine is the first method to achieve state-of-the-art OOD detection performance without access to positive labels. It extracts positive concepts from a large text corpus by combining visual-only sample consistency (via clustering) and zero-shot image-text consistency. Our experimental study reveals that ClusterMine is scalable across a plethora of CLIP models and achieves state-of-the-art robustness to covariate in-distribution shifts. The code is available at https://github.com/HHU-MMBS/clustermine_wacv_official.