Cross-modal Proxy Evolving for OOD Detection with Vision-Language Models

AI summary

Authors: Hao Tang, Yu Liu, Shuanglin Yan, Fei Shen, Shengfeng He, Jing Qin arXiv: 2601.08476v2 [cs.CV], 01 Apr 2026 Code: https://github.com/yuleoliu/CoEvo

Overview

• Research area: Computer vision — zero-shot out-of-distribution (OOD) detection using vision-language models (VLMs), specifically CLIP.
• Technical level: Advanced. The paper assumes familiarity with CLIP-style contrastive embeddings, cosine similarity scoring, AUROC/FPR95 metrics, and test-time adaptation.
• Scope (one sentence): The paper proposes CoEvo, a training- and annotation-free test-time framework that co-evolves both textual and visual proxy caches so that zero-shot OOD detection stays robust under distribution shift.

What This Paper Is About

Zero-shot OOD detection asks a model to decide whether an image belongs to a known set of classes or comes from something unseen, without any labeled OOD examples for training. Existing "negative-label" methods handle this by comparing an image against a fixed list of textual labels that are semantically unrelated to the known classes, but those lists never change at test time, so they cover the open semantic space sparsely and drift out of alignment with visual features when the test distribution shifts. The paper's goal is to keep both the text-side and image-side proxies continuously aligned with each incoming test sample, so the OOD score remains reliable without retraining or backbone updates.

Key Contributions

1. CoEvo framework. A zero-shot OOD detection framework that builds semantically aligned ID/OOD proxy caches at test time by jointly using visual and textual modalities — with no training and no annotation.
2. Proxy-aligned co-evolution mechanism. A sample-conditioned, bidirectional adaptation scheme: images guide the mining of new contextual textual negatives, and the updated textual proxies in turn reshape the visual decision boundary in a closed loop.
3. Calibrated dual-modal OOD score. A dynamically re-weighted fusion of the two modality scores, with a "cold-start asymmetry" in which textual proxies dominate early (λ ∈ [0.5, 1)) and visual proxies dominate after evolution (weight flipped to (1−λ) on text and λ on visuals).
4. State-of-the-art empirical results. On ImageNet-1K, the paper reports a 1.33% AUROC improvement and a 45.98% FPR95 reduction over strong negative-label baselines.

Main Findings

• ImageNet-1K main benchmark. CoEvo NegLabel reaches an average FPR95 of 10.22% and AUROC of 97.95% across iNaturalist, SUN, Places and Textures; CoEvo CSP reaches 10.94% FPR95 and 97.85% AUROC. These beat the closest baseline, AdaNeg (18.92% FPR95, 96.66% AUROC).
• Per-dataset wins. CoEvo NegLabel scores 99.81 / 0.53 (AUROC / FPR95) on iNaturalist, 98.68 / 4.42 on SUN, 95.80 / 23.51 on Places, and 97.48 / 12.42 on Textures. All methods in this table use a CLIP ViT-B/16 encoder.
• OpenOOD benchmark. Under Far-OOD, CoEvo CSP achieves 14.47% FPR95 and 96.70% AUROC, and the framework delivers the best average ID accuracy among training-free methods at 67.36%. Under Near-OOD the picture is mixed: CoEvo CSP obtains 66.88% FPR95 and 74.65% AUROC and slightly underperforms AdaNeg on AUROC, which the authors attribute to marginally reduced sensitivity to fine-grained OOD discrimination.
• Ablation — both modalities matter. Starting from NegLabel with no evolution (24.97% FPR95, 94.56% AUROC): textual evolution alone gives 21.77% / 95.38%; visual evolution alone gives 17.41% / 96.99%; combining both gives the best result at 10.22% / 97.95%.
• Score evolution beats static weighting. Flipping the fusion weights after proxy evolution (Eq. 11) outperforms retaining the pre-evolution weighting, with the gap peaking around λ = 0.8. In the low-λ regime the flipped rule favors stable textual cues; in the high-λ regime it favors the now-enriched visual proxies.
• Hyperparameter λ. Performance on ImageNet-1K rises with λ, peaks at λ = 0.8, then degrades slightly as λ approaches 1.0.
• Hyperparameter Top-N. Increasing N initially helps by injecting richer negatives, then saturates and can decline due to redundancy and noisier candidates; the authors set N = 5.
• Robustness to data imbalance. Using ImageNet-1K as ID and SUN as OOD, CoEvo NegLabel achieves FPR95 of 17.00 at 1:100, 6.70 at 1:10, 5.27 at 1:1, 5.76 at 10:1, and 14.77 at 100:1. CoEvo CSP scores 14.00, 7.50, 5.58, 6.15 and 15.38 for the same ratios.

Methodology in Plain English

The method keeps two "memory banks" — one for text, one for images — and updates them on the fly as test images arrive.

1. Setting up the text bank. Known class names are turned into CLIP text embeddings to form a positive queue that stays fixed, since the identity of known classes does not change. A negative queue is initialized from a large lexical corpus (following NegLabel, using 10K OOD class names, or CSP's 9,493), and unlike the positive queue this one is allowed to grow during inference.
2. Setting up the image bank. A positive visual queue holds up to L = 10 stored instances per known class, with the first slot seeded from the corresponding text embedding so the queue is meaningful before any images arrive. A negative visual queue mirrors the negative text queue and grows as OOD-looking images are seen, using a priority-queue rule that discards low-similarity or outdated entries.
3. Scoring an image. For a test image, cosine similarities against the positive and negative anchors are converted into a softmax-style ratio (temperature τ = 0.01), producing a textual OOD score and a visual OOD score. The visual score aggregates the stored instances per class through similarity-based attention controlled by β = 5.5. The two scores are fused into a preliminary multi-modal score using λ = 0.8.
4. Deciding whether to learn from this sample. An adaptive threshold δ is used with a margin γ: samples scoring above δ + γ(1−δ) are treated as ID, samples below δ − γ(1−δ) as OOD, and anything in between is skipped as too ambiguous.
5. Evolving the text bank. For an OOD-looking sample, the system retrieves the N = 5 closest unused entries from the corpus (near negatives) and appends them, tightening the boundary locally. For an ID-looking sample, it appends the 5 farthest entries, broadening coverage of the negative space. Only the negative queue is updated; the positive text queue stays fixed to avoid drift. This gating strategy follows AdaND (see the paper's Appendix A, not included in the provided content).
6. Evolving the image bank. The negative visual queue is expanded from ℝ^(M×L×D) to ℝ^((M+N)×L×D) to match the newly added text negatives. Incoming embeddings are assigned by softmax similarity, and when a slot is full, the stored exemplar with the highest entropy — i.e., the least confident one — is replaced only if the new sample is more confident, using entropy H(z) = −Σ z_i log z_i.
7. Re-scoring with flipped weights. After evolution, the unimodal scores are recomputed and fused with the opposite weighting: (1−λ) on the updated text score and λ on the updated visual score, because visual proxies are by then richer and locally sharper. In the default configuration, the preliminary score drives proxy updates while the post-evolution score drives the final ID/OOD decision.

All experiments use a CLIP ViT-B/16 backbone, batch size 128, the prompt template "The nice cls", and a single NVIDIA RTX 3090 GPU. No backbone parameters are updated at any point.

Why This Matters

Impact on research. The paper reframes zero-shot OOD detection as a bidirectional alignment problem rather than a one-sided adaptation problem, showing that letting textual negatives drift with the data — not just visual proxies — produces large empirical gains. It also shows that the relative reliability of the two modalities changes over the course of inference, which motivates the flipped weighting scheme. Both ideas are portable to other test-time adaptation settings built on vision-language models.

Real-world applications:

• Healthcare. The paper explicitly names healthcare as a high-stakes setting where overconfident predictions on unseen inputs carry substantial safety and reliability risk.
• Autonomous driving. Also named in the paper as a domain where rejecting unfamiliar inputs matters for robustness of downstream decisions.
• Open-world image classification and content moderation. Any deployed classifier that meets categories it was never trained on can use this scoring approach to flag and defer uncertain inputs.
• Safety monitoring in deployed vision systems. A training-free test-time method fits scenarios where retraining or collecting labeled OOD data is impractical.

Industry relevance. Because CoEvo requires no training, no prompt tuning, and no annotation, and reports results on a single consumer-grade GPU with a CLIP ViT-B/16 backbone, it is relatively cheap to bolt onto an existing CLIP-based pipeline as a post-hoc detection layer.

Future Directions

1. Closing the Near-OOD AUROC gap. CoEvo CSP slightly trails AdaNeg on Near-OOD AUROC (74.65% vs 76.70%), so improving fine-grained discrimination between semantically close classes and OOD inputs remains open.
2. Extending beyond one backbone. All reported experiments use CLIP ViT-B/16; whether the co-evolution mechanism transfers to other vision-language backbones or larger model scales is not reported and is a natural next test.
3. Controlling proxy growth. The negative text queue grows with every confident update, and the paper notes that a larger Top-N introduces redundancy and weakly aligned candidates. Pruning, merging, or bounding these caches is a practical open problem.
4. Sensitivity to the adaptive threshold. The gating depends on an adaptive threshold δ whose derivation lives in Appendix A (referenced as following AdaND) and is not included in the provided content; robustness of this choice across domains is a logical follow-up.

Target Audience

Researchers and engineers working on out-of-distribution detection, test-time adaptation, or vision-language models, particularly those who need zero-shot detection without labeled negatives. It is also useful for practitioners deploying CLIP-based systems in open-world settings who want a training-free drop-in scoring method. Readers without a background in contrastive vision-language models and OOD evaluation metrics will find the methodological sections dense.

Authors’ abstract

Reliable zero-shot detection of out-of-distribution (OOD) inputs is critical for deploying vision-language models in open-world settings. However, the lack of labeled negatives in zero-shot OOD detection necessitates proxy signals that remain effective under distribution shift. Existing negative-label methods rely on a fixed set of textual proxies, which (i) sparsely sample the semantic space beyond in-distribution (ID) classes and (ii) remain static while only visual features drift, leading to cross-modal misalignment and unstable predictions. In this paper, we propose CoEvo, a training- and annotation-free test-time framework that performs bidirectional, sample-conditioned adaptation of both textual and visual proxies. Specifically, CoEvo introduces a proxy-aligned co-evolution mechanism to maintain two evolving proxy caches, which dynamically mines contextual textual negatives guided by test images and iteratively refines visual proxies, progressively realigning cross-modal similarities and enlarging local OOD margins. Finally, we dynamically re-weight the contributions of dual-modal proxies to obtain a calibrated OOD score that is robust to distribution shift. Extensive experiments on standard benchmarks demonstrate that CoEvo achieves state-of-the-art performance, improving AUROC by 1.33% and reducing FPR95 by 45.98% on ImageNet-1K compared to strong negative-label baselines.