Revisiting Logit Distributions for Reliable Out-of-Distribution Detection

AI summary

Overview

Research area: Out-of-distribution (OOD) detection in computer vision, specifically post-hoc scoring functions for pre-trained models (vision-language models such as CLIP and vision-only classifiers).

Technical level: Intermediate. Readers should be comfortable with classification logits, softmax normalization, temperature scaling, and OOD evaluation metrics (FPR95, AUROC, AUPR).

Scope: The paper proposes and evaluates LogitGap, a training-free post-hoc OOD score that quantifies the average gap between a sample's maximum logit and its remaining logits, together with a truncated top-N variant and a strategy for choosing N.

What This Paper Is About

Models deployed in open-world settings see inputs that belong to none of their training classes, and these should be detected and rejected rather than classified confidently. Existing post-hoc scorers either ignore the non-maximum logits entirely (MaxLogit) or compress them through a softmax that discards absolute logit magnitudes (MCM). The paper's goal is a scoring function that explicitly exploits the relationship between the largest logit and the rest of the logit vector, so that in-distribution (ID) and OOD samples become more separable without any retraining.

Key Contributions

1. A new post-hoc scoring function, LogitGap. It is defined as the average difference between the largest logit and all other logits, using the entire logit space rather than only the maximum or its softmax probability.
2. A theoretical link between LogitGap and MCM. The paper proves (Theorem 4.1) that when the softmax temperature satisfies τ > 2(K−1) for a K-way task, the false positive rate of LogitGap is bounded by that of MCM, and it explains the result through information preservation and temperature sensitivity.
3. A training-free logit-selection strategy. Rather than scoring over all K classes, LogitGap-topN uses only the top N logits, with N chosen to maximize the difference between the mean top-N logits of ID and OOD samples, estimated using no more than 100 ID validation samples plus synthetic OOD data.
4. Broad empirical validation. Experiments cover zero-shot and few-shot CLIP-based detection, semantically hard OOD, covariate-shift OOD, and traditional OOD settings with models trained from scratch using cross-entropy loss.

Main Findings

• ID and OOD logits differ systematically. ID samples produce sharper, more peaked logit distributions with higher maximum logits, while OOD samples produce flatter distributions with elevated non-maximum logits. On CLIP ViT-B/16 with ImageNet-100 as ID and iNaturalist as OOD, this produces a consistently larger "logit gap" for ID data.
• Zero-shot gains over MCM. With ImageNet as the ID dataset, LogitGap reduces FPR95 by 3.65% compared to MCM; with ImageNet-100 as ID, the improvement reaches 5.78%. The reported average numbers on ImageNet/CLIP ViT-B/16 are FPR95 76.83, AUROC 76.43, AUPR 96.37 for LogitGap, versus 79.67, 73.59, 95.87 for MCM.
• Adaptive N helps, but not always. LogitGap* (adaptive N) typically improves over fixed-N LogitGap. In the ImageNet-100 results, LogitGap and LogitGap* are identical because the parameter search returned 20% of the number of categories, which is the fixed default.
• Complementary to few-shot methods. On ImageNet as ID, combining LogitGap with ID-Like improves FPR95 and AUROC by 7.39% and 7.08% respectively under the one-shot setting.
• Strong on semantically hard OOD. Alternating ImageNet-10 and ImageNet-20 as ID/OOD, LogitGap reduces FPR95 by 3.40% versus MCM when ImageNet-10 is ID, and by 8.40% in the reverse configuration. Observed values are FPR95 3.80 / AUROC 99.05 / AUPR 98.52 and FPR95 14.00 / AUROC 98.14 / AUPR 99.15.
• Robust under covariate shift. With ImageNet-R, ImageNet-A and ImageNet-Sketch as OOD, LogitGap reduces FPR95 by 2.20% (one-shot) and 1.17% (four-shot) for ID-Like.
• More shots are not always better. ID-Like shows higher FPR95 in four-shot than one-shot training in the covariate-shift setting; the authors attribute this to overconfident predictions from the strongly generalizing fine-tuned CLIP model.
• N has a broad stable range. Performance degrades when N is too small (too little discriminative information) or too large (noisy tail logits). LogitGap is stable and strong when N is between 20% and 50% of the number of classes.
• Nonlinear variants behave similarly. LogitGap_exp, LogitGap_square and LogitGap_sqrt perform comparably to the original LogitGap, with LogitGap_square giving the best reported average on ImageNet zero-shot (FPR95 74.08, AUROC 79.77, AUPR 94.15) versus LogitGap (74.92 / 79.41 / 93.87).
• Theory condition. The guarantee that LogitGap's FPR is no worse than MCM's holds when τ > 2(K−1); the proof is given in Appendix A.

Methodology in Plain English

The authors start by inspecting how logits are distributed for ID and OOD inputs. They sort each sample's logits in descending order and notice that ID samples have one clearly dominant logit with lower values elsewhere, whereas OOD samples look flatter. From this they define a score: subtract the average of the non-maximum logits from the maximum logit. Larger scores indicate ID-like peakedness.

Because the tail of the sorted logits (classes unrelated to the input) overlaps heavily between ID and OOD and adds noise, they truncate the computation to the top N logits, which makes the score equal to the top logit minus the mean of logits ranked second through N. To pick N without real OOD data, they use at most 100 ID validation samples, synthesize OOD-like samples by interpolating ID features and injecting random noise, and select the N that maximizes the gap between the mean ID score and mean synthetic-OOD score.

They then compare the method against MaxLogit, MCM, Energy, GL-MCM and TAG, plus the few-shot methods CoOp and ID-Like, on CLIP ViT-B/16 with ImageNet or ImageNet-100 as ID and NINCO, ImageNet-OOD, ImageNet-O, ImageNet-10/20, ImageNet-R, ImageNet-A and ImageNet-Sketch as OOD. Three metrics are used throughout: FPR95, AUROC and AUPR.

Why This Matters

Post-hoc OOD detection is attractive because it needs no retraining and no access to OOD data, so it can be dropped onto an already-deployed model. This paper shows that a large amount of discriminative signal sits in the non-maximum logits and is partly destroyed by softmax normalization, which is directly relevant to anyone building rejection or abstention behavior on top of frozen foundation models.

Real-world applications include:

• Autonomous driving, where perception systems encounter objects outside their training taxonomy and should flag uncertainty rather than mislabel them.
• Medical imaging, where scans from unlearned pathologies or atypical acquisition conditions must be routed to a human reader instead of being classified into a known disease category.
• Access control and biometrics, where spoofed or unenrolled identities should be rejected rather than matched to the nearest enrolled class.
• Open-world content moderation or product tagging, where inputs frequently fall outside the predefined label set.

Industry relevance: the method is training-free, adds only a sorting and averaging step on top of logits, requires at most 100 ID samples for its adaptive variant, and works as a drop-in complement to existing few-shot prompt-tuning pipelines.

Future Directions

• Selecting N without any ID data. The current strategy still assumes a small ID validation set and synthetic OOD, so a fully data-free selection rule remains open.
• Beyond a single global N. The paper uses one N for the whole task; per-sample or per-class N selection is not explored.
• Understanding when more few-shot examples hurt. The observed FPR95 degradation from one-shot to four-shot in ID-Like is attributed to overconfidence but not resolved.
• Traditional cross-entropy-trained models. The paper states it evaluates LogitGap in the traditional OOD setting with models trained from scratch using cross-entropy loss, but the detailed results of that section were not available in the provided content, so the size of those gains cannot be reported here.

Target Audience

Researchers and practitioners working on OOD detection, model reliability, and safe deployment of vision-language or vision-only classifiers. It is also useful for engineers who want a cheap post-hoc confidence score for a frozen model, and for students who want a compact example of connecting an empirical logit observation to a formal comparison between scoring functions.

Authors’ abstract

Out-of-distribution (OOD) detection is critical for ensuring the reliability of deep learning models in open-world applications. While post-hoc methods are favored for their efficiency and ease of deployment, existing approaches often underexploit the rich information embedded in the model's logits space. In this paper, we propose LogitGap, a novel post-hoc OOD detection method that explicitly exploits the relationship between the maximum logit and the remaining logits to enhance the separability between in-distribution (ID) and OOD samples. To further improve its effectiveness, we refine LogitGap by focusing on a more compact and informative subset of the logit space. Specifically, we introduce a training-free strategy that automatically identifies the most informative logits for scoring. We provide both theoretical analysis and empirical evidence to validate the effectiveness of our approach. Extensive experiments on both vision-language and vision-only models demonstrate that LogitGap consistently achieves state-of-the-art performance across diverse OOD detection scenarios and benchmarks. Code is available at https://github.com/GIT-LJc/LogitGap.