The Invisible Gorilla Effect in Out-of-distribution Detection

AI summary

Overview

Research area: Out-of-distribution (OOD) detection for deep neural networks in computer vision, with an emphasis on medical imaging and industrial inspection benchmarks.

Technical level: Advanced. The paper assumes familiarity with OOD detection terminology (AUROC, near-OOD, post-hoc vs. ad-hoc methods, Mahalanobis scoring) and uses PCA-based latent-space analysis.

Scope: A large-scale empirical study identifying and characterising a previously unreported bias — the "Invisible Gorilla Effect" — in which OOD detection performance depends on the visual similarity (specifically colour) between an OOD artefact and the model's region of interest (ROI).

What This Paper Is About

OOD detection methods are supposed to flag inputs the model cannot reliably handle, but prior work has shown their performance varies unpredictably across different types of artefacts — without explaining why. This paper identifies a systematic cause: for hard-to-detect (near-OOD) artefacts, detection performance improves when the artefact visually resembles the model's ROI (e.g. colour) and drops when it does not. The authors demonstrate this across 40 OOD methods, 7 benchmarks and 3 architectures, and test two mitigation strategies.

Key Contributions

1. Identification of the Invisible Gorilla Effect. The authors report a previously unreported bias in OOD detection where near-OOD artefacts that share visual similarity with the model's ROI are easier to detect than dissimilar ones, based on an extensive study of 40 OOD detection methods (3795 hyperparameter settings) across 7 benchmarks and 3 network architectures.

2. New colour annotations and counterfactuals. They manually annotated artefact colours in 11,355 images from three public datasets (e.g. ISIC), and generated colour-swapped counterfactuals of colour chart artefacts to rule out dataset bias. Annotations are released on the project's GitHub repository.

3. Method-level characterisation. They show that feature-based OOD methods suffer larger average AUROC drops than confidence-based methods as artefact colour varies, and use PCA-based latent analysis to show these colour shifts align with high-variance directions in the latent space.

4. A nuisance-subspace mitigation. They define a "nuisance subspace" of high-variance, colour-sensitive latent directions (span of k=5 principal components with the highest colour-discrimination scores) and show that projecting features orthogonally to it reduces the Invisible Gorilla Effect for several feature-based methods.

Main Findings

• Headline example: In a skin lesion classifier with a red lesion ROI (mean RGB (176,116,77)), Mahalanobis Score achieves a 31.5% higher AUROC when detecting OOD red ink (similar to ROI) compared to black ink (dissimilar) annotations.

• Performance drops are widespread: Across the ISIC benchmark, most of the 40 evaluated methods showed consistent drops in AUROC when artefacts were visually dissimilar from the ROI. The diffusion-based method DDPM-MSE was the only method that did not exhibit an AUROC drop across all three ISIC benchmarks.

• Feature-based methods are hit hardest: Mean AUROC drop on real ISIC benchmarks was 7.1 (μ) ± 1.8 (σ) pp for feature-based methods, versus 1.5 ± 1.1 pp for confidence-based methods. Ad-hoc methods showed 6.5 ± 6.2 pp and external methods 6.8 ± 7.7 pp. NAC and NuSA were exceptions to the feature-based pattern, though NAC performed poorly across all tasks.

• The effect tracks the ROI, not just the data: In CheXpert, with synthetic square artefacts covering 1% of the image, AUROC dropped by an average of 6.31% when square intensity shifted from +3 to −3 for models trained on hyperintense hearts. When models were retrained on hypointense-heart counterfactuals, the trend reversed, with an average AUROC drop of 4.28% when intensity shifted from −3 to +3.

• Not simply a generalisation trade-off: Classification accuracy and Mahalanobis OOD performance were only weakly correlated (r = 0.39). Accuracy drops on dissimilar artefacts were large (e.g. 20 pp for purple, 9 pp for black), indicating these cases remain clinically significant. A weak positive correlation was also found between a method's AUROC drop and its AUROC on similar-colour artefacts.

• Results replicate on MVTec: On pill and metal nut benchmarks using ResNet18, performance again increased when OOD ink artefacts were visually similar to the ROI, with feature-based methods (e.g. KNN, Δ = 20.64 on the combined S-D column; KDE, Δ = 13.65) generally showing larger drops than confidence-based methods (e.g. WeiPer, Δ = 1.19).

• Colour jitter augmentation is unreliable: Light jitter (brightness/contrast/saturation = 0.2) produced an ID accuracy change of −0.1 pp and heavy jitter (0.8) produced −5.5 pp. Effects were inconsistent: KNN's gap narrowed, CoP's widened, and DICE's trend reversed (light augmentation degraded similar-colour detection by 7.2 pp).

• Nuisance subspace projection works better: Spearman correlation between colour-sensitivity (I_k) and log variance of principal components was positive (ρ = 0.47, p < 1.5 × 10⁻⁴), indicating colour-sensitive directions are high-variance. Projecting features orthogonally to the nuisance subspace narrowed gaps: Mahalanobis went from 77.0 (similar) / 63.6 (dissimilar) to 77.5 / 75.8; FeatureNorm from 75.1 / 52.9 to 75.3 / 74.5; NAN from 75.6 / 48.5 to 75.3 / 76.8.

• Hyperparameter choice matters: Layer selection strongly affects Mahalanobis performance, and the authors note the Invisible Gorilla Effect would not be observed if only a single layer configuration were evaluated.

Methodology in Plain English

The team trained primary classifiers (ResNet18, VGG16, ViT-B/32) on three public datasets — CheXpert chest X-rays, ISIC dermatology images and MVTec-AD industrial images — using 25 random seeds across five repetitions of five-fold cross-validation. Training used AdamW with a learning rate of 1e-4 and weight decay 0.01, cross-entropy loss for 600 optimisation steps, batch size 256, and image augmentations that deliberately excluded colour perturbations so the model's ROI colour stayed intact.

They then assembled OOD test sets under three regimes: semantic shifts (unseen classes), real covariate shifts (naturally occurring artefacts like ink annotations and colour charts), and synthetic covariate shifts (square patches covering 1% of the image area). Artefacts and ROIs were segmented using the Segment Anything Model with interactive prompts, or ground-truth masks for MVTec ink, and mean RGB values were computed. Artefacts were labelled similar or dissimilar to the ROI by linear Euclidean RGB distance, with a benchmark-specific threshold; images containing multiple artefact colours were excluded.

To rule out dataset bias, they created colour-swapped counterfactuals — recolouring charts with per-channel mean shifting while preserving pixel-level variance and texture — changing similar charts (red, orange, yellow) to black (mean RGB (66,61,60)) and dissimilar charts to the average skin lesion colour (176,116,77). They also tested whether the ROI drives the effect by scaling heart-region intensities by one-third in CheXpert to create hypointense hearts and retraining.

For mechanistic insight, they pooled latent features, applied PCA, and measured how well each principal component discriminated similar from dissimilar artefacts, then correlated that with component variance. Finally, they tested two mitigations: colour jitter augmentation during training, and projecting features orthogonally to the nuisance subspace before applying feature-based detectors.

Why This Matters

This work challenges the common assumption that OOD difficulty is a monotonic function of global similarity to the training distribution. It shows that difficulty also depends on what the model attends to and how the detection method interacts with that attention — reframing how researchers should design and evaluate detectors.

Real-world applications:

• Medical imaging: Dermatology (ISIC) and radiology (CheXpert) models may silently fail to flag ink annotations, rulers, colour charts, pacemakers or other artefacts that differ in colour from the clinical region of interest.
• Industrial inspection: MVTec-AD defect detection, where unseen ink artefacts constituted a semantic shift, is affected by the same bias.
• Autonomous driving and other high-risk vision systems, which encounter real-world OOD inputs whose visual properties are not known a priori.
• Regulatory compliance: Recent US and EU guidance has highlighted the need for machine learning systems to handle OOD inputs, and this paper exposes a failure mode that such evaluations may miss.

Industry relevance: Colour jitter augmentation — a common off-the-shelf remedy — is shown not to reliably fix the problem and can cost ID accuracy (−5.5 pp under heavy jitter). The subspace projection mitigation offers a more consistent, targeted alternative for feature-based detectors, and the paper warns that evaluating a single hyperparameter configuration (e.g. one feature layer) can mask the effect entirely, with implications for how OOD benchmarks should be designed.

Future Directions

• Extend beyond colour. The authors focus on colour because it can be varied independently of shape and texture; whether the effect generalises to other similarity axes (shape, texture, brightness beyond the CheXpert intensity experiment) remains an open question.
• Generalise the mitigation. The nuisance subspace projection was computed on the ISIC colour-chart benchmark and evaluated on the ISIC ink benchmark; its transferability across datasets, architectures and artefact types needs further testing, along with ablation of the number of principal components (k=5 was used here).
• Explain confidence-based methods. Feature-based methods showed larger drops, but the mechanisms behind the smaller yet non-zero drops in confidence-based and external methods are less explored, including why DDPM-MSE uniquely showed no drop on all three ISIC benchmarks.
• Include foundation models. The authors deliberately excluded CLIP-style approaches to avoid data leakage and confounding pre-training effects; whether such models exhibit the Invisible Gorilla Effect under controlled conditions is left open.

Target Audience

Researchers and practitioners in OOD detection, distribution-shift robustness and safety-critical computer vision who want to understand why detection performance varies across artefact types. It is especially useful for those designing evaluation protocols for medical imaging or industrial inspection systems, for engineers choosing among 40 detector variants and their hyperparameters, and for anyone considering colour augmentation or feature-projection as robustness interventions. Some familiarity with OOD metrics (AUROC, AUCPR, FPR@95) and latent-space analysis is assumed.

Authors’ abstract

Deep Neural Networks achieve high performance in vision tasks by learning features from regions of interest (ROI) within images, but their performance degrades when deployed on out-of-distribution (OOD) data that differs from training data. This challenge has led to OOD detection methods that aim to identify and reject unreliable predictions. Although prior work shows that OOD detection performance varies by artefact type, the underlying causes remain underexplored. To this end, we identify a previously unreported bias in OOD detection: for hard-to-detect artefacts (near-OOD), detection performance typically improves when the artefact shares visual similarity (e.g. colour) with the model's ROI and drops when it does not - a phenomenon we term the Invisible Gorilla Effect. For example, in a skin lesion classifier with red lesion ROI, we show the method Mahalanobis Score achieves a 31.5% higher AUROC when detecting OOD red ink (similar to ROI) compared to black ink (dissimilar) annotations. We annotated artefacts by colour in 11,355 images from three public datasets (e.g. ISIC) and generated colour-swapped counterfactuals to rule out dataset bias. We then evaluated 40 OOD methods across 7 benchmarks and found significant performance drops for most methods when artefacts differed from the ROI. Our findings highlight an overlooked failure mode in OOD detection and provide guidance for more robust detectors. Code and annotations are available at: https://github.com/HarryAnthony/Invisible_Gorilla_Effect.