ICONIC-444: A 3.1-Million-Image Dataset for OOD Detection Research

AI summary

Overview

• Research area: Computer vision — out-of-distribution (OOD) detection, dataset construction, and benchmark design for industrial image classification.
• Technical level: Intermediate. The core ideas are accessible without deep background, but the evaluation metrics (FPR95, FPR99, AUROC) and the taxonomy of OOD detection methods assume some familiarity with classification models.
• Scope (one sentence): The paper introduces a 444-class, 3,132,406-image industrial dataset captured with a prototype sorting machine, defines four classification tasks with a structured near/far/extreme/synthetic OOD hierarchy, and benchmarks 22 post-hoc OOD detection methods on it.
• Note on completeness: The paper content supplied to this summary is truncated partway through Section 4.2 ("Analysis"). All later analysis, conclusions, and supplementary details are therefore not covered here, and nothing is inferred or filled in from outside the provided text.

What This Paper Is About

OOD detection research depends on datasets that contain in-distribution (ID) classes for training and clearly separated out-of-distribution classes for testing. The paper argues that the datasets commonly used for this — CIFAR, ImageNet, and fine-grained sets such as FGVC-Aircraft and CUB-200 — were not designed for OOD detection, and suffer from problems such as categorical and incidental contamination, small test sets, low resolution, and no control over how "far" OOD samples are from the ID data.

The authors address this by building ICONIC-444, a large-scale dataset captured with a prototype industrial free-fall sorting machine, and by organizing it into ID tasks of varying granularity plus a deliberately structured OOD hierarchy. Their goal is to give the community a contamination-free benchmark large enough for statistically meaningful evaluation, including an unusually strict metric (FPR99) that smaller datasets cannot support.

Key Contributions

1. ICONIC-444 dataset: A specialized industrial image dataset of 444 classes and 3,132,406 images (described as over 3.1 million), intended for contamination-free OOD detection evaluation. It is organized into two categories (Food with 387 classes in 12 groups, Non-Food with 57 classes in 7 groups) and 19 groups total. It is publicly available at https://github.com/gkrumpl/iconic-444.

2. Four reference ID tasks: Almond (7 classes), Wheat (12 classes), Kernels (29 classes), and Food-grade (324 classes), each paired with task-specific OOD test sets spanning near-, far-, extreme-, and synthetic-OOD. These mirror real-world classification challenges at increasing granularity and scale.

3. A structured OOD hierarchy: OOD samples are explicitly grouped by difficulty rather than pooled from arbitrary external datasets — near-OOD drawn from the Food category (excluding the task's ID classes), far-OOD consistently drawn from the Non-Food category, extreme-OOD drawn from four public datasets (ImageNet, iNaturalist using the curated version from prior work, Places365, and Textures), and synthetic-OOD covering 25 types of simulated failures such as camera failure, data corruption, and transmission errors.

4. A baseline benchmark of 22 post-hoc OOD detection methods across all four tasks and two architectures (ResNet18 and the Compact Transformer, CCT), each trained with three random seeds, using the OpenOOD benchmark library.

Main Findings

• ID classification is strong but calibration differs by architecture: Reported accuracy across the four tasks ranges from 93.18% to 96.38%. Per task (ResNet18 / CCT): Almond 95.53 ± 0.11 / 93.63 ± 0.50; Wheat 94.72 ± 0.06 / 93.77 ± 0.45; Kernels 96.38 ± 0.14 / 95.75 ± 0.20; Food-grade 94.31 ± 0.31 / 93.18 ± 0.04. Expected Calibration Error is markedly lower for the ResNet18 (0.44% to 1.51%) than for CCT (3.10% to 4.75%).

• OOD detection remains unsolved: The paper states that robust OOD detection is still an unsolved challenge even for seemingly simple scenarios such as far- and extreme-OOD, and the benchmark table is consistent with that claim at the level of strict criteria.

• Best average method is GRAM: In the table of average performance per method across three seeds, two architectures and four tasks, GRAM achieves the strongest results in the average column (FPR95 13.98, FPR99 23.47, AUROC 95.51). Methods are sorted by average FPR99, and GRAM also has the best near-OOD numbers (FPR95 35.74, FPR99 54.59, AUROC 88.13) and the best extreme-OOD numbers (FPR95 0.00, FPR99 0.00, AUROC 99.99).

• Near-OOD is by far the hardest category: Even the leading method only reaches AUROC 88.13 on near-OOD with an FPR99 of 54.59, while its far-OOD AUROC is 94.03 and its synthetic-OOD AUROC is 99.88. Several methods collapse on near-OOD — for instance GradNorm reports near-OOD FPR95 of 73.72 and FPR99 of 88.56, and RankFeat reports near-OOD FPR95 of 92.35, FPR99 of 97.69, and AUROC of 50.49.

• Some methods degrade badly overall: RankFeat has the worst average column scores in the table (FPR95 91.94, FPR99 96.60, AUROC 44.05), and ASH-b follows (FPR95 82.77, FPR99 90.87, AUROC 60.09). Score-based approaches such as MSP (FPR95 62.26, FPR99 81.45, AUROC 81.12), TempScale (58.92 / 77.21 / 81.57), and GradNorm (64.84 / 77.15 / 67.12) occupy the weaker end of the ranking in the average column.

• A stricter metric, FPR99, is proposed: Because rejecting 5% of good data is typically unacceptable in real-world applications, the authors add False Positive Rate at 99% True Positive Rate alongside the standard FPR95 and AUROC. They note FPR99 requires a large evaluation dataset to be statistically meaningful and argue ICONIC-444 is the first dataset to provide enough images for such analysis.

• OOD difficulty is visible in feature space: A t-SNE visualization for the Wheat task shows near-OOD samples spanning the largest area and overlapping heavily with ID clusters, while far- and extreme-OOD samples occupy tighter, more distant regions. Synthetic-OOD generally lies further from ID clusters, though some types — such as color-channel-permuted images — remain semantically close to ID data.

• Gap in existing benchmarks documented: The paper reports that NINCO contains 5879 images including 64 classes, and ImageNet-OOD contains 31807 images from 637 classes sourced from ImageNet-21k, which is frequently used to pretrain modern architectures and may therefore inflate measured OOD performance. Neither provides explicit control over ID/OOD granularity.

Methodology in Plain English

The authors chose industrial object sorting as the setting because it naturally combines controlled acquisition with genuine task complexity — sorting machines must reject foreign material while keeping valuable product. They built the dataset with a prototype free-fall sorting machine fitted with a 4K RGB line-scan camera operating at 56 μm/pixel, which exceeds typical human visual resolution even at close range.

Objects travel on a vibrating conveyor onto a chute, then fall freely while the line-scan camera images them mid-air against a blue LED backlight, with two white LED lights illuminating the front. The constant blue background makes automatic detection and cropping straightforward, producing images between 256×256 and 1024×1024 pixels depending on the expected object size for each class. Each image shows a distinct physical object captured only once. Classes were selected for size suitability (millimeters to centimeters), distinctness (unambiguous single-class membership), and bulk availability. Variability comes from natural products, from deliberately induced defects in 53 classes (for example pasta drilled, broken, or dotted with colored dots), and from rotation and falling-speed differences.

Quality control covered three areas: sourcing pure, uncontaminated materials (mostly store-bought food-grade items and grains from suppliers or plant breeders); strict acquisition protocol (one class handled at a time, machine inspected and cleaned after each run); and semi-automatic cleaning that removed objects that were too small, too dark, or touching other objects, supplemented by image classification models run in a cross-validation setup to flag suspicious samples for manual inspection.

For evaluation, images were resized to 256×256 and center-cropped to 224×224. Four ID tasks were trained with ResNet18 and CCT across three random seeds, and 22 post-hoc, sample-free OOD detection methods from the OpenOOD library were scored. For near- and far-OOD sets, 500 samples were randomly selected per class for balance; six classes were removed from the far-OOD set to create an OOD validation set of 10k samples for hyperparameter tuning; and the synthetic set uses 500 generated samples per type. Performance was reported with FPR95, AUROC, and the newly proposed FPR99.

Dataset scale relative to prior work (Table 1, classes / samples): CIFAR-10 10 / 60000, CIFAR-100 100 / 60000, ImageNet-1k 1000 / 1431167, FGVC-Aircraft 102 / 10200, Stanford Cars 196 / 16185, CUB-200 200 / 11788, North American Birds 555 / 48562, MVTec AD 15 / 5354, VisA 12 / 10821, ICONIC-444 444 / 3132406. Task-level ID splits reported in Table 2: Almond 7 classes (9.4k/1.3k/2.7k train/val/test), Wheat 12 classes (83.2k/11.8k/23.7k), Kernels 29 classes (365.3k/52.1k/104.4k), Food-grade 324 classes (291.6k/32.4k/97.2k). OOD set sizes reported are 380 classes (190.0k images) for Almond, 375 (187.5k) for Wheat, 358 (179.0k) for Kernels, and 63 (31.5k) for Food-grade.

Why This Matters

For research, the paper argues that progress in OOD detection is bottlenecked by benchmark quality, not only by method design. By supplying a contamination-free dataset with controlled difficulty levels and 3,132,406 images, it enables evaluation setups — particularly FPR99 — that were not statistically feasible on smaller sets. The authors also note that OOD detection

Authors’ abstract

Current progress in out-of-distribution (OOD) detection is limited by the lack of large, high-quality datasets with clearly defined OOD categories across varying difficulty levels (near- to far-OOD) that support both fine- and coarse-grained computer vision tasks. To address this limitation, we introduce ICONIC-444 (Image Classification and OOD Detection with Numerous Intricate Complexities), a specialized large-scale industrial image dataset containing over 3.1 million RGB images spanning 444 classes tailored for OOD detection research. Captured with a prototype industrial sorting machine, ICONIC-444 closely mimics real-world tasks. It complements existing datasets by offering structured, diverse data suited for rigorous OOD evaluation across a spectrum of task complexities. We define four reference tasks within ICONIC-444 to benchmark and advance OOD detection research and provide baseline results for 22 state-of-the-art post-hoc OOD detection methods.