Optimal Transport-Induced Samples against Out-of-Distribution Overconfidence

AI summary

Overview

Research area: Computer vision and trustworthy deep learning — specifically out-of-distribution (OOD) robustness and confidence calibration for deep neural network classifiers, approached through the mathematics of optimal transport (OT).

Technical level: Advanced. The paper assumes familiarity with semi-discrete optimal transport, Brenier's theorem, convex potentials, Laguerre cells/power diagrams, transport singularities, and standard OOD evaluation protocols.

Scope (one sentence): The paper proposes generating proxy OOD training samples from the singular boundaries of a semi-discrete optimal transport map computed in an autoencoder latent space, and using a confidence suppression loss on those samples to reduce overconfident predictions on unfamiliar inputs.

What This Paper Is About

Deep neural network classifiers trained under a closed-world assumption tend to give high-confidence predictions on inputs that do not belong to the training distribution, which is dangerous in open-world deployment. Existing fixes either filter out such inputs after training (test-time OOD detection) or train on heuristically generated proxy OOD samples that have no theoretical reason to sit where overconfidence actually occurs. This paper argues that the singular boundaries of a semi-discrete optimal transport map mark regions of semantic ambiguity, and constructs training samples ("optimal transport-induced OOD samples", or OTIS) near those boundaries to suppress overconfidence.

Key Contributions

1. A theoretical link between geometric singularities in semi-discrete optimal transport and the emergence of overconfident predictions on OOD inputs. The authors state that the Brenier potential becomes non-differentiable on a singular set when the target measure is multimodal or disconnected, and that the transport map is discontinuous across that set.
2. A new OOD overconfidence mitigation framework that regularizes model confidence using semantically ambiguous samples derived from OT-induced singularities. The pipeline encodes ID data into a latent space, solves a semi-discrete OT problem against a continuous Gaussian or uniform base distribution, identifies high-angular-deviation boundaries, interpolates between adjacent cell centroids, and decodes back to image space.
3. Empirical validation across multiple architectures and ID/OOD settings, reported as outperforming state-of-the-art baselines in mitigating OOD overconfidence without sacrificing ID accuracy.
4. A set of analyses covering the choice of singularity ranking, comparison against latent- and image-level interpolation, compatibility with test-time OOD detection methods (MSP, ODIN, ReAct), ID calibration (ECE), sample-set fidelity (FID), base distribution choice, and autoencoder depth.

Main Findings

• Low-resolution benchmarks: Across CIFAR-10, CIFAR-100, SVHN, MNIST, and FMNIST as ID datasets, the method reports the lowest OOD mean maximum confidence (MMC) in nearly all settings. The paper highlights 13.18% OOD MMC on SVHN (CIFAR-10 as ID) and 9.30% OOD MMC on SVHN (CIFAR-100 as ID), lower than baselines including OE and CCUd, which use large auxiliary datasets.
• Near-floor suppression on simple shifts: The paper reports near-optimal suppression values such as 10.00% on Noise and Uniform samples for several ID/OOD configurations in Table 1. The corresponding table entries for "Ours" include CIFAR-10 on Uniform (10.00%), CIFAR-10 on Adv. Noise (10.00%), and FMNIST on Uniform (10.00%).
• Adversarial shifts: The method reports 8.94% OOD MMC on CIFAR-100 (Adv. Noise, with CIFAR-100 as ID) and 15.09% OOD MMC on FMNIST (Adv. Samples, with FMNIST as ID), without handcrafted augmentations or external supervision.
• Confidence histograms (Figure 3): For ResNet-18 trained on CIFAR-10 and CIFAR-100, the distribution of maximum softmax confidence on OOD inputs shifts toward lower values compared with the baseline, reducing the prevalence of high-confidence predictions.
• ImageNet results (Table 2): With ResNet-50, the method attains the highest ID MMC (88.93%) and reduces OOD confidence, with the lowest OOD MMC on most benchmarks. It reports the best FPR95 on iNaturalist (49.16%) and OpenImage-O (63.96%). Test error is 26.43%, slightly higher than some baselines, which the authors describe as a favorable accuracy/calibration trade-off.
• Singularity ranking matters (Figure 5): Selecting the top 10% of boundaries by singular score consistently achieves the lowest OOD MMC across datasets. Performance degrades when lower-ranked boundaries are selected but remains better than random boundary selection (RanB).
• Interpolation baselines are weaker (Figure 5): Both latent-space interpolation (L-Inter) and input-space (image-level) interpolation (I-Inter), which do not use OT geometry, produce substantially higher OOD MMC than boundary-based sampling.
• Better OOD detection at inference (Figure 6): Applying MSP, ODIN, or ReAct on top of a model trained with the suppression loss yields ROC curves that dominate their respective baselines across four ID→OOD settings, with the improvement especially pronounced for CIFAR-10 → Textures_C.
• Sample fidelity (Table 3): Measured by FID to CIFAR-10, OTIS reaches 2.45, the smallest among compared sample sets (CEDA 7.25, ACET 7.49, CODES 3.18, VOS 5.28, OE 5.73), while inducing an MMC of 91.29% on those samples. The paper notes ACET attains the highest MMC (99.98%) but its Adv. Noise samples lie much farther from the ID manifold.
• ID calibration (Table 4): OTIS achieves the best expected calibration error on CIFAR-10 (1.88, in units of 10⁻²) and FMNIST (3.26), matching the best baseline on MNIST (0.14, tied with CEDA). On CIFAR-100 and ImageNet, several methods worsen ECE relative to the base model while OTIS stays close to the baseline.
• Base distribution and autoencoder depth (Table 5): Gaussian-5 and Uniform-5 give similar CIFAR-10 test error (7.31 vs 7.52) and ID MMC (96.48 vs 95.46), but Uniform-5 gives better OOD metrics. Uniform-5 generally offers the best OOD trade-off among Uniform-3, Uniform-5, and Uniform-7.
• FPR95 detail (Appendix Table 6): For CIFAR-10 with Adversarial Noise, the method reports 0.66% FPR95, described as far below VOS (94.12%) and CODEs (19.55%). The provided paper text is truncated at this point, so the rest of Table 6 is not reported here.

Methodology in Plain English

The pipeline has four conceptual steps:

1. Compress the data. An autoencoder encodes in-distribution images into a compact latent space (latent dimensionality 256 for the 28×28 and 32×32 benchmarks; 1024 for 224×224 ImageNet using a symmetric VGG-16 autoencoder). Working in this latent space makes the geometry more tractable than working directly on pixels.

2. Build an optimal transport partition. Each training image's latent vector becomes a support point of a discrete target measure. A semi-discrete OT problem is solved between a continuous base distribution (Gaussian or uniform) and these support points. The solution is the gradient of a convex piecewise-linear potential, which partitions the base distribution's domain into convex Laguerre cells (a power diagram). The scalar offsets of the potential are found by minimizing the squared difference between each cell's estimated μ-volume and a uniform target weight of 1/n, using Monte Carlo sampling to estimate volumes, subject to the normalization that the offsets sum to zero.

3. Find the ambiguous boundaries. Each pair of adjacent cells shares a boundary hyperplane. The authors score each candidate boundary by the angular deviation between the two support points, computed as the arccosine of their cosine similarity. Boundaries with the largest scores indicate sharper changes in transport direction and are more likely to be singular. The top 10% of boundaries by this score form the singular boundary set used for sample generation.

4. Generate OTIS and train. For each selected boundary, the mass centers of the two adjacent Laguerre cells are estimated by Monte Carlo. A latent point is drawn from the base distribution, inverse-distance interpolation weights are computed with respect to the two centers, and a smoothed transport extension combines the transports of the two centroids. The resulting latent vector is decoded back to image space to become an OTIS sample. During training, each batch is 50% in-distribution samples supervised by cross-entropy loss and 50% OTIS guided by a confidence suppression loss that sums (1/K)·log of the softmax probability over all K classes, encouraging the model to spread confidence evenly.

Training uses 200 epochs, the Adam optimizer, and a learning rate of 0.0001, with the OT problem solved following An et al. (2020) and the overall confidence suppression procedure following the default configuration of CCU (Meinke & Hein, 2020). Experiments run on a workstation with eight NVIDIA RTX 4090 GPUs. LeNet is used for MNIST, ResNet-18 for CIFAR-10, CIFAR-100, SVHN, and FMNIST, and ResNet-50 for ImageNet. Evaluation reports test error (TE), mean maximum confidence (MMC) on ID and OOD data, and FPR95.

Why This Matters

Impact on research. The paper reframes OOD overconfidence as a geometric phenomenon rather than purely a statistical or empirical one. Instead of adding noise, corrupting inputs, mixing classes, or pulling in external datasets, it derives training samples from a property of the transport map itself. This connects optimal transport regularity theory — specifically the behavior of the Brenier potential on singular sets — to the practical problem of confidence calibration, and offers a principled alternative to the heuristics that currently dominate proxy-OOD training. Because the method uses no auxiliary OOD dataset, it also sidesteps the licensing and overlap problems associated with large external corpora, a point the paper makes by contrasting against OE and CCUd (the latter trained on 80 Million Tiny Images with CIFAR-10/100 overlaps removed).

Real-world applications.

• Safety-critical perception: Autonomous driving, medical imaging triage, and industrial inspection systems where a confident wrong answer is worse than an abstention.
• Open-world deployment of classifiers: Production models that encounter classes or domains never seen in training (new product categories, novel sensor conditions, unfamiliar document layouts).
• Selective prediction and abstention: Downstream systems that route low-confidence cases to humans need confidence scores that actually drop on unfamiliar inputs.
• Adversarial and anomaly screening: The reported gains on Adversarial Noise and Adversarial Samples suggest relevance to input anomaly monitoring, not only to natural distribution shift.

Industry relevance. The method is applied to standard, widely used architectures (LeNet, ResNet-18, ResNet-50) and standard datasets, which lowers the barrier to adoption. It adds no external data requirement and no test-time change, so it can be integrated as a training-time regularizer into existing pipelines. The reported ImageNet trade-off — higher ID MMC (88.93%) and better FPR95 on several benchmarks, at a test error of 26.43% versus some lower-error baselines — is the kind of trade-off practitioners weigh when calibration matters more than a fraction of top-1 accuracy.

Future Directions

1. Extending beyond image classification. All experiments here are 2D image classification benchmarks. Whether OT-induced singular samples transfer to detection, segmentation, language, audio, or tabular models is not reported.
2. Removing the fixed 10% threshold. The paper uses the top 10% of boundaries by singular score and shows in Figure 5 that this is best among the tested options, but an adaptive or theoretically derived selection rule is left open.
3. Reducing dependence on the autoencoder. OTIS quality depends on a latent space produced by an autoencoder (5-layer symmetric encoder–decoder, or symmetric VGG-16 for ImageNet). How sensitive the framework is to autoencoder quality, latent dimensionality, or alternative representation learning objectives is not reported.
4. Scaling and cost analysis. The paper reports results on ImageNet but does not report runtime, memory, or scaling behavior of the Monte Carlo volume estimation and offset optimization as the number of support points grows.
5. Completing the reported evidence. The provided paper content is truncated during Appendix A.1, so the full FPR95 comparison (Table 6) and the "complete source code and reproduction instructions" promised to be released upon acceptance are not available here.

Target Audience

Researchers and graduate students working on trustworthy machine learning, OOD detection, uncertainty quantification, and calibration, especially those with an interest in optimal transport and its geometric structure. It is also relevant to machine learning engineers who need calibrated, deployable classifiers in open-world settings and who want a training-time regularizer that requires no external OOD dataset and no test-time modification. Readers without a background in semi-discrete optimal transport, Brenier potentials, or power diagrams will find Sections 2.2 and 2.3 dense and will benefit from reading the related-work discussion of OT applications first.

Authors’ abstract

Deep neural networks (DNNs) often produce overconfident predictions on out-of-distribution (OOD) inputs, undermining their reliability in open-world environments. Singularities in semi-discrete optimal transport (OT) mark regions of semantic ambiguity, where classifiers are particularly prone to unwarranted high-confidence predictions. Motivated by this observation, we propose a principled framework to mitigate OOD overconfidence by leveraging the geometry of OT-induced singular boundaries. Specifically, we formulate an OT problem between a continuous base distribution and the latent embeddings of training data, and identify the resulting singular boundaries. By sampling near these boundaries, we construct a class of OOD inputs, termed optimal transport-induced OOD samples (OTIS), which are geometrically grounded and inherently semantically ambiguous. During training, a confidence suppression loss is applied to OTIS to guide the model toward more calibrated predictions in structurally uncertain regions. Extensive experiments show that our method significantly alleviates OOD overconfidence and outperforms state-of-the-art methods.