Credal Ensemble Distillation for Uncertainty Quantification

AI summary

Authors: Kaizheng Wang, Fabio Cuzzolin, David Moens, Hans Hallez arXiv: 2511.13766v1 [cs.LG], 14 Nov 2025 | License: CC BY 4.0

Overview

Research area: Uncertainty quantification (UQ) in deep neural networks, specifically knowledge distillation and OOD detection.

Technical level: Advanced — the paper assumes familiarity with deep ensembles, Dirichlet-based models, and imprecise-probability concepts such as credal sets, probability intervals, and intersection probability.

Scope: The paper proposes credal ensemble distillation (CED), a framework that compresses a deep ensemble (DE) into a single classifier (CREDIT) whose output defines a credal set for uncertainty quantification, and validates it on OOD detection benchmarks.

What This Paper Is About

Deep ensembles quantify predictive uncertainty well but are expensive at inference because they require running multiple networks. Existing distillation methods that compress an ensemble into one model either output only a single softmax distribution (limiting aleatoric uncertainty estimation) or output a Dirichlet distribution (which lacks ground-truth training labels and has been criticized theoretically). This paper aims to distill a deep ensemble into a single model that instead outputs class-wise probability intervals defining a credal set, so that both aleatoric and epistemic uncertainty can be estimated at much lower inference cost.

Key Contributions

1. A new task and framework: The paper introduces credal ensemble distillation (CED), described by the authors as a novel task not previously explored — distilling a DE teacher into a single model that predicts a credal set.
2. The CREDIT student architecture: A modification of the final classification layer of any standard neural network backbone, replacing C output neurons with 2C + 1 nodes that predict an intersection probability in R^C, an interval length vector in R^C, and a scalar weight factor in R.
3. A distillation loss for credal labels: A loss combining a cross-entropy term between the teacher's and student's intersection probabilities with mean-squared-error terms on the interval length vector and the weight factor, compatible with temperature scaling.
4. Empirical validation: OOD detection experiments across dataset pairs and backbones (VGG16 trained from scratch, pre-trained ResNet50) showing superior or comparable UQ versus DE, ED, EDD*, and MCDO baselines, with substantially lower inference overhead than DE.

Main Findings

• Aleatoric uncertainty is capped in standard ED: The authors state that ensemble distillation produces a single predictive distribution capturing input-output randomness while assuming precise knowledge of that dependency, which limits its ability to quantify aleatoric uncertainty.
• CED improves epistemic uncertainty estimation: Table 1 shows CED "significantly and consistently" improves EU estimation over baselines on OOD detection, across dataset pairs and backbones.
• CED on VGG16 (CIFAR10 vs SVHN): EU AUROC 93.56±2.17 and EU AUPRC 96.09±1.72, versus DE at 89.99±0.79 and 93.78±0.67; TU AUROC 92.51±1.96 and TU AUPRC 95.21±1.52, versus DE TU at 91.53±0.72 and 95.09±0.49. Test accuracy 92.23±0.17 and ECE 6.71±0.18.
• CED on ResNet50 (CIFAR10 vs CIFAR10-C): EU AUROC 96.80±2.81, EU AUPRC 96.09±4.14; DE at 87.78±2.28 and 78.92±3.67. Test accuracy 91.77±0.74 and ECE 6.34±0.59.
• EU beats TU for CED in most cases: The paper reports that CED's EU estimates yield the best performance in most cases, which the authors argue highlights the importance of EU quantification for reliable OOD detection.
• MCDO's EU is unreliable here: MCDO shows EU AUROC 51.42±0.46 on VGG16 (CIFAR10 vs SVHN), which the authors attribute to limited model diversity.
• EDD was excluded from the main comparison: EDD trained under the same configurations as ED and CED was excluded due to substantially lower prediction accuracy: test ACC 74.56±2.02 and ECE 5.51±0.57 on VGG16, and ACC 61.15±8.47 and ECE 8.40±2.06 on ResNet50. Note that ResNet50-based MCDO results are not reported because no pre-trained models were available.
• Accuracy and calibration: Distillation enhances predictive accuracy of individual SNNs, and CED performs comparably to baseline distillation methods on accuracy and ECE. The authors note the ECE metric used is designed for single-probability predictions and that a principled extension to credal-set predictions is needed.
• Larger teacher ensembles do not clearly help the student: For DE teachers with M ∈ {5, 15, 25, 30}, the DE teacher improves with size but no clear trend is observed for CED or EDD*.
• Temperature scaling matters: Testing T ∈ {1, 2.5, 5, 10} on VGG16, T = 2.5 consistently yielded the best results, while excessively high values (T = 10.0) degraded performance.
• Efficiency: Inference time on the CIFAR10 test set with a single P100 GPU was 2.26±0.23 s for CED versus 5× (2.22±0.20) s for DE and 2.22±0.20 s for EDD*. Training time per epoch was 659.52±11.82 s for CED, 684.54±5.05 s for EDD*, and 5× (130.07±0.24) s for DE.
• Qualitative behavior: Kernel density plots show CED assigns substantially higher EU and TU values to OOD samples than to ID instances.

Methodology in Plain English

The pipeline has three stages.

Teacher. A deep ensemble of M standard neural networks each produces a softmax probability vector. For every class, the highest and lowest probabilities across the M members define an interval [lower, upper]. Taking the maximum over members for the upper bound and the minimum for the lower bound yields a set of probability intervals. Any normalized probability vector whose entries fall inside these intervals belongs to a credal set — a convex set of probability distributions. To get a single usable class prediction out of this set, the method computes an "intersection probability": p*_k = lower_k + β(upper_k − lower_k), where β = (1 − Σ lower_k) / (Σ Δp_k) and Δp_k is the interval length.

Student. CREDIT keeps any backbone unchanged except the final layer, which is enlarged to 2C + 1 outputs. A softmax over the first C logits gives the predicted intersection probability, a sigmoid over the next C gives the interval length vector, and a sigmoid on the last logit gives the weight factor. From these three pieces the probability intervals can be reconstructed (and clamped to [0, 1]), and the construction is shown to satisfy the condition for defining a valid credal set.

Training and uncertainty. The distillation loss is the sum of a cross-entropy between the teacher's and student's intersection probabilities, an MSE on the interval lengths, and an MSE on the weight factor — the last two equivalent to directly matching the reconstructed lower and upper bounds. Temperature scaling can be applied, with the loss scaled by T². At inference, the intersection probability gives the class prediction, and the full output vector recovers a credal set. Uncertainty is measured with generalized entropy: the maximum Shannon entropy over the credal set is the total uncertainty, the minimum is the aleatoric uncertainty, and epistemic uncertainty is the difference. These optimization problems are solved with standard solvers such as SciPy, and the paper states the overhead is marginal when C ≤ 10.

Experimental setup. The main experiments used VGG16 trained from scratch on CIFAR10, with an ablation on pre-trained ResNet-50. Fifteen SNNs were trained with different random initializations; fifteen DEs of size M = 5 were formed by randomly selecting fifteen distinct subsets from the SNN index set {1, ..., 15}, and fifteen students were distilled per method. Training used batch size 128, Adam with a learning rate initialized at 0.001 and reduced by 0.1 at epoch 80, 100 epochs, and T = 2.5. EDD* used a cyclic learning rate policy (cycle length 60), T = 10, and temperature annealing. MCDO with 10 forward passes and dropout rate 0.1 served as an additional baseline. OOD detection is treated as binary classification with uncertainty scores as the decision variable, evaluated by AUROC and AUPRC, with CIFAR10-C scores averaged over 15 corruption types and 5 severity levels.

Why This Matters

Impact on research. The paper reframes ensemble distillation as a problem of transferring imprecision, not just averaged probabilities. It connects deep learning distillation to imprecise-probability theory (credal sets, probability intervals, intersection probability) and argues that existing distillations — ED for single distributions and EDD for Dirichlet distributions — are limited in what they can say about aleatoric uncertainty, while Dirichlet-based methods have been criticized for departing from the theoretical tenets of epistemic uncertainty and for lacking meaningful quantitative interpretation. CED offers a different second-order representation without requiring ground-truth Dirichlet labels.

Real-world applications:

• Safety-critical perception and monitoring systems where out-of-distribution inputs must be flagged reliably.
• Medical image classification, which the authors state is explored as a case study in the Appendix.
• Deployed models on edge or embedded hardware, where a five-fold ensemble is too costly but a single distilled model fits.
• Any production classifier needing calibrated confidence and uncertainty-aware decision thresholds.

Industry relevance. The headline practical result is inference cost: a CED model runs in roughly the time of a single model (2.26±0.23 s) instead of the 5× ensemble cost, while improving OOD detection metrics in the reported benchmarks. That trade-off matters for latency- and memory-constrained deployment. The paper also notes CED is simpler to train than EDD* since it does not require a sophisticated learning rate scheduler or temperature annealing.

Future Directions

1. Scaling to many classes. Extending CED to classification with a significantly larger number of classes (e.g., 100 or 1000). The authors identify a key challenge: the softmax activation in the DE teacher produces probability values near zero for most classes, which could destabilize the regression component of the distillation loss and undermine UQ robustness.
2. Calibration-aware distillation. Integrating calibration considerations into the distillation strategy to achieve comparable or better calibration than the DE teacher.
3. Extending ECE to credal predictions. The paper notes that the ECE metric used is designed for single-probability predictions, and a principled extension to credal-set predictions is needed for a fair comparison.
4. Broader validation. Additional ablation studies on the ResNet18 backbone and the medical image classification case study are described as provided in the Appendix, suggesting further empirical ground to explore; the paper also observes no clear trend in student performance as teacher ensemble size grows from 5 to 30.

Target Audience

Researchers and practitioners working on uncertainty quantification, Bayesian and ensemble methods, and knowledge distillation — particularly those who need ensemble-grade uncertainty at single-model inference cost. It will also interest readers in imprecise probability and credal-set methods looking for a deep-learning application, and engineers deploying OOD detection or safety-critical classifiers under compute constraints. Readers should be comfortable with softmax classifiers, entropy-based uncertainty decomposition, and basic convex-set reasoning.

Authors’ abstract

Deep ensembles (DE) have emerged as a powerful approach for quantifying predictive uncertainty and distinguishing its aleatoric and epistemic components, thereby enhancing model robustness and reliability. However, their high computational and memory costs during inference pose significant challenges for wide practical deployment. To overcome this issue, we propose credal ensemble distillation (CED), a novel framework that compresses a DE into a single model, CREDIT, for classification tasks. Instead of a single softmax probability distribution, CREDIT predicts class-wise probability intervals that define a credal set, a convex set of probability distributions, for uncertainty quantification. Empirical results on out-of-distribution detection benchmarks demonstrate that CED achieves superior or comparable uncertainty estimation compared to several existing baselines, while substantially reducing inference overhead compared to DE.