Let Confidence Change, Not the Prediction: Prediction-Preserving Repair for Post-hoc Calibration

AI summary

Overview

Research area: Post-hoc confidence calibration for multiclass classifiers, with a focus on whether calibration changes the predicted class.

Technical level: Advanced. The paper uses convex optimization, KL divergence decompositions, Lagrange multipliers, and calibration metrics, though the core idea can be grasped without following the derivations.

Scope: The paper introduces CORD, a post-fit adapter that takes the output of any fitted multiclass calibrator and produces a repaired probability vector whose argmax exactly recovers the original classifier's top-1 prediction, evaluated on CIFAR-10/100 and ImageNet-1K.

What This Paper Is About

Post-hoc calibration is supposed to fix the confidence a classifier reports, but a multiclass calibration map can also change which class ends up as the top-1 prediction. Accuracy only records the net effect of these changes on correctness, so a small accuracy change can hide a large number of prediction revisions. The paper's goal is to separate the two concerns: let the calibrator be fitted freely without any prediction-preservation constraint, then repair its output afterward so the original prediction is exactly restored.

Key Contributions

1. A new diagnostic for prediction revision. The authors define the Top-1 Prediction Change Rate (TPCR), which counts the total fraction of examples whose top-1 prediction changes, and decompose those changes into (C^{+}) (incorrect to correct), (C^{-}) (correct to incorrect), and (C^{0}) (incorrect to a different incorrect class). Accuracy captures only the net balance of (C^{+}) and (C^{-}), while (C^{0}) is invisible to it.

2. CORD, a post-fit repair adapter. CORD is presented as the first post-fit adapter to impose exact prediction preservation by repairing a fitted calibrator's full probability vector. It uses only the original classifier output and the direct calibrated output, fits no additional supervised map, and introduces no user- or validation-tuned hyperparameter.

3. A principled one-dimensional repair family. For each input, CORD fixes the calibrated conditional distribution over all classes other than the original top-1 by renormalization, leaving only a single scalar (s_i) (the repaired mass on the original prediction) free. It shows this reconstruction uniquely minimizes (D_{\mathrm{KL}}(q_i ,|, \cdot)) on that slice, and derives a closed-form prediction-preserving interval for (s_i).

4. A calibration-split coordination rule. Because independent pointwise repairs can shift the mean mass assigned to original predictions, CORD projects the calibrator's mean mass (n^{-1}\sum_i b_i) onto the attainable mean interval and solves a single coupled program via one shared scalar (\eta^{\star}). CORD's only persistent data-dependent state is that scalar, stored in 8 bytes.

Main Findings

• Accuracy hides prediction churn. Applying Vector Scaling to a pretrained ResNet-50 on ImageNet-1K reduces accuracy by only 0.24 percentage points yet changes the original top-1 prediction for 5.74% of examples.

• Zero TPCR by construction. For every evaluated classifier–calibrator pair, CORD attains zero TPCR, recovering every original top-1 prediction and hence the original accuracy. Across dataset–calibrator pairs, direct-output TPCR spans 0.07% to 31.05% and top-1 accuracy change spans −10.06 to +0.22 percentage points.

• CORD lowers all three metrics on average. Across CIFAR-10, CIFAR-100, and ImageNet-1K, CORD lowers mean ECE, NLL, and Brier relative to the corresponding direct outputs. Mean paired reductions (Base minus Base+CORD) are: CIFAR-10 ECE 0.061 pp, NLL 0.0014, Brier 0.361 ×10⁻³; CIFAR-100 ECE 4.642 pp, NLL 0.5114, Brier 33.691 ×10⁻³; ImageNet-1K ECE 0.847 pp, NLL 0.0154, Brier 2.159 ×10⁻³. Bonferroni-adjusted simultaneous 95% bootstrap CIs lie above zero.

• Gains track direct-output TPCR. CORD's paired gains are largest when direct-output TPCR is high and remain small when few predictions are revised, which the authors relate directly to its construction.

• Beating minimal repair. A minimal pointwise repair that sets (s_i^{\min} = \Pi_{I_i}(b_i)) also attains zero TPCR and preserves the calibrated conditional distribution, yet CORD yields significantly lower ECE, NLL, and Brier on every dataset. On CIFAR-100, minimal repair gives ECE 5.144% / NLL 1.875 / Brier 0.4055 versus CORD's 3.912 / 1.705 / 0.4041.

• The aggregate target matters. In a mean-target ablation on CIFAR-100, the locally optimized independent variant performs worst in every column (ECE 5.265%, NLL 1.844, Brier 0.4056, |s̄−b̄| 2.808 pp), while CORD yields the lowest values throughout (3.912, 1.705, 0.4041, 0.579 pp) — including the smallest held-out deviation from the Base mean.

• Robustness under distribution shift. Fitting on clean CIFAR data and evaluating on CIFAR-10-C and CIFAR-100-C across severities 1–5, CORD maintains zero TPCR and yields positive mean paired reductions in ECE, NLL, and Brier at every severity. Mean direct-output TPCR rises from 1.65% to 4.58% on CIFAR-10-C and from 12.75% to 18.70% on CIFAR-100-C.

• Calibration-size sensitivity. On ImageNet-1K, CORD's mean paired reductions are largest with limited calibration data but persist across calibration-set fractions of 10%–100%. Mean direct-output TPCR declines with calibration-set size but remains nonzero at 100%, while CORD maintains zero TPCR throughout.

• Competitive with fit-time preservation. On CIFAR-10, where Base direct outputs revise fewer than 1% of top-1 predictions, the least-changing Base direct outputs often attain lower ECE, equal-mass ECE (ECE_EM), and kernel-smoothed smECE than fit-time baselines TS, IRM, AdaTS, TS–TvA, and MCCT-I; CORD lowers nearly every paired value while restoring zero TPCR.

• Low runtime overhead. Reported single-threaded CPU medians (with (q) from IROvA-TS): construction 0.268 s (CIFAR-10), 0.280 s (CIFAR-100), 1.384 s (ImageNet-1K); amortized repair 0.978, 1.996, and 25.64 microseconds per example respectively.

• Structural guarantees. Proposition 1 establishes that the repaired vector lies in the simplex, that (a(x)) is uniquely top-ranked, that the ratio (p̃_j/(1-p̃_{a(x)})) equals (q_j/(1-q_{a(x)})) for every (j \neq a(x)), and that the calibration-split masses average to (\mu).

Methodology in Plain English

The setup is simple to state. A classifier produces a probability vector (p^0). A calibrator, already fitted, produces a direct output (q). CORD takes the pair ((p^0, q)) and returns a third vector (p̃).

The first move is to shrink the space of possible repairs. CORD leaves the calibrated vector's relative allocation among all non-top-1 classes untouched — it renormalizes those entries into a conditional distribution α. That leaves exactly one free number per example: (s_i), the mass assigned to the original prediction. Every other coordinate is then determined.

The second move is to find which values of (s_i) keep the original class on top. Comparing (s_i) against ((1-s_i)\rho_i), where (\rho_i) is the largest entry of α, gives a threshold (\rho_i/(1+\rho_i)); any (s_i) above it works. The paper adds a tiny numerical offset (\epsilon_{\mathrm{num}} = 10^{-12}) to keep the interval open.

The third move is to choose (s_i) well rather than arbitrarily. Each example gets a local reference (g_i): the calibrated mass (b_i = q_{i,a_i}) when the calibrator already keeps the original class on top, or the average of (b_i) and the original classifier's mass (p^0_{i,a_i}) when it does not. CORD minimizes the total Bernoulli–KL divergence from these references, subject to each (s_i) staying in its interval and the average of the (s_i) equaling a target mean (\mu). That target mean is the calibration-split average of the (b_i), projected onto the interval of means that are actually attainable.

Because the only coupling between the (n) variables is one aggregate equality, a single Lagrange multiplier handles everything. CORD solves for one scalar (\eta^{\star}) by bisection (each per-example response has a closed form), then applies that same scalar to any new input pair. Nothing about the fitted calibrator is changed, and no labels are needed after the calibration split is used.

Evaluation covers seven calibrators (VS, SVS, MS, SMS, Dir-ODIR, IROvA, IROvA-TS), with MS, SMS, and Dir-ODIR excluded on ImageNet-1K because each has more than (10^6) fitted coefficients at (K = 1000). Metrics are ECE (15 equal-width bins), NLL, and the multiclass Brier score. Splits are 5,000 examples per split from each CIFAR test set and 25,000 per split from the ImageNet validation set, across five random seeds.

Why This Matters

Impact on research. The paper challenges an implicit assumption in calibration work: that preserving the top-1 prediction is a constraint the calibrator itself must satisfy. If preservation can be imposed afterward without harming calibration quality, the constraint can be lifted from fitting, which widens the space of admissible calibration maps. It also supplies TPCR as a measurement tool — a way to quantify prediction churn that accuracy structurally cannot see, since opposing revisions cancel and incorrect-to-incorrect revisions leave accuracy unchanged.

Real-world applications:

• Clinical decision support. If a calibrated diagnostic model changes its reported class, the confidence score now describes a different diagnosis than the one the clinician is acting on. CORD keeps the reported condition fixed while still improving the confidence number.
• Autonomous driving and robotics. Perception modules that report a class label plus a confidence need the label to be stable across a calibration update; a 5.74%-scale revision rate corresponds to real prediction switches.
• Regulated or audited ML systems. Where a deployed model's decisions have been validated, certified, or logged, silently revising the top-1 after a calibration change complicates audit trails and model-change reviews.
• Model monitoring and A/B testing. Comparing two classifiers or two calibrators is confounded when the calibration step itself alters predictions; CORD removes that confound.

Industry relevance. CORD is designed as a drop-in addition: it does not touch the fitted calibrator, requires no additional labeled data or auxiliary model, needs no hyperparameter tuning, stores only one 8-byte scalar, and costs microseconds per example at inference (25.64 μs/example reported for ImageNet-1K). That makes it practical for teams that already have a calibration pipeline and simply want the prediction-revision behavior to be a deliberate deployment choice rather than a side effect.

Future Directions

• Extending post-fit preservation beyond exact top-1. The paper opens what it calls post-fit preservation as a "distinct calibration design space" but only instantiates one objective; other preservation criteria (e.g., top-(k) set preservation, or bounded revision budgets) are left unexplored.
• Repair rules other than the Bernoulli–KL objective. The comparison with minimal repair shows the choice within the feasible family has a significant effect on ECE, NLL, and Brier, suggesting room for other objectives or target means.
• Better calibrated or label-aware target means. CORD's aggregate target is a single projection of the Base mean derived without labels; whether a validation-based or otherwise informed target would help is not resolved here.
• Measurement and adoption of TPCR. The paper argues that prediction-change frequency deserves separate reporting from accuracy, but whether TPCR becomes a standard reported quantity across the calibration literature remains open.

Target Audience

Researchers and practitioners in machine learning calibration and uncertainty quantification will get the most from this paper, particularly those who work with multiclass classifiers and need reported confidence to describe a stable prediction. It is also relevant to engineers deploying calibrated models in settings where the predicted class must remain fixed after a calibration update, and to anyone evaluating calibration methods who wants a measurement that distinguishes prediction churn from accuracy change. The optimization derivation makes the middle sections demanding, but the diagnostic argument in the introduction and the empirical tables are readable for those with a working knowledge of calibration metrics.

Note: the supplied paper content ends mid-sentence in the conclusion ("The broader desig..."), so the paper's complete closing discussion is not available for summarization. Several appendix items — complete classifier–calibrator tables, reliability diagrams, CIFAR-10/ImageNet-1K versions of Tables 3 and 4, absolute corruption results, and the Python implementation — are referenced but not included in the content provided.

Authors’ abstract

Post-hoc calibration corrects reported confidence, yet a multiclass calibrator can also change the associated top-1 prediction. Accuracy captures only the net effect of these changes on correctness, not how often predictions change; the Top-1 Prediction Change Rate (TPCR) instead measures this frequency. We propose Calibrator-Output Repair for Top-1 Decision Preservation (CORD), the first post-fit adapter to impose exact prediction preservation by repairing the full calibrated probability vector. From the original and calibrated outputs alone, CORD determines the mass assigned to the original top-1. The calibrated conditional distribution allocates the remaining mass over the other classes, yielding a repaired vector whose own argmax recovers the original prediction. On the calibration split, CORD coordinates the repaired masses to retain the calibrated outputs' mean mass on original predictions whenever attainable. The adapter alters neither the fitted calibrator nor its direct output, fits no additional supervised map, and requires no user- or validation-tuned hyperparameter. Across CIFAR-10/100 and ImageNet-1K, CORD attains zero TPCR by construction and lowers mean ECE, NLL, and Brier relative to the corresponding direct outputs in every dataset; paired gains persist under distribution shift and across calibration-set sizes. CORD thus removes the preservation constraint from calibrator fitting and assigns exact recovery of the original decision to subsequent output repair. Our code is available at https://github.com/labhai/CORD.