Calibration, Uncertainty, and Abstention

The word confidence often collapses several quantities. Those quantities have different meanings and different validation needs.

Each of them is checkable against something. A score is checked against an ordering. A probability is checked against observed frequencies. A support signal is checked against the data the model actually saw. A decision policy is checked against what happened to the cases it accepted and the cases it pushed elsewhere. The rest of this lesson keeps them apart by looking at measurements where the difference has already been quantified.

If many cases receive a predicted probability near 0.20, roughly one fifth should exhibit the event in a well-calibrated setting. Calibration is conditional on the evaluation population, time, and grouping.

A model can be calibrated overall. It can be miscalibrated within important subgroups or score ranges.

How much data a calibrator needs has also been measured. Learning curves published in 2005 compared Platt scaling and isotonic regression across eight classification problems. Isotonic regression is the more powerful method. For exactly that reason it “is easier for it to overfit when the calibration set is small”, wrote Niculescu-Mizil and Caruana. The curves also give the crossing point: “when there are 1000 or more points in the calibration set, Isotonic Regression always yields performance as good as, or better than, Platt Scaling”. Scikit-learn's calibration guide documents the same threshold from the other side. Isotonic “will perform as well as or better than ‘sigmoid’ when there is enough data (greater than ~ 1000 samples) to avoid overfitting”. Below that count, the flexible method is the riskier one.

Calibration needs its own predictions. They must not be predictions that were used to train the base model.

The trade is not a matter of taste, and the numbers for it are on the record. On CIFAR-100 a 110-layer ResNet had an expected calibration error of 16.53%, measured with M=15 bins. A 5-layer LeNet on the same data had 4.85%. The deeper, more accurate model was the badly calibrated one. “The average confidence of LeNet closely matches its accuracy, while the average confidence of the ResNet is substantially higher than its accuracy,” wrote Guo, Pleiss, Sun and Weinberger, who measured it in 2017.

The repair on that same ResNet ranks the methods in this table. Temperature scaling is the least flexible mapping available. It cut the error to 1.26%. Isotonic regression, which is more flexible, reached only 4.99%. Matrix scaling, more flexible still, produced 25.49% — worse than the 16.53% of doing nothing at all. Flexibility bought nothing here. Past a point it cost more than it returned.

None of this is permanent. Minderer and colleagues re-measured the same quantity across newer architectures in 2021. They reported that the most recent models, notably those not using convolutions, are among the best calibrated, and that the decay of calibration with model size or distribution shift is less pronounced in recent architectures. Measure the calibrator on your own scores. Do not memorise a ranking of methods.

Probability errors often stay invisible. They appear only when predictions are used for pricing, prioritization, or resource allocation.

The best-documented case is a proprietary risk score running inside hospital software. The Epic Sepsis Model was validated externally in 2021 across 27,697 patients and 38,455 hospitalizations, of whom 2,552 (7%) developed sepsis. Its hospitalization-level AUC was 0.63 (95% CI, 0.62-0.64). It failed to identify 1,709 patients with sepsis, 67% of them. It still generated alerts on 6,971 of the 38,455 hospitalizations, 18%. “This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis,” concluded Wong and colleagues in JAMA Internal Medicine. Both halves of that sentence matter. The ranking was weak and the probability scale was wrong. Neither was visible from the deployed alert itself.

An independent look at Epic's model v1.0 followed in 2024, in two county emergency departments, covering 145,885 encounters from 2023. Sensitivity was 14.7%. Positive predictive value was 7.6% within a 6-hour window. A score can be in production at scale for years, be measured once, and be found this far off.

Reliability diagrams show observed frequency against predicted probability, but binning choices can hide variation. Brier score combines calibration and discrimination components. Log loss strongly penalizes confident wrong predictions.

Use counts, uncertainty, and slice views alongside summary metrics.

The Brier score is older than machine learning, and it was built for exactly the problem this lesson has. It appeared in Monthly Weather Review in January 1950, in three pages, from Glenn W. Brier of the U.S. Weather Bureau. The point was not to summarise accuracy. It was to remove an incentive. “It is the purpose of this paper to discuss one situation where it appears to be possible to devise a verification scheme that cannot influence the forecaster in any undesirable way,” he wrote in the introduction. A forecaster cannot improve the score by hedging. Nearly forty years later, in 1988, NOAA's National Weather Service Central Region still described it as “one of the most widely used verification scores for probability of precipitation (POP)”. That is what a proper scoring rule is for, and why it belongs beside the diagram rather than after it.

The binning is not a presentation detail either. It changes the number. A 2019 paper showed that “popular recalibration methods like Platt scaling and temperature scaling are (i) less calibrated than reported, and (ii) current techniques cannot estimate how miscalibrated they are”. Its scaling-binning calibrator “requires O(1/ε² + B) samples”, where scaling methods need O(B/ε²). The same authors note that “an alternative estimator introduced in the meteorological community requires fewer samples (O(√B) instead of O(B))”. On CIFAR-10 and ImageNet their method achieves “35% lower calibration error than histogram binning”. A calibration figure nobody can verify is a claim, not a measurement.

Probability and deferral policies require their own development evidence.

The steps below are not one team's house style. Ten Good Machine Learning Practice guiding principles for medical device development were published jointly in October 2021 by the U.S. FDA, Health Canada and the UK's MHRA. Health Canada dates its posting 27 October 2021. Three of those numbered principles say what steps 3, 5 and 6 say. Principle 8 requires clinically relevant testing independent of the training set and covering important subgroups. That is step 3, validation on untouched data, written as a regulatory expectation. Principle 10 requires deployed models to be monitored in real-world use, with controls for overfitting, unintended bias and dataset drift. That is step 6.

Principle 7 is the one that governs abstention, and it is worth reading in full: “Where the model has a "human in the loop," human factors considerations and the human interpretability of the model outputs are addressed with emphasis on the performance of the Human-AI team, rather than just the performance of the model in isolation.” A deferral rule that has only been evaluated on the cases the model kept has not been evaluated against that principle at all.

A monotonic score can preserve ordering while absolute risk changes. Updating a calibrator may then be appropriate, when the conditional relationship is stable enough and new mature labels are available.

Do not recalibrate blindly when feature relationships, interventions, or target definitions also changed. Refitting a mapping from score to probability answers one question: has the probability scale drifted. It silently assumes the answer to another, which is whether the ordering underneath it still holds. When both have moved, a new calibrator will make the reliability diagram look repaired while the decisions stay wrong.

Terms such as risk, chance, and likelihood invite frequency interpretation. If the output is an uncalibrated margin or vote, call it a score and document the operating threshold.

Avoid confidence percentages that merely rescale an arbitrary model output.

The most disciplined version of this practice is public and tabulated. The IPCC's calibrated likelihood scale assigns numbers to its words: virtually certain 99-100%, very likely 90-100%, likely 66-100%, about as likely as not 33-66%, unlikely 0-33%, very unlikely 0-10%, exceptionally unlikely 0-1%. That table has been in the panel's uncertainty guidance for authors since 2010. Every author writing “very likely” is writing 90-100%, by rule.

Even that is not sufficient, which is the part worth carrying into your own reporting. A study of 25 samples in 24 countries and 17 languages found the words drifting on the way to the reader. “As predicted, laypeople interpret IPCC statements as conveying probabilities closer to 50% than intended by the IPCC authors,” its authors wrote in Nature Climate Change in 2014. Their measured fix is mechanical rather than rhetorical: printing the numerical range alongside the word increased agreement with the guideline. Define the term. Then print the number next to it anyway.