Calibration, Log Loss, Brier Score, and Reliability

If a system assigns 0.70 risk to many comparable cases, roughly 70% of those cases should experience the outcome under the evaluated conditions. That is a claim about groups of predictions, not certainty for one individual. Calibration is population- and time-dependent. A model can be calibrated overall and miscalibrated for important subgroups or score ranges.

In 2017 a 5-layer LeNet and a 110-layer ResNet were set against each other on CIFAR-100. The ResNet was the better classifier: 30.6% error against 44.9%. It was also the worse of the two at saying how sure it was. Its average confidence sat substantially higher than its accuracy, while LeNet's average confidence closely matched its own. Guo and colleagues ran that pairing, and it is what stands behind the line their paper is usually quoted for. They “discover that modern neural networks, unlike those from a decade ago, are poorly calibrated”, and they observe that “depth, width, weight decay, and Batch Normalization are important factors influencing calibration”. Their remedy was narrow and useful: “on most datasets, temperature scaling”, described as “a single-parameter variant of Platt Scaling”, “is surprisingly effective at calibrating predictions”.

Read that as an empirical claim about one generation of architectures, not as a law. Four years later the measurement was repeated at much larger scale: 180 models from 16 families, 79 ImageNet-scale datasets, 28 metric variants. For the newest, non-convolutional architectures the trend did not hold. Minderer and colleagues state the finding directly: “Our results suggests that there is no general trend for recent or highly accurate neural networks to be poorly calibrated compared to older or less accurate models.” Measure this property on your own population. Do not infer it from the model's vintage.

Log loss and Brier score penalize different shapes of error. Both sit inside a class with a formal guarantee attached, and Gneiting and Raftery set that guarantee out in 2007. The definition is the whole point of the class: “It is strictly proper relative to P if (1) holds with equality if and only if P = Q, thereby encouraging honest quotes by the forecaster.” The quadratic or Brier score is their first example, traced back to Brier in 1950; the logarithmic score is their third. Ferro proved the Brier score proper independently the same year, by showing that its expected contribution is minimised by issuing your true belief p. He also found the ensemble-size-adjusted version improper. It “should not be used in situations where it could be hedged”.

What an improper score does instead is not a thought experiment. The same review scores a real verification database: 16,015 sea-level-pressure forecast records over the North American Pacific Northwest, from a five-member University of Washington MM5 ensemble in 2000. Those forecasts had an RMSE of 3.30 mb against an ensemble spread of 2.13 mb, a ratio of r0 = 1.55. Each rule was then asked which inflation factor it preferred. The strictly proper rules were maximised at r = 2.18 (quadratic), 1.84 (spherical), 2.41 (logarithmic) and 1.62 (CRPS). Every one of them sits above the 1.55 already in the data. The two intuitively appealing but improper rules pointed the other way entirely: “The linear and probability scores were maximized at r = .05 and r = .02, thereby suggesting ignorable forecast uncertainty and essentially deterministic forecasts.”

A scoring rule is a set of instructions to whoever issues the probability. Log loss instructs by punishing confident probability assigned to the wrong outcome very heavily. Brier instructs through squared distance between the predicted probability and the outcome. The calibration view examines empirical frequency within score neighborhoods or bins; the decision view translates probabilities into actions under a utility or cost model. Two of the rules in that case study told the forecaster to shrink all uncertainty to nothing. They did it on real forecasts. That is why propriety is checked before a score is adopted, not after.

Probability quality cannot be reduced to one word. Three separate things can go wrong, and they fail independently.

Calibration asks whether predicted probabilities match observed frequencies. It is a group-level property, it depends on the population it is measured over, it can vary by subgroup, and it guarantees nothing about ranking. Discrimination asks whether higher-risk cases tend to receive higher scores. It is measured by ranking metrics, it can be strong while calibration is poor, it supports prioritization, and it sets no absolute risk for anyone. Sharpness asks whether predictions are concentrated away from uninformative values. It reflects the confidence distribution, it is useful only alongside calibration, it rewards overconfidence when read on its own, and it must be judged against outcomes.

So a model can pass on one axis and fail the decision anyway. It can be calibrated overall and miscalibrated exactly in the score range where the thresholds sit. It can separate the classes sharply and still be wrong about absolute risk. Measure each property on its own terms, then read the three together.

A plot can look different without the model changing. And in clinical prediction, producing one is not optional. TRIPOD+AI settled that: a 27-item, 52-subitem reporting checklist covering any published clinical prediction model, regression or machine learning alike, published in the BMJ in April 2024. Collins, Moons and colleagues built it through a two-round Delphi, 170 participants and then 200, working from an initial list of 65 candidate items and closing it at a consensus meeting in July 2022. Its item 12e turns the reliability diagram into a floor rather than an option: “It is generally expected that as a minimum, model discrimination and calibration (including calibration plots) are presented”. The checklist asks for a smoothed flexible calibration curve, observed against estimated.

That settles whether you draw the plot. Everything below is a choice the author still has to make, and defend.

Post-hoc methods are fitted models, so they need their own boundary. That boundary has a price you can put a number on. Kumar and colleagues measured it in 2019: “We find in this work that popular recalibration methods like Platt scaling and temperature scaling are (i) less calibrated than reported, and (ii) current techniques cannot estimate how miscalibrated they are.” Histogram binning escapes that problem and pays for it elsewhere. It “has measurable calibration error but is sample inefficient”, needing O(B/e^2) samples where scaling methods need O(1/e^2). Their own scaling-binning calibrator needs only O(1/e^2 + B). On CIFAR-10 and ImageNet it obtained “a 35% lower calibration error than histogram binning”. Roelofs and colleagues arrive at the same conclusion from the estimator side: the standard binned estimate of calibration error is statistically biased, and better estimators change which recalibration methods look effective.

So the calibration set is not free bookkeeping. It is a sample-size requirement that moves by an order of magnitude with the method you choose. And the number it produces has its own error bar.

The sequence follows from that. First, reserve calibration evidence: use validation or cross-validated predictions rather than the final test labels. Second, fit the mapping on that evidence alone — Platt scaling, isotonic regression, temperature scaling, or another justified method. Third, recheck discrimination, confirming that the mapping has not harmed ranking or class decisions unexpectedly. Fourth, evaluate independently, measuring proper scores and reliability on untouched cases. Fifth, monitor after release: prevalence, score distribution, delayed outcomes, subgroup calibration. A mapping fitted once describes the population it was fitted on.

Well-calibrated probabilities can support expected-cost decisions, risk communication, and threshold portability under stable conditions. They still depend on correct labels, representative data, and valid utility assumptions.

The degenerate case has a name and a long history. A climatological forecast is the base rate, issued to everyone. In Gneiting and Raftery's words, such forecasts are among the predictions that “they are calibrated by construction, but often lack sharpness”. Expected calibration error has exactly the same hole. It is not a proper scoring rule, so the useless model optimises it. Ovadia and colleagues spell that out: “Neither of these is a proper scoring rule, and thus there exist trivial solutions which yield optimal scores; for example, returning the marginal probability p(y) for every instance will yield perfectly calibrated but uninformative predictions.”

That is why the target is not calibration alone. Gneiting and Raftery state the goal as maximising sharpness subject to calibration: be as concentrated as the evidence allows, and be honest about the concentration. A calibrated model can be useless if it assigns nearly the same probability to everyone. A sharp model can be dangerous if the sharpness is unearned. Read calibration and discrimination together, on the population where the system will actually be used.