Proper Scoring Rules, Calibration, and Decision Theory

A scoring rule assigns a numerical loss to a predictive distribution after the outcome is observed. A proper scoring rule is minimized in expectation when the forecaster reports the distribution it actually believes. That is what ties probability estimation to incentives. Honest reporting is not asked for as a virtue. It is the strategy with the best expected score.

The split between calibration and sharpness is not a folk distinction. Gneiting and two colleagues set out the paradigm in 2007, in the Journal of the Royal Statistical Society. Calibration there is statistical consistency between the forecasts and the observations. Sharpness is the concentration of the predictive distributions. And the goal the paper proposes is “maximizing the sharpness of the predictive distributions subject to calibration”. The order of those two words is the whole content of the rule. Sharpness is what you maximize. Calibration is the constraint you may not spend to get it.

The principle outgrew probabilities. A 2020 paper by Marc-Oliver Pohle describes its own contribution as one that “generalizes the famous sharpness principle by Gneiting et al. (2007) from probabilistic to all types of forecasts”.

The measurement tradition is older still. Glenn W. Brier's paper on verifying forecasts expressed in terms of probability ran to three pages in Monthly Weather Review in January 1950. Log loss and Brier score are both proper under standard conditions, but they emphasize errors differently, and neither one is calibration. A forecast can be calibrated and uninformative. It can be sharp and systematically wrong.

Consider two binary classifiers that both classify 80 of 100 examples correctly.

Different losses elicit different predictive objects, and the choice quietly decides what your evaluation is able to notice.

Log loss is the negative log probability assigned to the observed outcome. It is strictly proper for full categorical probabilities, it connects directly to likelihood and cross-entropy, and it punishes confident mistakes hard. That last part follows from its sensitivity to near-zero probabilities, where −log p grows without bound.

Brier score is the squared error of probabilistic predictions. It is proper for probabilities, bounded in binary classification, and supports the calibration decompositions Brier's own 1950 paper set in motion. It penalizes extremes less sharply than log loss. That makes it more forgiving of a single catastrophic overconfidence, and less useful for detecting one.

Zero-one loss counts whether the chosen class is wrong. It evaluates a hard decision, ignores probability quality entirely, is not a proper score for full distributions, and depends on the threshold you happened to set. Two models with identical zero-one loss can differ arbitrarily in everything a proper score measures. That is exactly the situation the 80-of-100 example above describes.

A probability estimate passes through several layers before it becomes an action. First the predictive distribution. Then a calibration check, comparing stated probabilities with observed frequencies. Then a scoring rule, measuring distribution quality across cases. Then a utility or cost, translating outcomes and actions into decision value. Finally a decision rule, choosing an action, a threshold, or an abstention policy. Every layer can fail on its own, and a deployed system can fail at several at once without any of them raising an alarm.

The Epic Sepsis Model is the worked example, because it ran at hundreds of US hospitals before anyone outside the vendor scored it. Wong and colleagues validated it externally at Michigan Medicine, on 27,697 patients and 38,455 hospitalizations between 6 December 2018 and 20 October 2019. Sepsis occurred in 2,552 hospitalizations, 7% of the total. The model's hospitalization-level AUROC was 0.63 (95% CI, 0.62–0.64). Their conclusion in JAMA Internal Medicine names both failures at once: “This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis.”

The decision layer was measured separately, elsewhere, and failed on its own terms. A 2024 validation in JAMIA Open covered 145,885 emergency-department encounters at two Harris Health county hospitals. Within a 6-hour window it reported sensitivity of 14.7%, specificity of 95.3% and a positive predictive value of 7.6%. A specificity above ninety-five percent looks reassuring, and it is not the operative number. At that positive predictive value, most of what the alert fires on is not sepsis. At that sensitivity, most of the sepsis is not alerted on. Discrimination, calibration and the alerting decision were each measured. Each one failed.

A model output read as a probability calls for the sequence below. Define the event — outcome, horizon, population. Check probability semantics — confirm the model output can support a probabilistic interpretation at all. Select a proper score, matching its sensitivity to confident errors against operational needs. Assess calibration with reliability diagrams, calibration error, and slice checks. Add utility, translating probabilities into action-specific costs and benefits. Monitor after shift, rechecking base rates, calibration, and thresholds over time.

Step one carries more weight than its position suggests, because a score is silent about whether you scored the right event. A commercial risk algorithm applied to millions of patients ranked Black patients as equally risky as healthier White patients. Obermeyer and three co-authors published that in Science on 25 October 2019. The algorithm was not broken at the modeling layer. It was predicting a different quantity from the one the decision needed: “The bias arises because the algorithm predicts health care costs rather than illness, but unequal access to care means that we spend less money caring for Black patients than for White patients.”

The size of the gap between the scored event and the intended one is measurable. Correcting the target would have raised the share of Black patients auto-identified for extra help at the 97th-percentile threshold from 17.7% to 46.5%. Nothing in a proper score computed against predicted cost would have flagged that. Against predicted cost, the model was doing well.

The decision layer moved fast once the paper appeared. On the same day, New York's Superintendent of Financial Services Linda A. Lacewell and Health Commissioner Howard A. Zucker wrote jointly to UnitedHealth Group about Impact Pro. The model was accurate. The question was what it was accurate about.

A model that always predicts the base rate can be perfectly calibrated when evaluated on a stable population. It contains no case-specific discrimination. Sharpness describes how concentrated the predictive distributions are, and the 2007 paradigm is a demand for both: sharpness subject to calibration, not calibration alone.

Calibration is also group- and time-dependent, and a globally calibrated model may be miscalibrated in important slices. The sharper problem is that per-group calibration can be genuinely satisfied and still be incompatible with the fairness property people assume it delivers. ProPublica analyzed 7,214 Broward County defendants on 23 May 2016 and found that COMPAS's false-positive rate was 44.85% for Black defendants against 23.45% for White defendants. Northpointe maintained that the score was calibrated by group. Both positions can be correct at once.

That is a theorem, not a controversy. Alexandra Chouldechova proved it in 2017 and states it directly: “We demonstrate that the criteria cannot all be simultaneously satisfied when recidivism prevalence differs across groups.” Kleinberg and two co-authors reached the same wall independently that year, proving that outside highly constrained special cases no method satisfies all three fairness conditions simultaneously. When prevalence differs across groups, calibration is a choice made against other properties. It has to be argued for, not assumed.