Error Analysis, Calibration, and Abstention

Suppose an entity extractor loses five F1 points after it is deployed. The number is one number. The six candidates below are six different repairs, and the metric cannot say which one it is describing.

The annotation mechanism alone can produce a drop of exactly this size. That has been measured. In 2019 Geva and colleagues held the amount of training data fixed and changed one thing: whether the test examples came from annotators the model had already seen during training. On OpenBookQA, across all five annotator sets, “performance drops by up to 23 accuracy points”. The same experiment cost “up to 10 accuracy points in COMMONSENSEQA and 5 in MNLI”. No model weight changed. No preprocessing changed. No vocabulary changed. Only who had written the sentences. Their conclusion is plain: “This shows that the model does not generalize to examples generated by unseen annotators.”

Five points on MNLI is the drop this section opens with, and its cause was a property of the labour pool. A team that read that number as a modelling failure would have retrained a model that was never broken.

Classifying failures by mechanism prevents indiscriminate retraining. The layers below are not a filing cabinet for blame; they are the list of things that can be true at the moment the score falls.

The hardest pair to separate is data and annotation from representation and model. A shortcut sitting in the data looks, from the outside, exactly like competence in the model. Gururangan and colleagues measured that gap in 2018. Natural language inference asks whether a premise entails a hypothesis, so a system shown only the hypothesis should be guessing. Theirs was not: “Specifically, we show that a simple text categorization model can correctly classify the hypothesis alone in about 67% of SNLI (Bowman et. al, 2015) and 53% of MultiNLI (Williams et. al, 2017).”

Two thirds of one benchmark is reachable without the input the task is defined around. The reported score was partly a record of how the data was collected, not of inference ability. Every aggregate metric reports that as model performance. A bigger model, retrained, preserves it faithfully.

A classifier is calibrated when predictions assigned a confidence near 0.8 are correct about 80 percent of the time for comparable cases; calibration can differ by class, domain, language, document length, and time.

A ranking score, cosine similarity, or generative token probability is not automatically a calibrated probability of task correctness. Calibration requires an explicit target and held-out evidence.

This property is not inherited from accuracy. Guo and colleagues showed that in 2017: “Modern neural networks, unlike those from a decade ago, are poorly calibrated”. They trace the drift to “depth, width, weight decay, and Batch Normalization”. The repair turned out to be unexpectedly cheap. On most datasets temperature scaling, “a single-parameter variant of Platt Scaling”, is “surprisingly effective at calibrating predictions”. A model can become more accurate and less honest about its own confidence in the same training run.

The phrase “under conditions” is the load-bearing one, and Desai and Durrett put a measurement on it in 2020. They evaluated BERT and RoBERTa on natural language inference, paraphrase detection and commonsense reasoning, in-domain and out-of-domain. The same weights were well behaved in one setting and not the other: “when used out-of-the-box, pre-trained models are calibrated in-domain, and compared to baselines, their calibration error out-of-domain can be as much as 3.5x lower”. The repair is condition-specific too. Temperature scaling is what works in-domain; label smoothing is what works out-of-domain. So a calibration report describes a model and a condition jointly. Carried across the boundary it was measured on, it is a claim about a setting the team never tested.

Combining them requires care and validation — and all of them move in the same direction when conditions do.

Under dataset shift, no method keeps its uncertainty honest. That was the finding of the first large-scale comparison of predictive uncertainty under shift, run in 2019 by Ovadia and colleagues across image, text-adjacent and tabular tasks. Their Section 5, “Takeaways and Recommendations”, states it without qualification: “Along with accuracy, the quality of uncertainty consistently degrades with increasing dataset shift regardless of method.”

Post-hoc calibration is the clearest casualty, and the paper says exactly how it fails: “while temperature scaling achieves low ECE for low values of shift, the ECE increases significantly as the shift increases, which indicates that calibration on the i.i.d. validation dataset does not guarantee calibration under distributional shift”. A temperature fitted on a held-out split from the same distribution is a repair for the condition it was fitted in. It is not a durable property.

What held up best was disagreement between independently trained models — “Deep ensembles seem to perform the best across most metrics and be more robust to dataset shift” — with roughly five members capturing most of the gain. The signal in the comparison below that costs the most compute is the one that survives the move.

The output should be a prioritized set of hypotheses and experiments.

A rare unsupported medical instruction may deserve more attention than a common punctuation error; conversely, a severe-looking one-off example may not justify a broad model change.

Prioritize expected harm, frequency, confidence, detectability, fix cost, and possible regression; product owners and domain experts should participate in the decision.

Sort validation predictions by confidence. Then compute error rate as lower-confidence cases are deferred. Repeat by class and one important user slice.

Estimate human-review workload and correction accuracy at several thresholds. The best operating point may differ across labels.

What it costs to skip that estimate is on the record. Wong and colleagues published an external validation of the proprietary Epic Sepsis Model in JAMA Internal Medicine in 2021. They ran it over 38,455 hospitalizations at Michigan Medicine, of which 2,552 (7%) involved sepsis. Hospitalization-level AUC was 0.63 (95% CI, 0.62–0.64). At the alert threshold of 6, the operating point reads: “The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue.”

Read that as a point on a risk–coverage curve nobody plotted. Nearly one admission in five generated review work. Two thirds of the patients the tool existed to find were missed anyway. That is the workload of a low threshold with the misses of a high one. The authors' conclusion is one sentence: “This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis.”

An independent team reached the same place three years later. Writing in JAMIA Open in 2024, they ran the model over 145,885 emergency-department encounters at two county hospitals and reported, in a 6-hour window, “a sensitivity of 14.7%, specificity of 95.3%, positive predictive value of 7.6%, and negative predictive value of 97.7%”. Their verdict: “The ESPMv1 provides suboptimal diagnostic characteristics for undifferentiated patients in a county ED.” Every one of these numbers was available before deployment, from a validation set and a curve.