Error Analysis: Learn From the Examples the Model Misses

Epic's proprietary sepsis prediction model was already running in hospitals when an outside team tested it. Wong and colleagues validated it at Michigan Medicine and published the result in JAMA Internal Medicine in 2021. The study covered 27,697 patients and 38,455 hospitalizations between 6 December 2018 and 20 October 2019.

The headline was a single figure: a hospitalization-level area under the ROC curve of 0.63 (95% CI, 0.62-0.64). That number tells a team the model is weak. It does not tell them what to change. Two different failures were folded into it, and their fixes pull in opposite directions. The abstract records both in one sentence: “The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue.”

Lower the alerting threshold to recover the missed two thirds and the 18% alert burden grows. Raise it to spare the clinicians and the misses grow. The 0.63 is the average over that trade. It names neither half, and no amount of staring at it produces a next step. Counting the cases does.

Error analysis is the systematic inspection and grouping of cases where a model, threshold, or workflow behaved poorly. The goal is not to shame the model or collect anecdotes. It is to identify recurring causes whose fixes can be tested. The Michigan Medicine numbers are what that inspection produces: not a better summary of 0.63, but two separately countable populations — 1,709 missed patients and 6,971 alerts — each with its own candidate intervention.

Review should include correct cases near the threshold and surprising successes as well as obvious errors. Those examples reveal fragile boundaries, and they reveal useful signals the model may be exploiting.

One bad output may involve several layers simultaneously, and the layer decides the remedy. At the top is the observed product failure: the user, reviewer, or downstream process experiences a harmful or unhelpful result. Beneath it sits decision-policy failure, where threshold, ranking capacity, fallback, or routing turns a score into the wrong action — the layer the sepsis model's 18% alert rate lives in. Beneath that, model failure: the fitted mapping scores or predicts the case poorly. The two lowest layers are the ones teams reach for last and the ones most often responsible.

Representation failure means the inputs omit, distort, or leak the evidence needed for the task. Pneumonia CNNs trained on 158,323 chest radiographs from NIH, Mount Sinai and Indiana University looked strong at home. The best internal model scored AUC 0.931 (95% CI 0.927-0.936). On the external site the same model scored 0.815 (95% CI 0.745-0.885). Zech and colleagues published the reason in PLOS Medicine in 2018, and it was in the pixels rather than in the architecture: the networks identified the source hospital system for 99.95% of NIH and 99.98% of Mount Sinai radiographs. The paper's conclusion reads: “CNNs robustly identified hospital system and department within a hospital, which can have large differences in disease burden and may confound predictions.” The same pattern turned up later in COVID-19 radiograph classifiers, reported in Nature Machine Intelligence — models that “rely on confounding factors rather than medical pathology” and “fail when tested in new hospitals”. Capacity cannot repair an input that carries a proxy for the answer.

Data and target failure means sampling, labels, definitions, or collection do not support the intended claim. It reaches the reference benchmarks themselves. Northcutt and colleagues went through ten standard test sets and reported: “Errors in test sets are numerous and widespread: we estimate an average of at least 3.3% errors across the 10 datasets, where for example label errors comprise at least 6% of the ImageNet validation set.” The flagged candidates were then checked by people, and crowdsourcing confirmed 51% of them wrong. The consequence is not cosmetic. On corrected ImageNet, ResNet-18 beats ResNet-50 once the prevalence of originally mislabelled test examples rises by just 6%. The label errors were choosing the model. Beyer and colleagues re-annotated the same validation set and found “the original ImageNet labels to no longer be the best predictors of this independently-collected set”.

Sampling strategy determines which failure story the team sees. A random error sample estimates the mix of common error types, supports rough frequency estimates, needs enough examples per category, and may miss rare severe cases. An uncertainty or boundary sample examines cases near thresholds or with model disagreement, shows fragile decisions, guides labeling, and overrepresents ambiguous cases. A severity sample selects high-cost outcomes regardless of frequency, needs domain expertise, and may require retrospective search. A slice-targeted sample focuses on languages, sites, products, or other important conditions. It is not representative of overall frequency. It is the one that finds failure modes an average has already dissolved.

What a slice-targeted review can produce is a subclass nobody had thought to define. The false negatives of a chest-radiograph model were audited by eye, one image at a time. Oakden-Rayner and colleagues reported what the audit yielded: one variable, whether the radiograph contained a chest drain. The paper states it plainly: “While overall pneumothorax ROC AUC closely matches that reported in Rajpurkar et al. [6] at 0.87, pneumothorax ROC AUC was 0.94 on the subclass with chest drains, but only 0.77 on the subclass without chest drains.” On the CXR14 pneumothorax task, 80% of test-set pneumothoraces contained a drain. Positive predictive value was 0.90 on that subclass against 0.60 on the other. Every one of those numbers was inside the 0.87, and none of them was visible from it.

Two audits of commercial face systems make the same point at industrial scale. Buolamwini and Gebru measured up to 34.7% error for darker-skinned females in 2018, against a maximum of 0.8% for lighter-skinned males, across three commercial gender classifiers. NIST's Face Recognition Vendor Test Part 3 ran 18.27 million images of 8.49 million people through 189 mostly commercial algorithms from 99 developers. Its executive summary, in December 2019, put the result in one line: “Across demographics, false positives rates often vary by factors of 10 to beyond 100 times.” A single accuracy figure over either corpus would have been an average of that spread. It would have moved with the composition of the test set rather than with the system.

Consistent review makes qualitative evidence more reliable. Define the sample first, recording selection logic, date, model version, and threshold. Hide avoidable cues, blinding reviewers to model identity or previous judgments where possible. Record observations: expected output, available evidence, harm, and likely failure layer. Resolve disagreement by discussing ambiguous cases and updating the target or taxonomy definitions. Quantify categories with frequency, severity, and slice concentration, carrying uncertainty. Propose tests, tying each candidate fix to a prediction about which errors should change.

This is not merely good practice. In clinical AI it is a written reporting requirement. CONSORT-AI is the international reporting standard for randomised trials of AI interventions, published in Nature Medicine in September 2020. Behind it sit a 103-stakeholder two-stage Delphi, a two-day 31-stakeholder consensus meeting and a 34-participant checklist pilot. Its item 19 reads: “Describe results of any analysis of performance errors and how errors were identified, where applicable. If no such analysis was planned or done, explain why not”. Note what item 19 asks for that a metrics table cannot supply: how the errors were identified. That is the sampling decision from the previous section.

The compliance figure is the interesting part. The guideline's own authorship group later audited 65 published AI randomised trials and reported it in Nature Communications in 2024. Item 19 came out among the worst-reported AI-specific items, at 77% concordance against a checklist-wide median of 90% (IQR 77-94%). Their conclusion about exploring AI errors was that it “remains non-standard practice”. The step teams skip is the one written down as mandatory.

A frequent minor inconvenience and a rare catastrophic error should not compete on frequency alone. Four dimensions decide the queue.

Frequency asks how often the failure occurs under the target distribution: use representative samples, report uncertainty, separate global from slice frequency, and beware selection bias. The sepsis model's alerts on 18% of 38,455 hospitalizations is a frequency claim. The 67% miss rate is a claim about a different denominator, and the two are not comparable without saying so.

Severity asks what the cost or harm is when it occurs: consider the affected people, include downstream consequences, account for reversibility, and escalate safety-critical cases. A false-positive spread of 10 to beyond 100 times across demographic groups matters more where the false positive triggers an intervention than where it triggers a suggestion.

Tractability asks whether a plausible, testable intervention exists: new data may be collectable, feature pipelines may be repairable, policy ambiguity may need governance, and some ceilings are irreducible. A subclass label such as chest drain is tractable precisely because it can be annotated.

Confidence asks how strong the evidence for the diagnosis is: separate observation from hypothesis, replicate on another sample, use ablations and controlled tests, and keep competing explanations alive. Northcutt and colleagues sent algorithmically flagged label candidates to crowdsourced human validation, and 51% were confirmed wrong. The flag was a hypothesis. The 51% was the evidence. The Epic model's numbers survived a second team, 145,885 encounters and a different pair of hospitals.

Suppose adding attachment extraction is expected to reduce “missing document evidence” errors. Re-run the same category definition on fresh examples. Measure whether those errors fall without creating new harms elsewhere.

The chest-drain finding shows the loop closing on someone else's data. A second team tested the diagnosis in European Radiology and confirmed it: pneumothorax algorithms trained on public data are “strongly limited and biased by confounding thoracic tubes”. Then they acted on it. AUROC rose from 0.778 to 0.877 once in-image pleural annotations were added. The prediction was that fixing the representation would move the number for the subclass that had been carrying the failure. The number moved. Beyer and colleagues closed the same loop on labels, by re-annotating ImageNet rather than by training anything larger.

If the category does not change, the diagnosis was incomplete or the implementation did not affect the intended pathway. Record negative results; they improve the next hypothesis.