Where Is a Prediction Model Miscalibrated, and for Whom? Local Audits in Outcome-Learned Pockets

AI summary

Overview

Research area: Machine learning — model auditing, calibration, and algorithmic fairness, with a clinical/health-informatics application focus.

Technical level: Advanced. The paper assumes familiarity with calibration, random forests, community detection on graphs, and multiple-testing / bootstrap inference.

Scope: The paper proposes and evaluates an audit procedure that uses outcome-learned "pockets" (clusters of observations that are similar in how their features predict the outcome) to localize where a prediction model is miscalibrated and whether its average error differs between demographic groups inside those pockets, with simultaneous confidence intervals that allow for correlated observations within units.

What This Paper Is About

Overall averages, group-level averages, and ranking metrics like the AUC can hide subsets of observations where a model systematically over- or under-predicts, because concentrated errors barely move an average and errors of opposite sign cancel each other out. The authors build an audit that answers two questions from features, outcomes and fixed predictions alone: where is a model miscalibrated (which sets of similar observations it over- or under-predicts on average), and for whom (whether the average prediction error differs between a focal group such as Black patients or one sex and a reference group within the same set). The goal is to turn a reassuring average into statistical evidence, with intervals, about specific subpopulations — and to say honestly when the data are too thin to test.

Key Contributions

1. Reusable, outcome-learned audit maps. A discovery-only procedure learns pockets from the outcome — a random-forest-similarity map and a ridge-regression-weighted map — and freezes the rule that assigns observations to them. Because neither the group attribute nor any model's errors are inputs to the maps, several models can be audited from their predictions alone, even when their internals are proprietary or undisclosed. The forest map requires no rescaling or imputation of numeric inputs, and a sparse nearest-neighbor graph keeps it scalable.

2. Statistical evidence rather than bare flags. Each pocket's local calibration error and local calibration gap are estimated with simultaneous confidence intervals built from a single-step max-t procedure with a wild cluster bootstrap, so intervals hold jointly across pockets and allow correlation within units such as hospitals. The audit also reports an equivalence share — when a pocket's simultaneous interval lies within (−t, t), pocket-average errors of size t or larger are ruled out — and explicitly reports pockets too thin to test rather than merging them.

3. An evaluation that reports its losses. In 17 simulation scenarios with a gap planted for one demographic group, the combined audit located it better than the strongest alternative grouping in 15; in 15 scenarios with an error planted for everyone in one feature region, it localized the error better than every available alternative in 12 at the smaller of two error sizes. The paper names the settings where it loses, false-alarm rates, and coverage checks.

4. Three applications, two to deployed scores with documented problems. A care-management algorithm's public synthetic data, APACHE IVa ICU mortality predictions, and two recalibrated chest X-ray classifiers.

Main Findings

• Offsetting local errors behind clean averages (care-management data). In public synthetic data from the algorithm studied by Obermeyer et al., with its cost-based score mapped to a prediction of chronic illness, the audit found no detectable overall error but under-prediction of illness in the sickest pockets and over-prediction in the others — under-prediction for a third of patient-years. Black patients were over-represented in that third, and a Black–White gap remained within it, so the audit separates concentration of a group in a badly served pocket from a gap between groups who share a pocket.

• APACHE IVa over-prediction localized (exploratory). The audit located previously reported over-prediction of ICU mortality in a pocket of mostly ventilated stays and in a low-risk pocket holding half of the stays with an APACHE prediction, where predicted mortality was more than twice the observed rate; none was detected in a moderate-risk pocket. This held under hospital dependence and limited support.

• Chest X-ray classifiers. Two recalibrated classifiers showed offsetting local errors behind no detectable overall error.

• Where outcome-learned similarity wins. With a U-shaped risk term, forest pockets reached F1 0.34–0.45 against 0.09–0.13 for ridge-weighted K-means and risk bins; under masking (a positive shift in one half of the region, negative in the other, so the population-wide gap is zero) they separated the two halves (0.49 and 0.44, against 0.37 and 0.18). With linear risk the two maps scored alike (0.62–0.63).

• Where it loses. The forest lost when the affected region was the top of the most predictive linear feature, which ridge-weighted K-means slices exactly (0.73 vs 0.67; WiDS ventilated patients, 0.77 vs 0.58); when only one tail of a U-shaped feature was affected, where pockets mix affected and unaffected observations and the disparity-targeted method won (0.38 vs 0.23); and in hospital-disjoint WiDS, where most affected observations sat in untested pockets (F1 ≤ 0.29 for every method). For local error it lost to an error-targeted grouping (which bins a boosted model of one model's residuals) in two irrelevant-feature scenarios (0.17 vs 0.07; 0.75 vs 0.43) and in the binary one-tail region (0.69 vs 0.28).

• Two maps fail differently, so both are run. The combined audit was better than its two component maps in 9 scenarios, not distinguishable in 6, and worse in 2; halving each map's level costs 3–4 points where the forest alone carries the signal, while each map finds affected observations the other misses.

• Error control. For the between-group gap, the two 450-replicate binary designs stayed well below 5% at their calibrated levels (α = 0.025 with independent observations, 0.0125 with hospital units), and the continuous designs at nominal 5% had intervals that include 5%. For local error with independent observations, the combined audit flagged 3.1–5.3% of null replicates, and each map at most 3.3% under a weak null whose errors cancel inside every pocket.

• A documented failure of the level calibration. With 150 dependent units of unequal size, the combined audit flagged 41 of 450 null replicates (9.1%; Wilson interval 6.8–12.1%) for local error at the level calibrated for the gap, and 9–11% per map under the weak null; a retrospective estimate puts the needed single-map level near 0.0025. The authors' explanation is that a pocket's mean error, unlike a within-pocket gap, does not difference out shared unit-level variation.

• Coverage checks. Under a planted gap, forest-map simultaneous coverage among replicates with a tested quantity ranged from 138 of 150 replicates in linear masking (92.0%; Wilson interval 86.5–95.4%) to 100%, against a 95% target, and was 13 of 15 in hospital-disjoint WiDS (86.7%; 62.1–96.3%). For local error at the smaller shift, coverage was 94.0–97.5% for the combined audit and 93.0–97.5% for the forest map. Conditional checks on the real audit observations gave false-flag rates of 0–0.4% per map in the ICU data (level 1.25%) and 1.0–4.0% for the combined care-management audit, with simultaneous coverage of at least 96.0%.

Methodology in Plain English

• Split the data three ways, by unit, once. Whole units such as hospitals go to a development sample (40%, where assessed models are fit or existing scores are used), a discovery sample (30%, where pockets are learned), and an audit sample (30%, where errors are measured). Audit outcomes are read once, after models, maps, the support rule and inference settings are frozen. Discovery is itself split 80/20 into observations used to build the pocket graph and observations used to calibrate the rule that assigns new observations to pockets.

• Build the first map from a forest grown on the outcome. A random forest is trained to predict the outcome (200 trees, at least 20 observations per leaf, √p candidate features per split; 500 trees and p/2 in the continuous simulations). Two observations are similar when many trees send them to the same leaf, so features that matter for prediction drive the similarity and irrelevant features fade — a built-in feature weighting that K-means on standardized features lacks.

• Turn similarity into pockets. Storing all pairwise similarities is quadratic, so each observation links to roughly its 30 most similar partners, keeping a link only when mutual (at most 15n links). Pockets are the communities of that graph, found with a nested degree-corrected stochastic block model whose number of blocks is chosen by minimum description length. The hierarchy's level is chosen on discovery data only, picking the level with the most pockets projected to meet a stricter version of the audit's support rule (at least 150 audit observations per compared group, 75 in the continuous simulations, and 10 expected events and non-events for a binary outcome); ties go to the coarser level. Spectral clustering replaces the block model in the continuous-outcome simulations where the dense similarity matrix is affordable.

• Freeze assignment, allow abstention. Audit observations never enter the graph. Each pocket is stored as a profile of its members' leaf counts, and a new observation joins the pocket with the highest average similarity to its members. If its top affinity, or its lead over the second, falls below the 1st percentile on calibration observations, it abstains; abstention is reported by group.

• Add a second map for straight-line errors. A ridge map imputes and standardizes each feature, multiplies it by the absolute value of its ridge-regression coefficient (logistic for a binary outcome), and runs K-means into as many pockets as the forest map has. The default combined audit runs both maps and treats each as its own family at half the nominal level, bounding the overall false-flag chance by Bonferroni; observations flagged in opposite directions by the two maps are recorded as conflicts.

• Estimate two quantities per pocket. The local calibration error is the average of predicted minus observed outcome in a pocket; the local calibration gap is the difference in that error between focal and reference groups in the same pocket. The pocket "all" reproduces the usual overall check, so every audit also reports the result it refines.

• Test only where the data suffice. A quantity is tested only if each cell it compares has at least 100 audit observations (50 in the continuous simulations), at least 10 expected events and non-events for a binary outcome (computed from predictions, never outcomes), and, with hospital units, at least 15 distinct and 10 effective hospitals with no hospital holding more than 20% of the cell. The rule never looks at audit outcomes, so the tested set cannot be steered toward findings.

• Quantify uncertainty simultaneously. Standard errors sum contributions within each unit so correlated observations are allowed for. To control the chance of any false flag in a family (target 5%), every estimate divided by its standard error is compared with a single max-t critical value from a wild cluster bootstrap (B = 4,999 in the applications), where each unit gets one Mammen two-point weight that perturbs all its residuals together and residuals are centered at the null that every tested quantity is zero. For rare binary outcomes the standard errors are floored at the value the predictions imply under no error, and nominal levels (0.025 with independent units, 0.0125 with hospital units, 0.05 for continuous outcomes) are calibrated by simulation rather than assumed.

• Score localization against known truth. Simulations plant an error in a known region — a shift for focal-group observations there, or for everyone in a feature region — and score observation-level F1 over affected focal-group observations, with paired bootstrap intervals across replicates. Every comparator (K-means, Mahalanobis K-means, risk bins, coarsened exact matching variants, a disparity-targeted method, an error-targeted grouping, a truth-aware oracle) is learned on the same discovery data and put through the same support rule and inference, so only the grouping differs.

Why This Matters

Impact on research. The paper reframes calibration auditing as held-out inference over a finite set of pre-declared, outcome-learned subpopulations, with explicit false-flag targets and explicit statements of what the data can and cannot rule out. It also gives a clean conceptual separation that group-level fairness audits usually blur: a group can be badly served because it is concentrated where the model errs for everyone, or because the model errs differently for group members who are otherwise alike — and these call for different responses. Its negative results (where it loses, and the local-error failure under strong within-unit dependence) are reported rather than buried.

Real-world applications:

• Care-management enrollment. Health systems use commercial risk scores to select patients for extra care; the care-management audit shows how a score can look calibrated on costs while understating illness in the sickest patients, and shows where a demographic group concentrates in the under-predicted region versus where a gap persists inside it.
• ICU severity scoring and benchmarking. The APACHE IVa audit locates over-prediction of mortality in a mostly ventilated pocket and in a low-risk pocket holding half of scored stays, relevant to units that benchmark mortality with such scores.
• Regulatory and procurement review of proprietary models. Because the audit needs only features, outcomes and fixed predictions, a vendor's score can be audited without disclosure of its internals, and one frozen map can compare several candidate models on the same observations.
• Medical imaging deployment checks. The chest X-ray results show that recalibration can leave offsetting local errors intact, so an overall calibration check alone is not sufficient evidence of readiness.

Industry relevance. Any organization deploying risk scores under reporting or fairness obligations — health systems, insurers, lenders, and vendors selling scoring products — can use the procedure as an external audit that does not require model access, produces per-subpopulation statements with uncertainty, and reports which subpopulations it could not test rather than silently omitting them.

Future Directions

• Separate level calibration for local error under dependence. The audit's local-error statistic exceeded its 5% target with 150 dependent units of unequal size (9.1% versus a nominal 0.0125 per-map level), and the authors' retrospective estimate near 0.0025 for the single-map level is explicitly not yet confirmed on fresh replicates. A principled treatment of unit-level variation in pocket mean errors is the clearest open problem.

• Recovering errors the outcome-learned map cannot see. The pockets miss errors unrelated to the outcome and errors concentrated in one tail of a U-shaped feature, where groupings learned from a model's own residuals won. Combining outcome-learned pockets with error-targeted or disparity-targeted groupings, without giving up the frozen, model-agnostic map, is an obvious extension.

• Confirming the retrospective analyses. Several simulation rows reuse saved replicates retrospectively, the combined audit was specified after the single-map runs, and the ICU audit is described as exploratory; fresh seeds and prospective audits of deployed scores would test whether the reported patterns hold.

• Finer targets than pocket averages and broader comparison. The local error is a mean within a pocket, weaker than calibration at every predicted value, so opposite errors can still cancel inside a pocket; moving toward finer-grained targets, and toward multi-category group attributes audited as pre-declared focal–reference pairs (mentioned but not evaluated here), are natural next steps.

Target Audience

Researchers and practitioners in ML auditing, calibration and algorithmic fairness; biostatisticians and clinical informatics groups who benchmark or validate risk scores; and data and model-risk teams at organizations that must audit third-party or proprietary prediction models without access to their internals. Readers need comfort with confidence intervals, resampling inference and clustering to follow the method sections, though the framing questions and the applications are accessible to a wider audience.

Authors’ abstract

Overall and group-level averages, and ranking metrics such as the AUC, can hide sets of observations for which a prediction model systematically over- or under-predicts. We present an audit that asks where a model is miscalibrated (in which sets of similar observations it over- or under-predicts on average) and for whom (whether its average prediction error differs between demographic groups, such as by race or sex, within the same set). It groups observations into pockets using random-forest similarity, which weighs each feature by its importance for predicting the outcome, with a ridge-regression grouping alongside. Pockets are learned on one sample and tested on another, with simultaneous confidence intervals that allow for correlation within units such as hospitals; the audit needs only features, outcomes and fixed predictions. In 17 simulation scenarios with an error planted for one group, the combined audit located it better than the strongest alternative grouping in 15, with false-alarm rates near or below 5%; with a local error planted for everyone, it won in 12 of 15 scenarios at the smaller error size and lost when the error was unrelated to the outcome. In public synthetic data from the care-management algorithm studied by Obermeyer et al., it found no detectable overall error but under-prediction of illness in the sickest pockets and over-prediction elsewhere, and distinguished Black patients' concentration where the score under-predicted for everyone from a Black-White gap within pockets. For APACHE IVa, an exploratory analysis located its previously reported over-prediction of ICU mortality in a mostly ventilated pocket and a low-risk pocket, with none detected in a moderate-risk one. Two chest X-ray classifiers showed offsetting local errors behind no detectable overall error. Code: https://github.com/modenesi/pocket-audit