Thresholds, Costs, Utilities, and Capacity

A classifier may output 0.63. That number does not say whether to block, review, defer, or ignore. The action comes from a policy that combines the score with costs, capacity, eligibility, and sometimes other evidence. Treating 0.5 as a universal default confuses a mathematical convention with an operational decision.

Medicine formalized this in 2006. Decision curve analysis starts from the patient rather than from the classifier. Vickers and Elkin begin by assuming that the threshold probability of a disease or event at which a patient would opt for treatment “is informative of how the patient weighs the relative harms of a false-positive and a false-negative prediction”. That relationship “is then used to derive the net benefit of the model across different threshold probabilities”. “Plotting net benefit against threshold probability yields” what the authors call the decision curve. The apparatus is modest. The method “requires only the data set on which the models are tested”. What it returns is not one cutoff. It is a curve across all of them.

Several transformations can sit between a model and an action. One of them has been published in full by a regulator. Apple's Irregular Rhythm Notification Feature received a De Novo classification, DEN180042, on 8 August 2018, and the confirmation layer is written into the FDA decision summary: “If a sufficient number of tachograms are retrieved and classified to meet the notification threshold (5 of 6 sequential tachograms classified as irregular within a 48-hour period), a notification indicating that the heart rhythm has shown signs of AF will be displayed to the user.”

Read that as the diagram above. The model output is a per-tachogram classification. The decision policy is a rule over six of them inside a 48-hour window. The operational action is one notification to one person. The Apple Heart Study followed 419,297 participants over a median 117 days of monitoring. That policy notified 2,161 of them, 0.52%, with a positive predictive value of 0.84 (95% CI 0.76 to 0.92) for atrial fibrillation on ECG concurrent with a subsequent notification. Nothing in the classifier fixes the 0.52%. The layer above it does.

Cost matrices are useful. Real consequences are richer than two constants, and one recommended cutoff can turn an uncommon event into a permanent alarm.

Michigan Medicine's own hospital operations committee set its sepsis alert at a score of 6, inside the 5-8 range the model's developer recommends. Wong and colleagues then validated the Epic Sepsis Model against what that produced: 38,455 hospitalizations among 27,697 patients, containing 2,552 sepsis cases — 6.6% of hospitalizations. At that threshold the hospitalization-level AUC was 0.63 (95% CI 0.62-0.64). Sensitivity was 33%. Positive predictive value was 12%. The abstract puts the operational consequence plainly: “The ESM did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue.” A policy that fires on 18% of every admission is a staffing decision wearing a statistical costume. Ostermayer and colleagues repeated the exercise at Harris Health, across 145,885 encounters at the same score threshold of 6. Positive predictive value there was 7.6%.

One level up, a national guideline body has documented the same problem in its own rules. NICE sets urgent cancer referral at a 3% positive predictive value. An editorial in the British Journal of General Practice records where that number came from: “This was a reduction from the previous 5% threshold and was based on clinical consensus balancing diagnosing more cancers in a timely manner against overburdening the health service and potential implications for individuals (for example, unnecessary anxiety).”

The same editorial shows what the cutoff means once the population under it moves. The PPV of CA125 at the usual 35 U/mL threshold is 9%. The individualised risk of a woman whose CA125 is 35 U/mL is about 1%. A 45-year-old with CA125 38 U/mL has an estimated 0.6% ovarian cancer risk. A 70-year-old with the identical value has 3.1%. One test, one number, a five-fold difference in what it means.

Two population breast-screening programmes have measured what changes when a model score routes cases through a reading workflow rather than replacing the reader. The interesting differences are not accuracy differences. They are reviewer-hour differences.

When uncertainty is high or consequences are severe, abstention can be better than forced classification. The system can request more information, route to a specialist, or delay action. The Irregular Rhythm Notification Feature is an abstention policy read from the other end. Five of six sequential tachograms have to agree within 48 hours before anything is said to the user. Everything short of that is a decision not to decide yet.

Abstention must be evaluated for coverage, residual error, subgroup burden, delay, and fallback quality. It is not a free safety feature. The 67% of septic patients the Epic Sepsis Model did not identify at a score threshold of 6 were, from the alerting system's point of view, cases it declined to flag. The fallback pathway is what decided their outcome.

The threshold should be reproducible and reviewable. The FDA decision summary for the Apple notification feature is what the last step of this process looks like when it is done properly. The notification rule — 5 of 6 sequential tachograms classified as irregular within a 48-hour period — is written where a reader outside the company can find it, read it, and check the action volume it produced. NG12 is the same discipline applied to a committee's judgement rather than a device's firmware. The 3% bar, the body that agreed it, and the cases exempted from it are all on the record. Neither document tells you the threshold is correct. Both make it possible to argue about. That is the property a versioned policy is supposed to have.