Metric Portfolios, Acceptance Criteria, and Release Gates

A single headline number would have passed the Epic Sepsis Model. Wong and colleagues measured it from outside. Across 38,455 hospitalizations at Michigan Medicine the model's area under the curve was 0.63 (95% CI 0.62–0.64), against the vendor's claimed 0.76–0.83. The portfolio failure is in the next sentence of the abstract: “The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue.” That is Wong, Otles and colleagues, in JAMA Internal Medicine in 2021. Miss 67% of sepsis cases and alert on 18% of all hospitalizations at the same time, and you have a number needed to evaluate of 8. Eight alerts read for one case found.

A second, independent validation reached the same verdict on different patients. In two Houston county emergency departments, across 145,885 encounters, the model showed sensitivity of 14.7%, specificity of 95.3% and a positive predictive value of 7.6%. Ostermayer and colleagues concluded that it “provides suboptimal diagnostic characteristics”. Note which number is the high one. Specificity of 95.3% is the figure a dashboard leads with, and it is the figure that decides nothing here. Teams collect dozens of numbers and choose afterward which ones mattered. A portfolio decides first. It names each number's job before any of them are read.

Assign the roles before the numbers arrive and no measurement has to compete for headline status. The sepsis validations supply one of each.

The primary outcome is the quantity attached to the decision: whether patients with sepsis are identified. 67% were not. A guardrail is a harm, cost or degradation that must not exceed its limit. Alerts on 18% of all hospitalizations, a number needed to evaluate of 8, a positive predictive value of 7.6% — that is what a guardrail looks like when nobody wrote one down. A diagnostic explains behaviour and directs improvement without deciding anything. An AUC of 0.63 (95% CI 0.62–0.64) against a claimed 0.76–0.83 tells you the model does not discriminate as advertised, but the release decision was already made by the two numbers above it. An operational measure decides feasibility: latency, capacity, availability, cost, reviewer workload. It is where the clinician's working day meets the model.

The same four figures, unlabelled and side by side, are a dashboard. The specificity of 95.3% is the one that gets quoted in the slide. Labelled by role, they are the beginning of a decision rule, and 95.3% is visibly answering a question nobody needed answered.

The price of leaving the analysis unfixed until the results are in has been measured twice, in two disciplines, with the same design. Hand many independent teams one dataset and one hypothesis. Count how far they diverge.

Twenty-nine teams — 61 analysts — were given identical soccer red-card data and a single primary research question. Silberzahn, Uhlmann and colleagues reported the spread in 2018: “The independent teams’ estimated effects for the primary research question ranged from 0.89 to 2.93 in OR units (1.0 indicates a null effect); no teams found a negative effect, 9 found no significant relationship, and 20 found a positive effect.” Twenty of twenty-nine — 69% — certified the effect. The remaining 9 found nothing to certify. Nobody cheated and nobody saw different data.

Botvinik-Nezer and colleagues repeated the experiment in neuroimaging: 70 teams, one fMRI dataset, the same 9 ex-ante hypotheses, 70 different analysis pipelines. From the abstract, in Nature: “The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data.” Not two workflows in agreement out of seventy.

That is what a post hoc score is worth. Inventing the combination rule after seeing which model wins does not merely invite cherry-picking, weaken reproducibility and hide tradeoffs. It places the conclusion inside a 69%/31% band that the analyst's own choices control.

A precommitted hierarchy removes that latitude: primary claim, guardrails, tie-breakers, written before evaluation. It exposes failure directly and still allows a documented exception. Where no hierarchy commands agreement, a Pareto analysis is the honest alternative — show every candidate not dominated across the chosen dimensions, avoid an arbitrary scalarization, and make the stakeholders choose in public. All three responses to conflict are available. Only the first is chosen after the answer is known.

There is a dated, published example of a gate written as a policy rather than presented as a value. It is worth following step by step.

1. State the primary claim. The IDx-DR pivotal trial named two: sensitivity and specificity for detecting more-than-mild diabetic retinopathy, in primary-care offices, on the patients the device was meant to serve.

2. Set minimum guardrails before enrolment. The FDA's own De Novo summary records the numbers as they stood before any data existed: “Performance thresholds were defined at 85.0% for sensitivity and 82.5% for specificity”. Not a target. Not an aspiration. A rejection region fixed in the protocol.

3. Add slice floors. An aggregate can be strong while a slice is catastrophic, and the gap has been measured across two orders of magnitude. Buolamwini and Gebru audited three commercial gender classifiers in 2018: “We evaluate 3 commercial gender classification systems using our dataset and show that darker-skinned females are the most misclassified group (with error rates of up to 34.7%). The maximum error rate for lighter-skinned males is 0.8%.” NIST went wider in 2019 — 189 algorithms from 99 developers, 18.27 million images of 8.49 million people — and found one-to-one false-positive rates higher for Asian and African American faces than for Caucasian faces “from a factor of 10 to 100 times”. No global accuracy figure in either study was doing anything but hiding that.

4. Include uncertainty in the rule itself. The IDx-DR endpoints were tested one-sided at a 2.5% Type I error. So the one-sided 97.5% bound had to clear the threshold, not the point estimate. In 900 participants at 10 primary-care sites the observed values were 87.4% sensitivity and 89.5% specificity, and the abstract states the verdict in the form the protocol demanded: “The AI system exceeded all pre-specified superiority endpoints at sensitivity of 87.2% (95% CI, 81.8–91.2%) (>85%), specificity of 90.7% (95% CI, 88.3–92.7%) (>82.5%), and imageability rate of 96.1% (95% CI, 94.6–97.3%)” — Abràmoff and colleagues, npj Digital Medicine, 2018. Sensitivity came in at 87.2% against a bar of 85%, and the lower bound of its interval, 81.8%, sits below that bar. The specificity margin and the imageability margin are what carried the trial. A gate that read only the three point estimates could not have told you that.

5. Define dispositions. The FDA granted the De Novo request on 11 April 2018 — one disposition among release, limited rollout, revise, collect more data, and reject. A gate that can only say yes is not a gate. The section that follows shows what the missing dispositions cost.

Plot quality against latency, reviewer hours, memory, or monetary cost. A model is dominated when another option is at least as good on every relevant axis and better on at least one. Dominated candidates should leave the room before anyone debates value tradeoffs.

The axes are rarely hypothetical. For the Epic Sepsis Model both were already measured at Michigan Medicine. Discrimination of 0.63 (95% CI 0.62–0.64) on one axis. On the other, an alert for 6971 of 38 455 hospitalized patients — a number needed to evaluate of 8, which is reviewer hours in the plainest possible units. When a candidate sits low on quality and high on operational cost at the same time, the frontier does the arguing and the stakeholders are spared it.

The same view reveals the opposite case, where a tiny metric gain demands a large operational sacrifice. It makes that sacrifice visible as a quantity rather than as a preference.

A decision table names each criterion, unit, population, threshold, uncertainty rule, owner, and resulting action. Written before the results, it makes exceptions visible and stops a dashboard from becoming an invitation to rationalize the preferred model. Metrics remain essential. Governance decides how they become a decision.

What it costs when the only available disposition is ship-to-everyone has a date and a size. On 19 July 2024 a CrowdStrike Rapid Response Content update — a new version of Channel File 291 — went to Falcon sensors with no staged rollout. The Content Validator assessed the new Template Instances expecting the template to be given 21 inputs. The sensor's integration code supplied 20. The resulting out-of-bounds read crashed machines worldwide. The blast radius was measured from outside the company: “We currently estimate that CrowdStrike's update affected 8.5 million Windows devices, or less than one percent of all Windows machines.” — David Weston, Vice President, Enterprise and OS Security at Microsoft.

CrowdStrike's own root cause analysis, dated 6 August 2024, lists the missing gate as Finding 6, “Each Template Instance should be deployed in a staged rollout”, and describes the disposition the release process did not have: “New Template Instances that have passed canary testing are to be successively promoted to wider deployment rings or rolled back if problems are detected.” Canary, ring, rollback — three dispositions between release and reject, absent from the table on 19 July.

The base rate that tells you what a gate is for comes from the same Bing paper. Its third tenet is “We are poor at assessing the value of ideas”, and the evidence is a count: “Only one third of the ideas tested at Microsoft improved the metric(s) they were designed to improve”. The authors add that “Success is even harder to find in well-optimized domains like Bing”. A release gate that never blocks anything is governing a process in which most proposals should not ship. A gate that refuses is doing the job it was written for.