Post-Deployment Evaluation, Drift, and Delayed Labels

A fraud system’s average risk score rises after a new merchant category launches. The shift may reflect genuine risk, a benign change in traffic mix, a feature bug, or a policy change. Monitoring detects changes. Evaluation decides whether those changes matter for performance and decisions.

How far apart the two can sit is documented. The Epic Sepsis Model was proprietary and already widely implemented in hospitals when outsiders validated it, across 38,455 hospitalisations of 27,697 patients at Michigan Medicine. Wong and colleagues published the result in JAMA Internal Medicine in 2021. Sepsis occurred in 7% of those hospitalisations. The model raised an alert on 6,971 of them, 18%. By the dashboard, the system was working hard. Measured against outcomes, its area under the ROC curve was 0.63 (95% CI 0.62-0.64). At the recommended score threshold of 6, its sensitivity was 33%. Under “Conclusions and Relevance” the authors wrote: “This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis. The widespread adoption of the ESM despite its poor performance raises fundamental concerns about sepsis management on a national level.”

The finding held at other sites. Ostermayer and colleagues ran the same model over 145,885 encounters in two county emergency departments. Their 2024 report gives a sensitivity of 14.7% and a positive predictive value of 7.6% within a six-hour window. Alert volume was never the performance number. No amount of it substitutes for an outcome-anchored evaluation.

A risk model sends high-score applications to manual review. The review queue, not the world, is what the dashboard ends up describing. The bail literature calls this structure selective labels.

Each layer needs different evidence, and the five do not fail together.

Input drift is movement in feature distributions, missingness, volume, or source composition. The search behaviour underneath GFT is input drift, changing partly because its owner kept changing it. Score drift is model outputs moving even when the decision policy is unchanged. Decision drift is thresholds, rules, overrides, or capacity changing the actions taken. The Epic Sepsis Model’s recommended score threshold of 6 lives in this layer, not in the model. Concept drift is a change in the relationship between inputs and outcomes. It is where a treatment protocol can hide: asthma predicting lower pneumonia risk is a relationship manufactured by care, not by disease. Outcome drift is a change in prevalence, severity, cost, or user behaviour over time.

The layers matter because the same alert means different things at each. A score-drift alarm answers with a rollback. An input-drift alarm answers with a data contract. A concept-drift alarm answers with a re-examination of what the label means.

An evaluation system should name each clock.

Immediate proxies are fast and abundant: a click, a reviewer action, a system acknowledgment, a short-term behaviour. They are useful for incident detection. They can also be gamed, or induced by the treatment itself.

Delayed outcomes sit closer to the real objective: chargeback, readmission, failure, retention, adjudicated quality. They arrive slowly. They can be censored. They are where performance truth lives. Sepsis in 7% of hospitalisations is a delayed-clock quantity. It is the clock on which an area under the ROC curve of 0.63 and a sensitivity of 33% were measured. The 18% alert rate runs on a different clock entirely and answers a different question.

The third clock never strikes. Never-observed outcomes belong to cases that were not reviewed, treated, or followed. A defendant the judge declined to release generates no failure-to-appear record at all. These create selective labels, cannot be assumed negative, and need sampling or causal design. They put a hard ceiling on what monitoring is entitled to claim.

Short windows detect incidents quickly but produce noisy estimates. Long windows are stable but slow. Seasonal cycles, traffic volume, label delay, and repeated users should guide window length and baselines.

Use event-time alignment and backfill metrics when delayed outcomes mature. Mark provisional values so incomplete labels are not compared with finalized history.

The GFT count is a lesson in baselines as much as in windows. Overly high prevalence in 100 out of 108 weeks, from 21 August 2011 to 1 September 2013, is not a week-level spike that a short window would flag. It is a sustained level shift, visible only against a long baseline and a comparison series. A monitoring design that watches for jumps will not see a system that is simply high, week after week, for two years.

A dashboard without ownership does not manage risk. One regulator has written the ownership down. Section 3308 of the Food and Drug Omnibus Reform Act of 2022 added section 515C to the FD&C Act, now 21 U.S.C. 360e-4. A manufacturer may change an approved device without a supplemental application if the change is consistent with an approved predetermined change control plan. The plan is filed before the change, not after the incident.

FDA guidance on those plans for AI-enabled device software, issued in December 2024 and reissued in August 2025, asks the Modification Protocol to “identify any triggers for re-training (e.g., when the quantity of new data reaches a certain size or when a drift in data is observed over time)”. Drift there is not a discussion topic. It is a named trigger with a pre-declared response. On the monitoring duty itself the guidance states: “As part of a manufacturer's responsibility to ensure that devices remain safe and effective, FDA anticipates that manufacturers will monitor their device's safety (e.g., adverse events) and effectiveness (e.g., performance) over time as modifications are implemented consistent with their authorized PCCP.”

The operating loop is the same five moves whether or not a regulator is reading. Detect: monitor data contracts, features, scores, decisions, latency, and the outcomes actually available. Triage: classify the alert as instrumentation, population, model, policy, or label-process risk. Confirm: use sampled review, delayed labels, slices, and comparison with a control or baseline. Act: roll back, adjust policy, retrain, restrict scope, or collect evidence. Learn: turn incidents into regression tests, and update thresholds, owners, and runbooks.