Model Quality Monitoring, Drift, and Delayed Labels

A hospital readmission model produces scores today. The confirmed outcome arrives thirty days later. That thirty-day window is not a modelling convention someone chose. It is a payment rule.

Medicare judges hospitals on condition-specific 30-day risk-standardized unplanned readmission measures for six conditions: AMI, COPD, heart failure, pneumonia, CABG, and elective THA/TKA. The Hospital Readmissions Reduction Program behind those measures was created by section 1886(q) of the Social Security Act, and it began cutting payments on 1 October 2012, the first day of fiscal year 2013. The measures are calculated over a rolling performance period. CMS states the consequence on its own programme page: “The payment reduction is capped at 3 percent (that is, a payment adjustment factor of 0.97).”

That cap is not theoretical. In the fiscal year that began in October 2021, 2,499 hospitals were penalised, the average penalty was 0.64 percent, and 39 hospitals lost the full 3 percent. Those counts come from Jordan Rau's KFF analysis of the penalty data, “10 Years of Hospital Readmissions Penalties”.

So the team sees inputs, score distributions, overrides and workflow load immediately. True performance arrives thirty days later, for some patients only, and is then aggregated over a rolling window before it touches anything. Monitoring under delayed labels is an inference problem. Leading indicators tell the team where to look first. They do not certify degradation, and they do not certify safety.

A monitoring programme should state which claims are possible at each delay. The readmission measure sits in the third band, not the first. Thirty days pass before an outcome exists. A rolling performance period passes before it is aggregated. Only then does a payment adjustment apply, capped at 3 percent. Nothing observable on the day of the score can stand in for that.

A commercial risk-prediction algorithm ranked patients for enrolment in high-risk care management. Industry estimates put the reach of tools in its class at roughly 200 million people in the United States each year. An audit published in Science on 25 October 2019 found what it was really predicting: health care costs, used as a proxy for health need. Judged on the proxy, it worked. Obermeyer and colleagues state the result plainly in their abstract: “Thus, despite health care cost appearing to be an effective proxy for health by some measures of predictive accuracy, large racial biases arise.”

The size of the gap is the point. At the 97th-percentile threshold, removing the racial gap that proxy produced would, in the authors' simulation, raise the fraction of Black patients automatically identified for the programme from 17.7% to 46.5%; actually training on chronic conditions instead of cost moved the Black share of the highest-risk group from 14.1% to 26.7%. On the same day the study appeared, the New York State Department of Financial Services and Department of Health wrote jointly to David S. Wichmann, CEO of UnitedHealth Group Incorporated, about Optum's Impact Pro. The letter told him that “neither you nor any other healthcare or insurance entity may produce, rely on, or promote an algorithm that has a discriminatory effect.”

Treating every alert as “model drift” encourages the wrong fix. The three families of signal answer different questions, and only one of them is close to correctness. Even that one is not enough alone. A hospitalization-level AUC of 0.63 is an outcome-label statistic, and it still needed 1,709 missed patients and an 18% alert rate placed beside it before anyone could say what the model was doing to a ward.

Input drift may reflect seasonality, acquisition changes, source faults, or a legitimate population transition. Prediction shift can arise from input change, model routing, threshold change, or calibration. Outcome degradation can occur without large marginal feature shifts.

Google Flu Trends is the case where this was counted rather than asserted. It was built to predict CDC influenza-like-illness reports, and it drifted for two years while the model itself sat still. Lazer and colleagues wrote in Science in March 2014 that “GFT also missed by a very large margin in the 2011–2012 flu season and has missed high for 100 out of 108 weeks starting with August 2011”. By February 2013 it was predicting more than double the CDC's proportion of ILI doctor visits. They attribute the failure to “algorithm dynamics”. Google made 86 changes to search in June and July 2012 alone, so the inputs kept moving because the vendor kept changing the product that generated them. Their most deflating comparison is the baseline: even 3-week-old CDC data projected current flu prevalence better than GFT did.

The monitor should connect signals to a failure hypothesis and test the affected slices, time windows and decision pathways before choosing an intervention. A hundred misses in a hundred and eight weeks is a pattern. The diagnosis was upstream of the model.

Start with attribution and data validity before changing model parameters. The five steps below run from asset verification to the choice of intervention.

The last step is easier when its options were written down before the alert fired. FDA's final guidance on predetermined change control plans for AI-enabled device software functions was announced in December 2024 and reissued in August 2025. The notice of availability says what such a plan should contain: “This guidance recommends that a PCCP describe the planned AI-DSF modifications, the associated methodology to develop, validate, and implement those modifications, and an assessment of the impact of those modifications.” A manufacturer that has specified all three in advance can then make those changes without a new marketing submission for each one. Step 5 becomes a decision inside an agreed envelope, not an argument held under time pressure.

A proxy alert like the cost-for-need case in this lesson is easiest to see in a simpler setting: a product-ranking model judged day to day by click-through, with purchases arriving later. The shop, the dates, the rates and the return window below are invented.

“Feature distribution changed” is a precise statement. “The model is bad” is not justified until the evidence connects the change to behavior or outcomes under the intended population.

Supervisors have written down the same discipline. Revised interagency guidance on model risk management was issued jointly by the Federal Reserve, the OCC and the FDIC on 17 April 2026, superseding earlier versions from 2011 and 2021. Its section on ongoing model monitoring defines the activity against a named list of causes: “Ongoing model monitoring involves an evaluation of the extent to which a model is performing as expected given potential changes in products, exposures, activities, clients, data relevance, or market conditions.” Where a model no longer performs as expected, the guidance says the situation “may warrant overlays, adjustment, or redevelopment”.

Read the scope before borrowing the authority. The guidance says it is expected to be most relevant to banking organisations with over $30 billion in total assets. And it is supervisory guidance rather than a binding rule: the agencies state that it does not set forth enforceable standards or prescriptive requirements. A monitoring runbook should say what each signal can and cannot establish, and what evidence is needed next. That includes being exact about what a cited authority actually requires.