Distribution Shift: When the World Stops Matching the Dataset

Google Flu Trends was built to predict the CDC's influenza-like-illness reports from search queries, and for a time it did. Then it drifted, and it drifted in one direction: starting in August 2011 it missed high in 100 of 108 weeks, and by February 2013 it was predicting more than double the CDC's proportion of doctor visits for influenza-like illness.

The first version had fitted 50 million search terms to 1152 data points. With that much room, a term can match the target by coincidence rather than by mechanism, and some of the terms that survived the fit tracked the season rather than the disease. Lazer and colleagues took the model apart in Science in 2014. Their verdict is one sentence: “In short, the initial version of GFT was part flu detector, part winter detector,” they wrote.

The decisive test had already been run by the world. The first version completely missed the non-seasonal 2009 influenza A-H1N1 pandemic — flu that arrived without the winter the model had learned to read. Nothing about the model changed. What changed was whether the season and the illness still moved together.

Distribution shift is a mismatch between the process that generated the development data and the process running in deployment. The change can be in the inputs, in the outcomes, in the relationship between them, or in how any of it is measured.

Not every statistical difference matters. A feature can drift while predictions stay robust. Model quality can deteriorate with no dramatic marginal feature shift at all. That is why monitoring has to connect the changes it detects to behavior and decisions.

It is worth knowing how small a change already costs something. In 2019 Recht and colleagues rebuilt the CIFAR-10 and ImageNet test sets, following the original collection processes closely. The goal was to reproduce the distribution, not to move it. Their abstract reports what came back: “We evaluate a broad range of models and find accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet.” Following the recipe was not the same as reproducing the sample.

The sequel is the reason to be careful with any drop you measure. An independent MIT group went back over that replication and corrected for statistical bias in how the new test set had been assembled. Only an estimated 3.6% ± 1.5% of the original 11.7% ± 1.0% ImageNet drop was left unaccounted for. Most of the observed shift was an artifact of the rebuild. Sizing a gap is itself a measurement problem, with failure modes of its own.

These categories help ask what changed and which response is plausible. Covariate shift moves the inputs. Label shift moves outcome prevalence while class-conditional patterns stay relatively stable. Concept shift moves the relationship between inputs and outcomes. Measurement or policy shift moves how any of it is collected, labeled, defined, or intervened on. The first, second and fourth mostly change how a model should be aimed. The third changes what it learned.

The third category has an audited price. On 2 November 2021 Zillow Group announced it would wind down Zillow Offers, its iBuying arm, because the price-forecasting model's assumptions about the housing market no longer held. The Q3 2021 inventory write-down was approximately $304 million — $304,359 thousand in the Form 10-Q. The company expected a further $240-265 million of losses in Q4 and cut roughly 25% of its workforce. The houses had not changed. The mapping from a house's features to the price it would fetch had.

The company said as much in its third-quarter earnings release, in the words of co-founder and CEO Rich Barton: “We’ve determined the unpredictability in forecasting home prices far exceeds what we anticipated and continuing to scale Zillow Offers would result in too much earnings and balance-sheet volatility,” — concept shift stated as a business decision.

Note what that sentence does not say. It does not report a bug, a broken pipeline, or a data quality incident. The model kept working exactly as specified, on a relationship that had moved out from under it.

The environment includes technical and human systems. Time brings seasonality, trends, shocks and gradual evolution. Population brings new users, regions, sites, devices or languages. Behavior changes as people adapt to the product, to incentives, or as adversaries. Policy changes eligibility, review, pricing or labeling rules. Infrastructure changes sensors, pipelines, vendors, schemas and preprocessing.

Population is the entry on that list most often written down as a formality, so it is the one worth attaching measurements to. In December 2019 NIST published its face recognition vendor test on demographic effects. It had run 18.27 million images of 8.49 million people through 189 algorithms from 99 developers — an audit large enough that its finding is about the field rather than about one vendor. The executive summary puts it in one line: “Across demographics, false positives rates often vary by factors of 10 to beyond 100 times.”

An academic audit had found the same shape earlier. Buolamwini and Gebru tested commercial gender classifiers for the 2018 Gender Shades study. The classifiers erred on up to 34.7% of darker-skinned females, against a maximum of 0.8% for lighter-skinned males. A single headline accuracy figure hides a spread like that entirely. Serving a new population is therefore not an expansion of an evaluated system. It is deployment onto a distribution for which no number has been reported.

Random holdouts are weak evidence when deployment is inherently temporal or cross-domain. The split, not the model, decides which number you see. The same standard training run reports 93.2 percent on Camelyon17 in distribution and 70.3 percent on hospitals it never saw. A shuffle would have reported only the first. The steps below are a way of cutting the split along the boundary deployment will actually cross.

List realistic change scenarios in time, site, device, policy and population. Create stress splits that hold out later periods, new groups, environments or transformed conditions. Compare metrics, calibration, abstention and error categories by scenario. Locate the failure — features, model mapping, threshold, or workflow. Then set a boundary: restrict the launch or add a fallback wherever the evidence is weak.

Input distributions, missingness, score distributions, alert rates, and operator overrides can give early warning. Every one of them is a proxy rather than a substitute for delayed outcome metrics. Alert rate is the proxy that most resembles performance and is least equal to it.

The Epic Sepsis Model was validated externally at Michigan Medicine, across 27,697 patients and 38,455 hospitalizations between 6 December 2018 and 20 October 2019. The hospitalization-level AUC was 0.63 (95% CI, 0.62-0.64), against the 0.76-0.83 Epic reported in internal documentation. Wong and colleagues published that in JAMA Internal Medicine in 2021. Anyone watching the dashboard would have seen a model doing a great deal of work: “The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue,” report the authors. Alerts on 18% of hospitalizations, and two thirds of the cases the system existed to catch still missed.

A healthy monitoring design combines fast pipeline signals with slower labeled performance and real-world outcomes. Alert thresholds should reflect natural variability, seasonality, and actionability. No volume of alerts, high or low, substitutes for the delayed outcome that says whether the right patients were flagged.

The response should match the changed component. Concept shift is the case that resists the cheap repairs. Zillow's answer was the far end of this list: scope restriction taken to its limit, closing the line of business rather than retuning the model.