Empirical Risk, Selection, and Generalization

Feature selection, preprocessing, hyperparameter search, early stopping, threshold choice, and manual iteration all use data. They belong to the learning algorithm when analyzing generalization. A guarantee for a fixed predictor does not automatically apply after choosing among thousands of candidates on the same validation set.

This is not a classroom scruple. On 27 October 2021 three medical-device regulators — the US FDA, Health Canada and the UK MHRA — jointly published ten Good Machine Learning Practice guiding principles. The fourth is titled “Training Data Sets Are Independent of Test Sets”. It states: “Training and test datasets are selected and maintained to be appropriately independent of one another.” The principle does not stop at storing the files separately. It requires that all potential sources of dependence, including patient, data acquisition and site factors, be considered and addressed. Three regulators had to write that down because splits fail in practice, for reasons that never appear in the fit call.

The failure has been counted. Kapoor and Narayanan went looking for data leakage across the sciences and reported the tally in Patterns in 2023: “Through a survey of literature in fields that have adopted ML methods, we find 17 fields where leakage has been found, collectively affecting 294 papers and, in some cases, leading to wildly overoptimistic conclusions.” Their civil-war-prediction case study is the mechanism in miniature. The reported superiority of ML models over logistic regression disappeared once the leakage was corrected. An independent group then put a size on which channels matter. Working across four datasets in Nature Communications, they concluded that “Leakage via feature selection and repeated subjects drastically inflates prediction performance, whereas other forms of leakage have minor effects”.

Reproducible code is not enough. Feature selection run before the split is reproducible, and it is also the channel that moved the numbers most. The information flow from data to every design choice must be included.

A learning algorithm sees a finite sample and chooses a predictor using empirical risk. Deployment performance depends on population risk under the future data-generating process. Generalization is the gap between those two quantities. It is not a mysterious property stored inside a model. It is a relationship among data, algorithm, hypothesis class, loss, and target population.

The first task is to define the population and sampling mechanism that make the comparison meaningful. Every case in this lesson is a case where that definition was skipped and a number survived anyway, until someone measured it somewhere else.

Each estimate has a distinct role and a distinct exposure to adaptive reuse. Training risk is the objective value on the data used to fit parameters. It is optimistically biased for selection and useful mainly as an optimization diagnostic. Validation risk guides model and hyperparameter choices, which makes it part of the adaptive algorithm and liable to wear out through repeated use. Test risk estimates the locked procedure after selection. It should be consulted sparingly, and contamination invalidates it. Deployment risk is the actual loss under live conditions. It is the only one a patient, a customer, or a clinician ever meets.

The distance between the reported number and the deployed one has been measured on a system running in hospitals. Epic Systems reported that its proprietary Sepsis Model scored an AUC of 0.76–0.83. Wong and colleagues, writing in JAMA Internal Medicine in 2021, validated it externally on 27,697 patients and 38,455 hospitalizations at Michigan Medicine between December 2018 and October 2019. They measured a hospitalization-level AUC of 0.63 (95% CI, 0.62–0.64), with 33% sensitivity. The paper states it directly: “Our observed hospitalization-level model performance (AUC, 0.63) was substantially worse than that reported by Epic Systems (AUC, 0.76-0.83)”. Its conclusion is unhedged: “This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis.”

The finding survived a second group and different hospitals. A 2024 study in JAMIA Open ran the same test across 145,885 encounters in two county emergency departments. Sensitivity was 14.7%, PPV 7.6%. Its verdict: “The ESPMv1 provides suboptimal diagnostic characteristics for undifferentiated patients in a county ED.” Two independent deployments, one vendor-reported range, and a gap that did not close.

A report claiming that a model will perform well beyond its sample can be checked step by step. Two review teams ran that audit at scale, on the same subject, and neither found a survivor. Roberts and colleagues screened 2,212 studies of machine learning applied to COVID-19 chest radiographs and CT scans, and analysed 62 of them after quality screening. Their finding, in Nature Machine Intelligence on 15 March 2021: “Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases.” A separate living review in The BMJ reached the same place. Its 2020 update appraised 51 studies describing 66 prediction models and rated all of them at high or unclear risk of bias: “This review indicates that proposed models are poorly reported, at high risk of bias, and their reported performance is probably optimistic. Hence, we do not recommend any of these reported prediction models to be used in current practice.”

Hundreds of published models, two independent appraisals, and a verdict of probably optimistic in every case. The six steps below are the checks those reviewers were applying, in the order a reader can apply them.

Several adaptive steps can widen the gap between observed validation performance and future risk. The population distribution is the environment the system is meant to face. The sample is what a finite collection process actually produced. The hypothesis class bounds what may be chosen. Training and tuning use the sample repeatedly. The selected predictor is the single function that meets future data.

Each arrow is a place where information can flow the wrong way. Feature selection performed before the split moves it backwards, and across four datasets that was the channel found to inflate reported performance most. Site, device and patient dependence moves it sideways. That is why Guiding Principle 4 names patient, data acquisition and site factors by name.

A small regression problem illustrates how training error alone can mislead.

The real-data version of the same picture is a model that fits its own hospitals and loses ground elsewhere. A pneumonia CNN was trained on 158,323 chest radiographs from three institutions, pooling Mount Sinai and NIH data. Internally it scored an AUC of 0.931. At a third hospital it had never seen, it scored 0.815. Zech and colleagues, reporting in PLOS Medicine, summarised the pattern: “Pneumonia-screening CNNs achieved better internal than external performance in 3 out of 5 natural comparisons.” The polynomial makes the mechanism visible on a handful of points. The radiographs show what it costs at 158,323.