Data Leakage: When the Answer Sneaks Into the Question

In the spring of 2020, COVID-19 classifiers trained on chest X-rays were being published faster than anyone could check them, and each arrived with an accuracy figure. Gianluca Maguolo and Loris Nanni ran the test those figures cannot run on themselves. They blacked out the centre of the images, removing most of the lung field — the organ the model was supposed to be reading — and re-ran the classification. The results barely moved. “We show that similar results can be obtained using X-Ray images that do not contain most of the lungs”, they wrote in 2020.

What the models were reading was the border, the cropping, the processing — the fingerprint of where an image came from. Roberts and colleagues reproduced the effect in Nature Machine Intelligence with their own figures. From the non-lung region alone, the source of the Cohen and Kermany datasets could be identified at an AUC between 0.9210 and 0.9997. COVID-19 itself was “diagnosed” at an AUC of 0.68. Positive cases had been assembled from one collection and negative cases from another, so the fastest route to a high score was never pneumonia. It was telling the two collections apart.

Data leakage occurs when training or evaluation uses information, transformations, or dependencies that would not be available under the intended prediction conditions, so the resulting score answers an easier or different question.

The IJCNN 2011 Social Network Challenge shows how far apart those two questions can drift. Kaggle released an anonymised graph of 1,133,547 nodes and 7,237,983 edges and asked entrants to say which of 8,960 test edges were real. The winning team recognised the graph as Flickr, crawled Flickr themselves, and matched the anonymised nodes back to accounts. “Using our own crawl of Flickr, we were able to successfully de-anonymize 64.7% of the test edge-set”, they report. They won with a test AUC of 0.981. A separate group, describing the same contest independently, recorded the winners predicting “over 60% of edges which were identified” and classed the episode as leakage. The organisers had set a link-prediction problem. The winning method solved a lookup problem, and the leaderboard could not tell the difference.

Leakage is not limited to an obvious target column: it can enter through time, preprocessing, duplicate entities, selection rules, human review, or features that indirectly encode the outcome.

The categories overlap. But they guide different checks. Group leakage is the one a random split cannot see by construction, and the measured example is radiological.

A network was given 158,323 chest radiographs from three hospital systems — 112,120 from NIH, 42,396 from Mount Sinai and 3,807 from Indiana — and trained to name the hospital instead of the disease. It almost never missed. “CNNs were able to directly detect hospital system of a radiograph for 99.95% NIH (22,050/22,062) and 99.98% MSH (8,386/8,388) radiographs”, Zech and colleagues report in PLOS Medicine in 2018. A second network went finer and separated departments inside Mount Sinai: 5,805/5,805 inpatient and 449/449 emergency-department radiographs.

That would be a curiosity if the hospitals shared a case mix. They did not. Pneumonia prevalence was 34.2% at Mount Sinai against 1.2% at NIH and 1.0% at Indiana. So a model that did nothing but rank cases by the prevalence of the system they came from reached AUC 0.861 (95% CI 0.855–0.866) on the joint Mount Sinai–NIH test set. An image carries its own site membership. A random split scatters sites across every partition. The score that comes out is partly a measurement of prevalence. An independent review in npj Health Systems states the consequence plainly: “Zech et al. (2018) demonstrated that models trained on data from one hospital showed reduced performance on external datasets”.

The order of operations determines whether held-out data influences the model, and the size of that influence has been measured rather than assumed.

Two published microarray results had a handful of genes classifying tumours almost perfectly. Xiong and colleagues had selected three genes using all the tissue samples, then averaged Fisher's rule over 200 random 95/5% training/test splits. The average test error was 10.7% on the colon data and 0% on the leukaemia data, where the internal leave-one-out cross-validation error for those three genes was zero. Christophe Ambroise and Geoffrey J. McLachlan performed the selection inside an external 10-fold cross-validation instead, and separately inside a .632+ bootstrap. The estimated prediction error rose to about 15% on the colon data and about 5% on the leukaemia data. “Using two published data sets, we demonstrate that when correction is made for the selection bias, the cross-validated error is no longer zero for a subset of only a few genes”, they wrote in the Proceedings of the National Academy of Sciences in 2002.

Zero became about five per cent. 10.7% became about fifteen. Nothing about the classifier changed. Only the boundary moved. Kapoor and Narayanan give this failure its own entry in their taxonomy, L1.3, “Feature selection on training and test set”. Nor does it need labels to bite. Moscovich and Rosset showed that even unsupervised, label-free preprocessing carried out outside the training boundary “can introduce a substantial bias into cross-validation estimates”.

An event can occur at 10:00. It may become available to the prediction system only at 10:30, after processing. A corrected record may be backfilled days later. Historical tables often store the final value without preserving when the value became known. The costliest version of this is not a late timestamp, though. It is an early one: a field written down precisely because a human had already noticed the thing the model is being asked to predict.

The Epic Sepsis Model has been re-evaluated on exactly that point. Over 77,582 University of Michigan hospitalisations from 2018–2020, sepsis occurred 3,766 times, 4.9% of them. Kamran and colleagues separated the predictions the model issued before clinicians had acted from those it issued afterwards, using receipt of antibiotics, fluids, blood culture and/or lactate measurement as the marker that the team had already recognised sepsis. Their result, published in NEJM AI in 2024, reads: “ESM achieved an AUROC of 0.62 (95% confidence interval [CI], 0.61 to 0.63) when including predictions before sepsis criteria were met and in some cases, after clinical recognition. When excluding predictions after clinical recognition, the AUROC dropped to 0.47 (95% CI, 0.46 to 0.48).”

Moving the prediction moment back to before the moment a clinician acted took a deployed commercial model from 0.62 to below chance. The model did not change. A large part of the 0.62 was a record of what the treating team already suspected, reaching the feature set as an order for a blood culture.

Independent evaluations point the same way. A validation in JAMIA Open covered 145,885 ED encounters at 2 hospitals in 2023 and reported sensitivity of 14.7% within a 6-hour window. A methodological systematic review in npj Digital Medicine covering 91 studies found median AUROC of 0.886 and 0.861 at 6 and 12 hours before onset, falling to 0.783 under full-window external validation. It warns that “retrospective validation still carries the risk of label leakage”.

For each feature, record event time, ingestion time, update time, and the prediction cutoff. Point-in-time reconstruction should use the version genuinely available then — and for any feature a person can influence, record when the person first acted.

A leakage audit combines data lineage, domain knowledge, and controlled experiments. Step 3 in particular is not merely a good habit. Where the stakes are highest it is written down and signed. On 27 October 2021 the US FDA, Health Canada and the UK MHRA jointly published ten Good Machine Learning Practice guiding principles for medical device development. Principle 4 is titled “Training data sets are independent of test sets”. It reads in full: “Training and test datasets are selected and maintained to be appropriately independent of one another. All potential sources of dependence, including patient, data acquisition and site factors are considered and addressed to assure independence.”

Patient, acquisition and site are named explicitly, by three regulators, in a document about what may be sold to hospitals. The entity, time and site barriers below are that requirement expressed as work.

A fraud team may know outcomes only for transactions selected for investigation. A hiring system may observe job performance only for applicants who were hired. The labeled sample then reflects previous policies.

This is not repaired by deleting a column. The team must understand the selection process, gather exploration data where appropriate, or narrow claims to the population whose outcomes are observed. Narrowing is precisely what the Epic Sepsis Model re-evaluation did. Kamran and colleagues restricted the score to predictions issued before clinicians had recognised sepsis, and reported 0.47, not 0.62, as the answer to the question the model was bought to answer.