Split Design, Sampling, and Leakage Prevention

A deterministic hash of a stable entity key keeps existing groups in the same split as new data arrives. The hash rule, the salt and the group definition all have to be versioned.

Temporal splits require explicit boundaries, not relative phrases such as “last month,” which change meaning when the build runs. Preserve the actual cutoff dates and the label-maturity buffer.

When a key system changes, map old and new identities before assigning membership. Otherwise one real entity moves between sets after migration — the same person, device or account landing on both sides of a boundary that the code still believes it is enforcing.

Sometimes a new dataset needs a new evaluation cohort. Keep the old benchmark for continuity. Add a forward-looking test that reflects the changed population.

The split function can be correct while the rows are already contaminated. An entire literature can be built on rows in that state.

Machine learning was applied to detecting or prognosticating COVID-19 from chest radiographs and CT scans in 2,212 published studies. Roberts and colleagues, writing for the AIX-COVNET collaboration, screened all of them, kept 415 and reviewed 62 in depth. Their verdict, in Nature Machine Intelligence in 2021: “Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases.” One of those papers claimed external validation on a dataset that already contained the two datasets it had trained on. That is this lesson's theme in its purest form.

A second review of a different corpus reached the same verdict. Wynants and colleagues screened 4,909 titles, reviewed 51 studies describing 66 models, and rated every one of the 66 at high or unclear risk of bias. It ran in the BMJ in 2020. Two independent teams, two literatures, no usable model. The five routes below are how rows arrive in that condition.

Randomly assigning rows is appropriate only when rows are sufficiently independent and future use resembles random unseen examples from the same population. Many ML systems violate those assumptions. One customer, device, document, product or location can generate many correlated rows. Random splitting then lets near-duplicates or shared history appear on both sides. Time matters too. A model deployed next month should be tested on later data, not on a random mixture that hands training the future's categories and trends.

Split design belongs in data engineering, because grouping, time, source, geography and duplicates are all decided while the dataset is built. Once leakage enters the snapshot, metric code cannot undo it.

The target does not need to appear as a column for the protocol to be contaminated. Six routes recur. A feature can contain the outcome, or a downstream consequence of it: direct target leakage. Features, corrections or aggregates can use information from after the prediction cutoff: temporal leakage. Correlated examples from one entity or source can land on both sides: group leakage. Exact or near-identical records can cross the boundary: duplicate leakage. Statistics, vocabularies, selection or imputation can be fitted using evaluation data: preprocessing leakage. And the evaluation set can decide which data, features or model variants are kept at all: selection leakage.

The last two routes met in public at the IJCNN 2011 Social Network Challenge, run by Kaggle. Entrants received a scrubbed graph of 1,133,547 nodes and 7,237,983 edges and were asked to predict links. Narayanan, Shi and Rubinstein won it without solving that task. They matched the scrubbed graph against their own crawl of Flickr, de-anonymised 64.7% of the 8,960-edge test set, trained on the de-anonymised test edges, and posted a winning test AUC of 0.981. Their abstract says so directly: “By de-anonymizing much of the competition test set using our own Flickr crawl, we were able to effectively game the competition.” A KDD paper that same year catalogued the episode as leakage, reporting that the winners could “correctly predict over 60% of edges which were identified”.

No row was duplicated by the organisers. No column contained the answer. The held-out edges were simply reconstructible from a public source the protocol had not considered. A split withholds information only relative to everything else the modeller can reach.

A protocol is valid only when it matches the claim being made. A random row split holds out individual examples from one broad population: efficient when rows are independent and stationary, leaky through repeated entities or duplicates, weak for any claim about the future or a new domain — think independent manufactured items from one stable line. A grouped split keeps all examples from one entity or cluster together, so it tests generalization to unseen groups and blocks entity-specific memorization; it needs stable group identity and enough groups. Hold out patients, or authors. A temporal split trains on earlier periods and evaluates on later ones, which is what forecasting and future deployment actually look like: train through June, test on July. A domain-held-out split withholds a region, site, device family or market, and tests transfer beyond the domains observed. Leave one hospital or factory out. Choosing among them is choosing which claim you are entitled to make.

The domain column is the one most often skipped, and its cost has been measured. Pneumonia-screening CNNs did better on their home data than on outside data in 3 of 5 natural comparisons. Worse, a CNN could name which hospital system a radiograph came from with near-perfect accuracy: 22,050 of 22,062 NIH images, 8,386 of 8,388 Mount Sinai images, and, inside one hospital, 5,805 of 5,805 inpatient against 449 of 449 emergency-department images. Zech and colleagues published that in PLOS Medicine in 2018. Their conclusion is the reason a site must be held out rather than shuffled: “CNNs robustly identified hospital system and department within a hospital, which can have large differences in disease burden and may confound predictions.” This is not one unlucky model. An independent systematic review in 2022 found that 70 of 86 algorithms — 81% — lost accuracy on external data, with a median performance difference of -0.046.

The temporal column has a comparable number attached to it. Three published Android malware classifiers were re-evaluated on 129,000 apps spanning three years, under three rules: training strictly precedent to testing, consistent time windows for goodware and malware, and a realistic 10% malware rate in testing. Published F1 figures reached 0.99. The honest 24-month scores were AUT(F1,24m) of 0.58, 0.32 and 0.64. The study is TESSERACT, presented at USENIX Security in 2019.

The protocol should shape dataset assembly, transformation fitting and label maturity from the start. Five steps, in order. First, state the deployment claim: future period, unseen entity, new domain, or same-population generalization. Second, identify the dependence units — the entities, episodes, source assets, originals and temporal neighborhoods that must stay together. Third, assign stable membership, by hashing persistent groups or storing an explicit split table with versioned rules. Fourth, fit transformations inside training: vocabularies, imputers, selectors and supervised features, learned without evaluation rows. Fifth, audit overlap and coverage: duplicates, group intersections, time boundaries, label maturity and slice representation.

The second step is not an author's preference. Three regulators have written it down. On 27 October 2021 the US FDA, Health Canada and the UK MHRA jointly issued ten Guiding Principles for Good Machine Learning Practice for Medical Device Development. Principle 4 is titled “Training Data Sets Are Independent of Test Sets”. It states: “Training and test datasets are selected and maintained to be appropriately independent of one another.” The principle names the dependencies that have to be considered and addressed: patient, data acquisition and site factors. Those are precisely the units of step 2. They are also the three that let Zech's networks name the hospital. For a device developer in any of those jurisdictions, a split that puts one patient, one scanner or one site on both sides of the boundary is not a debatable modelling choice.