Training, Validation, and Test: Three Different Jobs

A split is not clerical housekeeping. It creates separate evidence for fitting, choosing, and making a final claim.

The three roles are not interchangeable, and in some fields that is not left to taste. Medical-device regulators have written it down. In October 2021 the US FDA, Health Canada and the UK's MHRA jointly issued ten guiding principles for good machine learning practice. The eighth requires performance evidence generated “independently of the training data set”. That sentence is the test set's entire job description. It is also the one job the training and validation partitions cannot do — not because they are lower-quality data, but because they have already been spent on decisions.

If deployment predicts future weeks, a chronological test is often more informative than a random historical split. If the system serves new stores, hold out stores. If it makes repeated predictions for known users, an event-level split may be appropriate — but that is a different claim from performance on new users.

The most thoroughly documented case of a model that fit history beautifully and then failed forward is Google Flu Trends. It was built to predict CDC influenza-like-illness reports by matching 50 million candidate search terms to 1,152 data points. That is an enormous space of candidate predictors searched against a very short outcome series. Retrospectively it looked excellent. Prospectively it did not. David Lazer and three colleagues reported in Science in 2014 that “GFT also missed by a very large margin in the 2011–2012 flu season and has missed high for 100 out of 108 weeks starting with August 2011”. The counted window runs from 21 August 2011 to 1 September 2013. The 2011–2012 season was overshot by more than 50%.

The comparison that settles the matter is the baseline, not the error. Over the out-of-sample period the mean absolute error was 0.486 for GFT against 0.311 for a simple lagged-CDC model. A system trained on 50 million candidate terms was beaten forward in time by the outcome variable's own recent history. No random split of the historical record would have shown that, because a random split lets the model borrow from weeks on both sides of the one it is predicting.

No split is universally correct. Write the deployment question, then create partitions that test that question.

Different deployment settings require different barriers between partitions. The barrier you choose determines the claim you are allowed to make.

The site barrier has been measured. A pneumonia-screening convolutional network trained on chest radiographs from Mount Sinai and the NIH scored an internal test AUC of 0.931 (95% CI 0.927–0.936). On an external test set from Indiana the same model scored 0.815 (95% CI 0.745–0.885, P = 0.001), and it performed worse externally in 3 of 5 natural comparisons. John Zech and five colleagues published that in PLOS Medicine in 2018.

Then they asked why, and the answer is the reason site holdouts exist at all: “CNNs were able to directly detect hospital system of a radiograph for 99.95% NIH (22,050/22,062) and 99.98% MSH (8,386/8,388) radiographs.” The network could name the hospital from the image almost perfectly. A random patient-level split leaves that signal on both sides of the barrier. There it is free to substitute for pneumonia whenever the two happen to correlate — which they do, because base rates differ between hospitals.

A validation set is expected to influence development. A test set is intended to provide a final check after choices are made. If the team keeps redesigning the model after every test result, the test becomes another validation set.

Large organizations may use staged holdouts, private challenge sets, temporal refreshes, or an evaluation service that returns limited information. The purpose is not secrecy for its own sake. It is to preserve evidence that has not shaped the system.

The measured damage is smaller than the fear, which is worth knowing before rebuilding a pipeline over it. A 2019 meta-analysis went through “data from 120 machine learning competitions on the popular Kaggle platform”. Its seven authors used the platform's own design: a public leaderboard scored throughout the contest, and a private test set scored once at the end. Despite up to 35,000 submissions per competition, the classification competitions “show little to no signs of overfitting”, with potential effect sizes “typically small (e.g., less than 1% classification accuracy)”. The exceptions teach the actual lesson. The outlier competitions “have pathologies such as non-i.i.d. data splits or (effectively) small test sets”, two of them with private test sets “of size 209 and 100”. Repeated peeking bites hardest where the holdout was too small, or too correlated with the training data, to begin with.

The second measurement is larger and stranger. Benjamin Recht and three colleagues did not analyse leaderboards. In 2019 they rebuilt the CIFAR-10 and ImageNet test sets by re-running the original collection processes, then re-scored the published models. Accuracy fell by 3–15% on CIFAR-10 and 11–14% on ImageNet — after years in which those same test sets had been the target everyone tuned against. The obvious explanation turns out to be the wrong one. They write: “Our results suggest that the accuracy drops are not caused by adaptivity, but by the models' inability to generalize to slightly "harder" images than those found in the original test sets.”

The story does not end there, and the ending is the part worth carrying. A separate group remeasured the selection frequencies used to assemble the replacement images. Once statistical bias in the replication is corrected, they reported in 2020, only about 3.6% of the original 11.7% ImageNet drop remains unaccounted for. Most of the first careful measurement of a holdout effect was an artefact of how the replacement set had been built. Rebuilding a test set is itself a sampling design. It can go wrong in exactly the ways the original did.

Write the split policy before extensive exploration whenever possible. The five moves below are Guiding Principle 4 turned into a checklist. Name the claim, name the sources of dependence, then decide in advance who is allowed to look at the answer and when.

Small datasets make fixed holdouts noisy. Using all the data for both fitting and evaluation does not solve the problem. Cross-validation, grouped resampling, repeated splits, or carefully designed temporal backtests can use data more efficiently.

The Kaggle meta-analysis shows the danger from the other side. Across 120 competitions the overfitting effect was typically under 1% accuracy. The competitions that did misbehave were the ones whose private test sets held 209 and 100 examples. A tiny holdout mostly supplies noise, and noise is what an adaptive process learns to exploit first.

Later evaluation paths will cover these methods in depth. The beginner principle is simple: every performance claim needs examples that did not influence the fitted parameters or the choices being judged.

When a test metric looks implausibly high, investigate the split before celebrating. The checks below were skipped at the scale of an entire literature, and the resulting review is the strongest available argument for running them.

A systematic review searched 2,212 studies of machine learning applied to COVID-19 detection and prognosis from chest radiographs and CT scans. It kept 415 after screening and 61 after quality review. It found none of the models of potential clinical use. Michael Roberts and the AIX-COVNET collaboration at Cambridge published it in Nature Machine Intelligence in 2021.

A recurring cause was the public 'Frankenstein' dataset — a dataset assembled from other datasets and re-released under a new name, so that a team believing it had trained on one corpus and tested on another had done neither. The review states the mechanism plainly: “This repackaging of datasets, although pragmatic, inevitably leads to problems with algorithms being trained and tested on identical or overlapping datasets whilst believing them to be from distinct sources.” Its remedy is this lesson's split-unit rule in a reviewer's words. Authors should “state how they ensured that images from the same patient were not included in the different dataset partitions, such as describing patient-level splits”.

How deep the source signal ran is measurable. Blacking out the central lung region of the same public X-ray images — deleting the anatomy the models were supposed to be reading — still left the source datasets distinguishable, the lowest ROC-AUC being 0.92. Maguolo and Nanni ran that test at the University of Padova in 2021. Whatever those classifiers had learned, it survived the removal of the lungs.