Dataset Partitions and Test-Set Governance

The dataset was designed to contain nothing. Ten thousand samples, ten thousand attributes — n = 10,000, d = 10,000 — with every attribute drawn independently of a uniformly random label. Then the ordinary loop ran over it. Score the variables, keep the ones that look good on the holdout, score again, keep k = 500.

What came out was published in 2015 by Dwork, Feldman and colleagues, in a section headed “No correlation between labels and data”: “In this scenario no classifier can achieve true accuracy better than 50%. Nevertheless, reusing a standard holdout results in reported accuracy of over 63% for k = 500 on both the training set and the holdout set (the standard deviation of the error is less than 0.5%).”

Their own Thresholdout mechanism, run on the identical data, held the reported accuracy at the true 50%. The gap between 63% and 50% is not a modelling difference. It is a difference in how often, and how freely, the holdout was allowed to answer.

No gradient touched those labels. The selection process adapted to their noise, and the noise was all there was. So test governance is about people and decisions as much as code. Look at a test set often enough — redesign the metric, review the errors, negotiate a release — and it quietly becomes validation data.

Leakage can occur before, during, or after model fitting. The second path on this list has a documented instance, with dates attached.

On 2 June 2015 the ILSVRC organisers published what the ImageNet test server had recorded. Between 28 November 2014 and 13 May 2015, at least 30 accounts belonging to a Baidu team had submitted at least 200 times. The limit was two submissions per week. The organisers asked the team to refrain from submitting to the evaluation server or the challenge for the next 12 months.

The announcement spells out what the extra submissions buy: “To make this clear, by exploiting the ability to test many slightly different solutions on the test server it is possible to 1) select the best out of a set of very similar solutions based on test performance and achieve a small but potentially significant advantage and 2) choose methods for further research and development based directly on the test data instead of using only the training and validation data for such choices.” Ren Wu replied on behalf of Baidu’s Heterogeneous Computing Team.

MIT Technology Review covered the episode two days later, under the headline “Why and How Baidu Cheated an Artificial Intelligence Test”. Tom Simonite named the margin at stake: a claimed error rate of 4.58% against Google’s 4.82%.

The advantage the organisers describe is the same one measured on random data in the holdout experiment. Neither requires anybody to look at a label.

Confusion begins when one partition is asked to serve several decisions at once. Training data fits parameters and any learned preprocessing. Validation data supports model, threshold and configuration choices, and becomes less neutral each time it is used. Test data estimates performance after the choices are frozen.

In one industry that separation is not a matter of taste. In October 2021 the FDA, Health Canada and the UK MHRA jointly issued ten Good Machine Learning Practice guiding principles for medical device development. All three regulators publish identical text.

The fourth of the ten is titled “Training Data Sets Are Independent of Test Sets”. It reads: “Training and test datasets are selected and maintained to be appropriately independent of one another. All potential sources of dependence, including patient, data acquisition, and site factors, are considered and addressed to assure independence.”

Note what the principle names. Not rows and columns — patient, data acquisition, and site. The dependence the three regulators ask developers to hunt for is the kind that survives a random split of the file: the same patient imaged twice, the same scanner, the same hospital. A partition boundary drawn on row indices does not address any of them.

Test credibility can be preserved without making evaluation impossible. There are worked mechanisms to copy here, not principles to admire.

The first is a leaderboard that answers less. Blum and Hardt named the failure it fixes: “As participants are allowed to repeatedly evaluate their submissions on the leaderboard, they may begin to overfit to the holdout data that supports the leaderboard.” Their 2015 paper calls the mechanism the Ladder.

The evidence came from Kaggle’s Photo Quality Prediction competition. A holdout of 12,000 samples: 8,400 behind the private leaderboard, 3,600 behind the public one. 200 teams produced 1,830 submissions, of which the authors parsed 1,785 successfully.

A “boosting attack” of k random submissions drives the standard mechanism’s leaderboard error to Ω(sqrt(k/n)). The Ladder rounds the score and releases a new number only when a submission improves significantly on the best so far. That buys leaderboard error O((log(kn)/n)^(1/3)) — an exponential improvement in k. A third bound in the paper, Ω(sqrt(log k / n)), is a different quantity: the minimax lower bound on what any estimator can achieve.

The strongest policy is to hold the test set where it cannot be consulted at all, because it does not exist yet. DeepMind’s account of CASP14 describes the arrangement: “Crucially, CASP chooses protein structures that have only very recently been experimentally determined (some were still awaiting determination at the time of the assessment) to be targets for teams to test their structure prediction methods against; they are not published in advance.”

The result under that protocol was not close. The Protein Structure Prediction Center’s own CASP14 group ranking, by summed z-score, places AlphaFold2 — group 427 — at 244.02 across 92 domains. The next-best group, BAKER, scored 90.82. DeepMind reported a median of 92.4 GDT across all targets and 87.0 GDT on free-modelling targets. A score like that cannot have been obtained by tuning against the answers.

A lightweight written policy beats an informal promise. Each step below has a published instance behind it, not just a rationale.

Assign roles. Separate model development from final evaluation where feasible. CASP is the extreme form: the assessment is run by the Protein Structure Prediction Center, not by the entrants, and the ranking table is theirs.

Freeze the protocol. Version population rules, labels, metrics, slices and thresholds, and settle the independence question first. Guiding Principle 4 asks for exactly that. It also names the sources of dependence to be considered and addressed — patient, data acquisition, and site factors — rather than leaving them to the developer’s imagination.

Limit feedback. Return only the information the predeclared decision needs. Blum and Hardt showed that the amount returned is itself the tunable parameter: rounding the score and withholding non-significant improvements moves the guarantee from Ω(sqrt(k/n)) under attack to O((log(kn)/n)^(1/3)).

Track adaptations. Log which design changes were influenced by test results, and how many times the test set was queried. The ILSVRC organisers could state the count — at least 200 submissions from at least 30 accounts — because the evaluation server kept the record.

Retire compromised evidence. Refresh or externally validate when reuse has materially shaped the system. GSM1k cost 1,205 newly written problems. It bought a correction of up to 8% on models everyone had been ranking by the old benchmark.

Test data is not a permanent oracle. Its evidentiary value declines as teams adapt to its cases, metrics and quirks. The decline has been measured: a true 50% reported as over 63% on data with no signal in it, 0.931 falling to 0.815 when the hospital changes, up to 8% when the arithmetic problems are new ones.

The goal is not secrecy for its own sake. It is an honest estimate of performance after the important choices have already been made. Where the stakes justify it, that means a written rule, agreed before the first experiment runs.