Experiments, Randomness, and Reproducibility

A team changed a feature pipeline and saw validation F1, a single score that combines precision and recall (higher is better), rise from 0.71 to 0.74. A rerun with a different seed returned 0.70. The old system sometimes reached 0.73.

A logged run can still be scientifically weak if the comparison is confounded.

A training run is one execution with a particular dataset, configuration, seed and environment. It produces artifacts and metrics, it may be useful for debugging, it can be lucky or unlucky, and on its own it does not identify a cause.

A controlled experiment is a planned comparison that changes a defined factor while holding relevant conditions stable. It starts from a hypothesis, uses comparable evidence and budget, accounts for randomness, and ends with a bounded conclusion.

Exploratory search is a third thing: many changes tried at once to discover promising directions. It is useful for ideation, it increases selection bias, it needs fresh confirmation, and it should not be reported as one preplanned test.

Randomness enters through more than a single seed. Data sampling decides which examples enter a split, batch, bootstrap sample or augmentation. Initialization sets the starting parameter values and so the optimization path. Execution order covers batch order, asynchronous updates and parallel operations. Hardware and libraries contribute numerical kernels, precision and implementation versions. And the measurement sample — a finite validation or test set — creates uncertainty even for a model that never changes.

The hardware-and-library source is not folklore. The two most widely used pieces of deep-learning infrastructure say so in their own documentation. PyTorch's reproducibility note opens with the warning itself: “Completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms. Furthermore, results may not be reproducible between CPU and GPU executions, even when using identical seeds.”

NVIDIA's cuDNN developer guide is equally plain. Across different GPU architectures, no cuDNN routine guarantees bitwise reproducibility. Even on one architecture the guide names the exceptions: cudnnConvolutionBackwardFilter (ALGO_0 or ALGO_3), cudnnConvolutionBackwardData (ALGO_0), cudnnPoolingBackward (CUDNN_POOLING_MAX), cudnnSpatialTfSamplerBackward and cudnnCTCLoss. They are nondeterministic “because they use atomic operations in a way that introduces truly random floating point rounding errors”. Most of them are backward-pass kernels, which training calls on every step whenever they are selected. A seed cannot reach them.

Reproducibility is a property of the whole experimental procedure, not only the model file. It also has a written definition rather than a house preference. The U.S. National Academies of Sciences, Engineering, and Medicine wrote it in a congressionally mandated report in 2019: “Reproducibility is obtaining consistent results using the same input data; computational steps, methods, and code; and conditions of analysis.” NISO adopted that exact definition in 2021 and turned it into four badges a publisher can award: Open Research Objects (ORO), Research Objects Reviewed (ROR), Results Reproduced (ROR-R) and Results Replicated (RER). Reproducing and replicating are separate achievements. The badge list keeps them apart.

The six steps that follow are that definition made operational. Version the question: record hypothesis, primary metric, slices and decision rule. Freeze the evidence: identify dataset snapshot, split logic, labels and feature code. Freeze the procedure: store model, hyperparameters, seed policy, budget and stopping rule. Record the environment: capture code commit, dependencies, hardware and execution settings. Preserve outputs: keep logs, curves, checkpoints, predictions and analysis artifacts. Rebuild independently: rerun from the recorded inputs and compare the result within expected variation.

NeurIPS 2019 ran that request as an experiment on a whole field. All 6,743 submissions had to answer the Machine Learning Reproducibility Checklist. The code policy told authors what was expected: “In short, we expect (but not require) accompanying code to be submitted with accepted papers that contribute and present experiments with a new algorithm.” About 40% of papers linked code at submission and 74.4% did by the camera-ready deadline, against under 50% at NeurIPS 2018. Surveyed at the end of the review period, reviewers answered yes 5,298 times when asked whether code had been provided, and 2,255 times when asked whether they had looked at it. Then 173 accepted papers were claimed for independent reproduction by participants at 73 institutions, producing 84 reviewed reports. Eight authors published the tally in 2021. The written request coincided with a sharp rise in shared code.

The five F1 scores per system below are invented, though they include the figures the opening reported: the team from the start of this lesson reruns its comparison properly.

Repeated training runs estimate variation caused by fitting randomness. Confidence intervals or resampling over held-out examples estimate uncertainty from the finite evaluation sample. A project may face both.

The second source is easy to confuse with a third: a change in what is being sampled. For ICML 2019, Recht and three co-authors rebuilt the CIFAR-10 and ImageNet test sets by repeating the original collection protocol, then re-measured fixed, already-trained models on the new samples. Every model lost accuracy. Their introduction: “The accuracy drops range from 3% to 15% on CIFAR-10 and 11% to 14% on ImageNet. On ImageNet, the accuracy loss amounts to approximately five years of progress in a highly active period of machine learning research.” No model changed. Only the sample it was judged on changed. Model rankings were almost exactly preserved, which is why adaptivity to the old test set was not the explanation; the authors pointed instead to new images slightly “harder” than the originals — a shift in the sample, not the finite-sample error that confidence intervals describe.

Then the measurement itself was audited. An independent group at MIT and Berkeley corrected statistical bias in the replication procedure and found that only 3.6% ± 1.5% of the original 11.7% ± 1.0% ImageNet drop remained unaccounted for. Even the measured drop had to be re-measured.

When data contains groups or time dependence, ordinary independent resampling can be misleading. The resampling unit should respect users, sites, sessions, or periods.

Bouthillier and sixteen co-authors modelled the whole benchmarking pipeline for MLSys 2021. They found that “variance due to data sampling, parameter initialization and hyperparameter choice impact markedly the results”. Their recommendation runs against instinct. Instead of freezing everything except the factor under test, randomise more of it. Holding the extra sources fixed narrows the spread you measure without making the estimate any more honest. The paper reports that “adding more sources of variation to an imperfect estimator approaches better the ideal estimator at a 51 times reduction in compute cost”. They then measured the error rate of detecting improvements “on five different deep-learning tasks/architectures”, before turning any of it into advice. The two pieces of advice in this lesson answer different questions: pairing the seeds of two systems removes noise from the comparison between them, while randomising more sources gives an honest estimate of how far any one result could move.

A concise experiment record reduces retrospective storytelling. Declaring the hypothesis, the metric and the decision rule before the result is visible has a measurable price, and another field has already paid it in public.

Large cardiovascular trials stopped finding what they used to find. Kaplan and Irvin reviewed all 55 large NHLBI-funded cardiovascular randomized trials from 1970-2012 with direct costs over $500,000 a year, and published the count in 2015. Their Results section: “17 of 30 studies (57%) published prior to 2000 showed a significant benefit of intervention on the primary outcome in comparison to only 2 among the 25 (8%) trials published after 2000” (chi-square = 12.2, df = 1, p = 0.0005). Of the variables they examined, one tracked the shift: pre-registration on ClinicalTrials.gov, which requires the outcome to be declared before the data are seen. It was strongly associated with the trend toward null findings. Industry co-sponsorship was not. The authors' own conclusion is careful: prospective declaration of outcomes “may have contributed” to that trend.

If predeclaration is the reason, it did not make the interventions worse. It removed the room to choose a positive outcome after the fact. A template like the one below is the same instrument at project scale.