Experiments, Randomness, and Reproducibility

A team changed a feature pipeline and saw validation F1 rise from 0.71 to 0.74. A rerun with a different seed returned 0.70. The old system sometimes reached 0.73.

A logged run can still be scientifically weak if the comparison is confounded.

A training run is one execution with a particular dataset, configuration, seed and environment. It produces artifacts and metrics, it may be useful for debugging, it can be lucky or unlucky, and on its own it does not identify a cause.

A controlled experiment is a planned comparison that changes a defined factor while holding relevant conditions stable. It starts from a hypothesis, uses comparable evidence and budget, accounts for randomness, and ends with a bounded conclusion.

Exploratory search is a third thing: many changes tried at once to discover promising directions. It is useful for ideation, it increases selection bias, it needs fresh confirmation, and it should not be reported as one preplanned test.

Randomness enters through more than a single seed. Data sampling decides which examples enter a split, batch, bootstrap sample or augmentation. Initialization sets the starting parameter values and so the optimization path. Execution order covers batch order, asynchronous updates and parallel operations. Hardware and libraries contribute numerical kernels, precision and implementation versions. And the measurement sample — a finite validation or test set — creates uncertainty even for a model that never changes.

The hardware-and-library source is not folklore. The two most widely used pieces of deep-learning infrastructure say so in their own documentation. PyTorch's reproducibility note opens with the warning itself: “Completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms. Furthermore, results may not be reproducible between CPU and GPU executions, even when using identical seeds.”

NVIDIA's cuDNN developer guide is equally plain. Across different GPU architectures, no cuDNN routine guarantees bitwise reproducibility. Even on one architecture the guide names the exceptions: cudnnConvolutionBackwardFilter (ALGO_0 or ALGO_3), cudnnConvolutionBackwardData (ALGO_0), cudnnPoolingBackward (CUDNN_POOLING_MAX), cudnnSpatialTfSamplerBackward and cudnnCTCLoss. They are nondeterministic “because they use atomic operations in a way that introduces truly random floating point rounding errors”. Those are backward-pass kernels. Training runs through them on every step. A seed cannot reach them.

Reproducibility is a property of the whole experimental procedure, not only the model file. It also has a written definition rather than a house preference. The U.S. National Academies of Sciences, Engineering, and Medicine wrote it in a congressionally mandated report in 2019: “Reproducibility is obtaining consistent results using the same input data; computational steps, methods, and code; and conditions of analysis.” NISO adopted that exact definition in 2021 and turned it into four badges a publisher can award: Open Research Objects (ORO), Research Objects Reviewed (ROR), Results Reproduced (ROR-R) and Results Replicated (RER). Reproducing and replicating are separate achievements. The badge list keeps them apart.

The six steps that follow are that definition made operational. Version the question: record hypothesis, primary metric, slices and decision rule. Freeze the evidence: identify dataset snapshot, split logic, labels and feature code. Freeze the procedure: store model, hyperparameters, seed policy, budget and stopping rule. Record the environment: capture code commit, dependencies, hardware and execution settings. Preserve outputs: keep logs, curves, checkpoints, predictions and analysis artifacts. Rebuild independently: rerun from the recorded inputs and compare the result within expected variation.

NeurIPS 2019 ran that request as an experiment on a whole field. All 6,743 submissions had to answer the Machine Learning Reproducibility Checklist. The code policy told authors what was expected: “In short, we expect (but not require) accompanying code to be submitted with accepted papers that contribute and present experiments with a new algorithm.” About 40% of papers linked code at submission and 74.4% did by the camera-ready deadline, against under 50% at NeurIPS 2018. Reviewers reported that code was provided for 5,298 submissions, and that 2,255 of them looked at it. Then 173 accepted papers were claimed for independent reproduction by participants at 73 institutions, producing 84 reviewed reports. Eight authors published the tally in 2021. Asking for the record, in writing, moved the field.

Repeated training runs estimate variation caused by fitting randomness. Confidence intervals or resampling over held-out examples estimate uncertainty from the finite evaluation sample. A project may face both.

The second source has been measured at full scale. For ICML 2019, Recht and three co-authors rebuilt the CIFAR-10 and ImageNet test sets by repeating the original collection protocol, then re-measured fixed, already-trained models on the new samples. Every model lost accuracy. Their introduction: “The accuracy drops range from 3% to 15% on CIFAR-10 and 11% to 14% on ImageNet. On ImageNet, the accuracy loss amounts to approximately five years of progress in a highly active period of machine learning research.” No model changed. Only the sample it was judged on changed. Model rankings were almost exactly preserved, which is why adaptivity to the old test set was not the explanation.

Then the measurement itself was audited. An independent MIT group corrected statistical bias in the replication procedure and found that only 3.6% ± 1.5% of the original 11.7% ± 1.0% ImageNet drop remained unaccounted for. Even the measurement of the measurement error had to be measured.

When data contains groups or time dependence, ordinary independent resampling can be misleading. The resampling unit should respect users, sites, sessions, or periods.

Bouthillier and sixteen co-authors modelled the whole benchmarking pipeline for MLSys 2021. They found that “variance due to data sampling, parameter initialization and hyperparameter choice impact markedly the results”. Their recommendation runs against instinct. Instead of freezing everything except the factor under test, randomise more of it. Holding the extra sources fixed narrows the spread you measure without making the estimate any more honest. The paper reports that “adding more sources of variation to an imperfect estimator approaches better the ideal estimator at a 51 times reduction in compute cost”. They then measured the error rate of detecting improvements “on five different deep-learning tasks/architectures”, before turning any of it into advice.

A concise experiment record reduces retrospective storytelling. Declaring the hypothesis, the metric and the decision rule before the result is visible has a measurable price, and another field has already paid it in public.

Large cardiovascular trials stopped finding what they used to find. Kaplan and Irvin reviewed all 55 large NHLBI-funded cardiovascular randomized trials from 1970-2012 with direct costs over $500,000 a year, and published the count in 2015. Their Results section: “17 of 30 studies (57%) published prior to 2000 showed a significant benefit of intervention on the primary outcome in comparison to only 2 among the 25 (8%) trials published after 2000” (chi-square = 12.2, df = 1, p = 0.0005). One variable separates the two eras. It is pre-registration on ClinicalTrials.gov, which forces the outcome to be declared before the data are seen. That was strongly associated with the shift to null findings. Industry co-sponsorship was not.

Predeclaration did not make the interventions worse. It made the earlier positive rate impossible to reproduce. A template like the one below is the same instrument at project scale.