Randomness, Seeds, and Reproducible Comparisons

Training can vary through several independent mechanisms, and a seed reaches only some of them. The data pipeline, the model, the kernels underneath, the ordering of distributed workers, and the software and hardware the whole thing runs on each carry their own randomness or nondeterminism.

The layers are not ranked by size. Four of them were measured separately on CIFAR-10 and came out almost indistinguishable: an accuracy standard deviation of 0.23% for parameter initialization, 0.25% for data shuffling, 0.23% for data augmentation and 0.22% for cuDNN kernel nondeterminism. Summers and Dinneen ran that comparison. This is a stack, not a hierarchy. Pinning the top layer leaves the others moving by about as much.

Each level answers a different question. The middle two are not this lesson's private vocabulary — the US National Academies fixed them in 2019, and NISO attached separate badges to them in 2021. The definitions section below sets both out.

The cost gap between the levels is real and measurable. Picard's controlled rerun of a single CIFAR-10 configuration took 10,000 runs and roughly 83 V100-hours. MLPerf's rules settle for 5 runs on vision tasks and 10 on everything else. The level you can afford determines the claim you are entitled to make.

Capture the state needed to interpret both success and failure: data identity, software identity, training state, random-stream policy, and the per-seed results including the runs that failed.

The record is what makes a range reportable instead of a mean. A spread of 1.82 points on CIFAR-10 was a statement Picard could make only because he kept the results of all 10,000 seeds rather than the best of them. A standard deviation of validation performance of 0.061 on MRPC, 0.069 on RTE and 0.101 on CoLA was a statement Dodge and his co-authors could make only because they kept 2,100 fine-tuning runs. A record that stores only the winning checkpoint can report neither number. A paper written from that record has nothing to put after its mean.

The appropriate design depends on cost and expected variability. Two papers have measured what happens when the number of seeds is chosen by habit instead.

Ten runs of TRPO on HalfCheetah-v1, identical hyperparameters, nothing varying but the random seed. Split them into two groups of five, average each group's learning curve, and the two halves of one identical experiment come out statistically different. The Figure 5 caption reads: “The average 2-sample t-test across entire training distribution resulted in t = −9.0916, p = 0.0016.” Five seeds against five seeds, same code, same hyperparameters, and the standard test says these are two different methods. Henderson and his co-authors ran that experiment, and their 2018 AAAI paper states the problem plainly: “non-determinism in standard benchmark environments, combined with variance intrinsic to the methods, can make reported results tough to interpret”.

Atari 100k later got the same treatment at 100 runs per game, against published results on that benchmark that had mostly used 3 or 5. For SPR the median score difference between 5 runs and 100 runs was +0.03 points, about 36% of the improvement SPR had reported over DrQ(ε). The few-run number and the improvement it claims are of the same order. The 95% confidence intervals on sample medians only tightened at 50 to 100 runs. Agarwal and his co-authors did that work.

More is not automatically the answer. Bouthillier and his co-authors modelled the whole benchmarking process and supply the counterweight: “a biased estimator with more source of variation will give better results, closer to the ideal estimator at a 51× reduction in compute cost”. Spread the budget across the sources of variation before spending it all on repetitions of one.

Exact repeatability asks whether the same environment and state reproduce the same result. The next term up is not this lesson's coinage. In 2019 the US National Academies of Sciences, Engineering, and Medicine fixed it in their report Reproducibility and Replicability in Science, in a numbered conclusion: “For this report, reproducibility is obtaining consistent results using the same input data; computational steps, methods, and code; and conditions of analysis.” Replicability, in the same conclusion, is the different thing — consistent results across studies that each collected their own data.

NISO adopted that definition verbatim in 2021, in its recommended practice on reproducibility badging. It quotes the definition as “We define reproducibility to mean computational reproducibility—obtaining consistent results using the same input data, computational steps, methods, code, and conditions of analysis”, and attaches it to the ROR-R “Results Reproduced” badge. The separate RER “Results Replicated” badge is reserved for an independent study aimed at the same scientific question. Two badges, because they certify two different claims.

Statistical robustness is a third question again: whether the conclusion survives ordinary stochastic variation. A bitwise repeatable run can still support a fragile claim based on one seed. No badge is awarded for that.

Written down, a recipe fixes ingredients, temperatures, tools, and timing, and nothing else. Another kitchen may produce a similar dish without identical molecules or oven fluctuations.

That is the distinction worth keeping, and it is the same one the National Academies wrote into their conclusion and NISO turned into two badges. Exact replay is one claim. A reproducible method is a second. A conclusion that survives ordinary variation is a third. A training record should say which of the three it supports. A reader who is told only that the code runs has been told the weakest of the three.

Using matched data order and corresponding seeds can make differences between two methods easier to detect. The pairing must preserve valid independence assumptions for the analysis.

Separating the factors on purpose looks like this. BERT was fine-tuned 2,100 times across four GLUE tasks — 625 runs each on MRPC, RTE and CoLA, laid out as a 25 × 25 grid of weight-initialization seeds against data-order seeds, and 225 on SST. Nothing varied but the seed controlling the data order and the seed controlling the final classification layer, which holds 0.0006% of the model's parameters. Those two seeds moved validation performance by a standard deviation of 0.061 on MRPC, 0.069 on RTE and 0.101 on CoLA, with weight initialization and data order contributing comparably. The grid is what makes that last statement measurable rather than assumed. Seed search alone raised BERT's RTE validation score from the published 70.0/70.4 to 77.3. Dodge and his co-authors published all of it in 2020.

What is at stake in an unpaired comparison is the ranking itself. Reimers and Gurevych evaluated 50,000 LSTM networks over five sequence-tagging tasks in 2017, and found the seed alone producing statistically significant differences (p < 10⁻⁴). Their abstract puts the consequence in one sentence: “For two recent systems for NER, we observe an absolute difference of one percentage point F1-score depending on the selected seed value, making these systems perceived either as state-of-the-art or mediocre.”

Do not discard runs because one method fails under a seed. Failure rate is part of the method's robustness and should be reported.

An industry benchmark body has written exactly that into its rulebook. MLPerf fixes the run count in advance: “Vision tasks require 5 runs to ensure 90% of entries from the same system are within 5%; all other tasks require 10 runs to ensure 90% of entries from the same system are within 10%.” The score is then computed by dropping the fastest and the slowest time and taking the arithmetic mean of the rest. Mattson and colleagues, describing the benchmark in 2020, record that for MiniGo the authors saw considerable variability across runs even with the random seed fixed.

The current MLCommons training rules keep the scheme and set a minimum number of runs per benchmark: 3 for LLM MoE pretraining; 10 for text-to-image, small-LLM pretraining, LLM fine-tuning and recommendation; and 40 for the deprecated medical image-segmentation benchmark. The same drop-fastest-and-slowest scoring survives. The submission score is “intended to represent the median expected result across a large number of runs”, and non-convergence is handled by rule rather than by discretion: one non-converging run may be treated as the slowest and dropped, while more than one invalidates the result.

That is the failure rate written into the scoring. The run count is chosen before the runs. The outliers are trimmed by a rule that applies to both ends. And the second failure costs the submission rather than being quietly excluded.