Validation, Early Stopping, and Checkpoint Choice

A run may produce hundreds of candidate states. The rule that picks one of them can be written down in advance, and it does not have to be the argmax of a curve.

The 2017 Transformer paper wrote its rule down. Every architecture variation in it was scored on a development set — Table 3 says the metrics are “on the English-to-German translation development set, newstest2013” — and the translation models it reported were not single checkpoints at all. Vaswani and colleagues state the policy in Section 6.1: “For the base models, we used a single model obtained by averaging the last 5 checkpoints, which were written at 10-minute intervals. For the big models, we averaged the last 20 checkpoints.” The cadence is stated. The window is stated. The operation is stated. A reader can reproduce the selection without ever being told which individual checkpoint scored highest, because no individual checkpoint was chosen.

That is one answer among several. Below, the same question asked five other ways.

Evaluating every few steps can identify short-lived peaks, but it also increases compute and the chance of choosing noise. Validating rarely may miss a meaningful transition, or spend budget long after the model started getting worse.

Cadence should match metric variability, schedule phases, and cost. Report how many checkpoints were considered.

The cost of considering many has been measured, and the failure has a name. Overtuning is overfitting to the validation estimate at the level of selection rather than at the level of weights. A 2025 reanalysis of hyperparameter-optimization benchmark data, by Schneider and colleagues, found it more common than previously assumed and usually mild: roughly 60% of runs showed none at all. The tail is the part that matters. “In approximately 10% of cases, overtuning leads to the selection of a seemingly optimal HPC with worse generalization error than the default or first configuration tried.” Severity concentrates in the small-data regime, which is exactly where frequent validation on a small held-out split is most tempting.

Ordinary fine-tuning shows the same thing. Dodge and colleagues fine-tuned BERT hundreds of times per task in 2020, “varying only the random seeds”, and released “training and validation scores for 2,100 trials”. On small datasets “many fine-tuning trials diverge part of the way through training”. Stopping the weakest runs early is a budget decision, and their paper offers “best practices for practitioners to stop training less promising runs early”.

For inference, normalization buffers, tokenizer or preprocessing versions, and calibration parameters may be essential. For resume, optimizer, scheduler, scaler, sampler, and random states also matter. Label every checkpoint by its purpose. “Best model” is incomplete when the surrounding state and selection metric are absent.

Smoothing changes how long a real turn takes to show, and how often a stop fires on noise.

A minimum-improvement threshold is a claim about how large a change has to be before it is not fluctuation. So it has to clear something measurable, and the fluctuation has been measured. Bouthillier and colleagues modelled every source of it in a 2021 benchmark across five deep-learning tasks and architectures. Data sampling, parameter initialization and hyperparameter choice each move benchmark results materially. They move them enough that an estimator deliberately randomising more of those sources approximates the ideal exhaustive estimator better than a lower-variance one that holds them fixed, at 51× less compute. Their second recommendation states the consequence: “Deciding of whether the benchmarks give evidence that one algorithm outperforms another should not build solely on comparing average performance but account for variance.”

An early-stopping rule makes that comparison in miniature, once per validation event, against whichever of those sources the run leaves free. A raw-metric rule with no declared delta is that comparison made without the variance term.

Write them before the run begins. Each control below is a separate decision. Leaving one unstated does not remove it — it fixes it at whatever the framework's default happens to be, chosen by nobody and reported by no one.

Learning rates, loss weights, augmentation, patience, architecture, and checkpoint choice are all influenced by validation results. Repeated access therefore adapts the project to that set.

A final test or later temporal holdout is needed for a less biased estimate after decisions are frozen. Even that evidence can be exhausted if reused for another round.

The Netflix Prize was built around that split, and its organisers said why. About 3 million withheld ratings formed a qualifying set. Half of it scored the public leaderboard: RMSE “is computed immediately and automatically for a fixed but unknown half of the qualifying set (the “quiz” subset)”, wrote Bennett and Lanning, both of Netflix, describing the design in 2007. The other half, the test subset, “is not reported and is used by Netflix to identify potential winners of a Prize”. A third subset, the probe, kept its labels and was attached to the training data so teams could check themselves offline. Yehuda Koren, whose team took the $1M Grand Prize, gave the reason for the second, silent half in his own report on the winning solution: “This precludes clever systems which might “game” the competition by learning about the Quiz set through repeated submissions.” Every entrant got a validation set to develop against. The prize was decided on a set nobody ever saw a score from.

Every entry below is fixed before the run launches, not after the curves arrive.

Step 5 is not only a research convention. Regulators have written it down. On 27 October 2021 the US FDA, Health Canada and the UK MHRA jointly issued ten Guiding Principles for Good Machine Learning Practice for Medical Device Development. Guiding Principle 4 is “Training data sets are independent of test sets”, and it spells out what independence obliges a developer to do: “Training and test datasets are selected and maintained to be appropriately independent of one another. All potential sources of dependence, including patient, data acquisition and site factors are considered and addressed to assure independence.” The verbs are active. Dependence is to be enumerated and addressed, not assumed absent because two files were split by a random seed. A checkpoint chosen by looking at the final set is a source of dependence like any other.