The Training Loop: Predict, Measure, Update

Implementations differ, but most parameter-learning procedures contain these conceptual stages.

During a forward computation, the model uses its current parameters to produce outputs. Training adds a feedback and update process on top of that, one that changes the parameters themselves. Inference usually performs only the output computation and leaves them fixed.

That distinction is why a deployed model does not automatically learn from every request it serves. Online updates require an explicit pipeline, an objective, safeguards, and new evidence, none of which appears by itself.

Microsoft built the exception and then published the post-mortem. Tay was released on Twitter on 23 March 2016 and did learn from its interactions with users. It tweeted more than 95,000 times within 16 hours. It was taken offline inside the first day. Peter Lee, then a Microsoft corporate vice president, gave the company's account on 25 March 2016: “In the first 24 hours of coming online, a coordinated attack by a subset of people exploited a vulnerability in Tay.” He wrote that Tay would return only when Microsoft could better anticipate malicious intent. The update loop was the easy half to build. The objective, the evidence and the safeguards around it were the half that was missing. The loop ran regardless — sixteen hours and ninety-five thousand tweets of it.

The same final training loss can accompany very different learning behavior.

The amber pattern below carries a warning the picture cannot show. It is a heuristic with published counter-examples. A ResNet18 was trained on CIFAR-10 with label noise, using Adam for up to 4,000 epochs. Test error fell, rose, and then fell again. The abstract names the effect: “double descent occurs not just as a function of model size, but also as a function of the number of training epochs”. Nakkiran and colleagues published that in 2020. The same paper identifies regimes where quadrupling the number of training samples hurts test performance. The model-size version of the curve had been established independently by Belkin and colleagues in 2019. So a run whose validation error has been climbing for a while may be in the middle of the second descent rather than past its best checkpoint. Stopping at the first divergence remains a sensible default. It is not a law, and the counter-example has a citation.

Useful run records make failures reproducible rather than mysterious.

The second item on this list has a price attached. Ten trials of one algorithm on one task were run with identical hyperparameters, varying only the random seed. The trials were split into two groups of five, and each group was averaged. A 2-sample t-test separated the two average curves at t = -9.0916, p = 0.0016 on HalfCheetah. Henderson and colleagues put it this way in 2018: “Particularly for HalfCheetah, it is possible to get learning curves that do not fall within the same distribution at all, just by averaging different runs with the same hyperparameters, but different random seeds.” Same code, same data, same settings, two contradictory and publishable pictures. The only variable that distinguished them is the one a run record most often omits. Agarwal and colleagues later showed the same problem across the field, arguing that results from a small number of runs cannot be reported as bare point estimates.

Training may stop after a fixed budget, when validation improvement becomes negligible, when a target criterion is reached, or when instability appears. The rule should be defined before the final test is examined.

An entire industry does exactly that, in public, and calls it a benchmark. MLPerf Training, run by MLCommons, fixes the quality first and measures the clock second: “MLPerf Training measures the time it takes to train machine learning models to a standard quality target in a variety of tasks including image classification, object detection, NLP, recommendation, and reinforcement learning.” For ResNet-50 on ImageNet the standard quality target is 75.90% top-1 accuracy. A submission is not scored on throughput, and not stopped at a fixed epoch budget. Training is stochastic, so each benchmark is run several times, the highest and lowest results are dropped, and the rest are averaged. That still leaves roughly ±2.5% variance on the imaging benchmarks. In the v1.1 round published on 1 December 2021, with 14 submitting organisations and over 185 peer-reviewed results, Graphcore reported reaching that 75.9% target on an IPU-POD16 in 28.3 minutes and 38 epochs. The 28.3 minutes means something only because the 75.90% was written down first.

The lowest training loss is rarely the only goal. Teams also care about validation behavior, calibration, latency, fairness, robustness, and cost. A stopping rule written around training loss alone will happily stop at the wrong place.

How long to keep going is the same class of decision. An entire field got it wrong for several years. Hoffmann and colleagues at DeepMind reported the size of that mistake in March 2022. Large language models had been trained on far too little data for their size. Model size and training tokens should be scaled in equal proportion: double the parameters, double the tokens. Their demonstration was Chinchilla. It has 70 billion parameters and was trained on 1.4 trillion tokens, at the same compute cost as the 280-billion-parameter Gopher. It outperformed Gopher and GPT-3 (175B), Jurassic-1 (178B) and Megatron-Turing NLG (530B), and reached 67.5% on the MMLU benchmark. A quarter of the parameters, four times the data, the same bill. The stopping rule was the design decision.

Before changing the architecture, verify that the loop is solving the intended problem.

A correct implementation can efficiently reduce a poorly chosen loss on an unrepresentative dataset. Training success is therefore conditional on framing, labels, features, and evaluation.

The largest documented case of that sentence runs on about 200 million people a year in the United States. An algorithm used to help manage their care scored patients by predicted health-care cost rather than by illness. Nothing in the loop malfunctioned. It reduced the loss it was handed, on the target it was handed. Cost was a stand-in for the thing anyone actually wanted, which was sickness. Obermeyer and colleagues measured the gap between the two in Science in 2019. Only 17.7% of the patients the algorithm assigned to extra care were Black, against 46.5% if the score were unbiased.

The consequence arrived by letter, dated 25 October 2019. New York's Department of Financial Services and its Department of Health wrote jointly to United Health Group Incorporated about the Impact Pro algorithm: “We call on you to immediately investigate these reports and demonstrate that this algorithm is not racially discriminatory or to cease using Impact Pro (or any other data analytics program) if you cannot demonstrate that it does not rely on racial biases or perpetuate racially disparate impacts.” A regulator can order a company to stop running a loss function. It cannot debug one.

The loop is powerful precisely because it is literal. It will optimize exactly what you wrote down. That is why the broader learning system has to make sure the feedback signal corresponds to the behavior people actually need. No stage of the loop failed here. The objective did, and the loop pursued it at scale, in production, for years.