Learning Curves and Capacity Diagnostics

A flat validation metric can result from underfitting, a saturated metric, label noise, threshold choice, or an optimization failure. A growing train-validation gap can reflect overfitting or a validation distribution mismatch.

Each of those alternatives is a measured quantity somewhere, not a mood. The test sets of ten of the most-used vision, language and audio benchmarks carry at least 3.3% label errors on average, and at least 6% of the ImageNet validation set is mislabeled. Those figures come from an audit by Northcutt and colleagues, presented at NeurIPS 2021. A separate team re-annotated that same ImageNet validation set in 2020 and found the gains of recent classifiers substantially smaller under the new labels. A curve that stops moving near the top of such a benchmark may be reporting the ceiling of the labels rather than the limit of the model.

Learning curves become useful when several axes and controlled interventions tell a consistent story.

Train matched recipes on nested dataset sizes and plot performance against unique evidence and compute. If gains continue, collecting more data may be worth paying for. If curves saturate, target quality, model bias, or metric ceilings deserve attention.

Nine researchers at Baidu Research measured exactly that in 2017, across machine translation, language modeling, image processing and speech recognition. Generalization error fell as a power law of training-set size in all four. They report “power-law generalization error scaling across a breadth of factors”. The architecture changes they tried moved the curve down without tilting it: “Further, model improvements only shift the error but do not appear to affect the power-law exponent.” A NeurIPS 2022 paper on data pruning opens by treating that power-law fall-off as the widely observed baseline. If a better model buys the intercept and not the slope, the data axis is the one that has to be paid on.

Compute keeps its own accounts. Hoffmann and colleagues at DeepMind trained over 400 language models, from 70 million to over 16 billion parameters, on 5 to 500 billion tokens. Their conclusion: “we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled”. Table 1 of that 2022 paper sets Chinchilla, 70 billion parameters trained on 1.4 trillion tokens, against Gopher's 280 billion parameters trained on 300 billion tokens. The smaller model reports 67.5% average accuracy on MMLU. Epoch AI pulled on the result in 2024 and found the paper's own parametric fit poorly matched the reconstructed data, while its corrected estimates remained consistent with the Chinchilla scaling policy. That is what an anchored claim looks like after someone has leaned on it.

No single horizontal axis describes every resource. Optimizer steps, examples or tokens consumed, wall-clock or compute spent, and unique training-set size answer different questions. A plan can be defensible on one axis and wrong on another.

Chinchilla and Gopher are the published illustration. Gopher was trained at 280 billion parameters on 300 billion tokens. Chinchilla was trained at 70 billion parameters on 1.4 trillion tokens. Read on the parameter axis, one model is far larger. Read on the token axis, the other consumed far more evidence. It is the smaller one that Hoffmann and colleagues describe as the more compute-optimal of the two.

Choose the test that separates competing explanations.

Step 2 has a strong published prior behind it. Zhang and colleagues reported at ICLR 2017: “Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data.” Explicit regularization did not stop the fit. Neither did replacing the real images with unstructured random noise. Arpit and colleagues confirmed that memorization capacity at ICML the same year, while showing that networks nevertheless learn simple patterns first. So when a modern network will not drive training loss down on a small verified sample, the evidence points at the implementation or the optimizer well before it points at capacity.

Step 5 has a measured floor. David Picard trained one fixed 9-layer ResNet recipe on CIFAR-10 across 10,000 random seeds — about 83 hours of V100 time. Final validation accuracy ran from 89.01% to 90.83%. That is a 1.82% spread produced by the seed alone. Fine-tuning a pretrained ResNet-50 on ImageNet, roughly 0.5% separated min from max over 50 seeds. Of the CIFAR-10 spread he writes: “Such difference is widely considered as significant in the community - to the point of being an argument for publication at very selective venues - whereas we know here that this is just the effect of finding a lucky/cursed seed.” Bouthillier and sixteen co-authors modelled the whole benchmarking process at MLSys 2021 and concluded that improvements smaller than that variance will not generalize. A curve pattern that does not survive re-seeding is not yet a finding.

These artifacts can send a team after the wrong fix, and the size of the effect is not always small. Even rebuilding an evaluation set is a biased measurement. Correct for that bias and only an estimated 3.6% ± 1.5% of an 11.7% ± 1.0% ImageNet accuracy drop is left unaccounted for.

No clinician reads a temperature on its own. Blood pressure, symptoms, medication timing, and lab results arrive with it, because one trend rarely identifies the cause by itself.

The chart earns its authority the way a set of training curves does. Each signal is weak alone. The diagnosis comes from reading several of them against a change someone deliberately made.

These are starting interpretations, not automatic labels. Two of the four have published counter-examples attached to them.

When both curves are poor, undercapacity is the least likely reading for a modern network. State-of-the-art convolutional classifiers fit a completely random labeling of the training data, and did so with explicit regularization in place. A training loss that will not fall is therefore evidence about the pipeline before it is evidence about the model.

When loss improves while the metric stays flat — or worse, when the two point in opposite directions — the fix is not to trust the loss. Take a 110-layer ResNet with stochastic depth, trained on CIFAR-100 with a 10x learning-rate drop at epoch 250. After that drop the test negative log-likelihood overfits for the rest of training while test error keeps falling. Guo and colleagues, who ran it in 2017, put the result this way: “Surprisingly, overfitting to NLL is beneficial to classification accuracy. On CIFAR-100, test error drops from 29% to 27% in the region where NLL overfits.” The best validation loss and the best validation error occurred at different epochs. A run stopped on the loss curve would have shipped the worse classifier. Minderer and colleagues revisited the same calibration question on newer architectures at NeurIPS 2021.

Nested samples should preserve time, groups, and coverage. Random subsets can hide the difficult conditions that larger datasets add.

Trustworthiness is measurable as well, and it has a ceiling. The NeurIPS 2021 label audit put at least 3.3% label errors on average across the test sets of ten of the most-used vision, language and audio benchmarks, and at least 6% of the ImageNet validation set mislabeled. On corrected ImageNet labels, ResNet-18 outperforms ResNet-50 once the prevalence of originally mislabeled test examples rises by just 6%. Northcutt, Athalye and Mueller state the consequence: “Surprisingly, we find that lower capacity models may be practically more useful than higher capacity models in real-world datasets with high proportions of erroneously labeled data.” A scaling curve drawn against labels that wrong can rank two capacities in the wrong order. Adding rows of the same quality moves the curve toward that ceiling rather than past it.