Learning Curves and When More Data Helps

A learning curve plots performance against the amount of training evidence. The evaluation policy stays fixed.

The procedure has a canonical published run. In 2001 Banko and Brill, at Microsoft Research, trained four learners on nested subsets of a one-billion-word corpus: Winnow, a perceptron, naive Bayes and a simple memory-based learner. The task was confusion-set disambiguation. Every one of those subsets was scored against the same held-out evaluation — 1 million words of Wall Street Journal text kept out of training. Averaged over ten confusion sets, accuracy was still climbing at the largest subset. That subset was three orders of magnitude beyond the largest corpus previously used for the problem. Their own summary of the plot is a single sentence: “Note that the curves appear to be log-linear even out to one billion words.”

The curve also moved the conclusions, not just the scores. Complementarity between two of the learners fell from 0.2612 at 10^6 words to 0.1612 at 10^9 words. At the small end the two methods were failing on noticeably different examples. At the large end they were failing on much the same ones. A comparison run only at 10^6 words would have reported a difference that had largely dissolved by 10^9. That is what the ladder of subsets is for. One dataset size gives you one point on a curve, and no way to know which part of it you are standing on.

Curves do not prove a cause, but they narrow the investigation.

Read each pattern as a statement about the range you actually measured. A curve still rising at the largest subset — as Banko and Brill's was at one billion words — is evidence that the sizes you tried were data-limited. It is not a promise that the next decade of collection behaves the same way. A curve that flattens is evidence about the representation, the labels or the metric at those sizes. It is not yet evidence about the problem itself.

More rows help most when they add independent variation relevant to deployment, and frames from the same video, repeated windows from one machine, or templated messages may increase compute without increasing the diversity of situations learned.

In medical devices this is not advice, it is a published expectation. Three regulators — the U.S. Food and Drug Administration, Health Canada and the UK Medicines and Healthcare products Regulatory Agency — jointly issued 10 Guiding Principles for Good Machine Learning Practice for Medical Device Development in October 2021. Guiding Principle 4 names the dependencies a row count hides: “Training and test datasets are selected and maintained to be appropriately independent of one another. All potential sources of dependence, including patient, data acquisition, and site factors, are considered and addressed to assure independence.” Three named axes: the same patient, the same scanner, the same hospital. Each one has to be addressed rather than counted around. Principle 3 asks that the intended population be “sufficiently represented in a sample of adequate size”. Principle 8 asks for test plans that generate performance information “independently of the training data set”.

Track distinct entities, sites, time periods, devices, annotators, and outcome types alongside row count. Sampling design determines the information gained.

Data acquisition becomes more efficient when linked to a diagnosed gap.

The coverage gap has the best-documented worked example. Three commercial gender classifiers were tested in 2018, and the error rate ran as high as 34.7% for darker-skinned females against a maximum of 0.8% for lighter-skinned males. Buolamwini and Gebru did not stop at the symptom. They measured the benchmarks the field was scoring itself on, and then built the missing data: “We find that these datasets are overwhelmingly composed of lighter-skinned subjects (79.6% for IJB-A and 86.2% for Adience) and introduce a new facial analysis dataset which is balanced by gender and skin type.” The slice was named, quantified, and collected against. That is the full sequence this section is asking for.

A federal agency then found the same class of gap at scale. NIST's Face Recognition Vendor Test Part 3, published in December 2019, ran 18.27 million images of 8.49 million people through 189 mostly commercial algorithms from 99 developers. Its finding: “Across demographics, false positives rates often vary by factors of 10 to beyond 100 times.” False negative effects varied by factors usually below 3. Two error types, two very different magnitudes of disparity. The gap between them is itself an argument for diagnosing before collecting, since the two point at different data.

Data work is expensive enough to require an explicit expected benefit.

Write the expected benefit down as a number before the first collection contract is signed, because the published numbers are sobering. A checkpoint that says “we will collect 30× and expect the curve to keep its slope” is a testable claim. “We need more data” is not.

A smooth curve fitted to past gains may fail when the next data source differs, or when the remaining errors are irreducible. Report projections as uncertain scenarios, not guaranteed returns.

The decay has a measured shape. Hestness and colleagues at Baidu Research measured learning curves for six deep network models across four domains: machine translation, language modeling, image classification and speech recognition. Their 2017 finding was that generalization error falls as a power law in training-set size. The exponents came out empirically between -0.07 and -0.35, far shallower than the -0.5 or -1 that theory predicts. The exponent is what decides whether the next order of magnitude of data is worth buying. It is also the one thing better engineering did not move: “Improved model architectures and optimizers can improve the power-law intercept, but not the exponent”. A better architecture shifts the whole curve down. It does not make the curve fall faster.

And a projection fitted to learning curves can be followed by an entire field, then overturned by a redesigned experiment. In January 2020 OpenAI concluded that “optimally compute-efficient training involves training very large models on a relatively modest amount of data”. Kaplan and colleagues put numbers on it: an exponent of 0.73 on parameters against 0.27 on tokens, nearly three to one in favour of size. Two years later Hoffmann and colleagues at DeepMind re-ran the sweep, with the learning-rate schedule matched to each run's token count. Their three methods returned 0.46 to 0.50 on parameters and 0.50 to 0.54 on tokens. Near-equal, not three to one. Their diagnosis is about experimental design rather than about data: “First, the authors use a fixed number of training tokens and learning rate schedule for all models; this prevents them from modelling the impact of these hyperparameters on the loss.” The curve was real. The runs underneath it had each been trained in a way that misrepresented what the data axis was worth. The field scaled the wrong axis for two years on the strength of it.

The important result may be that more data is no longer the most efficient intervention. Better labels, features, workflow, or problem framing can dominate the next improvement.