Underfitting: When the Model Cannot Learn Enough

A forecast that predicts nearly the same demand every day is the textbook picture of a model that never learned the structure in front of it. Training and validation errors are both high. The residuals underpredict the same weekdays again and again. The question is how often that happens to competent people. It has been counted.

The M5 Accuracy competition ran on 42,840 hierarchical daily Walmart sales series. It drew 7,092 participants on 5,507 teams from 101 countries. Only 2,666 teams, 48.4%, beat a Naive forecast. Only 1,972, 35.8%, beat seasonal Naive. Just 415, 7.5%, beat ES_bu, the best of the simple benchmarks. The full results were published in the International Journal of Forecasting in 2022. Stephan Kolassa reduced them to one line: “only 7.5% of teams manage to beat an extremely simple benchmark!”

The structure was there to be found. The winning submission improved on that same benchmark by 22.4%. Most of the field had modern methods and the same data. It produced fits that a seasonal rule of thumb could match or better.

A model underfits when its fitted behavior remains too simple, or otherwise inadequate, for the training task. High training error is a common sign. It is most convincing when a reasonable baseline, or plain human inspection, reveals structure the model misses.

Poor training performance does not prove that the model family is too small. That is not a caution. It is a measured fact. Standard image-classification convnets trained by SGD drive training error to zero on CIFAR-10 with the labels replaced by random noise. Zhang and colleagues showed that in 2017, and their abstract says it in one line: “state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data”. Arpit and colleagues confirmed the same memorisation capacity independently the same year: “deep networks are capable of memorizing noise data”. A model family that can memorise pure noise and still cannot fit your real training set has a data, objective or optimisation problem. It does not have a capacity problem.

The opposite reading fails just as often. High error can be the task's own ceiling. The COMPAS recidivism tool uses 137 features. On 1,000 Broward County defendants it scored 65.2% accuracy. A crowd of non-expert humans scored 67.0%. Julia Dressel and Hany Farid reported that in Science Advances in 2018, and added: “We further show that a simple linear predictor provided with only two features is nearly equivalent to COMPAS with its 137 features.” A separate group reproduced the result independently on the same Broward County data — 7,214 records, 6,907 complete — using certifiably optimal short rule lists. When two features do the work of 137, the error that remains is not waiting for a bigger model.

The diagnosis should move through the full learning system, tier by tier, from the outside in.

The problem ceiling comes first. The target may be ambiguous, or simply unavailable from the inputs. Zillow Group announced on 2 November 2021 that it would wind down Zillow Offers. It took an approximately $304 million Q3 inventory write-down, forecast a further $240–265 million of Q4 losses, and cut about 25% of its workforce. The stated ground was that home prices were less predictable than its pricing models assumed. A company with the data, the capital and the engineers concluded that the tier it had hit was this one.

Data and labels come second, and they carry a measured floor. The field's own benchmarks have been audited for it. Northcutt and colleagues published the count in 2021: “Errors in test sets are numerous and widespread: we estimate an average of at least 3.3% errors across the 10 datasets, where for example label errors comprise at least 6% of the ImageNet validation set.” An independent re-annotation of that same ImageNet validation set in 2020 found the original labels no longer the best predictor of careful human annotation. On the benchmark the field trusts most, roughly one validation label in sixteen is wrong. That share of the residual error is unreachable by any model.

Representation comes third: useful structure may be absent or encoded poorly. Model capacity comes fourth: the family may be unable to express the required relationship. Optimization comes last: the procedure may fail to find a good solution that already exists inside the available family. Only the fourth tier is fixed by a bigger model. It is the tier practitioners reach for first.

Training and validation behavior provide clues, not automatic verdicts. Three patterns are worth separating. Underfitting typically leaves training and validation both weak and relatively close, with systematic residual patterns, little gain from extra training, and simple examples mispredicted. Overfitting shows strong training against substantially worse validation, a gap that grows with training or complexity, and rare training details dominating the fit. An optimization failure shows weak or unstable training performance despite plausible capacity: loss that will not fall even on a tiny sample, curves that oscillate or stay flat, gradients or scales that look abnormal.

The third pattern is the one that gets misread as the first, and there is a clean measurement of it. A 56-layer plain network on CIFAR-10 reaches higher training error than a 20-layer one. On ImageNet the 34-layer plain net records 28.54% top-1 error against 27.94% for the 18-layer plain net, and it carries higher training error throughout training. The ResNet paper reported those numbers in 2016. Its caption to Figure 1 states the point flatly: “The deeper network has higher training error, and thus test error.”

That cannot be a capacity limit. The shallower net's solution space is a subspace of the deeper one's. The deeper model could in principle copy the shallower model's answer and stop. It failed to reach it. A separate group later gave an account of why deep plain feedforward nets are hard to optimise, in a 2017 paper on shattered gradients. Adding layers to that network raised training error. Adding still more would not have lowered it.

Move from basic correctness toward added flexibility, one controlled change at a time.

First, verify the target: inspect examples and confirm the inputs contain plausible evidence at all. Second, beat the baseline. Check whether the model learns more than a constant or a simple seasonal rule. In the M5 Accuracy competition only 415 of 5,507 teams cleared ES_bu, so this step eliminates a great deal. Third, fit a tiny sample. The upper bound this test relies on is the memorisation result: convnets trained by SGD reach zero training error on CIFAR-10 with random labels, and Arpit and colleagues confirmed that “deep networks are capable of memorizing noise data”. If your flexible model cannot drive loss down on a handful of real examples, the learning loop, data pipeline or objective is broken. Memorising a handful of examples is the one thing such a model is known to be able to do.

Fourth, inspect residual patterns for missed calendar, group, nonlinear or interaction structure. Fifth, add one capability — a feature, an interaction, depth, or a different model family — and remeasure. Sixth, confirm held-out value: lower training error must also improve the validation behavior the project cares about.

Different diagnoses call for different changes, and the last one on the list is a decision rather than an engineering step. Zillow Group took it in public. On 2 November 2021 it announced it would wind down Zillow Offers: an approximately $304 million Q3 inventory write-down, $240–265 million of forecast Q4 losses, about 25% of its workforce cut. Rich Barton, the company's co-founder and CEO, gave the reason in the results press release: “We've determined the unpredictability in forecasting home prices far exceeds what we anticipated and continuing to scale Zillow Offers would result in too much earnings and balance-sheet volatility.” The remedy for a problem ceiling is to change the decision the model supports, not the model.

Reducing severe underfitting is progress. The model has begun to capture observed structure. The improvement only becomes useful when it transfers to held-out cases and supports the decision the project was framed around.

A richer model can move a project from underfitting into overfitting, and the memorisation results say how far that can go. The same networks that fit CIFAR-10 with random labels to zero training error will happily drive your training loss down without learning anything transferable. Pair capacity changes with validation curves, slice checks, and error inspection. And when the residual error looks like the Fragile Families ceiling, or Zillow's, ask whether the task is predictable at all.