Debugging a Model That Will Not Learn

A model that does not learn may have wrong labels, misaligned examples, a detached graph, frozen parameters, a constant feature, an invalid reduction, or a broken metric. Optimizer tuning sits late in that chain. Start with tests that can falsify entire categories of failure quickly. Random knob changes spend compute without improving understanding.

Start with the labels. In 2021 three researchers re-examined the test sets of the 10 most commonly used vision, NLP and audio benchmarks, and reported this: “Errors in test sets are numerous and widespread: we estimate an average of at least 3.3% errors across the 10 datasets, where for example label errors comprise at least 6% of the ImageNet validation set.” Those are not speculative flags. Crowdworkers confirmed 51% of the algorithmically flagged candidates as genuine errors. The part to keep is what the errors do to a decision. On corrected labels ResNet-18 overtakes ResNet-50 once the prevalence of originally mislabelled test examples rises by just 6%, and the same audit reports a comparable crossover between VGG-11 and VGG-19. The benchmark never said the smaller model was better. The label noise did.

Leakage does the same thing to conclusions. A survey of fields that had adopted machine-learning methods put a count on it: “Through a survey of literature in fields that have adopted ML methods, we find 17 fields where leakage has been found, collectively affecting 294 papers and, in some cases, leading to wildly overoptimistic conclusions.” Kapoor and Narayanan published that in 2023, with a taxonomy of eight leakage types. The count is itself a small lesson in checking. Their 2022 preprint had said “17 fields where errors have been found, collectively affecting 329 papers” and offered “a fine-grained taxonomy of 8 types of leakage”; peer review revised the paper count down to 294. In their own civil-war-prediction reproducibility study, every paper claiming that complex machine learning beats logistic regression failed to reproduce once the leakage was fixed. A pipeline fault reads like a modelling fault. It survives peer review. It costs far more time than the optimizer ever will.

A good debugging suite includes both expected success and expected failure. The negative control is the half that gets skipped. It is also the half with a formal statistical form. Ojala and Garriga set it out in 2010: “The first test assess whether the classifier has found a real class structure in the data; the corresponding null distribution is estimated by permuting the labels in the data.” That is Test 1 of a named pair of permutation tests, not a rule of thumb. scikit-learn ships it as sklearn.model_selection.permutation_test_score, which “Permutes targets to generate 'randomized data' and compute the empirical p-value against the null hypothesis that features and targets are independent”, citing Ojala and Garriga as the test it implements. A label shuffle that still scores above chance is a rejected null with a p-value attached. It is not a hunch about leakage.

What happens when nobody runs the negative control is on the record at scale. Roberts and the AIX-COVNET collaboration reviewed the machine-learning models built to detect or prognosticate COVID-19 from chest radiographs and CT scans, over studies from 1 January to 3 October 2020. Their abstract: “Our search identified 2,212 studies, of which 415 were included after initial screening and, after quality screening, 61 studies were included in this systematic review. Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases.” The published version in Nature Machine Intelligence, and Cambridge's own release, give 62 for that final count. Either way, not one model that survived screening was of potential clinical use.

The mechanisms are the point. One model had learned the patient's posture. MIT Technology Review reported it in 2021: “Driggs's group trained its own model using a data set that contained a mix of scans taken when patients were lying down and standing up. Because patients scanned while lying down were more likely to be seriously ill, the AI learned wrongly to predict serious covid risk from a person's position.” Other models keyed on the text font a hospital used to label its scans. Derek Driggs describes “Frankenstein data sets, which are spliced together from multiple sources and can contain duplicates”. Every one of those models would have passed a tiny-batch fit. What fails them is a label shuffle, a simple baseline, or an hour spent looking at raw examples. Dr Michael Roberts, in the Cambridge release: “Any machine learning algorithm is only as good as the data it's trained on”.

Save parameters, run one verified batch, apply one update, and compare changed tensors. Confirm that frozen parameters stay fixed and intended modules move.

Repeat with zero loss or detached output to verify the test itself. Unit tests should fail when the learning signal is deliberately removed.

The positive control rests on an unusually firm result: networks fit random labels. Zhang and colleagues showed it, and published at ICLR in 2017: “Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization, and occurs even if we replace the true images by completely unstructured random noise.” Three clauses matter separately. The fit reaches zero training error. Explicit regularization does not prevent it. And it survives throwing away the images altogether. Arpit and colleagues reproduced the noise-memorization result independently at ICML in 2017, and added the timing: “While deep networks are capable of memorizing noise data, our results suggest that they tend to prioritize learning simple patterns first.”

So a network that cannot memorize a handful of manually verified examples is failing at a task two independent groups have shown to be both easy and early. Capacity is rarely the explanation. The graph, the targets, or the data path is.

Move upward only after the lower layer passes. The ordering is not a stylistic preference. It matches where defects are actually found in real deep-learning code. Islam and colleagues, at Iowa State University, examined 2,716 Stack Overflow posts and 500 GitHub bug-fix commits across Caffe, Keras, TensorFlow, Theano and Torch. They found that “data bug and logic bug are the most severe bug types in deep learning software appearing more than 48% of the times”, and that “major root causes of these bugs are Incorrect Model Parameter (IPS) and Structural Inefficiency (SI) showing up more than 43% of the times”. The mass of real faults sits in the data and in the model definition, which are the lower rungs of this stack. It does not sit in the update rule at the top. That is where debugging attention usually starts.

Stop when a test fails and isolate that layer. Each step is chosen so that its outcome is evidence about one layer of the stack. A failure then narrows the search instead of merely changing the loss curve.

Each symptom suggests a different next test, and the first of them is a counted category rather than folklore. A 2018 study sorted 175 real TensorFlow bugs — 87 collected from Stack Overflow, 88 from GitHub — into four symptom classes. One of them is “Low Effectiveness”, meaning extraordinarily poor accuracy or loss. Three symptom types together covered 161 of the 175 bugs (92%), and 159 root causes (90.9%) fell into six TensorFlow-related categories. The same study recorded how practitioners decide a program is defective in the first place: “In general, the accuracy is expected to show an increasing trend across iterations and the loss is expected to a decreasing trend. If no clear trend of increasing or decreasing is observed among several iterations, the TF users consider the model as buggy.” The grammatical slip is in the original.

Record the suspected cause, predicted observation, experiment, result, and next conclusion. This prevents circular tuning and repeated tests.

Include disproved hypotheses and failed interventions. A clean history helps another engineer reproduce the diagnosis and avoids rediscovering the same dead ends.

Two of the largest public training runs kept precisely this artefact and then published it. BigScience kept a public chronicle of the BLOOM 176B run: 19 dated entries between 11 March and 4 July 2022, each recording a suspected cause, the test that reproduced it and the fix. One entry, “2022-03-24 grad clip TP sync bug fixing”, documents a defect that no loss curve would ever have surfaced: “We discovered a bug in BF16Optimizer that didn't clip gradients in TP ranks > 1, leading to layer norm weights and biases getting out of sync. Mathematically they all should have the same layer norm weights since the TP splitting is virtual. This doesn't impact the model while it's training, as each TP rank just learns its own weights.” The ledger also names the test written to pin it down, test_layer_norm_consistent. Elsewhere the same log records 7.3 hours lost to a CUDA crash, answered by halving the checkpoint interval, and a throughput drop from 149 to 140 TFLOPs localized by binary search over nodes. Elimination, written down, rather than tuning.

Meta AI treated the log as a deliverable in its own right. The OPT-175B paper says so in its abstract: “We are also releasing our logbook detailing the infrastructure challenges we faced, along with code for experimenting with all of the released models.” The metaseq repository's index to those chronicles puts it plainly: “Here we have included our full logbook used while training the OPT-175B model, along with a series of notes written to summarize the process and communicate some of the challenges we faced along the way”. The log was part of the release, not a byproduct of it.