Skip to content
AI.info

Neural networks

Debugging and Testing Neural Networks

Build a disciplined neural debugging playbook using tiny-batch overfits, unit tests, invariance checks, gradient checks, ablations, and failure localization.

By the end you can

Visual

Five failure classes before any tuning

Begin by deciding what kind of evidence would distinguish the main hypotheses. Five classes cover nearly everything that goes wrong: data and target (wrong examples, labels, masks, splits, units, or availability); implementation (shape, graph, mode, indexing, reduction, or export bugs); optimization (unhealthy scale, learning rate, gradient flow, or update behavior); capacity and bias (the architecture cannot express or efficiently learn the needed relationship); and evaluation (metric, threshold, slice, leakage, or baseline that does not match the decision).

That list is not a hunch about where bugs live. It has been counted twice, from real defects. Humbatova and colleagues built a taxonomy of real faults in deep learning systems in 2020, from a corpus they describe plainly: “We have manually analysed 1059 artefacts gathered from GitHub commits and issues of projects that use the most popular DL frameworks (TensorFlow, Keras and PyTorch) and from related Stack Overflow posts.” They enriched the taxonomy with structured interviews with 20 researchers and practitioners, then validated it with a survey of 21 more developers. Thirteen of the fifteen fault categories had been experienced by at least half the participants. Islam and colleagues classified a separate corpus a year earlier: 2,716 Stack Overflow posts and 500 GitHub bug-fix commits across Caffe, Keras, TensorFlow, Theano and Torch. Data bugs and logic bugs came out the most severe types, appearing more than 48% of the time. Incorrect model parameters and structural inefficiency were root causes more than 43% of the time.

So the mathematics of the model is the least likely place for the defect to be. And the first class on the list is the one with a measured error rate. Northcutt and colleagues audited the test sets of ten of the most-used vision, language and audio benchmarks in 2021: “Errors in test sets are numerous and widespread: we estimate an average of at least 3.3% errors across the 10 datasets, where for example label errors comprise at least 6% of the ImageNet validation set.” Of the candidates their algorithm flagged, 51% were confirmed erroneous by crowdworkers. The rate is large enough to invert a verdict. On corrected ImageNet labels ResNet-18 outperforms ResNet-50, once the prevalence of originally mislabelled test examples rises by just 6%. Beyer and colleagues re-annotated the same ImageNet validation set with a more robust human-annotation procedure, and independently found the gains of recently proposed classifiers substantially smaller under the cleaner labels. A wrong label in the test set does not announce itself as a bug. It arrives as a score.

FigureHierarchy · 5 levels

  • Data and target

    Wrong examples, labels, masks, splits, units, or availability.

    • Implementation

      Shape, graph, mode, indexing, reduction, or export bugs.

      • Optimization

        Unhealthy scale, learning rate, gradient flow, or update behavior.

        • Capacity and bias

          Architecture cannot express or efficiently learn the needed relationship.

          • Evaluation

            Metric, threshold, slice, leakage, or baseline does not match the decision.

Key idea

Overfit a tiny batch before scaling

A sufficiently expressive network should usually drive training loss very low on a handful of clean examples. A failure there suggests a broken target, loss, graph, update, mode, or capacity assumption.

Success does not prove generalization. It establishes that the end-to-end machinery can memorize a controlled sample.

Fitting is not evidence of learning, and the distinction has been quantified. Deep networks are “capable of memorizing noise data” — labels carrying no information whatever. On real data the same networks “tend to prioritize learning simple patterns first”. Arpit and colleagues showed both in 2017. The two cases can be told apart by how the optimization behaves. They cannot be told apart by whether the training loss goes down, since both go down. The same work also found that explicit regularization tuned to slow memorization of noise need not cost anything on real data. That is the useful corollary. A control that hurts only the noise case is a control worth keeping.

This is exactly why the tiny-batch test has to be read as plumbing. A pipeline that memorizes twelve examples has shown you that gradients reach the parameters and that the target is connected to the loss. It has shown you nothing about whether the twelve examples carry signal. As the label-error audits show, some fraction of them will not.

Tiny-batch overfitting is a plumbing test, not a quality benchmark.

Comparison

Four test styles for neural software

One assertion that the loss decreases is not enough. Neural code needs four different kinds of test, each covering a surface the others miss. Unit tests check a module on small known tensors: shapes and values, boundary behavior, train/eval semantics, deterministic fixtures. Property tests verify broad invariants across generated inputs: permutation or translation behavior, finite outputs, monotonic constraints, mask independence. Gradient tests compare analytical or autodiff results with numerical checks, on custom operations, shared parameters, stop-gradient boundaries and small deterministic models. System tests exercise data-to-decision behavior: serialization and reload, batch-size changes, exported runtime parity, latency and fallback paths.

The gradient column is the one people describe as advanced and skip. It is also the one both major frameworks ship as a first-class API. PyTorch's torch.autograd.gradcheck defaults to eps=1e-06, atol=1e-05 and rtol=0.001, and the reference documentation attaches a condition to those numbers: “The default values are designed for input of double precision. This check will likely fail if input is of less precision, e.g., FloatTensor.” JAX ships the equivalent, jax.test_util.check_grads, documented as “Check gradients from automatic differentiation against finite differences.”, which raises an AssertionError when the two disagree beyond atol/rtol. The practical consequence is a trap with two doors. Run the check on float32 tensors at default tolerances and it will likely fail on correct code. Loosen the tolerances until it passes and you have converted a derivative test into a test of the tolerance. Cast the fixture to double precision instead, and keep the defaults.

A single run is not a measurement either. Variance from data sampling, parameter initialization and hyperparameter choice “impact markedly the results” that comparisons are built on. Bouthillier and colleagues modelled the whole benchmarking process in 2021 to show it, across five deep-learning tasks and architectures. Their counter-intuitive recommendation is to randomize more of those sources rather than hold them fixed. Adding variation to an imperfect estimator “approaches better the ideal estimator at a 51× reduction in compute cost”. A test that pins one seed measures that seed.

FigureComparison · 4 columns

Unit tests

Check a module on small known tensors.

  • Shapes and values
  • Boundary behavior
  • Train/eval semantics
  • Deterministic fixtures

Property tests

Verify broad invariants across generated inputs.

  • Permutation or translation behavior
  • Finite outputs
  • Monotonic constraints
  • Mask independence

Gradient tests

Compare analytical or autodiff results with numerical checks.

  • Custom operations
  • Shared parameters
  • Stop-gradient boundaries
  • Small deterministic models

System tests

Exercise data-to-decision behavior.

  • Serialization and reload
  • Batch-size changes
  • Exported runtime parity
  • Latency and fallback paths

Use deliberately simple baselines as sensors

A constant predictor, linear model, frozen random features, or rules engine can expose data and metric problems. If a sophisticated network cannot beat the baseline, complexity has not earned its place.

If a suspicious baseline performs extremely well, investigate leakage, duplicated entities, temporal contamination, or target shortcuts before celebrating.

The reverse failure — nobody ever measures the deployed model against the outcome it claims to predict — has a documented case with numbers. Epic's proprietary sepsis prediction model was validated by outsiders at Michigan Medicine, on 27,697 patients across 38,455 hospitalizations, and the result appeared in JAMA Internal Medicine in June 2021. Area under the curve was 0.63 (95% CI, 0.62-0.64). Sensitivity was 33%. The model failed to identify 1,709 of the 2,552 patients who developed sepsis — 67% of them — while generating alerts on 18% of hospitalizations. Wong and colleagues wrote their conclusion in one clause: “the ESM has poor discrimination and calibration in predicting the onset of sepsis”.

The finding then survived replication by strangers. An independent group ran the same model version at two county emergency departments, on 145,885 encounters in 2023. Their numbers, reported in JAMIA Open in 2024: sensitivity 14.7%, specificity 95.3%, positive predictive value 7.6%, negative predictive value 97.7%. Both audits used instruments any team could have pointed at the model on day one: a held-out outcome, a confusion matrix, an alert rate. Nothing about them is sophisticated. They were simply never read.

A baseline is not merely a competitor; it is a diagnostic instrument.

Case

Leakage was found in 17 fields and 294 papers

The failure has been counted. Leakage has turned up in 17 fields, “collectively affecting 294 papers and, in some cases, leading to wildly overoptimistic conclusions”, under “a detailed taxonomy of eight types of leakage”. Kapoor and Narayanan reported that in Patterns in 2023, after surveying reviews of machine-learning practice across research fields. They then reproduced one such literature themselves — civil war prediction, where complex models were believed to vastly outperform logistic regression — and found that “when the errors are corrected, complex ML models do not perform substantively better than decades-old LR models”. The baseline was not a sensor anyone had bothered to read.

One field was audited in real time, by two teams that did not coordinate, and it reached the same verdict. Roberts and colleagues searched EMBASE, MEDLINE, bioRxiv, medRxiv and arXiv for COVID-19 diagnosis and prognosis models built from chest radiographs and CT scans, published between January and October 2020. They identified 2,212 studies, kept 415 after initial screening and 61 after quality screening. The verdict, in Nature Machine Intelligence in 2021: “Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases.” Not the weakest models. None of the sixty-one that had already survived a quality screen.

The parallel audit covered prediction models generally, and it landed in the BMJ on 7 April 2020. Wynants and colleagues screened 4,909 titles and included 51 studies describing 66 prediction models. They rated every one of the 66 at high or unclear risk of bias, and recommended against clinical use. Two systematic reviews, two search strategies, two literatures, one answer. The models were not failing because the architectures were wrong. They were failing because the evaluation was, and the evaluation is the part nobody had tested.

Position

A benchmark delta is not progress until a baseline and a second seed have had a go at it

Progress in this field is reported as the difference between two numbers, and the difference is usually the part that has been checked least. Three of this lesson's results say why, and they compound.

Start with the baseline. Kapoor and Narayanan found leakage in 17 fields, collectively affecting 294 papers and in some cases producing wildly overoptimistic conclusions, under a taxonomy of eight distinct types. Then they reproduced one of those literatures themselves. Civil war prediction was the case, and complex models were believed to vastly outperform logistic regression there. Correcting the errors left the complex models not substantively better than decades-old logistic regression.

Then the seed, which has now been documented three times in two subfields. Bouthillier and colleagues found that variance from data sampling, initialisation and hyperparameter choice markedly affects the results comparisons are built on. They recommended randomising more of those sources rather than holding them fixed, which approached the ideal estimator at a 51× reduction in compute. Three years earlier, Henderson and colleagues had shown in 2018 that non-determinism plus intrinsic method variance makes reported results hard to interpret, and that improvements over prior state of the art cannot be judged without significance metrics. Three years later, Agarwal and colleagues opened an Outstanding Paper at NeurIPS 2021 with the same diagnosis: “Most published results on deep RL benchmarks compare point estimates of aggregate performance such as mean and median scores across tasks, ignoring the statistical uncertainty implied by the use of a finite number of training runs.” On the Atari 100k benchmark, conclusions drawn from point estimates diverged substantially from those drawn from a proper statistical analysis. They found the same discrepancies in prior comparisons on the ALE, Procgen and the DeepMind Control Suite.

And then the test set itself. CIFAR-10 and ImageNet were rebuilt from scratch, following the original collection procedures. Recht and colleagues reported the result in 2019: “We evaluate a broad range of models and find accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet.” The counterweight is the honest half of that result — the rankings were preserved. Yadav and Bottou independently reconstructed the full MNIST test set of 60,000 digits, the 50,000 never distributed plus the familiar 10,000. Twenty-five years of reuse had produced the same pattern. Misclassification rates shift; classifier ordering and model selection hold.

So a reported improvement is a hypothesis carrying three open questions. Against which baseline? Across how many sources of variance? And with what keeping the test set out of the training data? An absolute number is worth roughly eleven to fourteen points less than printed on a fresh draw from the same pipeline. A rank order is worth more than that. And the uncomfortable part of the civil war reproduction is not that the simple model held up. It is that the comparison had been there the whole time, and the baseline was not a sensor anyone had bothered to read.

A number compared only with last year’s number has not been compared with anything.

Example

High-value debugging experiments

Each experiment should eliminate several hypotheses at low cost. Run them on a frozen case, one at a time. Write down what each outcome would rule out before you look.

  • Shuffle labels: performance should collapse toward chance on held-out data.
  • Repeat one example: verify the loss and gradient direction match a hand-calculated expectation.
  • Disable augmentation and stochastic layers: confirm deterministic reproduction of one step.
  • Replace a complex block with identity or a linear layer: localize where learning breaks.
  • Swap the real model for a tiny known implementation: test the surrounding pipeline independently.
  • Reload a checkpoint and compare outputs bytewise or within a stated numerical tolerance.
  • Evaluate train examples in evaluation mode: separate memorization from mode-related mismatch.

Logs should support a hypothesis, not create a landfill

Record loss components, learning rate, gradient norms, update ratios, activation summaries, throughput, and the data identifiers needed for reproduction. Attach step and checkpoint versions.

Avoid collecting every tensor continuously. Sample detailed traces around failures and keep lower-cost summaries for routine monitoring.

The identifiers matter more than the tensors. At least 3.3% of test labels in the ten most-used benchmarks are wrong. If a defect turns out to be a label, the only thing that lets you prove it is a log that can name the exact example, split and version the model was scored against.

Analogy

Debugging a factory line by isolating stations

Defective products appear at the end of a factory line. Inspecting only the final defect count cannot reveal whether the raw materials, one station, or the quality measurement is wrong.

Neural stages are trained jointly and can compensate for one another. Replacing one block may therefore change the behavior of the rest.

The analogy also carries the failure mode the audits keep finding. A widely deployed sepsis model came in at 33% sensitivity, with alerts on 18% of hospitalizations, when Wong and colleagues finally measured it. The broken station was not in the network. It was the inspection bench at the end of the line, which nobody had calibrated against a real defect.

Localize the earliest divergence from expected behavior, then test one hypothesis at a time.

Steps

A diagnosis-first debugging loop

Resist the urge to launch a broad hyperparameter sweep before the system passes controlled checks. The loop has five steps and the order is the point.

First, freeze a failing case: save the batch, code, seed, environment, and checkpoint. Second, state competing hypotheses, separating data, graph, optimization, capacity and evaluation causes — the same five classes Humbatova and colleagues distilled from 1,059 artefacts, and that a second corpus study reached independently. Third, run the cheapest discriminator: assertions, tiny batches, ablations, or numerical checks such as torch.autograd.gradcheck on a double-precision fixture. Fourth, repair the earliest cause, changing one boundary and rerunning the same case. Fifth, restore realistic conditions, reintroducing scale, stochasticity, augmentation and the deployment runtime gradually.

Step three is where sweeps get substituted for thought. A sweep changes many boundaries at once, and that is precisely the design that makes a result uninterpretable. When the comparison also rests on a single seed, three of the papers above have each shown it cannot support the conclusion drawn from it.

FigureProcess · 5 steps

  1. 1. Freeze a failing case

    Save the batch, code, seed, environment, and checkpoint.

  2. 2. State competing hypotheses

    Separate data, graph, optimization, capacity, and evaluation causes.

  3. 3. Run the cheapest discriminator

    Use assertions, tiny batches, ablations, or numerical checks.

  4. 4. Repair the earliest cause

    Change one boundary and rerun the same case.

  5. 5. Restore realistic conditions

    Reintroduce scale, stochasticity, augmentation, and deployment runtime gradually.

Key idea

When to stop debugging the model and redesign the task

If labels are inconsistent, the decision arrives before relevant evidence, or deployment incentives reward the wrong behavior, architecture tuning cannot solve the core problem.

Escalate to data collection, workflow redesign, abstention, rules, or human review when the learning task itself is not defensible.

The COVID-19 reviews are the clearest instruction on when to stop. Roberts and colleagues screened 2,212 studies down to 61 and found none of potential clinical use. Wynants and colleagues rated all 66 models they included at high or unclear risk of bias. No amount of further architecture work on any single one of those models would have moved it into clinical use. What failed was the data collection and the evaluation design, not the fitting. Recognising that class of failure early is the highest-leverage debugging skill there is, and it is the only one that ends with writing less code.

The best neural fix is sometimes to change the problem boundary rather than the network.

Key takeaways