Testing Machine Learning Systems

A service can have excellent code coverage and still hand another component a number in the wrong unit. The Mars Climate Orbiter was lost on 23 September 1999. The Mishap Investigation Board's Phase I Report, published on 10 November 1999, put the cause in one sentence: “The MCO MIB has determined that the root cause for the loss of the MCO spacecraft was the failure to use metric units in the coding of a ground software file, “Small Forces,” used in trajectory models.” Thruster impulse data was delivered in pound-seconds. The Software Interface Specification required newton-seconds. The specification existed and named the unit; nothing about it was ambiguous. The navigation software underestimated the trajectory effect by the conversion factor 4.45, and the spacecraft arrived roughly 170 km lower than planned. The US General Accounting Office described the same units error in 2002.

Every function in that chain could have passed its own unit test. What no test asserted was the agreement between two components about what a number meant. ML systems inherit that failure mode and add several of their own, because a learned component also has a data contract, an output contract and a distribution it was fitted on. They need ordinary software testing plus tests for distributions, invariants, statistical tolerance, workflow, and recovery. Exact expected predictions are sometimes useful. They are not the whole strategy.

Fast local tests provide breadth. Fewer integrated tests cover the seams where operational failures hide.

The layers are not a matter of taste, and someone has already counted them. Breck and four colleagues at Google published a rubric in 2017, and its abstract promises exactly that: “we present 28 specific tests and monitoring needs”. They come grouped in four areas — features and data, model development, ML infrastructure, and monitoring. The scoring is deliberately unforgiving. Half a point for a test performed manually, a full point for one that runs automatically. The final score is the minimum across the four sections, so strength in one area cannot pay for a weak one. The authors met with 36 teams from across Google, and reported what they found: “in a survey of several dozen teams at Google, none of these tests was implemented by more than 80% of teams”. A pyramid nobody fills in completely is the normal case, not the failing one. Naming the layers tells you which one you have chosen to leave empty.

Each method detects a different class of failure.

Metamorphic testing is not a fallback for when labels are missing; it finds things labels do not. In 2018 a tool called DeepTest generated inputs with nine realistic image transformations and built its oracle out of metamorphic relations rather than labels. It ran at three top-performing models from the Udacity self-driving car challenge. The results section is blunt: “In total, DeepTest detects 6339 erroneous behaviors across all three models.” Manual review judged 130 of those cases false positives. The models had already scored well on the challenge. That is the only reason the number is interesting.

Statistical tests are where a vendor's held-out metric stops counting as evidence about your population. Wong and colleagues took the proprietary Epic Sepsis Model and validated it at Michigan Medicine, on 27,697 patients and 38,455 hospitalizations. They measured an AUC of 0.63 (95% CI 0.62–0.64), against the 0.76–0.83 range the developer had reported. The results section of their abstract gives the operational shape of that gap: “The ESM did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue.” Their conclusion, in JAMA Internal Medicine in 2021, was that “the ESM has poor discrimination and calibration in predicting the onset of sepsis”. A separate team then evaluated the same vendor model across 145,885 encounters in two county emergency departments during 2023, and measured 14.7% sensitivity, 95.3% specificity and 7.6% PPV. Two independent measurements, two settings, the same verdict about a number that had shipped as a specification.

CrowdStrike published its own root cause analysis on 6 August 2024, and it describes a test suite that was green for a reason unrelated to correctness. The “What Happened” section states: “In summary, it was the confluence of these issues that resulted in a system crash: the mismatch between the 21 inputs validated by the Content Validator versus the 20 provided to the Content Interpreter, the latent out-of-bounds read issue in the Content Interpreter, and the lack of a specific test for non-wildcard matching criteria in the 21st field.”

Model outputs vary across hardware, seeds and library versions while remaining acceptable, and that variation has been measured rather than assumed. Sara Hooker and three colleagues trained ResNet-50 on ImageNet ten times, in 2022, and reported top-1 accuracy of 76.58% ± 0.10. Implementation-level noise alone produced 14.68% predictive churn between runs — roughly one prediction in seven flipping between models that a summary table calls identical. Their abstract names the asymmetry directly: “While top-line metrics such as top-1 accuracy are not noticeably impacted, model performance on certain parts of the data distribution is far more sensitive to the introduction of randomness.” Pham and colleagues found the same effect independently in 2020. Implementation-level factors alone moved accuracy by up to 2.9%, per-class accuracy by up to 52.4% and training time by up to 145.3% across identical training runs. Of the 901 researchers and practitioners they surveyed, 83.8% were unaware of or unsure about that variance.

A test that pins one exact float is asserting a number that moves for reasons the production contract does not care about, while saying nothing about the per-class collapse the same noise can hide. Better targets include schema, monotonicity, invariance, bounded sensitivity, calibration tolerance, slice behavior, resource limits, and fallback semantics. The property should follow from the production contract. A tolerance of ±0.10 on an aggregate is not a tolerance on the slice that matters.

Translate each clause into the cheapest test that can detect a meaningful violation.

Three regulators have already published a version of this exercise. In October 2021 the US FDA, Health Canada and the UK MHRA jointly issued 10 guiding principles for Good Machine Learning Practice for Medical Device Development. Principle 4 is “Training Data Sets Are Independent of Test Sets”. Principle 8 states: “Testing Demonstrates Device Performance During Clinically Relevant Conditions: Statistically sound test plans are developed and executed to generate clinically relevant device performance information independently of the training data set.” Principle 10 carries the obligation past release, to monitoring deployed models.

Read as a contract-to-test mapping, that is the five steps below, with an issuing agency behind each one. Independence of the test set is a clause you can violate silently. Clinically relevant conditions are the deployment population, not the convenient one. Principle 10 is the admission that release testing expires. A team outside medical devices is not exempt from the logic — only from the audit.

Tests are easiest to maintain when their names state the property and the consequence. “Prediction changed” is weaker than “threshold decision changed for a protected fixture under the approved policy.”

Review the suite after incidents, and write the general mechanism rather than the escaped example. CrowdStrike's own mitigation list is the template. Not a test for the Template Instance that shipped on 19 July 2024, but tests with non-wildcard criteria for every field, runtime array bounds checks, and staged deployment — the contract, the failure mode and the blast radius, each given its own test. A regression test that only reproduces the one input which escaped is a test of history.