Mathematical Foundations Capstone: Audit a Learning System

A manufacturer wants to predict whether a machine will fail within seven days. Each machine produces irregular sensor streams, maintenance records, operating loads, and site metadata. The proposed system aggregates recent measurements into a vector, standardizes features, and trains a neural classifier with cross-entropy. When predicted risk exceeds 0.7, a technician is sent to inspect the machine. Failures are rare, maintenance changes later observations, labels arrive with delay, and several sensors share calibration drift. The device it runs on uses mixed precision and must respond within fifty milliseconds. Your task is not to choose a fashionable model. It is to audit whether the mathematical claims connecting data, score, threshold, and action are defensible.

Nothing in that chain is hypothetical, and nothing in this lesson is a cautionary parable. Every checkpoint below has already failed somewhere in public and been written down. A sepsis score externally validated on 38,455 hospitalizations. A units mismatch in one ground software file that moved a spacecraft's first periapsis from a planned 226 km to an estimated 57 km. A 110-layer network whose accuracy improved while its probabilities got worse. A pneumonia model whose area under the curve fell from 0.931 to 0.815 the moment it met a new hospital. A gradient tensor in which 67% of the values were already zero. A police risk model that had scored 399,412 individuals before it was decommissioned on 1 November 2019. The audit is the discipline that catches those before deployment rather than after.

A single number can support very different operational claims. The distance between those claims is measurable rather than philosophical.

Accuracy rose and calibration fell, and somebody measured exactly that. Guo and Pleiss, with two co-authors, stated the finding directly in 2017: “While neural networks today are undoubtedly more accurate than they were a decade ago, we discover with great surprise that modern neural networks are no longer well-calibrated.” Their first figure sets a 5-layer LeNet beside a 110-layer ResNet on CIFAR-100, with the reliability diagram for each. Error of 44.9 for the LeNet against 30.6 for the ResNet. The substantially more accurate network is the substantially worse calibrated one, its average confidence sitting well above its accuracy. The repair was small. Temperature scaling, a single-parameter variant of Platt scaling, was usually enough to recalibrate it. Accuracy and calibration are separate properties of the same fitted model, and one can be fixed without retraining for the other. The three columns below rest on that.

The relationship does not even run reliably in one direction. Minderer and colleagues re-ran the question on newer architectures in 2021 and reported that “the most recent models, notably those not using convolutions, are among the best calibrated”. So an auditor may not infer calibration from an accuracy figure. Nor from the model's generation or architecture family. It has to be measured on the model in hand, under deployment conditions.

Apply that to the proposal. If the 0.7 cut is to be read as a 70% chance of failure within seven days, someone must produce a calibration measurement on the deployment population, with the population and time window named. If the score is only used to order machines against inspection capacity, calibration is not automatically required. Ranking quality must still be validated, and the scale may shift at every retraining. If it is an engineered alarm with no probabilistic semantics, it can still be useful. But the document must say so, and stop reporting it as a risk probability.

Define the prediction timestamp, seven-day outcome window, sensor availability cutoff, and maintenance status. Without those boundaries, future leakage can enter through aggregated features. Record sensor units and calibration versions. Standardization statistics must be fitted on allowed training periods and evaluated for site-specific shifts. Irregular streams require a stated aggregation rule. Mean, maximum, slope, and missingness each preserve different information and create different invariances.

The instruction to record units reads as clerical hygiene until it is priced. The Mars Climate Orbiter was lost in 1999. Its mishap investigation board gives the reason: “The MCO MIB has determined that the root cause for the loss of the MCO spacecraft was the failure to use metric units in the coding of a ground software file, “Small Forces,” used in trajectory models.” Thruster performance data was written to the SM_FORCES file in English rather than metric units. The navigation software therefore underestimated the trajectory effect of the thruster firings by a factor of 4.45 — one pound force is 4.45 newtons. The first periapsis was meant to be 226 km. Post-hoc navigation estimates put it at 57 km. The report records that “The minimum periapse altitude considered survivable by MCO is 80 km”. The U.S. General Accounting Office, an independent oversight body, later described the same events: “A NASA mishap investigation team determined that the loss was due to the mistaken use of English rather than metric units in the navigation software”. It also recorded that the orbiter “cost $75 million to develop” and was launched December 11, 1998.

Nothing there was a modeling failure. The trajectory mathematics was right, the software ran, and every number it produced was internally consistent. One interface carried a quantity whose unit was assumed rather than recorded, and a constant factor of 4.45 propagated silently through everything downstream. A sensor table with units and calibration versions per feature is the artifact that makes that failure visible before the model is fitted rather than after the spacecraft is gone. It is the same artifact that catches a pressure channel recalibrated mid-year, or a load feature whose vendor changed scale between sites.

Produce a document another technical team can challenge and reproduce. Each row of it should be checkable by someone who does not trust you. A claim, the quantity that would support it, the assumption that quantity rests on, and the measurement that was actually taken, with its population and its denominator written next to it. Where a measurement was never taken, the row says so rather than borrowing confidence from a neighbouring row.

Each checkpoint can invalidate a claim even when the others are correct. Each has a documented failure attached to it rather than a hypothetical one. Checkpoint 1 is the SM_FORCES file: a representation whose units were assumed. Checkpoint 4 is the gap between an error of 30.6 and a confidence level that does not match it. Checkpoints 5 and 6 are chest radiographs. Zech and colleagues found a network that identified the source hospital system for 22,050 of 22,062 NIH radiographs and 8,386 of 8,388 Mount Sinai radiographs, and an area under the curve of 0.931 on internal test data that fell to 0.815 at a hospital system the model had not been trained on. DeGrave and colleagues then closed the obvious escape route, adding that “evaluation of a model on external data is insufficient to ensure AI systems rely on medically relevant pathology”. Checkpoint 7 is an activation-gradient histogram in which “67% of values are zero” before the intended arithmetic has a chance to run.

Mathematical maturity in ML is not being able to recite more symbols. It is being able to notice which symbol is missing, which assumption has been smuggled in, and which conclusion exceeds the evidence. Linear algebra tells you what the representation can express. Calculus and optimization tell you how the fitted parameters respond. Probability and statistics tell you what uncertainty statements mean. Information theory and learning theory connect coding, loss, capacity, and generalization. Numerical analysis tells you whether the calculation you intended survived the code that ran it. A trustworthy practitioner moves fluently between these layers while keeping a formal result, an empirical observation, and an operational decision apart.

In practice the maturity shows up as small, unglamorous refusals. It is the difference between reporting an area under the curve of 0.63 and reporting what the 18% alert rate and the 7% yield are each a proportion of. It is declining to call a coefficient of 4.45 a rounding detail. It is measuring calibration on the model in hand instead of inferring it from an error rate of 30.6.

These checks connect abstract mathematics to testable artifacts. Each of them has a published failure behind it that shows what the check is for.

The review should make every conditional claim traceable to a test or owner. Each question below has a version that was answered wrongly in public.

Different forms of evidence answer different questions and should not be substituted for one another. The substitution has been audited at scale, twice, by different teams using different appraisal instruments.

One review went through the machine learning literature on COVID-19 chest radiographs and CT. Its search “identified 2,212 studies, of which 415 were included after initial screening and, after quality screening, 62 studies were included in this systematic review”. Roberts and colleagues published the result in Nature Machine Intelligence in 2021. The verdict on those 62: “Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases.” Not some. None.

A living systematic review in The BMJ reached the same place independently, with a different team, a different modality and a different appraisal tool. Its original version, of 7 April 2020, screened 2,696 titles and appraised 27 studies describing 31 prediction models, all rated at high risk of bias. The first update of the same review, online by 1 June 2020, reports that “4909 titles were screened, and 51 studies describing 66 prediction models were included”, all at high or unclear risk of bias. Wynants and colleagues state the conclusion plainly: “we do not recommend any of these reported prediction models to be used in current practice”.

Every one of those models had published performance numbers. That is precisely the point of the four columns below. An equation shows what follows from definitions. A theorem gives a conditional guarantee under stated hypotheses. A simulation tests behaviour under a chosen synthetic process. None of the three is deployment evidence. Two independent appraisals, of 62 and of 66 models, found zero fit for clinical use — and they found it by asking what the reported numbers were measurements of.

The final deliverable should not be a collection of formulas. It should state which claims are supported, which assumptions remain untested, and which failure modes require monitoring or fallback. A strong memo distinguishes a mathematical guarantee from an empirical estimate and from an operational judgment. It also names conditions that would invalidate approval.

Mathematical maturity appears when a reviewer can recommend a simpler model, a different experiment, or that nothing be deployed at all. Chicago is what that last recommendation looks like when it arrives late. The Strategic Subject List assigned a risk score to 399,412 individuals as of July 2018. Its successor, the Crime and Victimization Risk Model, assigned a risk tier to 313,513 as of March 2019, on $3.8 million in federal grants. A score, a threshold and an intervention — the same three objects as the proposal in this capstone. RAND evaluated the 2013 pilot and published in 2016. Saunders and colleagues examined the 426 people on that list against a matched comparison group, with an ARIMA and propensity-score design, and found that “Individuals on the SSL are not more or less likely to become a victim of a homicide or shooting than the comparison group, and this is further supported by city-level analysis.” On the outcome the list existed to change, the effect was null. On a different outcome it was not: “The treated group is more likely to be arrested for a shooting”.

The City of Chicago Office of Inspector General recorded the rest, in an advisory of 23 January 2020. It found that “CPD's risk scores and tiers were unreliable as they were not updated regularly and the quality of the data was, in certain respects, poor”. It set down the counts: “As of July 2018, 399,412 individuals had an SSL risk score and as of March 2019, 313,513 individuals had a CVRM risk tier”. And it found that neither CPD nor RAND ever evaluated versions 2 through 5 of the models. The one published evaluation covered a version that had been superseded four times. CPD decommissioned the program on 1 November 2019. Every element of the audit in this lesson was available years earlier: an outcome definition, a sampling unit, a comparison group, an evidence table with one row per model version. A memo that had written them down would have said not ready. It would have said it before 399,412 people had scores.