Capstone: Build and Defend a Classical ML System

Every feature must be available by the nightly cutoff, and labels mature only after the outcome horizon closes.

The gap between those two clocks is where most of this lesson lives. A row frozen at 02:00 cannot know what a technician wrote at 09:00. A row scored tonight cannot be scored for calibration until fourteen days plus reporting delay have passed. Every anchored failure in the sections that follow is a case of one of those two clocks being read as if it were the other.

One row represents one active unit at one nightly cutoff. Repeated rows from the same unit are correlated — same compressor, same store, same duty cycle, same technician. Stores share technicians, climate, and maintenance policy.

The split should move forward in time and keep later rows from the same units out of earlier evaluation windows when leakage or near-duplicates would result. The alternative is to pool all the rows and split them at random. That alternative has been measured, and it is not a stylistic preference.

Shuffling rows makes a model look better than it is. Saeb and colleagues reviewed 62 sensor-based clinical-prediction studies in 2017: 28 had split record-wise, 34 subject-wise. Among the 47 whose reported accuracies could be extracted, the median classification error was 5.60% for the record-wise papers against 13.00% for the subject-wise ones (P<0.01, two-tailed Wilcoxon rank sum test) — more than twice as high. Their abstract says it in one line: “Using both a publicly available dataset and a simulation, we found that record-wise CV often massively overestimates the prediction accuracy of the algorithms.” The published literature of an entire field is split down the middle by a single line of splitting code. The half that shuffles reports itself roughly seven error points better.

The result reproduced. Tougui and colleagues ran it in 2021 with different data and different pipelines: “The record-wise division and the record-wise cross-validation techniques overestimated the performance of the classifiers and underestimated the classification error.” Record-wise splitting understated error by roughly 20% for their SVM pipeline and by more than 25% for their random-forest pipeline, relative to subject-wise splits.

A refrigeration unit is a subject and its nightly rows are records. Write the row contract so that the unit, not the row, is the thing that is split.

Every feature needs an availability rule, unit, missingness meaning, and owner. The availability rule is the load-bearing one. It is what stops a field being computed from information the nightly cutoff could not have had.

The selection plan should state what each challenger is expected to fix. It should also state what each one will get wrong about probabilities. The queue you are building is cut at a capacity, and a distorted probability moves the cut.

Some model families distort probabilities as a matter of course. Niculescu-Mizil and Caruana compared ten learning algorithms on eight classification problems in 2005, training on 4000 cases and calibrating on independent samples of 1000. Their abstract: “We show that maximum margin methods such as boosted trees and boosted stumps push probability mass away from 0 and 1 yielding a characteristic sigmoid shaped distortion in the predicted probabilities.” The RBF SVM among the four candidates is a max-margin method. Its raw decision values are not probabilities, and they will carry that sigmoid. Random forests, neural nets and bagged trees were the best calibrated before any calibration step at all.

Which calibrator to fit is then a question of how much matured label you have, and the same paper answers it with a threshold: “When the calibration set is small (less than about 200-1000 cases), Platt Scaling outperforms Isotonic Regression with all nine learning methods”, while “When there are 1000 or more points in the calibration set, Isotonic Regression always yields performance as good as, or better than, Platt Scaling.”

Unscheduled service events are rare. Count the matured positives available inside each rolling fold before choosing. That count, not preference, decides between Platt scaling and isotonic regression for this pipeline.

Protect the final evidence. Allow model development.

Two of the six steps below are the ones the anchored failures turn on. Step 2, rolling-origin evaluation, is where the subject-wise discipline of the row contract has to be enforced: a fold that lets later rows of a unit appear beside its earlier ones is the record-wise split that produced 5.60% instead of 13.00%. Step 3, fitting every transformation inside folds, includes the calibrator. Platt scaling or isotonic regression is fitted on the fold's own matured labels, never on the evaluation window it will be scored against.

The company can inspect only a fixed number of units each day, so the model ranks eligible units. Units with severe telemetry gaps are deferred to a data-quality workflow rather than scored normally.

The threshold is set by visit capacity and expected service avoidance, then checked for site and unit-model disparities. That last clause is the whole of the section. There is a documented case of what it catches.

A commercial risk-prediction algorithm ran precisely this design over, by industry estimates, roughly 200 million people in the United States each year. “Patients above the 97th percentile are automatically identified for enrollment in the program” — the high-risk care management program. Patients above the 55th percentile were referred to their physician. Obermeyer and colleagues dissected the algorithm in Science on 25 October 2019, across 6079 self-identified Black patients and 43,539 White patients, 11,929 and 88,080 patient-years.

At that 97th-percentile cutoff, Black patients had 26.3% more chronic illnesses than White patients — 4.8 against 3.8 distinct conditions, P<0.001. Two patients with the same score did not have the same health. The queue was ranked correctly on what the model had been trained to predict. It was ranked wrongly on the thing the program existed to do. From the abstract: “Remedying this disparity would increase the percentage of Black patients receiving additional help from 17.7 to 46.5%.” Same capacity, same cutoff rule, a different set of people in the queue.

The consequence was not academic. On the day of publication the New York State Department of Financial Services and Department of Health wrote jointly to UnitedHealth Group, demanding the company investigate or cease using the algorithm, Impact Pro.

So: for every candidate threshold, report the visit budget it consumes and who is in the queue at it — by store, by unit model, by climate zone, by telemetry quality. A cutoff is a policy. Its subgroup composition is part of the specification, not a follow-up study.

Technicians need concise evidence such as trend, alarm recency, and comparable past units, plus clear uncertainty and missing-data flags. Model owners need global reliance, calibration, stability, and drift diagnostics.

Neither audience should receive causal claims about a feature without intervention evidence. There is a record of what that rule protects against. A rule-based learner trained on real pneumonia data found a pattern that Caruana and colleagues reported in 2015: “On one of the pneumonia datasets, the rule-based system learned the rule “HasAsthama(x) ⇒ LowerRisk(x)”, i.e., that patients with pneumonia who have a history of asthma have lower risk of dying from pneumonia than the general population.”

The pattern was true in the data. It was true because asthmatic pneumonia patients were admitted directly to the ICU. Read as an association it is correct. Read as advice it kills people.

Deployment closes the loop, and Zachary Lipton, at Carnegie Mellon, named the mechanism: “asthma was predictive of lower risk of death. This owed to the more aggressive treatment these patients received. But if the model were deployed to aid in triage, these patients would then receive less aggressive treatment, invalidating the model.”

What the same team did next is the part worth copying. The multitask neural nets were the most accurate models they had — AUC 0.86 against 0.77 for logistic regression on one dataset — and they were judged too risky to field, because a model nobody can read cannot be searched for the other rules of that kind. Logistic regression was deployed instead. Accuracy lost to auditability, deliberately, in writing.

So the technician-facing card names the evidence and stops. It does not say that a feature causes the failure, and it never implies that removing the feature would remove the risk. And the model-owner package must be readable enough that an asthma-shaped rule could actually be found in it.

A nightly batch job creates features, validates coverage, scores eligible units, writes a versioned queue, and records the decision. Monitoring covers data freshness, score distribution, calibration after label maturity, visit yield, missed events, and queue capacity.

Rollback restores the previous pipeline and threshold. A rule baseline remains available if data-quality gates fail.

Google engineers put the proportions on a diagram in 2015. Their central figure is captioned “Only a small fraction of real-world ML systems is composed of the ML code, as shown by the small black box in the middle. The required surrounding infrastructure is vast and complex.” In the text: “only a tiny fraction of the code in many ML systems is actually devoted to learning or prediction”. Their argument is that the resulting debt is hard to detect because it lives at the system level rather than the code level — configuration, data dependencies, undeclared consumers, hidden feedback loops, boundary erosion. It is paid late: “it is common to incur massive ongoing maintenance costs in real-world ML systems”. The runbook above is not overhead around the model. It is most of the system.

What happens when that surrounding infrastructure is not monitored has a count. Google Flu Trends was a deployed nightly nowcast, and Lazer and colleagues examined it in Science in 2014. Their figure caption reads: “From 21 August 2011 to 1 September 2013, GFT reported overly high flu prevalence 100 out of 108 weeks.” It overshot the 2011–2012 season by more than 50%. At its worst it predicted more than double the CDC's proportion of doctor visits for influenza-like illness. The authors name two causes: “big data hubris” — 50 million search terms fitted to 1152 data points — and “algorithm dynamics”, changes to the surrounding service that silently changed the model's inputs. For 108 weeks a system that was wrong in the same direction almost every week kept publishing.

Google stopped publishing its own current estimates in August 2015: “Instead of maintaining our own website going forward, we're now going to empower institutions who specialize in infectious disease research to use the data to build their own models”, handing the raw signals to CDC and university partners. Note the failure mode precisely. Nothing in the model broke. The inputs were changed by the product around it. That is why the monitoring list above starts with data freshness and score distribution rather than with accuracy — accuracy arrives fourteen days late, and by then the queue has already been dispatched fourteen times.

The artifacts should let another team reproduce and challenge the system.

The last of them, the production runbook, is not a house convention. Its stages are an international standard. ISO 13374 covers condition monitoring and diagnostics of machines, and MIMOSA implements it as the Open Systems Architecture for Condition-Based Maintenance — OSA-CBM, current release 3.3.1, dated 29 June 2010. MIMOSA states the relationship on its own specification page: “ISO-13374, Condition Monitory and Diagnostics of Machines, defines the six blocks of functionality in a condition monitoring system, as well as the general inputs and outputs of those six blocks.” (The typo “Monitory” is in the original.)

The blocks themselves are listed by Hernandez and colleagues, at the National Institute of Standards and Technology, in an ASME Standards and Certification white paper: “The cited functional reference model is based on ISO 13374 and the Open Systems Architecture for Condition-Based Maintenance (OSA-CBM) specification [10]. The functional elements include (bottom to top): data acquisition, data manipulation, state detection, health assessment, prognostic assessment, and advisory generation.”

Read that sequence against the nightly job: ingest, build features, detect state, assess health, forecast the fourteen-day horizon, issue an advisory. The standard ends where this capstone ends — at advisory generation, not at a score. And the blocks have defined inputs and outputs so that vendors compete block by block rather than on whole systems. That is the same reason your submission is six separable artifacts rather than one notebook.