Capstone: Design an Honest Learning Experiment

A home-repair company wants to reduce repeat visits. Before an appointment it may send a specialist, reserve extra time, or load uncommon parts, and each of those interventions costs enough that not every visit can receive them.

The capstone is a decision dossier, not only a trained model.

Two of these eight artifacts are not this course's invention. Three medicines regulators already require them. In October 2021 the US Food and Drug Administration, Health Canada and the UK Medicines and Healthcare products Regulatory Agency jointly published “Good Machine Learning Practice for Medical Device Development: Guiding Principles”, 10 numbered principles in all. Artifacts 2 and 4 — the evidence design and the split — are principle 4, “Training Data Sets Are Independent of Test Sets”. It reads: “Training and test datasets are selected and maintained to be appropriately independent of one another. All potential sources of dependence, including patient, data acquisition, and site factors, are considered and addressed to assure independence.” Principle 8 requires testing under clinically relevant conditions, independently of the training set.

Artifact 7 is principle 10. Health Canada's publication of the same jointly issued principles states it this way: “Deployed models have the capability to be monitored in 'real world' use with a focus on maintained or improved safety and performance.” Dataset drift and the risks of re-training are named there as things a deployed model is controlled for. Not things a team discovers later.

Three regulators wrote down the deployment boundary and the monitoring plan as requirements. A dossier that omits them is not merely thinner than this course would like. It is missing what a regulator already asks for.

The first deliverable should make the product mechanism visible.

One example represents one scheduled visit at 18:00 on the previous day. Inputs must be available by that cutoff. Later technician edits, final parts used, completion status, and follow-up notes are excluded from features.

The target might be “avoidable repeat visit within seven days,” but the team must define avoidable, handle cancellations, and allow the outcome window to mature. A second human-reviewed target may be needed to separate genuine complexity from preparation failure.

The reason to spend a week on that definition is that the substitution becomes invisible the moment the model runs. One commercial risk-prediction algorithm in US health care was applied, by industry estimates, to roughly 200 million people each year. Its designers did what the home-repair team will be tempted to do. They used the outcome the records made easy to compute: cost. Ziad Obermeyer and three colleagues dissected the algorithm in Science in 2019, using 6,079 patients who self-identified as Black and 43,539 who self-identified as White. Their abstract names the mechanism: “The bias arises because the algorithm predicts health care costs rather than illness, but unequal access to care means that we spend less money caring for Black patients than for White patients.”

The size of that substitution is measurable, and it is not small. Correcting the label would have raised the share of Black patients flagged for extra help from 17.7% to 46.5%. The paper appeared on 25 October 2019. That same day the New York State Department of Financial Services and Department of Health wrote jointly to UnitedHealth Group, demanding that the company “immediately investigate these reports and demonstrate that this algorithm is not racially discriminatory or to cease using Impact Pro”. Choosing a convenient proxy is not a modelling shortcut. It is a product decision with a regulator on the other end of it, and “avoidable repeat visit” is exactly that kind of choice.

The capstone should state what the system must beat and where it must transfer.

A score that skips this step can sit in production for years before anyone measures it against the workflow it displaced. Epic's proprietary Sepsis Model was validated from outside the company at Michigan Medicine, on 27,697 patients and 38,455 hospitalizations between 6 December 2018 and 20 October 2019. Wong and colleagues published the result in JAMA Internal Medicine on 21 June 2021. Hospitalization-level area under the curve was 0.63 (95% CI, 0.62-0.64). Of 2,552 patients with sepsis, the model failed to identify 1,709 — 67% of them. Their conclusion: “The widespread adoption of the ESM despite its poor performance raises fundamental concerns about sepsis management on a national level.”

Read the comparison in that sentence carefully. It is not against a rival model. It is against the adoption — the clinical practice the score was already embedded in, which nobody had measured it against. That is the historical baseline below, and it is the one most often left unbuilt. Reconstructing what dispatchers already do is tedious and produces no new technology.

Extra preparation capacity is limited. Evaluate precision and completion benefit among the top-ranked visits, recall of avoidable repeats, cost per prevented repeat, and workload by region and appliance type.

Also measure calibration if the score informs expected resource use, and report confidence intervals using visits or customers as the resampling unit — never duplicated records, which will make the interval look tighter than it is.

The same Epic Sepsis Model shows what a metric looks like once it is stated at the operating point rather than in the aggregate. At the score threshold of 6 it generated alerts on 6,971 of 38,455 hospitalizations — 18% of admissions — with sensitivity 33%, specificity 83% and positive predictive value 12%. The 18% is the alert budget the hospital actually paid. The 12% is what the clinician sees when an alert fires. Neither number is visible in an AUC of 0.63.

An independent group later validated version 1 of the same model on 145,885 emergency-department encounters at two county hospitals during 2023. Within a six-hour window they reported sensitivity 14.7%, specificity 95.3%, PPV 7.6% and NPV 97.7%, and concluded that “The ESPMv1 provides suboptimal diagnostic characteristics for undifferentiated patients in a county ED”. A specificity of 95.3% beside a PPV of 7.6% is the arithmetic every capacity-limited ranking has to publish. The rate reads as excellent, and the queue the specialist actually works through is still mostly wrong. Your evaluation table should force that pair of numbers into the same row.

The dossier should include likely failure modes before results are known.

Google Flu Trends is the counted version of most of them. The original system fitted 50 million search terms to 1,152 data points, and it missed the non-seasonal 2009 influenza A-H1N1 pandemic completely. Lazer and Kennedy, with two colleagues, named the shortcut it had learned instead, in Science in 2014: “In short, the initial version of GFT was part flu detector, part winter detector.” After the 2009 rebuild the system “has missed high for 100 out of 108 weeks starting with August 2011”. A simple projection from CDC data, lagged by two to three weeks, did better — a baseline of the kind this capstone requires, quietly outperforming the model. An independent group had reported both failures already, in PLOS Computational Biology in October 2013.

One hundred of 108 weeks is what an unmonitored shortcut costs after the launch metrics were good. Every row below is a way of becoming a winter detector.

Each stage answers a question before the project earns more complexity. A stage that fails is the plan working, not the plan collapsing: the whole point of the sequence is that the cheap question is asked first and the expensive commitment is made last.

Use a concise document, tables, and diagrams rather than a long narrative.

The six artifacts below are a working subset of a standard that already exists. TRIPOD+AI is a reporting standard for prediction-model studies, published in the BMJ on 16 April 2024. It contains 27 main checklist items and 52 subitems, covering title, abstract, introduction, methods, open science practices, patient and public involvement, results and discussion. It is not one group's preference. It came out of a two-round Delphi survey completed by 170 and then 200 participants, and a 28-participant consensus meeting held on 5 July 2022. Collins and colleagues write in the abstract: “TRIPOD+AI provides harmonised guidance for reporting prediction model studies, irrespective of whether regression modelling or machine learning methods have been used.”

Map the one-page brief, the dataset specification and the evaluation table onto its numbered items, and the blanks in a dossier appear within an hour. That is faster than any reviewer will find them, and much faster than a pilot will. The risk register has an external anchor of its own. Principle 10 of the ten guiding principles issued jointly by the FDA, Health Canada and the MHRA requires that deployed models be monitored in real-world use. Re-training risks such as dataset drift are named there as things to control, not assume.

A strong submission does not promise that the model will work. It shows that the team knows which claim it is testing, which evidence could support it, which failure modes could invalidate it, and which alternatives remain available.

The hallmark of learning maturity is not confidence before the experiment. It is a design that can produce a trustworthy yes, no, or “only under these conditions.”

The expensive version of a stop decision is the one taken late. On 2 November 2021 Zillow Group announced that it would wind down Zillow Offers, its algorithmic home-buying business, and cut approximately 25% of its workforce. It recorded a write-down of inventory of approximately $304 million in the third quarter, because it had bought homes at prices above its own later estimates of future selling prices. It expected a further $240 million to $265 million of losses in the fourth. In the results announcement filed with the SEC that day, co-founder and CEO Rich Barton said: “We've determined the unpredictability in forecasting home prices far exceeds what we anticipated and continuing to scale Zillow Offers would result in too much earnings and balance-sheet volatility”

That recommendation was correct. It arrived after the houses. The whole apparatus of this capstone — the reconstructed baseline, the pre-mortem, the staged roadmap, the red team — exists so that the same sentence can be written at stage two. There it costs a baseline reconstruction, rather than $304 million of inventory and a quarter of a company's staff.