Capstone: Build and Defend an Unsupervised System

The project contains linked models and human decisions rather than one clustering call. A measurement layer, where sensor quality, time windows, hardware metadata, maintenance events and missingness define what an observation even is. A representation layer, where scaling, robust summaries, sequence features, embeddings and reduction determine which relations are visible at all. A structure layer, where regime clusters, soft membership, local anomalies and change points produce candidate evidence. A decision layer, where engineers inspect profiles, alerts, transitions and supporting time-series context. And an operations layer, where versioning, drift checks, queue budgets, fallback, retraining and retirement either sustain the system or quietly stop existing.

The operations layer is the one written last and read never. Google Flu Trends is the published record of what that costs. GFT was held up as the exemplary big-data nowcast, and it drifted against the ground truth it was built to predict for two years without anything stopping it. Lazer and colleagues put the count in Science in 2014, in the caption to the figure “GFT overestimation”: “From 21 August 2011 to 1 September 2013, GFT reported overly high flu prevalence 100 out of 108 weeks.” It overshot the 2011–2012 level by more than 50%. A hundred wrong weeks out of a hundred and eight is not a subtle drift. It is a monitored production system running two years past the point where any drift check should have fired.

The ending is the decision stage 6 of this capstone asks you to defend. On 20 August 2015 Google announced that it would stop running its own public Flu Trends website and would instead supply the signal to Columbia, Boston Children's Hospital/Harvard and the CDC. The owner chose retirement over repair. Repair was not impossible. Kandula and Shaman's independent reanalysis in 2019 noted that “several studies have previously reported large errors in GFT estimates of ILI in the US”, and concluded that “An 80% reduction in error during 2012/13, when GFT had large errors, shows that extreme failures of GFT could have been avoided.” The failure was avoidable and was not avoided. No layer of the system owned the job of noticing.

A cold-chain company operates twelve thousand refrigeration units across several climates and hardware generations. It wants stable operating regimes for fleet analysis and an alert queue for unusual behavior before costly failures. No authoritative regime labels exist. Failures are rare, maintenance changes the data, and sensors drift. Your task is to design an unsupervised system that helps engineers without inventing machine states or overwhelming the review team.

Both halves of that sentence are load-bearing, and the second half is the one students treat as a courtesy. The sections below replace the word “overwhelming” with published numbers: what an unmanaged alert queue measures at in an intensive care unit, what a regulator says a control-room queue is allowed to contain, and what an alerting model that passed its vendor's evidence measured at when two outside groups counted.

A polished notebook is not enough. Reviewers need artifacts they can rebuild the work from and argue with. The alert evaluation is the artifact most often waved through, and there is a careful measurement of what waving it through produces.

In a single month, the physiologic monitors in five adult ICUs at UCSF Medical Center produced 2,558,760 unique alarms. The month was March 2013. The study ran 31 days and covered 461 adult patients across 77 beds. Drew and colleagues reported the burden in PLOS ONE in two sentences: “There were 381,560 audible alarms for an audible alarm burden of 187/bed/day. 88.8% of the 12,671 annotated arrhythmia alarms were false positives.” The annotations were made by nurses, on real patients, on a system nobody had described as broken.

The rate is not an outlier. The Joint Commission, an independent accreditor, estimates that between 85 and 99 percent of alarm signals do not require clinical intervention. Its 2013 alert on medical device alarm safety reports 98 alarm-related events in its Sentinel Event database between January 2009 and June 2012. Eighty of them were deaths. An alert system evaluated on whether its alerts are interesting can reach 187 audible alarms per bed per day and still be reported as working. An alert system evaluated at operating capacity cannot. That is the difference the alert-evaluation artifact exists to produce, and it is why every item below has to arrive as a document with numbers in it rather than a section heading in a notebook.

Each stage produces an artifact and a stop condition before the next layer is added. Frame and freeze: write the decision, unit, cutoff, populations, costs, queue budget and prohibited claims. Audit measurement: test missingness, duplicates, sensor drift, maintenance leakage, exposure and hardware comparability. Establish baselines: robust summaries, rules, PCA, simple clustering, simple anomaly scoring. Add complexity conditionally: density, sequence, graph or nonlinear methods only where a measured gap remains. Validate jointly: stability, source-space checks, domain review, alert outcomes, slices, held-out assignment. Decide and govern: deploy, pilot, explore-only or stop, then define monitoring, fallback and retirement.

Stage 1 is where students write “queue budget” and leave it. It does not have to be left. The UK Health and Safety Executive publishes a quantified one for control-room operators: “Alarm rate targets: the long-term average alarm rate during normal operation should be no more than one every ten minutes; and no more than ten displayed in the first ten minutes following a major plant upset”. One alarm per operator per ten minutes in steady state. Ten in the first ten minutes of a crisis. Those are numbers a reviewer can hold your design against.

The budget is published, endorsed and largely unmet, which is the part worth internalising before you promise to meet it. Reising and Montgomery benchmarked 37 unique operator consoles against it for the Abnormal Situation Management Consortium in 2005. About one-third of the consoles met the normal-operation rate. Only 2 of the 37 came close to meeting the post-upset limit. Write your queue budget in stage 1 as a rate, and assume you are in the majority that misses it until stage 5 measures otherwise.

The capstone requires at least three credible designs, including a simple baseline.

A profile-and-rule baseline — robust fleet summaries plus engineering thresholds — is easy to explain and audit, provides a non-ML comparator, may miss interacting patterns, and can remain the production fallback. A partition-and-score pipeline — PCA or domain features, clustering, then within-regime anomaly scores — separates operating context from rarity and supports cluster-specific thresholds, but adds reducer and cluster lifecycle risk and requires stable held-out assignment. A sequence representation pipeline — learned window embeddings, density or neighbor models, transition monitoring — can capture temporal shape, needs careful training-boundary control, may be harder to interpret, and must justify its compute and maintenance cost. A hybrid evidence system — rules, regimes, local scores and human escalation combined — allows complementary failure detectors and supports explicit abstention, but requires a score fusion and alert policy and demands end-to-end evaluation.

Each design is a different alert queue. Compare them at the rate they would actually produce, not at the rate that makes the strongest one look best.

Your memo should state what the system can and cannot infer, which evidence supports each claim, how alternatives compared, and what would reverse the recommendation. Unresolved risks belong in it, not buried in an appendix. A strong capstone may recommend a narrow pilot, a simpler system, or no deployment. The quality lies in the trace from measurement to decision and in the honesty of its boundaries. Kleinberg does not read his impossibility result as a dead end. Relaxations of the three properties “expose some of the interesting (and unavoidable) trade-offs at work in well-studied clustering techniques”. He names single-linkage, sum-of-pairs, k-means and k-median among them. Naming the trade-off is a stronger defence than claiming none.

Match each product claim to independent evidence and a failure threshold. Regimes are repeatable: resampling, temporal holdouts, hardware slices and alternative preprocessing, reporting assignment agreement, split and merge behavior, unmatched new units and a stated minimum stability. Regimes are interpretable: medoids, profiles, blinded engineer review where possible, recorded disagreements, counterexamples, transition context, no causal names, documented unassigned cases. Alerts are useful: queue yield, lead time, repeated-alert burden, misses and intervention outcomes, evaluated at operating capacity, with delayed labels handled carefully and compared against rules. The system remains safe: drift, coverage, fallback, escalation, audit trail and retirement readiness, with input support monitored, every artifact versioned, rollback tested and operational owners assigned.

The “alerts are useful” row is the one with a documented body count of counterexamples, and the Epic Sepsis Model is the clearest. It was widely implemented and proprietary, and its evidence was the vendor's. Then Wong and colleagues validated it from outside, at Michigan Medicine, across 38,455 hospitalizations of 27,697 patients, and published in JAMA Internal Medicine in 2021. The area under the curve was 0.63, 95% CI 0.62–0.64. Their abstract states the operational consequence directly: “The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue.” Of 2,552 sepsis patients, 1,709 were missed. Of every hundred admissions, eighteen produced an alert.

Then a second group repeated the exercise, which is what independence means in this matrix. Ostermayer and colleagues evaluated version 1 of the same commercial model across 145,885 encounters at two county emergency departments in 2023, and published in JAMIA Open. Within a 6-hour window they measured sensitivity 14.7%, specificity 95.3% and positive predictive value 7.6%. A different research group, different institutions, a different journal, the same deployed model, the same conclusion. Specificity of 95.3% sounds like a passing grade until you multiply it by the number of encounters that are not sepsis. 7.6% is what reaches the clinician. Your matrix should be built so that a reviewer could produce those three numbers about your system without your help.

Try to break the interpretation before users build policy around it. Remove a dominant feature and check whether the taxonomy collapses when one measurement family changes. Shuffle nuisance metadata and test whether site, device or logging batch secretly drives the separation. Simulate new hardware and measure out-of-support assignments and alert inflation. Inject maintenance feedback and check whether interventions manufacture false regime changes or apparent model success. Ask for a simpler explanation and compare every complex finding against robust descriptive analysis.

Step 2 is the one with a documented cost. Duke University's chemotherapy-prediction method had a published “validation” behind it: Bonnefoi et al. (2007), a substudy of the European trial EORTC 10994/BIG 00-01, co-authored by Duke's Joseph R. Nevins. Baggerly and Coombes re-analysed the microarray data forensically in 2009. Examining high pairwise correlations produced three clear blocks. Plotting array run date by index produced the same three blocks. The third block, all of whose patients were treated with TET, had been run on a different scanner from the first two. Their section 4.1 is titled “Treatment is confounded with run date”, and its finding is one sentence: “There is perfect confounding of run date with treatment.” The structure everyone had been reading as biology was the processing metadata.

The check nobody ran was the cheap one, and it was not run in time. The Institute of Medicine's 2012 report on translational omics records that “Between October 2007 and April 2008, three cancer clinical trials were launched at Duke University” and that investigators later “concluded that the omics-based tests used in the three clinical trials were invalid. They terminated the clinical trials, and began the process of retracting the papers describing the development of the tests.” Patients had been assigned to treatment arms on the strength of three clusters that were three scanner blocks. Shuffling nuisance metadata takes an afternoon. The version of this failure that reaches a refrigeration fleet only costs money.

A defensible project anticipates challenges from engineering, operations, safety, and finance. The operations and exit answers are the ones students improvise, and there is now a published template for what a change plan is expected to contain.

The US Food and Drug Administration wrote it. Its final guidance on predetermined change control plans for AI-enabled device software functions was issued on 4 December 2024 and reissued on 18 August 2025. It is expressly a nonbinding recommendations document. The Federal Register notice announcing it names the three components: “This guidance recommends that a PCCP describe the planned AI-DSF modifications, the associated methodology to develop, validate, and implement those modifications, and an assessment of the impact of those modifications.”

Planned modifications, methodology, impact assessment. FDA reviews the PCCP as part of a marketing submission, so the modifications it describes can be implemented without an additional marketing submission for each one. That is the point worth carrying into a capstone that is not a medical device. A change plan written and reviewed in advance is what buys you the right to change the model later without renegotiating your evidence from scratch. “We will retrain when performance degrades” is not that plan.

Submit a concise decision memo supported by reproducible technical evidence. The memo names the chosen system, the rejected alternatives, the strongest supporting result, the strongest counterexample, and the unresolved risk that matters most.

Attach the dataset contract, experiment registry, validation matrix, representative examples, alert-capacity analysis, monitoring plan, and retirement checklist. A reviewer should be able to reconstruct both the result and the reasoning that limited the claim. The alert-capacity analysis is not the appendix. It is the one artifact that distinguishes a system a team can operate from one that will be measured, two years later, by someone outside the building.