Capstone: Design, Run, Diagnose, and Defend a Training Program

A commercial fleet operates electric delivery vehicles across climates and routes.

The first action should separate competing causes rather than change several knobs.

Every added component should defeat a simpler alternative under the same temporal protocol. On this kind of data the classical rung is not a courtesy rung.

Across a standard set of 45 tabular datasets, tree-based models such as XGBoost and Random Forests were still state of the art on medium-sized data of roughly 10,000 samples. That benchmark was published in 2022, and its authors released the hyperparameter search behind it: 20,000 compute hours of raw results per learner. The same conclusion came independently from Shwartz-Ziv and Armon, testing on the deep models’ own ground: “Our study shows that XGBoost outperforms these deep models across the datasets, including the datasets used in the papers that proposed the deep models.” XGBoost also required much less tuning, which is the second cost the fine-tuned rung has to absorb.

Thirty days of telemetry aggregated into engineered windows is exactly the terrain both benchmarks describe. The boosted baseline is the number the sequence model has to beat, not the number it is shown next to for form.

Define the scoring timestamp, prediction horizon, unit of analysis, label delay, exclusions, and what information exists at decision time. Separate training labels from the actions that maintenance teams took after earlier scores. The model contract also specifies the three output heads, acceptable latency, queue policy, and abstention behavior. Any change to these elements creates a new experimental question.

Leakage is the most thoroughly surveyed failure in applied machine learning. Kapoor and Narayanan surveyed it across disciplines in Patterns in 2023. They “find 17 fields where leakage has been found, collectively affecting 294 papers”. Their taxonomy names “eight types of leakage” running “from textbook errors to open research problems”. They also re-ran one case in detail: civil war prediction. Once the errors were corrected the complex models “do not perform substantively better than decades-old LR models”.

A scoring timestamp is what keeps that outcome away from this project.

The submission should let another team reproduce the reasoning, not merely rerun code. The first two items are the ones teams treat as classroom ceremony. They are also the two a regulator now asks for by name.

The FDA's final guidance of December 2024 tells manufacturers of AI-enabled device software what a predetermined change control plan has to contain: “This guidance recommends that a PCCP describe the planned AI-DSF modifications, the associated methodology to develop, validate, and implement those modifications, and an assessment of the impact of those modifications.” The incentive is exact. Changes made in line with the pre-authorised plan can be made without a new marketing submission.

A charter written before the runs, and a contract that records what was actually trained, buy the same thing here — the right to change the system later without relitigating the whole evaluation.

The training signal must preserve each task without hiding operational priorities. The evaluation row is where that discipline is won or lost, because on rare events the familiar summaries stay flat while decision quality collapses.

A worked example published in PLOS ONE in 2015 makes the gap visible. Accuracy is 0.6 on both the balanced and the imbalanced sample. Precision falls from 0.6 to 0.33. Saito and Rehmsmeier's reason for preferring precision-recall plots on skewed data is stated in the abstract: “We show here that the visual interpretability of ROC plots in the context of imbalanced datasets can be deceptive with respect to conclusions about the reliability of classification performance, owing to an intuitive but wrong interpretation of specificity.”

The formal version had already been proved in 2006 by Davis and Goadrich: a curve dominates in ROC space if and only if it dominates in PR space, and an algorithm optimising area under the ROC curve is not guaranteed to optimise area under the PR curve.

Confirmed failures in this fleet are rare and the queue is capped. So the evaluation suite is written as PR curves and recall at queue capacity, not as an average the training loss can quietly satisfy.

The numbers are starting hypotheses, not universal defaults.

A model card alone cannot carry the decision. Operational fit is the step that a strong-looking model most often fails.

The Epic Sepsis Model, a proprietary score deployed at hundreds of US hospitals, was validated externally across 38,455 hospitalizations at Michigan Medicine and reported in JAMA Internal Medicine in 2021. Its area under the ROC curve was 0.63 (95% CI, 0.62–0.64). The number that decided its usefulness was not the AUC: “The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue.” It missed 1,709 of 2,552 sepsis patients while alerting on nearly a fifth of everyone admitted.

An independent 2024 validation at two Harris County, Texas emergency departments found a sensitivity of 14.7% and a positive predictive value of 7.6% for sepsis within six hours.

Read that against this fleet's constraint. A scorer that flags 18% of the fleet cannot be run by depots that can inspect two percent, whatever its curves look like in the notebook.

Do not launch the final run until each earlier gate produces credible evidence.

A strong capstone may conclude that the temporal model does not justify its complexity, that labels are too selective, or that depot coverage prevents a fair launch. Those are successful findings when supported by disciplined evidence.

The most expensive version of that conclusion was reached by a board rather than by a researcher. On 2 November 2021 the board of Zillow Group decided to wind down Zillow Offers, the home-buying business whose pricing forecasts drove purchases. The company had bought homes above its own estimates of future selling prices. It recorded a $304.4 million inventory write-down for the quarter ended 30 September 2021, expected a further $240–265 million of charges in the fourth quarter, and cut roughly 25% of its workforce.

The reasoning given to shareholders was not about a model that could be tuned: “Ultimately, we determined that further scaling up Zillow Offers is too risky, too volatile to our earnings and operations, provides too little opportunity for return on equity, and serves too narrow a portion of our customers.” Risk, volatility, return and scope. That is the vocabulary a capstone memo should also be able to speak.

The final judgment should explain what was learned, what remains uncertain, and which next action has the highest expected value. Training excellence includes knowing when not to continue.