How machines learn
A Diagnosis-First Strategy for Improving Models
Integrate learning curves, baselines, error analysis, data audits, and experiment discipline into a practical decision framework for model improvement.
By the end you can
- Classify a learning project's dominant bottleneck before proposing a remedy
- Choose interventions that target framing, data, representation, optimization, capacity, evaluation, or workflow
- Estimate expected value, cost, and risk for competing improvements
- Create an iterative plan with explicit hypotheses, checkpoints, and stopping rules
Example
The team that changed the architecture before opening the data
A defect classifier missed scratches on reflective metal. The team proposed a deeper network, but error review showed that inspectors labeled reflective scratches inconsistently and the camera overexposed that product line. The architecture was the one layer nobody had evidence against.
The repair sequence that followed is the whole method in miniature: read the errors, name the layer they implicate, change that layer, and hold everything else fixed.
- A larger model could fit the inconsistent labels more closely without learning a stable concept.
- Camera correction addressed the missing visual evidence.
- A labeling workshop clarified which reflections counted as defects.
- A targeted collection added difficult reflective examples.
- The original architecture improved substantially after data and process repair.
- Architecture research became a lower-priority experiment rather than the default response.
Case
Data cascades: 92% prevalence, reported by the practitioners themselves
That ordering — evidence first, architecture last — is the exception rather than the rule.
A data cascade is a compounding downstream failure set off by an upstream data problem. In 2021 six researchers went looking for them, reporting “data practices in high-stakes AI, from interviews with 53 AI practitioners in India, East and West African countries, and USA”. The cascades turned out to be “pervasive (92% prevalence), invisible, delayed, but often avoidable”. The paper's title is the diagnosis: “Everyone wants to do the model work, not the data work”.
Invisible and delayed are the operative words. A cascade surfaces as a model that will not improve, months after the decision that caused it. That is precisely the moment at which a team reaches for a bigger model.
Visual
Seven places where progress can be blocked, and the one that stopped eleven clinics
Most projects have several weaknesses. One or two usually dominate the next decision.
The last node is the one teams least expect to be the answer. Google Health built a diabetic-retinopathy screening system that was accurate in the laboratory, and in 2020 researchers followed it into eleven clinics in Thailand. The model was not the failure. Its own image-quality threshold rejected more than a fifth of the retinal images nurses captured — about 21%, largely because of poor clinic lighting. Cloud upload over slow clinic internet turned an instant result into a queue. Their abstract names the mismatch: “We find tensions between the model's thresholds for data quality, and the quality of data that arise from an imperfect, resource-constrained environment.”
No change to the network would have moved that deployment. The bottleneck sat in the Workflow row. It was found only by watching the system work in the clinics.
Framing
The target, unit, timing, action, or success criterion is wrong or ambiguous.
Evidence
Sampling, labels, coverage, leakage, freshness, or data quality limits the claim.
Representation
Inputs omit or distort the stable signal.
Optimization
The fitting procedure fails to use the available model family.
Capacity and regularization
The model is too restricted or too responsive to noise.
Evaluation
Metrics, threshold, split, uncertainty, or slices do not support the decision.
Workflow
The intervention, review, latency, cost, or fallback prevents product value.
Comparison
0.931 inside the hospital, 0.815 outside it: symptoms have competing explanations
Avoid one-to-one rules such as “validation gap means add dropout.” Two of the four columns below have public, measured instances, and they are worth holding in mind as the shape of the real thing.
Strong training, weak validation. A pneumonia model trained on 158,323 chest radiographs from three hospital systems scored AUC 0.931 (95% CI 0.927–0.936) on internal pooled Mount Sinai–NIH test data. At Indiana University it scored 0.815 (95% CI 0.745–0.885, P = 0.001). Zech and Badgeley, whose cross-sectional study appeared in PLOS Medicine in 2018, put the mechanism in the same abstract: “CNNs were able to directly detect hospital system of a radiograph for 99.95% NIH (22,050/22,062) and 99.98% MSH (8,386/8,388) radiographs.” Part of what the network had learned was which hospital it was looking at. The same failure mode turned up independently in Scientific Reports in 2024, where ResNet18 models predicted from knee X-rays alone whether a patient avoids refried beans (AUC 0.63) or beer (AUC 0.73). A shortcut does not announce itself in the validation number. It announces itself when the partition changes.
Good offline, weak product outcome. The Epic Sepsis Model is deployed at hundreds of US hospitals. Wong and Otles validated it externally at Michigan Medicine, on 27,697 patients and 38,455 hospitalizations, and published the result in JAMA Internal Medicine in 2021. Epic Systems had reported 0.76–0.83. They measured AUC 0.63 (95% CI, 0.62–0.64). The model alerted on 6,971 of 38,455 hospitalizations — 18% — and missed 1,709 of the 2,552 patients who had sepsis, 67% of them. Their conclusion is one sentence: “This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis.” A second team repeated the exercise across two county emergency departments and 145,885 encounters, reporting in JAMIA Open in 2024 a sensitivity of 14.7% and a positive predictive value of 7.6%. The remedy neither team points to is a better architecture.
Weak training and validation
The system is not fitting useful structure.
- Missing signal or bad representation
- Capacity too low
- Optimization or implementation failure
- Ambiguous or noisy target
Strong training, weak validation
Observed fit does not transfer.
- Overfitting or shortcut
- Shift between partitions
- Duplicate or group contamination
- Validation too small or noisy
Good offline, weak product outcome
The model metric does not translate into value.
- Threshold or action is wrong
- Users ignore the output
- Latency or review capacity blocks use
- Metric optimizes an incomplete proxy
Performance decays after launch
The evidence-to-use bridge changed.
- Data or concept shift
- Pipeline or feature failure
- Feedback loop
- Policy or label change
$304 million: the technically strongest idea may not be the best next investment
Compare interventions by expected impact on the diagnosed error, implementation time, data requirements, serving cost, maintenance burden, reversibility, and risk, remembering that a modest feature repair can dominate a major architecture project.
Estimate ranges rather than false precision. The purpose is to make assumptions explicit and choose an experiment whose result will inform the next decision.
The same arithmetic can come out against an entire product. On 2 November 2021 Zillow Group shut down Zillow Offers, its algorithmic home-buying business, and filed the announcement with the SEC. The company recorded an inventory write-down of approximately $304 million in the third quarter, expected a further $240–265 million of losses in the fourth, and cut roughly 25% of its workforce. The stated reason is not a technique it had failed to try. “We’ve determined the unpredictability in forecasting home prices far exceeds what we anticipated and continuing to scale Zillow Offers would result in too much earnings and balance-sheet volatility,” said Rich Barton, co-founder and CEO of Zillow Group. Forecast error the business could not price is a cost line, not a research topic. That is the judgement this section asks for, made at the scale of a company.
Steps
A practical order of operations: 2,212 studies screened, none usable
The sequence favors cheap falsification before expensive expansion. Step 1 is not a formality, and a whole literature has shown what skipping it produces.
A systematic review published in Nature Machine Intelligence in 2021 screened 2,212 machine-learning studies of COVID-19 imaging that had appeared between 1 January and 3 October 2020. It kept 415 after initial screening and 62 after quality screening. Of the 62 that survived, not one was fit for the clinic: “Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases.” A separate team retrieved 629 articles, included 44, and reached the same verdict later in 2021 in JMIR Medical Informatics.
Two review teams, working separately over the same field, arrived at a usable-model count of zero. Steps 1 and 2 are what those 62 and 44 papers passed over, and they cost days rather than quarters.
1. Validate the claim
Confirm target, prediction time, split, metric, and baseline.
2. Verify the pipeline
Inspect examples, labels, preprocessing, and tiny-sample learning.
3. Read the errors
Build a taxonomy and identify concentrated failure modes.
4. Locate the bottleneck
Choose the layer most likely to explain the priority errors.
5. Design one experiment
Predict which metric and error category should change.
6. Measure side effects
Check slices, calibration, latency, cost, and new harms.
7. Decide and document
Adopt, reject, collect evidence, narrow scope, or stop.
Key idea
39.1% downgraded: a good process can conclude that machine learning is not the answer
A project should stop or reframe when the target lacks measurable signal, collection would be disproportionate, or a simpler rule meets the need. Stop, too, when error costs remain unacceptable. Stop when the workflow cannot use the output.
The most public version of that decision was taken in four days. Ofqual's interim report on summer 2020 awarding, published on 13 August 2020, records what its standardisation model did to 718,276 A level centre assessment grades. It adjusted 35.6% down by one grade, 3.3% down by two, and 0.2% down by three or more: 39.1% downgraded against 2.2% raised, with 58.7% left unadjusted. The report states the limit that produced it: “However, there is no statistical model that can reliably predict grades for particularly small groups of students.” On 17 August 2020 the regulator abandoned the calculated grades. The statement from Roger Taylor, Chair, Ofqual, announced that each student would receive their centre assessment grade or the moderated grade, whichever was higher.
Ending an unsupported model project protects resources and users. It is evidence of disciplined engineering, not failure.
The goal is a better decision system, not a model at any cost.
Analogy
An analogy: treating a diagnosis rather than a symptom
A clinician prescribes the same medicine for every fever, without checking infection, inflammation, environment, or measurement error. The symptom is real, but the treatment is unjustified.
Model improvement requires the same diagnostic discipline. Machine-learning bottlenecks can be tested through controlled data and system changes; medical causality carries biological complexity that no such test reaches.
Example
Organize improvement as a learning portfolio
A mature backlog contains experiments at several levels rather than twenty architecture variants. Each of the cases above is one such experiment someone eventually ran — usually later than they should have.
- Framing experiment: compare a ranking workflow with a forced binary decision; an 18% alert rate like the Epic Sepsis Model's is a framing question before it is a threshold question.
- Data experiment: add independently sampled cases from the weakest site, as Zech's team did by testing at Indiana University instead of pooling all three hospital systems.
- Label experiment: double-annotate ambiguous cases and test a revised policy.
- Representation experiment: add a valid pre-decision feature, or remove an unstable proxy such as the site signal that let a pneumonia network name its hospital 99.95% of the time.
- Model experiment: vary capacity under fixed data and evaluation.
- Workflow experiment: change threshold, escalation, or review interface while holding the model fixed — the retinopathy deployment's image-quality threshold was a workflow parameter, not a model weight.
Keep decisions connected to what was learned
For every iteration, record the hypothesis, evidence, result, uncertainty, affected slices, cost, and decision. Link negative findings to the backlog. The same failed idea should not be repeatedly rediscovered.
That is the team-scale version of what the COVID-19 reviews found missing at field scale: 62 papers and 44 papers repeating flaws that each individual group had no register to warn them about. A project becomes faster when it accumulates explanations, not merely model files.
Example
Questions that determine the next move
Before approving an improvement project, answer these questions in writing.
- Which observed error category or product outcome should change?
- What mechanism connects the proposed intervention to that change?
- What evidence would falsify the hypothesis — and would it be an internal score or an external site?
- What else must stay constant for the comparison to be interpretable?
- Which slices or harms could worsen?
- What is the decision if the effect is small, unstable, or expensive?
Key takeaways
- Improvement begins by locating the dominant bottleneck across framing, evidence, representation, optimization, capacity, evaluation, and workflow. In eleven clinics in Thailand the blocked layer was workflow, with about 21% of images rejected by the system's own quality threshold.
- The same symptom has several causes, so symptom-to-remedy rules are unreliable: AUC 0.931 internally and 0.815 at Indiana University was a shortcut, not a capacity problem.
- Interventions should state a mechanism, predicted error change, falsifying evidence, controlled conditions, and side effects. 62 reviewed COVID-19 imaging models and 44 more failed on exactly those points.
- Expected impact must be weighed against implementation time, data needs, serving cost, maintenance, reversibility, and risk. Zillow priced that trade at approximately $304 million written down and roughly 25% of its workforce.
- A disciplined project may narrow scope, use a simpler method, redesign the workflow, or stop entirely, as Ofqual did four days after 39.1% of A level centre assessment grades were downgraded.
- An experiment register should preserve hypotheses, results, uncertainty, negative findings, costs, and decisions. Data cascades were found to be “pervasive (92% prevalence), invisible, delayed, but often avoidable”, and invisible failures are the ones a register catches.