From a Real Problem to a Learning Task

“AI that prevents late deliveries” is what a logistics manager asks for, and it sounds like one well-posed request. It is not. That single sentence hides several different problems, and several decisions that a model cannot make alone.

The business outcome, the model target, and the operational action are related. Teams routinely treat them as one thing. That is how a project ends up optimizing an easy score that does not improve the outcome anybody wanted. They are three layers, not one.

A commercial algorithm that decides which patients get extra medical care shows what the collapse costs. Its authors call it “one of the largest and most typical examples of a class of commercial risk-prediction tools that, by industry estimates, are applied to roughly 200 million people in the United States each year”. Four researchers dissected it in Science in 2019, and all three layers are visible in it. The outcome wanted was health need. The proxy the data could supply was next year's health-care cost. The action was automatic identification for enrolment in a care-management program at the 97th percentile of risk score. At that threshold, Black patients carried 26.3% more active chronic conditions than White patients — 4.8 against 3.8 distinct conditions, P<0.001 — at the same score. Remedying that gap would have raised the Black share of auto-identified patients from 17.7% to 46.5%.

The abstract says where the gap comes from: “The bias arises because the algorithm predicts health care costs rather than illness, but unequal access to care means that we spend less money caring for Black patients than for White patients.”

The decisive demonstration is that the middle layer can be swapped while everything else is held still. Retrained on other labels, the Black share of the highest-risk group moved from 14.1% for the cost predictor to 26.7% for a predictor of the number of active chronic conditions. Same patients, same machinery, a different proxy. The action reached a different population.

Regulators moved the same day the paper appeared. New York State's Department of Financial Services and Department of Health wrote jointly to UnitedHealth Group Inc. about the Optum Impact Pro algorithm. The desired outcome, the model proxy and the operational action are three separate lines in a framing document for one reason. The second can be wrong while the first two look fine.

The prediction unit is the entity or event represented by one row at inference time. It might be an order, account, machine, image, message, patient visit, or user-session pair. A vague unit produces vague labels and unreliable splits.

The same business problem can use different units. Predicting whether an order will be late at checkout is not the same design as updating risk at each package scan. The second creates multiple observations per order. That forces careful handling of time and identity.

Choosing an output form changes the data, loss, evaluation, interface, and downstream decision. A category, a number, a ranking, and a generated artifact are not four dialects of one task. Each commits the team to a different kind of evidence and a different failure mode. Each also leaves a person with a different thing to do when the output arrives.

A learning task is justified only relative to a baseline. The baseline may be a current manual process, a simple rule, a historical average, a search interface, or a redesigned form that prevents missing information.

Google Flu Trends is the cleanest demonstration. Its training target was the CDC's own influenza-like-illness reports. From August 2011 it ran high in 100 of the 108 weeks that followed, and in February 2013 it predicted more than double the CDC's proportion of doctor visits for ILI. Four researchers then set it against the dullest baseline available, the CDC's own lagged numbers, in Science in 2014: “Even 3-week-old CDC data do a better job of projecting current flu prevalence than GFT”. A three-week-old average is not a sophisticated competitor. It was never entered in the comparison. That is the point.

The second reason to name the baseline is arithmetic about review capacity. Michigan Medicine externally validated the proprietary Epic Sepsis Model over 27,697 patients and 38,455 hospitalizations, from 6 December 2018 to 6 October 2019, with sepsis in 2,552 hospitalizations (6.6%). The hospitalization-level AUC was 0.63 (95% CI 0.62-0.64). The study appeared in JAMA Internal Medicine in 2021. At an alert threshold of 6: “The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%).” The baseline here was the clinicians. Against them the model contributed 183 sepsis patients (7%) they had not already recognised, at a number needed to evaluate of 8, while consuming 18% of all admissions as alerts.

The last reason is that the fancy alternative is often untested rather than better. A 2019 reproducibility study could reproduce only 7 of 18 neural top-n recommendation algorithms published at top conferences, and reported: “For these methods, it however turned out that 6 of them can often be outperformed with comparably simple heuristic methods, e.g., based on nearest-neighbor or graph-based techniques.” A year later, four researchers at Google re-ran the neural collaborative filtering experiments and found that a properly tuned dot product substantially outperformed the learned MLP similarity that had displaced it.

If a rule catches nearly every actionable case and is easy to maintain, a model may add cost without adding value. The reverse also holds. A model may be useful even with modest accuracy, provided it prioritizes a queue that nobody could inspect in full. That is a claim about the queue and the reviewers. It has to be written down and tested, not assumed.

A useful brief is short. Short enough that someone will actually challenge it before data engineering begins. Six moves: state the decision, naming who acts, when, and with what options; define one prediction unit and the exact observation cutoff; choose the output form; identify the target and how long it takes to arrive; record the constraints on latency, privacy, review capacity, fairness and cost; and compare the current baseline against at least one simpler intervention.

That shape is not a house style. On 27 October 2021 the U.S. Food and Drug Administration, Health Canada and the UK Medicines and Healthcare products Regulatory Agency jointly issued ten guiding principles for machine learning in medical devices. They are non-binding, intended to inform the development of Good Machine Learning Practice rather than to impose requirements. Three of them are the first two steps of the brief.

Principle 1, “Multi-Disciplinary Expertise Is Leveraged Throughout the Total Product Life Cycle”, is step 1: “In-depth understanding of a model’s intended integration into clinical workflow, and the desired benefits and associated patient risks, can help ensure that ML-enabled medical devices are safe and effective and address clinically meaningful needs over the lifecycle of the device.” The workflow and the risks come before the model, in the regulators' text as in the brief.

Principle 4, “Training Data Sets Are Independent of Test Sets”, is step 2. The unit and the cutoff are what make that independence real, and the principle names patient, data-acquisition and site factors as sources of dependence to be considered and addressed. It is the same failure the COVID reviews found in the field: sets that overlap through a factor nobody declared.

Principle 6, “Model Design Is Tailored to the Available Data and Reflects the Intended Use of the Device”, is steps 3 to 6 compressed into one line. The output, the target, the constraints and the alternatives are all answers to what the data can support and what the thing is for. Three national regulators converged on this order independently of any course.

These signals have one thing in common. Each of them usually means the team is trying to force machine learning onto a process nobody has specified properly.

The third signal below has a named, dated, numbered instance. The mid-1990s Cost-Effective HealthCare pneumonia study compared eight statistical and machine-learning methods on 14,199 patient records — 9,847 for training, 4,352 for testing, 46 variables per patient including a yes/no asthma field. One model said something that would have killed people. As Rich Caruana and his co-authors recount: “On one of the pneumonia datasets, the rule-based system learned the rule “HasAsthama(x) ⇒ LowerRisk(x)”, i.e., that patients with pneumonia who have a history of asthma have lower risk of dying from pneumonia than the general population.” (The spelling of "HasAsthama" is as printed in the paper.)

The rule was correct about the data and lethal as a policy. Asthmatic pneumonia patients were routinely admitted directly to the ICU, so the label had already absorbed the treatment decision the model was being asked to inform. The most accurate models on this problem were the neural nets — AUC 0.86 against 0.77 for logistic regression on one dataset — and they were judged too risky to field, precisely because nobody could read what else they had learned. Logistic regression was used instead. Accuracy was never the question. What the target had quietly recorded was.