ML Data Engineering as an Evidence System

These examples show why model choice is only one part of the engineering problem. The last of them is not a thought experiment.

The Epic Sepsis Model runs at hundreds of US hospitals. Two teams have validated it externally. Wong and colleagues at Michigan Medicine went first, in JAMA Internal Medicine in 2021. Ostermayer and colleagues repeated the exercise in 2024, in two Texas county emergency departments, across 145,885 encounters in 2023. They found a sensitivity of 14.7% and a positive predictive value of 7.6%. Their abstract closes on one sentence: “The ESPMv1 provides suboptimal diagnostic characteristics for undifferentiated patients in a county ED.”

Nothing in either study is a statement about the algorithm. Both are statements about which patients, which encounters, which time windows and which recorded outcomes the system was fed.

A trained model is usually a small artifact at the end of a long data path. Along that path, events are generated, transported, stored, joined, filtered, labeled, transformed, split, versioned, and finally consumed.

Most silent failures happen before optimization begins. A missing event, a reused identifier, a late label, or a future-looking join can create impressive metrics that collapse in production. That last failure has been counted, not merely warned about. Kapoor and Narayanan went through the reviews published in fields that had taken up machine learning, and summarised what they found in Patterns in 2023: “Through a survey of literature in fields that have adopted ML methods, we find 17 fields where leakage has been found, collectively affecting 294 papers and, in some cases, leading to wildly overoptimistic conclusions.” Seventeen fields, 294 papers, and eight types of leakage, temporal leakage and illegitimate features among them. None of those papers went wrong at the optimizer.

ML data engineering makes those upstream assumptions explicit and testable. It treats data as a maintained product with consumers, service expectations, change control, and recovery procedures.

The discipline is broader than cleaning. It connects business meaning, time, identity, statistics, software, and operations so that a model sees the evidence the team believes it sees.

The structural version of that argument came out of Google in 2015. Sculley and nine colleagues drew the model as a small black box in the middle of the page, and the caption says why the drawing looks like that: “Only a small fraction of real-world ML systems is composed of the ML code, as shown by the small black box in the middle. The required surrounding infrastructure is vast and complex.” The debts the paper names accumulate out there in the surrounding boxes, not in the model. Boundary erosion, entanglement, hidden feedback loops, undeclared consumers and data dependencies are all on the abstract's list.

The stages are connected. Each stage still answers to a different owner and fails in a different way.

The shape of that path is not a house convention. Amershi and eight colleagues at Microsoft studied software teams there building AI applications, and published what they saw in 2019; the paper took Best Paper. Their Figure 1 draws nine stages, and its caption states both the split and the loops: “The nine stages of the machine learning workflow. Some stages are data-oriented (e.g., collection, cleaning, and labeling) and others are model-oriented (e.g., model requirements, feature engineering, training, evaluation, deployment, and monitoring). There are many feedback loops in the workflow.”

Three of the nine — collection, cleaning, labeling — are data work that happens before there is a model to speak of. Their abstract names that data work as the first of the three ways in which AI differs fundamentally from prior software domains.

The same team can use all three patterns. Critical ML systems need the guarantees of a maintained data product.

Before selecting tools, trace one prediction from the real-world event back to every data dependency.

The walk has a published target list. Kapoor and Narayanan's leakage taxonomy sorts the failures into eight types, temporal leakage and illegitimate features among them. What one of those types is worth has since been measured in a single field. Working across four datasets and over 400 pipelines in neuroimaging, Rosenblatt and colleagues reported in Nature Communications in 2024 that leakage through feature selection inflated the reported correlation by Δr = 0.03–0.52, and that small datasets made it worse: “Leakage via feature selection and repeated subjects drastically inflates prediction performance, whereas other forms of leakage have minor effects.”

An inflation of 0.52 in correlation is not a rounding artifact. It is the difference between a publishable result and nothing. Neither of those two failures is visible in the feature table. Both are decided upstream, in what was joined and when.

Model-ready does not mean that a table can be loaded into a notebook. It means the team can explain the row, the target, the time boundary, the allowed sources, and the transformation version. The dataset should be reproducible from named inputs or a preserved snapshot. Its known limitations, intended uses, restricted uses, and quality checks should be available to downstream users. A mature team should also answer operational questions about ownership, freshness, consumers, change impact, and rollback.

That list is not only a house standard. On 27 October 2021 three regulators — the US FDA, Health Canada and the UK's MHRA — jointly published “Good Machine Learning Practice for Medical Device Development: Guiding Principles”. Ten numbered principles, of which three are squarely about the data layer: principle 3, representativeness of the intended patient population; principle 4, independence of training and test sets; principle 5, reference datasets built by best available methods, with the limitations of the reference understood. A fourth, principle 2, names “data quality assurance, data management” and “data authenticity and integrity” among the software-engineering and security fundamentals.

Principle 4 is worth reading in full, because it is an evidence contract rather than a metric: “Training and test datasets are selected and maintained to be appropriately independent of one another. All potential sources of dependence, including patient, data acquisition, and site factors, are considered and addressed to assure independence.” Site factors, in the language of three regulators, is the failure Zech and colleagues measured.

What the absence of such a contract costs was measured twice on the same literature. Roberts and colleagues screened 2,212 studies of machine learning for COVID-19 from chest radiographs and CT scans. Initial screening left 415, quality screening left 62, and in Nature Machine Intelligence in 2021 they reported: “Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases.” Wynants and colleagues ran the same exercise for the BMJ, in a living review whose literature search closed on 7 April 2020. They reviewed 51 studies describing 66 prediction models, rated every one at high or unclear risk of bias, and closed: “Hence, we do not recommend any of these reported prediction models to be used in current practice.”

Two review teams, two publishers, 66 models on one side and 62 surviving studies on the other, and not one recommendation between them. Not because the architectures were wrong, but because of how the data had been assembled.

Together, those answers form the definition of done for the data layer. Accuracy cannot compensate for uncertainty about what the training examples actually represent.