Intended Use, Units, Populations, and Harm

Row counts hide how much independent evidence a table actually holds. A pneumonia-screening study put that gap on the public record. Zech and colleagues trained convolutional neural networks on 158,323 chest radiographs and published the result in PLOS Medicine in 2018. Those images came from 47,392 patients at three hospital systems. The NIH Clinical Center supplied 112,120 images from 30,805 patients, Mount Sinai Hospital 42,396 from 12,904, and the Indiana University Network for Patient Care 3,807 from 3,683.

Nobody asked the networks to learn where an image came from. They learned it anyway. “CNNs were able to directly detect hospital system of a radiograph for 99.95% NIH (22,050/22,062) and 99.98% MSH (8,386/8,388) radiographs.” Site provenance was worth learning because pneumonia prevalence differed sharply between the sites. A model that knew nothing but the site could score well on the pooled data. The same shortcut reappeared in chest radiography during COVID-19: a 2021 paper in Nature Machine Intelligence found detectors selecting site provenance over signal.

Read the row count as evidence and this dataset looks like 158,323 independent facts about pneumonia. Read the unit and it is three.

A single project can involve several legitimate units, and the Zech study contains all four at once. The observation unit is one chest radiograph. The prediction unit is the image-level pneumonia score the network emits for it. The decision unit is the patient — one of the 47,392 people whose care changes when that score crosses a threshold. One patient may contribute several images. The analysis unit is the level at which the evidence is actually independent, and here that is the hospital system. There were three: the NIH Clinical Center, Mount Sinai Hospital and the Indiana University Network for Patient Care.

The four numbers attached to those units are 158,323, 158,323, 47,392 and 3. A report that names only the first invites its reader to imagine the last.

Generalization requires a defensible relationship among three populations. A device that sits on millions of fingers shows how far apart they can drift. Pulse oximeters read poorly on patients with darker skin, and the UK Department of Health and Social Care's independent review of equity in medical devices, published in March 2024, locates that failure not in an algorithm but in the population the devices were built on: “One of the main reasons for the poorer performance of pulse oximeters for patients with darker skin tones is that they have tended to be tested and calibrated on light-skinned individuals, whose readings are then taken as the norm.”

That is the three-population gap stated as a fact rather than a caution. The observed sample was the set of calibration readings actually collected. The sampling frame was the volunteers who could be recruited into calibration studies, which skewed light-skinned. The target population was everyone whose blood oxygen would later be read by the device. Nothing in the calibration data announced the difference. The reader had to know to ask.

Both halves of the correction are now written down by regulators. Recommendation 2 of the review asks that “Validation of devices should be conducted in the intended use population and setting, such as at home or in an ICU”. On 6 January 2025 the FDA proposed updated recommendations of its own. They would increase the number of clinical study participants and require performance data across a range of skin pigmentations, assessed two ways: a subjective method, the Monk Skin Tone Scale, and an objective one, calculation of individual typology angle.

Many standard calculations assume observations are independent or weakly dependent. Repeated measurements, shared devices, households, geographic clusters, and coordinated behavior violate that simplification.

Three regulators have written down which dependencies count. In October 2021 the U.S. FDA, Health Canada and the UK MHRA jointly issued ten guiding principles for good machine learning practice in medical device development. Guiding Principle 4 is titled “Training Data Sets Are Independent of Test Sets”, and it does not leave the sources of dependence to the reader's judgment: “Training and test datasets are selected and maintained to be appropriately independent of one another. All potential sources of dependence, including patient, data acquisition, and site factors, are considered and addressed to assure independence.”

Patient, data acquisition, site. Those are the three levels that turned 158,323 rows into far less than 158,323 pieces of evidence in the chest radiograph case. Principle 3 covers the other half of the contract. It requires that data sets be “Representative of the Intended Patient Population” so that “results can be reasonably generalized to the population of interest”.

Dependence does not make evaluation impossible. It changes the split unit, resampling method, confidence interval, and interpretation of effective sample size.

A useful evaluation report says all of this near the beginning, and one field has already turned it into a numbered checklist item. CONSORT-AI, published in Nature Medicine in 2020, adds 14 new items to the trial-reporting standard. It was developed through a two-stage Delphi with 103 respondents, a consensus meeting with 31 stakeholders and a pilot with 34 participants. It refuses to let eligibility be a single question. Item 4a(i) asks for it at the level of participants; item 4a(ii) says: “State the inclusion and exclusion criteria at the level of the input data.”

The split exists because the two questions come apart in practice, and the size of the gap has been measured. Martindale and colleagues reviewed 65 AI randomised trials in Nature Communications in 2024. Participant-level criteria were reported in 98% of trials, input-data-level criteria in only 74%, against a median overall concordance of 90%. Nearly every trial says who was eligible. A quarter of them never say which data were.

Steps 2 and 4 below are that missing quarter. Naming the target, stating inclusion at both levels, identifying the dependence structure, recording exclusions and choosing the analysis unit is not a private discipline. In at least one field it is a published item with a compliance rate attached to it.

A metric without a population statement invites the reader to imagine broader coverage than the data support. Naming the evaluation unit and the exclusions is not administrative detail. It is part of the scientific claim. Every case in this lesson is the same failure seen from a different angle: 158,323 radiographs that were three sites, a calibration frame of light-skinned individuals used to certify a device for everyone, a gender-imbalanced training set whose cost fell on the smaller group, and 50 million search terms that were still one train.

The remedy has a name and a format. Mitchell and colleagues proposed model cards in 2018: “short documents accompanying trained machine learning models” that carry “benchmarked evaluation in a variety of conditions, such as across different cultural, demographic, or phenotypic groups” and that “disclose the context in which models are intended to be used”. Intended use is written down rather than assumed by the reader.

Later lessons use these definitions to design splits, confidence intervals, slices, and online experiments.