Sampling, Representation, and Measurement Bias

A remote-care model predicts respiratory deterioration from wearable data. The first question is who is wearing the device.

A national survey answers it, and the answer is not the one the pipeline assumes. The Health Information National Trends Survey asked US adults about wearable use in 2019 and 2020. The result appeared in JAMA Network Open in 2023: “In this cross-sectional study based on a representative sample of 9303 US adults in 2019 and 2020, 18% with established CVD and 26% at risk for CVD reported using wearable devices compared with 29% of the general population.”

The instrument is least often on the people the model exists to protect. Among at-risk adults, older age (OR 0.35), lower educational attainment (OR 0.35) and lower household income (OR 0.42) were each independently associated with lower use. Now feed that population into a pipeline that scores a missing reading as a normal one. People with older devices, irregular connectivity, or jobs that prevent continuous wear come out looking healthier. Their silence is being read as health.

Between the target population and the analytic dataset sit three filters: who could be sampled at all, who took part and stayed, and how they were measured. Every one of them quietly removes people. Two of them have been counted.

The sampling frame comes first — the operational list or process through which examples can be observed. Clinical deep-learning studies in the US draw their patients from three states: California, Massachusetts and New York. A JAMA research letter established that in 2020. It screened 2,606 studies and kept the 74 that trained clinical deep-learning diagnostic algorithms. Of the 56 with a geographically identifiable US cohort: “Forty of 56 studies (71%) used a patient cohort from at least 1 of these 3 states. Among the remaining 47 states, 34 did not contribute any patient cohorts, and the remainder contributed between 1 and 5 cohorts.” That is the frame for a whole literature. It is a map with holes in it that no one drew on purpose.

Participation and retention come second — who enters, remains, withdraws, or becomes unreachable. Roughly 9.2 million people aged 40–69 living within 25 miles of one of 22 UK Biobank assessment centres were invited to take part. 5.5% did. The resulting cohort ran to 500,000 people. Its all-cause mortality was 46.2% lower in men and 55.5% lower in women aged 70–74 than in the general population of the same age. The investigators said it plainly in 2017: “UK Biobank is not representative of the sampling population; there is evidence of a “healthy volunteer” selection bias.”

The measurement process is the third filter — sensors, surveys, records, annotators, and proxies that produce values. The analytic dataset is what survives filtering, missingness handling, deduplication, and inclusion rules. Five stages. A reader of the finished table sees only the last one.

Representation parity counts who appears. Measurement equivalence asks whether a variable means the same thing for them. Outcome coverage asks whether the rare events were recorded at all. They fail separately, and passing one says nothing about the other two.

Parity is a count, and the count can actually be done. Buolamwini and Gebru did it in 2018 for two face benchmarks: “We find that these datasets are overwhelmingly composed of lighter-skinned subjects (79.6% for IJB-A and 86.2% for Adience) and introduce a new facial analysis dataset which is balanced by gender and skin type.” IJB-A was 4.4% darker-skinned female. The new dataset was the Pilot Parliaments Benchmark: 1,270 individuals drawn from six national parliaments, evidence bought rather than assumed. On it the worst-classified group reached a 34.7% error rate, against 0.8% for lighter-skinned males. Note what the count does and does not do. Counting the composition of a benchmark takes clarity about the target population. It establishes nothing about whether anything was measured correctly. Not every task needs population-proportional sampling, and where it does, weighting is part of the answer.

Equivalence is a different test, and instruments that work in every engineering sense can fail it. A University of Michigan cohort compared pulse oximetry with arterial blood gas. Among patients whose oximeter read 92–96%, “an arterial oxygen saturation of less than 88% was found in 88 of 749 arterial blood gas measurements in Black patients (11.7%; 95% confidence interval [CI], 8.5 to 16.0) and in 99 of 2778 measurements in White patients (3.6%; 95% CI, 2.7 to 4.7)”. Sjoding and colleagues reported that in the New England Journal of Medicine in 2020. A multicentre replication reproduced the gap: 37,308 paired measurements from 7,342 White and 1,050 Black patients, 17.0% against 6.2%. Both groups are present in the dataset, in numbers. The recorded variable simply does not mean the same thing for both. That is the differential error this check exists to catch, and catching it takes domain and measurement expertise rather than a proportions table.

Outcome coverage is the third question, and the one that survives both. Were the relevant and rare events — severe deterioration episodes, in the remote-care case — observed at all? It looks at task evidence rather than at who appears. It can require oversampling or targeted collection, and it can conflict with privacy or cost. A cohort can be proportionally drawn, measured with a validated instrument, and still contain almost none of the events the model is being asked to predict.

Defining the target population is the step everyone assumes somebody else did. Reconstructing how the examples were actually selected usually shows otherwise.

First, define the target population: the people, contexts, devices, time periods, and decision points the system will make claims about. Second, reconstruct selection — eligibility, recruitment, participation, retention, and filtering. That is the trace that turns 9.2 million invitations into a 5.5% participation rate and a cohort that is not its own sampling population. Third, validate measurement: construct validity and error across the conditions the instrument actually meets, not the conditions of the bench test. Fourth, model missingness. Absence, device failure, nonparticipation and informative missingness are four different things. A pipeline that cannot tell them apart will score all four as normal. Fifth, bound the claim — through weighting, targeted collection, uncertainty, or restricted deployment.

Step five has a worked precedent rather than a slogan. Reweighting the data you already hold cannot reach the people you do not. The Pilot Parliaments Benchmark's 1,270 purpose-collected individuals are what paying for the missing evidence looks like.

Representation concerns whether the data cover the populations and situations relevant to the decision. Measurement bias concerns whether the recorded variables represent the intended construct comparably across people and contexts. A dataset can grow to millions of rows and stay systematically incomplete. The row count for that claim is on the record.

Chest-radiograph classifiers were trained on three public datasets. MIMIC-CXR: 371,858 images from 65,079 patients. CheXpert: 223,648 images from 64,740 patients. NIH ChestX-ray14: 112,120 images from 30,805 patients. Then on all three combined, a multi-source set of 707,626 images from 129,819 patients. The result appeared in Nature Medicine in 2021: “We find that classifiers produced using state-of-the-art computer vision techniques consistently and selectively underdiagnosed under-served patient populations and that the underdiagnosis rate was higher for intersectional under-served subpopulations, for example, Hispanic female patients.” The label the models were selectively handing out was “No Finding”. It is the same shape of failure as the wearable pipeline — absence of a signal presented as absence of risk — at the scale that was supposed to make it impossible.

A reviewer should examine six things: how the examples were selected, who never responded, who survived to the end of the study, how the instruments behaved, what data went missing, and what the administrative records actually covered. Judge all six against the people the system will meet and the decision it will make, not against abstract demographic quotas alone.

The coverage table is dull to build. It is also the artifact that shows where a target population quietly became a convenience sample.

The instrument test is no longer a matter of professional taste either. A regulator has now written the enrolment down. Draft guidance on pulse oximeters, issued by the US Food and Drug Administration in January 2025, recommends this: “We recommend that you conduct the study as described in Annex EE of ISO 80601-2-61 Second edition 2017-12 (Corrected version 2018-02) in a diversely pigmented group of 150 or more healthy participants.” The rest of the recommendation is quotas, because that is what a measurement-validity study is made of.

Every model carries an implicit claim about who it was measured on. Writing that claim down is what turns it into something a reviewer can refuse. The claim has a shape, and each part of it is checkable: this population, reached through this frame, retained at this rate, measured with this instrument under these conditions. A reading of 92–96% meaning what it says — or meaning something different depending on whose finger it was taken from.

Define when sampling and measurement fairness requires the team to redesign, restrict, remedy, or retire the system.