Labels, Targets, and the Limits of Ground Truth

Give 115 pathologists the same breast biopsy slides and you do not get one diagnosis. You get a distribution. Joann G. Elmore and colleagues ran that test and counted the result: 115 practising pathologists in eight U.S. states, 240 cases, 6,900 interpretations in all, published in JAMA in 2015. Their own summary of what came back: “In this study of pathologists, in which diagnostic interpretation was based on a single breast biopsy slide, overall agreement between the individual pathologists' interpretations and the expert consensus–derived reference diagnoses was 75.3%, with the highest level of concordance for invasive carcinoma and lower levels of concordance for DCIS and atypia.” Nothing here is annotation failure. It is what a label looks like when you measure it.

The real-world condition, the observation procedure, the annotator, and the storage system can all differ. A target column records the output of that chain. It is not a record of reality without mediation.

Good dataset design therefore documents who labeled what, using which evidence, at which time, and under which policy. Without that history, a model may learn the quirks of the labeling process.

The chain can break at its first link, before any annotator sees anything. A pulse oximeter on a finger reports an oxygen saturation, and that reading can be checked against an arterial blood gas measurement from the same patient. Sjoding and colleagues made that comparison and published it in the New England Journal of Medicine in 2020. They took the paired readings where the oximeter showed a reassuring 92–96% and asked how often the arterial measurement was nonetheless below 88%. That is occult hypoxemia: a patient recorded as adequately oxygenated who was not. In the Michigan cohort, “an arterial oxygen saturation of less than 88% was found in 88 of 749 arterial blood gas measurements in Black patients (11.7%; 95% confidence interval [CI], 8.5 to 16.0) and in 99 of 2778 measurements in White patients (3.6%; 95% CI, 2.7 to 4.7)”. In a multicenter cohort the same comparison gave 17.0% against 6.2%. No annotator was involved at any point. The instrument wrote the value. The U.S. Food and Drug Administration issued a safety communication and, on 6 January 2025, proposed updated recommendations for evaluating pulse oximeter performance across skin pigmentations. A dataset assembled from those charts inherits the same error, and a model trained on it learns to reproduce it.

Labels mature through stages, and using them too early can convert uncertainty into systematic error. The stage at which a label freezes is set by policy rather than by nature, and in regulated industries you can read the policy.

Default on a loan is defined on two limbs. The bank judges the obligor unlikely to pay in full, or — in the Basel Framework's own words, at paragraph CRE36.68 — “The obligor is past due more than 90 days on any material credit obligation to the banking group.” The same paragraph's footnote lets a supervisor substitute a figure of up to 180 days for retail and public-sector obligations. The two-limb definition is written into EU/UK law as Article 178(1) of the Capital Requirements Regulation, which likewise permits competent authorities to replace the 90 days with 180 for retail exposures secured by residential property and for public sector entities.

So a borrower past due more than 90 days but not yet 180 is a default in one jurisdiction's training data and a performing loan in another's, on the same day, on the same facts. The timeline below is not a description of how evidence accumulates on its own. It is a description of decisions somebody made, and the last of them has an issuing body and a number.

The target format should retain distinctions that matter to decisions and evaluation. Abstention is usually the column that gets designed away first. In U.S. mammography it is the one the law requires.

Every mammogram report has to carry an overall final assessment drawn from a fixed ordered set: “Negative”, “Benign”, “Probably Benign”, “Suspicious”, “Highly suggestive of malignancy”, with “Known Biopsy-Proven Malignancy” and “Post-Procedure Mammogram for Marker Placement” added by the amendments that took effect on 10 September 2024. That rule is 21 CFR 900.12(c)(1). It is an ordinal target with an official vocabulary, published by the regulator rather than invented by the modelling team.

Beside it sits the abstention, and it is mandatory. Paragraph (c)(1)(v), as it read until 10 September 2024: “In cases where no final assessment category can be assigned due to incomplete work-up, "Incomplete: Need additional imaging evaluation" shall be assigned as an assessment and reasons why no assessment can be made shall be stated by the interpreting physician”. The incomplete assessment is now split into “Incomplete: Need additional imaging evaluation” and “Incomplete: Need prior mammograms for comparison”, and the American College of Radiology's BI-RADS system, which the categories mirror, names both as Category 0.

A radiologist who cannot tell yet does not record a negative and does not leave the field blank. The uncertainty has its own code, and the reason for it has to be written down.

Treat target design as a measurement project rather than a column-selection task. The last step, versioning the definition, is the one most often skipped. Rhee and colleagues measured what skipping it costs.

Two labelling procedures were run over the same hospitalizations: insurance claims codes on one side, EHR-based clinical criteria on the other. The study covered 173,690 sepsis cases among 2,901,019 adult admissions at 409 hospitals in 2014, with 7,801,624 admissions in the trend analysis, and appeared in JAMA in 2017.

Labelled by claims, sepsis incidence was rising 10.3% per year (95% CI 7.2% to 13.3%). Labelled clinically, the same admissions were flat: +0.6% per year (95% CI −2.3% to 3.5%). In-hospital mortality fell 7.0% per year by claims against 3.3% clinically. One set of patients, two procedures, an epidemic in one and no change in the other.

The Centers for Disease Control and Prevention built its Adult Sepsis Event surveillance definition on that work. Its Hospital Toolkit for Adult Sepsis Surveillance, of May 2018, is blunt about the alternative: “These analyses demonstrate that coding practices are likely vulnerable to biases from increasing sepsis awareness and financial incentives (higher reimbursement for sepsis coding), and therefore unreliable for surveillance purposes.” A team that changes its label definition without recording the change will read a difference of that size as a change in the world.

Arrest, approval, suspension, and treatment labels often reflect who was examined, which policy applied, and whose behavior was visible, so predicting those labels may reproduce an institution rather than measure the underlying risk or need.

A widely deployed risk tool predicted future health-care costs as a proxy for health need. The label was not illness; it was spending. Obermeyer and colleagues dissected the tool in Science in 2019 and found that at a given risk score, Black patients were considerably sicker than White patients. Reformulating the label to predict illness rather than cost raised the share of Black patients flagged for extra help from 17.7% to 46.5%. That is a change of target, not of algorithm or training data.

Within days New York's Department of Financial Services and Department of Health named the product, UnitedHealth's Impact Pro, in a joint letter to UnitedHealth Group Incorporated, and told the company to stop using it or justify it. Linda Lacewell and Howard Zucker wrote: “These discriminatory results, whether intentional or not, are unacceptable and are unlawful in New York.”

Before scaling supervision, ask whether the target should be predicted at all. Consistency is not enough. A technically consistent label can still be ethically or strategically unsuitable.