Labels, Ground Truth, and Annotation Pipelines

No source is automatically authoritative. Each one records a particular measurement process.

An observed outcome — loan repayment after twelve months — records the result of interest directly. But it arrives late, it can be censored, and the intervention meant to act on it can change it. An expert annotation — a radiologist marking a suspicious finding — reaches nuanced concepts at the cost of disagreement and fatigue, so it needs calibration, blind overlap and adjudication. An operational decision — an application was manually rejected — is cheap to collect at scale. It measures what was done rather than what was true, and it carries policy, capacity and historical bias with it. A proxy or weak label — complaint keywords standing in for service failure — fills the gap when direct labels are scarce, and introduces noise both known and unknown.

The fourth column has a priced example. A commercial risk algorithm covering 43,539 White and 6,079 Black patients did not predict illness. Its label was future health-care cost, a convenient proxy, so the model ranked patients by what the system would spend on them rather than by how ill they were. Switching the label would have raised the share of Black patients referred for extra help from 17.7% to 46.5%, with the architecture untouched. Obermeyer and colleagues dissected the system in Science in 2019, and their abstract states the general case: “We suggest that the choice of convenient, seemingly effective proxies for ground truth can be an important source of algorithmic bias in many contexts.” Regulators treated it as one. The New York Department of Financial Services and Department of Health wrote jointly to UnitedHealth Group Incorporated about the same system. A proxy is a choice about what the target means. It is answerable as such.

A queue of unlabeled items is not yet an annotation system. Five steps turn it into one. Define the evidence boundary: state what annotators may see and which contextual cues must remain hidden. Write decision-focused guidelines with positive, negative, borderline, abstain and exclusion examples. Calibrate reviewers on shared examples, with feedback, before production labeling begins. Add blind overlap: assign a sample to several reviewers to estimate agreement and systematic differences. Then adjudicate the conflicts, update the rules carefully, and keep the guideline version attached to every label.

Step three is the one teams treat as optional, and ImageNet priced it. The two annotators whose top-5 error the challenge reports differed mainly in how much practice they had before starting: annotator A1 worked through 500 practice images, annotator A2 through 100. The guidelines were identical. Preparation, not the task, is what separated them — and the gap it produced is larger than the gap between the trained annotator and the best model of the day.

Agreement is useful, but a high value can reflect an easy or imbalanced sample. Report class prevalence, abstention, disagreement patterns, adjudication rate, and performance on expert-reviewed gold examples.

Analyze quality by reviewer, source, class, difficulty, language, site, device and time. Systematic disagreement on one slice can reveal guideline gaps or missing context.

A low agreement figure need not condemn the collection either, and information retrieval has the longest record on this. Independent assessors judging TREC topics agreed on only about a third of the relevance judgments — an overlap of 0.36. The system rankings those judgments produced were almost identical, at a correlation of 0.890. Ellen Voorhees of NIST reported that in 2000, and her abstract draws the conclusion: “The high correlations indicate that the comparative evaluation of retrieval performance is stable despite substantial differences in relevance judgments, and thus reaffirm the use of the TREC collections as laboratory tools.” Twenty-five years later another team re-annotated the TREC 2019 Deep Learning track for SIGIR 2025 and reproduced the pattern with worse agreement still: assessor overlap of 0.15, with a system-ranking correlation of τ = 0.897 against her 0.890. The agreement statistic more than halved. The decision the labels support did not move. Report both, and say which decision the label is meant to serve before quoting an agreement coefficient.

For delayed outcomes, measure label maturity and revision rates. A preliminary label may be useful for monitoring and unsuitable for final evaluation.

Finally, audit construct validity: does the measured label represent the decision concept? Perfect agreement on the wrong proxy is still a poor target.

These problems cannot be fixed by changing the model architecture alone. Epic’s proprietary sepsis model is the case in full. Its external validators could not use the vendor’s target, because the target was proprietary. So they built a sepsis label themselves, from the CDC clinical surveillance definition and ICD-10 codes. Measured against that label, across 38,455 hospitalizations of 27,697 patients, the model reached an area under the ROC curve of 0.63 (95% CI, 0.62–0.64). It alerted on 18% of hospitalizations. It failed to identify 1,709 patients with sepsis (67%). Wong and colleagues published that in JAMA Internal Medicine in 2021, concluding that “the ESM has poor discrimination and calibration in predicting the onset of sepsis”. A second team repeated the exercise in 2024 over 145,885 emergency-department encounters, with a label of their own, and found a 6-hour-window sensitivity of 14.7%. Nothing in the architecture changed between those evaluations and the deployment. The label did, and the label is what decided what the number meant.

A fraud label may depend on customer disputes, investigator decisions and waiting periods. A medical label may depend on tests, coding practices and follow-up. When one team needed a sepsis label for 38,455 hospitalizations, they built it from the CDC clinical surveillance definition and ICD-10 codes — and that construction is the definition of sepsis their evaluation actually used. An image label may reflect annotator guidelines and how much of the object could be seen. Calling these values ground truth should not hide their provenance.

Labels can be delayed, missing, disputed, censored, policy-dependent, or wrong. How the target is defined shapes the model’s behavior. Predicting “case sent to review” learns an operational policy. Predicting “confirmed harmful outcome” aims at a different concept. Predicting future health-care cost, as one widely used risk algorithm did, aims at neither illness nor need.

A label system therefore needs the same engineering discipline as features: contracts, timestamps, versions, quality metrics, ownership, and feedback about disagreements.

The final value should keep its source, timing, policy and review history. Define the concept, unit, classes, horizon, exclusions and uncertainty states. Collect outcomes, annotations, proxies and source metadata under versioned rules. Review by measuring agreement, auditing slices, adjudicating conflicts and investigating unclear examples. Publish with a label version, confidence, timestamps and permitted uses attached to the dataset. Learn from model errors and downstream outcomes to refine guidelines, without contaminating evaluation.

The first step is the expensive one. The gap between 17.7% and 46.5% in the health-care risk algorithm was settled at definition time, when cost was chosen to stand in for illness. No care taken at collection, review, publication or learning recovers a target that was aimed at the wrong concept. The later steps are where the record is kept: which guideline version, which reviewer, how much practice, and what the adjudicators decided about the 597 items that turned out to have more than one right answer.