Data Annotation as Measurement

AI summary

Overview

Research area: AI safety and ethics / data quality — specifically the design, evaluation, and documentation of data annotation processes for machine learning.

Technical level: Intermediate. There is no mathematics or modeling; the paper is a conceptual reframing supported by qualitative evidence, but it assumes familiarity with annotation workflows, inter-rater reliability, and dataset construction.

Scope (one sentence): The paper argues that data annotation should be treated as a measurement problem — requiring concept definition, operationalization, application, and interrogation of reliability and validity — rather than as a problem of maximizing annotator agreement.

Note on source material: the provided paper content is truncated mid-sentence in Section 5 (Face Validity). Sections after that point, including any later discussion or conclusion and the appendices, were not available and are not summarized here.

What This Paper Is About

Modern AI systems are built and evaluated on annotated data, yet annotation quality is usually judged by a single signal: whether multiple annotators assign the same label. The authors argue this is a mistake, because agreement among annotators can occur while those annotators systematically misinterpret the concept they were asked to capture. The goal of the paper is to make the decision points inside annotation processes visible, to distinguish the different sources of annotation problems, and to supply measurement-theory concepts that let teams assess whether annotations are not just consistent but actually correct.

Key Contributions

1. A visible map of annotation processes. Drawing on a semi-systematic literature review (N=132) and semi-structured interviews (N=10), the authors lay out the four stages of annotation — Task Design, Annotator Management & Annotation Collection, Quality Assessment, and Quality Improvement & Adjudication — and enumerate the key decision points within each stage (Table 2).

2. A taxonomy of five distinct sources of annotation issues. The authors define error, ambiguity, impossibility, subjectivity, and annotator identity as separate sources (Table 3), each with its own appropriate intervention. Problems that look identical at the outcome level (a mismatch between annotation and "ground truth") can arise from different places in the process and therefore require different fixes.

3. Reframing annotation as measurement. The authors apply measurement theory to annotation, arguing that annotation should follow the stages of systematization, operationalization, application, and interrogation, and that teams should evaluate reliability and validity rather than agreement alone.

4. Practical guidance translating measurement theory into annotation practice. The paper begins to show how specific aspects of reliability and validity — including test-retest reliability and face validity — can be folded into everyday annotation workflows.

Main Findings

• Agreement is not validity. Agreement is easy to compute and is a useful indicator of reliability, but treating it as evidence of validity is a mistake: annotators may agree while applying the same flawed assumptions or the same reading of ambiguous instructions. Interview participants themselves acknowledged that high agreement does not imply high correctness (P1, P7, P8, P10).

• Correction requires process change, not outcome inspection. Correcting issues at scale means changing the annotation process and re-annotating; annotating processes are under-specified and under-documented, with key decision points treated as invisible defaults rather than interrogated choices.

• "Ground truth" is itself nebulous. It refers to the true value of the underlying concept, may be unknown or contested for any instance, and the assumption that a single ground truth exists may be flawed. Proxies such as expert-produced "gold labels" are commonly used instead.

• Five different sources of annotation issues (Table 3), each with distinct solutions:

  • Error — the annotator did not follow instructions; address by retraining annotators, giving feedback, or filtering annotators. Participants framed errors as good-faith mistakes from fatigue or over-reliance on heuristics, more often than as task "gaming."
  • Ambiguity — the annotation instructions do not contain enough information to determine the annotation; address by improving instructions or refining the schema.
  • Impossibility — the data itself does not contain enough information; address by re-processing or filtering data.
  • Subjectivity — the annotation depends on implicit values, beliefs, opinions, or assumptions; address by improving instructions or refining the schema. This affects even tasks that sound objective, such as identifying skin tone in an image.
  • Identity — the annotator's identity does not match the desired annotator population; address by filtering annotators. An example is AI-generated annotations used for a task intended to elicit human preferences.

• Mismatched interventions are harmful. Prior work (Zhang et al. 2025) found annotation managers treat all low-agreement annotations as errors; when large-scale low agreement actually indicates ambiguity or impossibility, rejecting and declining to pay those annotators is both an objectionable labor practice and ineffective at solving the problem.

• Selecting the annotator population is the dominant management decision. Every participant reported this as the biggest decision point in annotator management — deciding between human experts, human crowdworkers, AI, or a combination. Participants noted AI annotators tended to make fewer errors, but their errors were often less explainable (P2, P5), and cost and time constraints prevented regular use of expert human annotators (P1–P3, P6, P7, P9, P10).

• Participants typically assess quality in three ways: manual review of a subset of annotations to see whether they pass the "sniff test" (P1–P5, P8–P10); calculating inter-rater reliability (P1–P3, P6, P7, P9, P10); and calculating agreement between annotations and gold labels (P1, P2, P7, P10).

• Publishing incentives pull teams toward agreement. Participants prioritized agreement even when they doubted it was the best quality indicator, summarized by P10 as "a gap between what is publishable and what a peer reviewer will say [is] good enough, and what we actually all should be doing or considering as a research community."

• Unit-of-annotation choices force explicit tradeoffs. P6, annotating trial transcripts for gender stereotyping, expanded the unit from sentence to a larger span to raise inter-rater reliability, which then made it impossible to count how many times a reference occurred — the team had to decide what it was prioritizing.

• Multiple IRR metrics exist and none is clearly right. Prior work identifies 22 published IRR metrics, many mathematically equivalent; popular ones include percent agreement, Cohen's kappa, and Krippendorff's alpha. There is no clear basis for choosing among them, nor a clear threshold for "good" IRR.

• Test-retest reliability is nearly absent from the literature. Of more than 80,000 papers in the ACL Anthology, just 56 (0.07%) report measuring test-retest reliability. It may help distinguish disagreement caused by subjectivity from disagreement caused by random error, and may be a better quality signal than IRR for subjective tasks — but there are no agreed best practices for computing it, such as how long to wait between rounds. Very few interview participants measured it; some were unaware it existed (P1: "Like, check over your own work? I don't know that that's ever been done."), while P7 called it "low-hanging fruit for quality [assessment]…of the annotator" and P10 noted that "as you code data…your affinity with the data and your understanding of the construct will change."

• Face validity checks are widespread but informal. Nearly every interview participant reported manually reviewing a subset of annotated instances, but these checks are usually undocumented and unpublished. In a rare documented exception, Harvey et al. (2025a) identified issues in a crowdsourced corpus of tweets written in African American English and 'Standard' American English and chose not to use the dataset.

Methodology in Plain English

The authors used two complementary qualitative methods, developed iteratively.

A semi-systematic literature review. They searched the ACM, ACL, IEEE, and NeurIPS databases for papers containing annotation terms (annotat*, label*, coding) together with quality terms (subjectiv*, objectiv*, ambigu*, eval*, valid*, reliab*, agree, bias*, ground truth, error*), requiring both kinds of keyword in the title or abstract. They filtered papers by title and then by abstract against three inclusion criteria: the work must be about annotation specifically, must evaluate/critique/improve annotation outcomes or propose/document annotation practices. They supplemented the database search with a natural-language search for "barriers or best practices in data annotation" using AI2 Asta, and reviewed the citations of surfaced papers to reduce selection bias. In total they reviewed 132 publications spanning 1972 to 2026, sorting them into process-focused papers (proposing best practices or surveying current practices) and outcome-focused papers (evaluating existing annotated data), and attempted to map outcome evaluation approaches onto aspects of reliability and validity.

Ten semi-structured interviews. Because the literature lacked narrative accounts of what decisions teams actually make, the authors added interviews conducted between November and December 2025 — 60 minutes each, over Zoom, recorded, with informed consent, IRB review, and a $50 gift card for each participant. Recruitment used purposive and snowball sampling; all participants had served as annotators, all but one had also served as annotation managers, and they worked across education, law, and general-purpose ML, all based in North America. Participants described each project's goal, data medium, scale, annotator population, and their processes in detail, focusing on decision points and challenges, and were asked what action they took at each and what effect they thought it had.

Analysis. The team ran reflexive thematic analysis with an inductive-deductive coding approach. The first author, in discussion with

Authors’ abstract

Modern AI systems depend on annotated data, but annotation is rarely treated as the act of measurement that it is. Instead, annotation quality is commonly reduced to agreement: if multiple annotators assign the same annotation to a data instance, the annotations are taken to be high-quality. Yet agreement does not establish whether annotations validly capture the underlying concept they are meant to represent. In this paper, we argue that data annotation should be understood as a measurement problem. Like other forms of measurement, annotation requires defining a concept, operationalizing it through an instrument, applying that instrument, and evaluating the reliability and validity of the resulting measurements. Drawing on a literature review of annotation quality research (N=132) and semi-structured interviews with annotation team members (N=10), we develop a framework for diagnosing and correcting annotation issues. First, we map key decision points across annotation processes - including task design, annotator management, quality assessment, quality improvement, and adjudication - that shape annotation outcomes. Second, we identify five distinct sources of annotation issues: error, ambiguity, impossibility, subjectivity, and annotator identity. Annotation problems that appear similar at the level of outcomes often require different process-level interventions based on their sources. Finally, we translate measurement theory into practical guidance for annotation teams, showing how assessments of reliability and validity can move beyond agreement alone. By reframing annotation as measurement, we offer a conceptual foundation for improving the quality of annotated data used in AI research and practice.