Same Output, Different Gold: Measuring How Reference Choice Moves a Multilingual Benchmark Score

AI summary

Overview

Research area: Natural Language Processing — evaluation methodology, benchmark auditing, and multilingual annotation quality (presented at the TAE workshop, "Trust-AI-Eval: Can We Trust AI Evaluation?").

Technical level: Intermediate. The statistics (paired bootstrap, interaction tests, Holm correction) assume some familiarity with benchmark metrics, but the core argument is stated in plain terms.

Scope: A single-paper measurement study that holds a system output fixed and swaps the gold reference in a six-language PII span-extraction benchmark, quantifying how far the score and the ranking of two outputs move as a result.

What This Paper Is About

A benchmark score is the comparison of a system output against a reference, and the research community scrutinizes the output far more than the reference — which is treated as a fixed property of the dataset. But a reference is the product of an annotation pipeline: people labelled items, disagreed, and some rule resolved those disagreements. This paper asks how much of a reported score is actually a property of that resolution rule rather than of the system being ranked, by measuring what happens when one defensible reference is exchanged for another on identical items.

Key Contributions

1. A reusable procedure for measuring reference sensitivity that requires no new annotation — only an annotation record that was retained rather than discarded, plus scoring against more than one reference with the scored output held fixed.
2. Evidence that an aggregation default is an evaluation bias, not an annotation detail. The pipeline resolved annotator disagreements by keeping Annotator A when the adjudication step did not fire, in 98.7% of such cases, making the shipped gold a partial copy of one of the outputs being compared. The authors note this is testable in two lines of code.
3. A quantified measurement on a six-language PII span-extraction benchmark, with bootstrap confidence intervals throughout: pooled score movement of 2.00 to 4.95 F1 points depending on which output is scored, up to 7.55 in the worst language, and a paired interaction of +2.95 points that survives correction for multiple testing.
4. The "reference-sensitivity band" — a proposed reporting convention giving the movement that reference choice alone produces at the top of its confidence interval (6.3 F1 points under Strict matching, 5.2 under Relaxed, or 3.4 computed from Annotator B's delta), to be published beside the headline score.

Main Findings

• The shipped gold is not neutral between annotators. The delivered gold reproduces Annotator A's span set exactly on 83.4% of prompts versus Annotator B's on 76.1%, a skew of +7.3 points that appears in every one of the six languages and ranges from +2.7 (Polish) to +12.9 (Swedish). The reviewer gold shows a pooled skew of +0.04 (A 70.55%, B 70.51%), with four of six languages within 1 point of neutrality.

• Swapping the reference moves the score. With the output held fixed, Annotator A scores 91.62 against the delivered gold and 86.67 against the reviewer gold — a 4.95-point difference (95% CI [3.23, 6.27]). Annotator B moves 2.00 points (CI [0.26, 3.37]). The effect is largest in Hindi (7.55, CI [4.90, 10.91]) and Chinese (7.06, CI [4.40, 9.86]), and 13 of the 24 language–system–rule deltas exclude zero.

• Swapping the reference also moves the comparison between outputs. The paired interaction — the margin under the delivered gold minus the margin under the reviewer gold, recomputed inside each bootstrap resample — is +2.95 points pooled (CI [+1.97, +3.87]) and remains significant at p < 0.01 after Holm correction over 14 tests. Hindi (+5.27, CI [+2.64, +8.05]) and Portuguese (+4.42, CI [+2.53, +6.45]) also survive; Polish, Chinese and Arabic show no interaction distinguishable from zero, and Swedish falls just outside after correction (p = 0.08).

• One language's margin changes sign. In Hindi, Annotator A leads by 3.97 points under the delivered gold (CI [+0.65, +7.14]) and trails by 1.30 under the reviewer gold (CI [-4.39, +1.55]). The authors are explicit that the reverse leg is not itself significant: what is significant is the interaction, so they claim only that a significant win becomes a non-significant loss, not that B is better.

• Swedish is a clean null, and it supports the account. The movement there is 0.78 points (CI [-7.42, +6.58]). Swedish is the one language whose reviewer gold is itself skewed toward A (by +5.3 points); where both references share a preference, exchanging them cannot move the score. This bounds the claim — the effect is a property of how these particular references were built, not a law.

• The effect traces to an undocumented default. On 94.9% of audited prompts the shipped reference is character-identical to one of the two annotator labellings. Where the third pass did fire on a disagreement, the gold reproduced B (42.3%) about as often as A (38.5%); the net skew arises entirely in the unadjudicated branch (10.6% of prompts pooled, 16.5% in Arabic).

• Adjudication rates are not uniform across languages. The third pass fired on 25.1% of Polish prompts but 61.3% of Hindi ones, and Hindi triggers adjudication on roughly 40% of prompts where annotators had already agreed, against 3–5% elsewhere.

• Annotator disagreement is mostly about boundaries, not entity identity. Pooled inter-annotator micro-F1 is 79.6 strict and 82.2 relaxed, ranging from 71.7 (Arabic) to 87.8 (Chinese) — a 16-point spread across languages annotated under one protocol. Once boundaries match, annotators assign the same type 95.6–97.6% of the time.

• Roughly one gold label in five does not survive expert re-annotation. On the audited subset, the reviewer's labelling differs from the delivered gold on 22.0% of prompts, from 17.3% in Portuguese to 29.9% in Arabic.

• The decontamination check is inconclusive. On the 135 prompts where the delivered gold copies neither annotator, A's pooled advantage moves from +2.83 to -0.09 (CI [-4.52, +4.51]) and the interaction falls to +0.10 (CI [-2.81, +2.94]) — consistent with self-scoring, but with 13 to 30 items per language the subset cannot detect an effect of the reported size. The authors state they cannot separate the two effects.

• Several reported margins are the same order as the reference noise. Under the delivered gold, every margin between the two annotators (2.83 pooled, magnitudes 1.46 to 4.59 across languages) falls below the 6.3-point band, and three of seven exceed the stricter 3.4-point one.

• Two scoring conventions are flagged as voiding metrics. A pre-release merge map collapses 25 fine-grained PII types into 17 (placing national identity numbers, tax identifiers and permanent account numbers into one "tax number" class), making a known annotator confusion unobservable. Separately, the original evaluation located predicted spans by exact string search, so across 24 reported result blocks the "partial match" and exact-match metrics are numerically identical in every one.

Methodology in Plain English

The usual experimental design varies the system and fixes the reference. The authors invert it: they take a fixed labelling, treat it as the system output, and vary which of the other available labellings they call gold. Any movement in the score is then attributable to reference choice alone, because nothing else changed.

This requires a dataset where the same items carry several independent, equally defensible references. They use the retained annotation record of a production multilingual PII benchmark covering six languages and 24,939 deduplicated prompts (roughly 4,100 per language). Normally such records are discarded; this one keeps four labellings: Annotator A, Annotator B, the aggregate shipped as gold, and an independent expert re-annotation of a sample. All four coexist only on the "audited subset" of 2,628 prompts, so every reference-choice measurement is restricted to it.

Scoring uses span F1 under two matching rules: Strict, where start, end and type must all match, and Relaxed, where type is ignored but exact boundaries are still required. Neither credits partial overlap, and the authors say so rather than claiming a decomposition they cannot make.

Uncertainty comes from a paired prompt-level bootstrap over 2,000 resamples with a fixed seed, stratified by language and resampling whole prompts because spans inside a prompt are not independent. The ranking question is tested as an interaction — the margin under one reference minus the margin under the other, computed inside the same resample — rather than by comparing two separate significance verdicts, a practice the authors explicitly reject by citing Gelman and Stern (2006).

Because the delivered gold often equals one of the outputs being scored, the authors report both a naive variant (what a practitioner would actually obtain) and a decontaminated variant that drops every prompt where the gold copies either labelling. The naive variant is the headline result; the decontaminated one checks how much of it is circular. All analyses are repeated under four duplicate-handling policies, a type-merge artefact and a duplicate defect in the record are reported in the appendices, and the authors note that the one property of the reviewer sample they can check — annotator agreement — shows the sample is easier than the full corpus in every language (pooled 84.5 vs 79.6 strict), so selection is more likely to understate the effect than inflate it.

Why This Matters

Impact on research. The paper reframes a benchmark score as measuring a system and a reference, and argues the second term can be larger than the margins a leaderboard reports. It gives evaluation researchers a cheap, annotation-free procedure for quantifying that term, and it makes a broader methodological point: a pipeline default is an evaluation decision whether or not anyone documented it. The authors also stress that the magnitude is a property of the pipeline, not a universal constant — what transfers is the procedure.

Real-world applications:

• Leaderboard and benchmark maintainers can report a reference-sensitivity band beside each score, so readers can tell whether a reported improvement is larger than the movement the benchmark's own reference choice produces.
• Annotation operations teams can audit their aggregation rule for positional bias by checking how often the gold reproduces each annotator's labelling exactly — a neutrality test the paper describes as two lines of code.
• Evaluation auditors and procurement reviewers comparing vendors on multilingual benchmarks can check whether per-language adjudication rates and agreement levels are reported, since a 16-point agreement spread across languages indicates the same protocol is not being applied uniformly.
• PII and compliance tooling evaluation, where an incorrect label conflates boundary convention with genuine error, benefits from knowing that most annotator disagreement is about where a span begins and ends rather than whether a string is PII.

Industry relevance. The dataset is a production benchmark from a commercial annotation pipeline, and the failure identified is a mundane implementation default rather than an exotic methodological flaw. Any organization that resolves annotation ties by position rather than adjudication is exposed to the same defect, and the fix costs nothing beyond retaining records most pipelines already produce and score against more than one reference.

Future Directions

• Test whether the effect transfers to model outputs. The scored outputs here are human annotation passes, not model predictions, because those are the only outputs the record supports. Whether reference sensitivity has the same magnitude or pattern for model outputs is left as an open empirical question — and annotators A and B share guidelines, vendor and protocol, so their outputs may be more correlated than independently developed systems.
• Replicate across other benchmarks and pipelines. The authors explicitly claim the method transfers but not the number, since a pipeline with a neutral aggregation rule would show a smaller band. Applying the procedure to other retained annotation records would establish how common positional aggregation defaults are.
• Establish what a neutral aggregation rule would yield. The decontamination subset (135 prompts, 13–30 per language) is too small to determine whether an annotator advantage persists once self-scoring is removed; a larger independently adjudicated sample would settle whether the interaction survives a neutral pipeline.
• Extend the measurement beyond boundary-agnostic classification and preference judgement. The paper distinguishes its setting — multilingual span extraction where a label error is a differing boundary rather than a substitution — from prior work, leaving open how reference sensitivity behaves for other task families and for partial-overlap matching, which this study's exact-boundary rules cannot decompose.

Target Audience

Evaluation researchers and benchmark maintainers who report or consume leaderboard scores; annotation and data-operations leads who design adjudication and aggregation rules; NLP practitioners interpreting multilingual span-extraction results, particularly in PII and compliance settings; and methodologists interested in how evaluation protocols themselves become objects of measurement. Readers without a statistics background will follow the argument but will need to take the bootstrap and interaction machinery on trust.

Authors’ abstract

A benchmark score compares a system output against a reference, and methodological attention falls almost entirely on the first term. We measure the second. The retained annotation record of a six-language benchmark for personally identifiable information contains two independent annotator labellings, the aggregate shipped as gold, and a reviewer gold from independent expert re-annotation of a sample. Using it, we hold the scored output fixed and exchange the reference. The score moves by 4.95 F1 points for one output and 2.00 for the other (95% CIs [3.23, 6.27] and [0.26, 3.37]), and by 7.55 in the worst language. The comparison between two outputs moves as well: the paired interaction between reference and system is +2.95 points (CI [+1.97, +3.87]), survives correction for multiple testing, and changes one language's margin outright. The cause is an undocumented aggregation default that usually kept one annotator when adjudication did not fire, making the shipped reference a partial copy of an output being scored. We report the resulting reference-sensitivity band, show how to compute one from any retained annotation record, and argue that the quantity belongs beside the score.