Human Evaluation, Rubrics, and Annotation Reliability

One reviewer may prefer brevity, another factual detail, and a third a friendly tone. Their disagreement can reflect an underspecified task rather than careless work.

Human evaluation starts by defining the construct and separating properties that can conflict. That choice is not cosmetic. Rescore the same outputs under a different protocol and the leaderboard can come back in a different order.

That is what happened in 2021 to the top WMT 2020 systems in two language pairs. Professional translators replaced crowd workers. MQM error annotation replaced a rating. Full document context replaced the isolated segment. Freitag and colleagues published the rescoring. Nothing about the systems had changed. The protocol had.

Two pairs of published studies did what a pilot is supposed to do: they changed one part of the protocol and watched the verdict move.

Format changes how hard the task is and how raters read it. The four are not interchangeable views of one underlying quality. Absolute ratings, pairwise preferences and rankings ask for a verdict. Error annotation asks what is wrong, where, and how badly.

The WMT rescoring is the demonstration that the choice decides the outcome. Swap the crowd rating for MQM error annotation by professional translators reading full documents, and the same top WMT 2020 systems in two language pairs come back in a substantially different ranking. Freitag and colleagues, 2021. The unit alone does part of that work: in 2018 Läubli and colleagues found the preference for human over machine translation was stronger when raters judged whole documents rather than isolated sentences.

Choose the format that matches the claim you intend to make, and report which one you used. A reader cannot reconstruct it from the scores.

A rubric connects abstract goals to examples, and it has to say what happens when the examples run out. Two diabetic-retinopathy studies mark the two ends of that decision. One resolved grader disagreement by simple majority over 7–8 ophthalmologist grades per image: Gulshan and colleagues, JAMA, 2016. The other re-graded the same task by adjudication to full consensus among retinal specialists: Krause and colleagues, Ophthalmology, 2018.

Both are defensible rules. They are not the same rule, and the second changed what the measured algorithm performance could reach.

Blind model identity where possible, randomize output order, balance left–right presentation, control context length, and avoid revealing scores from other raters. Track fatigue and learning over the session.

Oncology has already written those precautions into a rulebook. Tumour assessments generally should be verified by central reviewers blinded to study treatments — that is the FDA's December 2018 endpoints guidance. The same passage offers a cheaper route: “Alternatively, a random sample-based blinded central review auditing approach could be used with a detailed auditing plan prespecified including a strategy to detect potential assessment bias.”

How often does the independent re-read actually change the answer? Across 49 randomized Roche oncology trials and over 32,000 patients, blinded independent central review and local evaluation reached the same statistical inference in 87% of progression-free-survival comparisons, with a BICR/LE hazard-ratio ratio of 1.044 in double-blind studies. Lian and colleagues published that in The Oncologist in 2024. The point is not that blinding is usually redundant. It is that the disagreement rate is a measurable quantity, and someone measured it.

For sensitive topics, recruit raters with relevant language, cultural, or professional expertise and compensate them appropriately. Convenience annotators may not represent affected users.

Treat the protocol as an experiment, and write down in advance what oncology writes into its guidance: who re-reads, blinded to what, and what fraction is audited. The FDA's December 2018 endpoints guidance asks that tumour assessments generally be verified by central reviewers blinded to study treatments, with a prespecified random-sample audit as the stated alternative. What the second reading buys has been counted: the same statistical inference in 87% of progression-free-survival comparisons, and a BICR/LE hazard-ratio ratio of 1.044 in double-blind studies, across 49 randomized Roche oncology trials and over 32,000 patients. Lian and colleagues, The Oncologist, 2024.

Step five is where the reference standard is decided rather than discovered. A simple majority over 7–8 ophthalmologist grades per image and adjudication to full consensus among retinal specialists are two answers to one disagreement. Gulshan and colleagues took the first route in 2016. Krause and colleagues took the second in 2018, on the same task, and the adjudicated grades allowed substantial improvements in algorithm performance. Choose the rule before the ratings arrive, and report it beside the result.