NLP Metrics for Extraction, QA, Translation, and Summarization

The answers “Rome,” “the city of Rome,” and “Rome, Italy” may all be correct. A fluent paragraph can also contradict its source in a single word. Text evaluation has to decide which variations are acceptable and which meaning differences are harmful. Automatic metrics are proxies. Tokenization, references, normalization and task assumptions all shape what they report.

NIST ran an official machine translation evaluation in 2005. Its own results document said BLEU “has been found to generally rank systems in the same order as human assessments”. Callison-Burch and colleagues went back to the Arabic-to-English track and checked that claim against the human scores. “In particular, the entry that was ranked 1st in the human evaluation was ranked 6th by Bleu.” Across the seven entries the metric accounted for almost none of the human judgments: R² = 0.14 for adequacy, R² = 0.002 for fluency with the outlier included. Their 2006 paper also showed that BLEU “vastly underestimates” a rule-based system, Systran, which three judges rated on 300 French sentences. The metric and the judges were not ranking the same thing.

The metric is not even one number. Post showed that in 2018. “BLEU is in fact a parameterized metric whose values can vary wildly with changes to these parameters.” He measured “differences as high as 1.8 between commonly used configurations”, and blamed the tokenization and normalization applied to the reference. Two papers can report BLEU and not be reporting the same measurement.

Task contracts determine what counts as correct. Changing the contract moves the achievable score more than changing the model does. SQuAD 2.0 changed one in 2018. Its authors took the original reading-comprehension set, added 53,775 crowd-written unanswerable questions, and collected an average of 4.8 human answers per question. Now a system had to decide whether the passage supported an answer at all. The results moved: “The best model, DocQA + ELMo, achieves only 66.3 F1 on the test set, 23.2 points lower than the human accuracy of 89.5 F1.” A system that always abstains scored 48.9 F1 on the same task. That is the floor any new number on that leaderboard has to be read against. The same model looks strong or weak depending on which contract scores it.

Each metric means one narrow thing. Each was certified on a much smaller body of evidence than its later use implies.

BLEU's warrant is one section of one paper. In 2002 Papineni and colleagues compared the metric against human ratings of 5 machine-translation systems on 250 Chinese-English sentence pairs. Two panels of 10 judges did the rating, and “None of the human judges was a professional translator”. Their conclusion: “The high correlation coefficient of 0.99 indicates that BLEU tracks human judgment well.” With the bilingual panel it was 0.96. Three years later NIST made BLEU the sole official measure of its 2005 MT evaluation. The same document warned that BLEU “does not have the power to distinguish subtle differences in high quality translations.”

ROUGE travelled the same road on a shorter timetable. Chin-Yew Lin validated the package on DUC 2001–2003 and published it in 2004. Pearson correlations with human judgments reached 0.99 for ROUGE-2 on DUC 2002 100-word single-document summaries. For ROUGE-1 on DUC 2001 they fell to 0.76. He concluded that a “correlation of high 90%” was hard to achieve for multi-document summarization. NIST had already announced that DUC 2004 tasks 1–4 would be graded “solely by means of ROUGE (ISI's Recall-Oriented Understudy for Gisting Evaluation, alias RED) automatic (n-gram) matching”. The caveat and the adoption are separated by weeks.

A reference set is a claim about what is correct. The claim can be false. CoNLL-2003 is the standard English named-entity benchmark. The CrossWeigh audit went through it in 2019: “We are able to identify label mistakes in about 5.38% test sentences, which is a significant ratio considering that the state-of-the-art test F1 score is already around 93%.” Four years later Rücker and Akbik corrected 7.0% of all English CoNLL-03 labels. On the corrected set models reached 97.1% F1. The share of correct predictions that the old annotation had been counting as errors fell from 47% to 6%. Nearly half of what a leading system was being penalised for was the reference, not the model. A span F1 near the ceiling is partly a measurement of the annotation.

Ask raters to judge one property at a time: adequacy, fluency, factual support, helpfulness, style, or safety. Pairwise preference can be easier than assigning absolute scores. It still depends on ordering, interface, and reviewer expertise.

Two protocols run over the same translations need not produce the same winner. In 2021 Freitag and colleagues at Google Research re-rated the top WMT 2020 submissions in English–German and Chinese–English. Professional translators did the rating, using MQM, over more than 100k segments. The ranking that came back differed substantially from the one WMT's crowd workers had produced. In English–German the human translation Human-A came 2nd under MQM and 4th under the WMT crowd score. The WMT21 Metrics Shared Task report drew the consequence: “In both language pairs, DA ranks the human translations below many MT systems, demonstrating again that expert-based evaluation is needed to generate a reliable ground truth for metric development for high quality language pairs.” That shared task adopted MQM as its golden standard.

Measure agreement, adjudicate ambiguous cases, and sample difficult slices. A majority vote does not automatically create objective truth. The protocol is part of the result, not a wrapper around it.

SQuAD shipped with a human number attached in 2016. Its authors reported a logistic regression model at 51.0% F1 against human performance of 86.8% on the same questions. They read that gap as “a good challenge problem for future research”. A rubric without a human baseline cannot say whether a score sits near the ceiling.

Combine inexpensive coverage with targeted semantic review. Find out what a metric delta is worth before shipping on it. In 2021 Kocmi and colleagues at Microsoft compared metrics against 2.3 million sentence-level human judgements covering 4,380 systems. BLEU agreed with the human verdict on 74.6% of 3,344 system pairs. COMET agreed on 83.4%. Their first best-practice recommendation is a single sentence: “Do not use BLEU, it is inferior to other metrics, and it has been overused.” Mathur and colleagues had already shown in 2020 that even for BLEU differences of 3 to 5 points, about a quarter of system pairs are judged by humans to be of similar quality. A one-point improvement over a published number is not a result. It is a hypothesis awaiting a human protocol and a significance test.