NLP Evaluation: Metrics, Human Judgment, and Behavioral Tests

Different outputs produce different units of error and different ways of aggregating them. Two of these families have had their default metric measured and found wanting, in public, with the size of the evidence recorded.

Generation. BLEU was tested in 2021 against 2.3 million sentence-level human judgements covering 4,380 systems. Kocmi and colleagues at Microsoft ran that comparison, and their abstract ends on the deployment consequence: “Lastly, we show that the sole use of BLEU impeded the development of improved models leading to bad deployment decisions.” The WMT organisers reached the same verdict independently the following year, and put it in the title of their metrics report: “Stop Using BLEU – Neural Metrics Are Better and More Robust”. A reference-overlap score is not neutral bookkeeping. It selects which models get shipped.

Retrieval. “Incomplete judgments” is a measured limit, not a caveat. In 2004 Buckley and Voorhees tested the Cranfield assumption on three TREC collections and 17 progressively reduced judgment sets, thinned down to 1 percent: “We show that current evaluation measures are not robust to substantially incomplete relevance judgments.” MAP, P(10) and R-precision moved as the qrels thinned, while their new bpref-10 measure kept Kendall tau above 0.9 down to the 25 percent qrels on TREC-8. The same paper prices the instrument. On TREC-8, a MAP difference of 0.040 — 9.7 percent of the best run's score — is needed for 95 percent confidence in the conclusion. Leaderboard decimals imply something far finer than that.

The modern form is stranger. Arabzadeh and colleagues pooled the top results of MS MARCO leaderboard runs over 500-plus queries and put them to crowd assessors. The assessors preferred a neural ranker's top result to the single labelled relevant item often enough to make the ranker score “better than perfect” under MRR.

Higher layers should not erase diagnostic component evidence below them.

The behavioral and slice layer earns its place because a single aggregate can average a population into invisibility. Five commercial speech recognisers — Amazon, Apple, Google, IBM, Microsoft — were measured on 19.8 hours of interviews with 42 white and 73 black speakers, matched on age and gender. Koenecke and colleagues published the result in PNAS on 23 March 2020: “We found that all five ASR systems exhibited substantial racial disparities, with an average word error rate (WER) of 0.35 for black speakers compared with 0.19 for white speakers.” The gap persisted on identical phrases, so it is not an artefact of what the two groups happened to say. A vendor reporting one pooled WER over that same audio would report a number that is arithmetically correct and describes no user.

The slice below the slice is linguistic. Martin and Tang worked through 100 hours of spoken African American English in 2020 and traced part of the mechanism: habitual “be” and the words around it are more error prone than non-habitual “be”. That is the level at which a fix can be specified. The aggregate never reaches it.

A compact suite can expose failures that thousands of average examples dilute. HANS is one. Built in 2019, it is a controlled test set constructed so that three syntactic shortcuts — lexical overlap, subsequence, constituent — are guaranteed to give the wrong answer. McCoy and colleagues state the result plainly: “We find that models trained on MNLI, including BERT, a state-of-the-art model, perform very poorly on HANS, suggesting that they have indeed adopted these heuristics.” The suite is small and adversarial by construction. It separates two systems that the held-out set scores identically.

Gardner and 25 co-authors generalised the method in 2020. They built contrast sets for 10 diverse NLP datasets by minimally perturbing test items — changing as little as possible while changing the correct label — and measured performance drops of up to 25 percent against the original test sets. The perturbed items were near-neighbours of examples the models had already answered correctly.

A vague instruction such as “rate quality from one to five” mixes support, completeness, clarity, style, and preference. Raters may reward fluency, or infer facts from memory instead of checking the supplied source.

Define separate criteria, anchor examples, qualifications, source access, blinded order, randomization, repeated items, adjudication, and agreement. Report uncertainty and disagreement rather than averaging them away. How much the procedure decides has been measured, twice, on the same outputs.

Microsoft claimed human parity for Chinese–English translation. In 2018 Läubli and colleagues re-ran the evaluation and changed one thing: the unit of judgement. They used 55 documents and 2x120 sentences from the WMT 2017 test set, rated by professional translators recruited on ProZ with an average of 13.7 years of experience. Judged as isolated sentences for adequacy, machine and human translation were statistically indistinguishable (x=86, n=189, p=.244). Judged as whole documents, the human translation was significantly preferred (x=104, n=178, p<.05), and preference for the machine output fell from 50 to 37 percent. Their conclusions: “Raters showed a markedly stronger preference for human translations when evaluating at the level of documents, as compared to an evaluation of single, isolated sentences.” Parity was a property of the evaluation window, not of the system. Toral and colleagues reached the same conclusion independently that year, by varying rater expertise, inter-sentential context and source-side translationese.

The second lever is the raters, and it moves the ranking too. Freitag and colleagues rescored the top WMT 2020 systems in two language pairs for Transactions of the ACL. The annotations came from professional translators “with access to full document context”. That produced “a substantially different ranking of evaluated systems from the one established by the WMT crowd workers”. It exhibited “a clear preference for human over machine output”. The same abstract holds the more awkward result: “automatic metrics based on pre-trained embeddings can outperform human crowd workers”.

The program should make claims, evidence, and decision thresholds traceable.

“Recall is 90 percent” is incomplete without the class, population, threshold, sample size, time period, and uncertainty. “NDCG improved” needs candidate depth, judgments, query mix, and index version.

The missing denominator has been priced. Card and colleagues measured the statistical power of standard NLP experiments in 2020: “For machine translation, we find that typical test sets of 2000 sentences have approximately 75% power to detect differences of 1 BLEU point.” Power of 75 percent means a real one-BLEU improvement goes unseen a quarter of the time, on the sample size the field treats as standard. The same paper reports several GLUE tasks too small for most comparisons to state of the art to be adequately powered. Mathur and colleagues showed the complementary failure that year: metric-versus-human correlations are driven by outlier systems, and they quantify the resulting type I and type II errors in pairwise system ranking.

A useful report pairs aggregates with slices, calibration, failure examples, cost, latency, coverage, and known limitations. It also states which decisions the evidence does not justify, including the differences the test set was never large enough to resolve.

Choose a claim such as “the summarizer is more accurate” or “the multilingual router is fair.” Define the construct, population, comparison, metrics, human criteria, slices, uncertainty, and release threshold.

Then list three ways the proposed measures could be gamed or misinterpreted. A hypothesis-only shortcut, a shifted unit of judgement and a swapped candidate order are three that have already been demonstrated on published benchmarks. Add one product outcome that could contradict the benchmark result.