Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

AI summary

Overview

Research area: Natural language processing, specifically machine translation (MT) evaluation and the use of large language models (LLMs) as automatic quality annotators.

Technical level: Intermediate. The paper uses standard evaluation statistics (Krippendorff's alpha, F1) but explains them clearly and focuses on empirical comparison rather than new model architecture.

Scope: A large-scale comparison of LLM annotations against human annotations for two MT quality schemes — Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA) — across 70 language pairs, multiple domains, and both score-level and error-span-level agreement.

What This Paper Is About

Human evaluation of machine translation is the gold standard, but it is expensive, slow, and unreliable, while automatic metrics struggle to generalize and are hard to interpret. The authors ask whether LLMs can stand in for human annotators by measuring how closely LLM annotations agree with human annotations under two widely used annotation schemes. Their conclusion is that neither side looks reliable yet: LLMs match or beat humans in a few settings, but agreement stays low in most language pairs, domains, and annotation types.

Key Contributions

1. A head-to-head evaluation of LLM vs. human annotation agreement under two schemes. The authors compare LLM-human agreement with human-human agreement for MQM (fine-grained, taxonomy-based, score computed from error weights) and ESA (simpler span annotation with scores entered directly), covering both scores and error spans for each scheme.
2. Coverage beyond prior work. Results span a long-context test set of 70 language pairs, the WMT23 MQM data (English-to-German and Chinese-to-English, three independent annotation rounds per segment), and the WMT25 ESA data (14 language pairs across 5 domains). The MQM results cover the full MQM taxonomy.
3. A dual diagnosis of failure modes. The paper identifies what challenges humans (fine-grained MQM annotation and low-resource language pairs) separately from what challenges LLMs (minor errors, wrong language variants, and error span annotation, especially rare error categories).
4. A methodological caution and a new use case. The authors argue LLM-human agreement superiority over human-human agreement should not be read as proof LLMs can replace humans when the human baseline is weak, and they show LLMs can be used as diagnostic tools to flag problematic human annotation rounds.

Main Findings

• LLMs sometimes match or beat human annotators. For MQM scores on WMT23, Gemini 2.5 Flash-human agreement for Chinese-to-English is on par with human-human agreement, while human-human agreement is significantly better for English-to-German. For ESA scores, Gemini 2.5 Flash-human agreement exceeds human-human agreement for 4 of 12 language pairs in the Literary domain, and both Gemini 2.5 Flash and Qwen3.5-397B exceed human-human agreement for English-to-Japanese across all domains. On the Long-Context Web MQM test set, Gemini 2.5 Flash and GPT-OSS-120B agreement exceeds human-human agreement for most language pairs in the X-Y direction.

• But agreement is unreliable in most settings. Almost all MQM score agreement on WMT23 falls below the 0.667 Krippendorff's alpha threshold for tentative agreement. For ESA scores, agreement is below 0.667 for nearly all language pairs and domains. Agreement varies substantially across annotation schemes, language pairs, and domains, and across the four LLMs (Gemini 2.5 Flash, GPT-OSS-120B, DeepSeek-V3.1, Qwen3.5-397B).

• Minor errors are the hardest for LLMs. Human-human agreement on minor error counts substantially exceeds LLM-human agreement for both WMT23 language pairs, with the larger margin for Chinese-to-English. The authors note this is consequential: as MT systems improve, minor errors become the main factor distinguishing closely ranked systems.

• LLMs fail to penalize wrong language variants. For English-to-Egyptian Arabic, human-human agreement on ESA scores is above 0.90 for all domains, while LLM-human agreement is negative, indicating systematic disagreement. Manual inspection shows most translations (60%) are in the wrong Arabic variant (Modern Standard Arabic instead of Egyptian Arabic); humans scored these zero as instructed, while LLMs assigned high scores despite explicit prompt instructions.

• Error span annotation is the weakest area. On WMT23 MQM, the best human-human span overlap F1 is 0.51, with all four LLMs significantly lower (0.39 to 0.44). On WMT25 ESA, human-human span overlap F1 is 0.45 versus 0.30 to 0.31 for LLMs, and character overlap is 0.43 for humans versus 0.16 to 0.18 for LLMs.

• Rare error categories collapse to near-zero. Many lower-frequency MQM categories reach zero F1 for all LLMs, such as Currency format, while higher-frequency categories such as Mistranslation show the closest agreement (GPT-OSS-120B-human reaches 0.37, on par with human-human 0.37, on WMT23).

• Detecting error-free segments is hard for everyone. On WMT23 MQM, all agreements on the No error category remain at or below 0.15. On WMT25 ESA, human-human agreement is 0.47, while DeepSeek-V3.1-human agreement is only 0.05 — the model is consistently weak on this measure across test sets.

• ESA and MQM behave differently. For ESA scores, human-human agreement significantly exceeds LLM-human agreement for the majority of language pairs and domains for most LLMs, whereas MQM score results show a more mixed picture. The authors suggest ESA's simpler scheme, where annotators enter scores directly rather than having them calculated from weighted span counts, may reduce disagreement.

• LLM-generated span indices were broken. At least 80% of spans extracted from LLM-provided indices did not match the corresponding span text in the LLM output across MQM and ESA data sets. The authors re-aligned spans by string matching before computing any span metrics.

• Humans struggle too, and LLMs can reveal it. Human-human MQM score agreement is negative or close to zero for English-to-Hindi and English-to-Thai on the Long-Context Web MQM set, and Gemini 2.5 Flash agrees much more with one human round than the other — evidence that one round has quality problems. Human-human agreement is also remarkably low for English-to-Maasai, consistent with its low-resource status.

Methodology in Plain English

The authors gathered three sources of human quality annotations: the WMT23 MQM data for two language pairs with three independent annotation rounds per segment; their own internal "Long-Context Web MQM" test set of multi-paragraph web text translated by human translators and a mix of neural and LLM-based MT systems; and the WMT25 ESA data covering 14 language pairs across 5 domains (the speech domain was excluded due to YouTube access restrictions). Their internal MQM set was annotated by professional annotators through a vendor, with two independent rounds and different annotators per round.

Four LLMs — Gemini 2.5 Flash, GPT-OSS-120B, DeepSeek-V3.1, and Qwen3.5-397B — were given adapted versions of the human guidelines and asked to produce annotations. All inference used temperature = 0 and top_p = 1; open-weight models were self-hosted and Gemini 2.5 Flash ran through Vertex AI in April–May 2026. Prompting was zero-shot because few-shot examples did not improve agreement. Open-weight models had reasoning disabled except GPT-OSS-120B, which ran with reasoning_effort = low, and output token limits were set per task (4096 for WMT23 MQM and WMT25 ESA, 8192 for Long-Context Web MQM, raised to 16384 for DeepSeek-V3.1). Guided decoding forced the required output format. Segments were dropped if any model failed to return a parseable annotation: 1.1% of WMT23 MQM, 5.1% of Long-Context Web MQM, and 7.3% of WMT25 ESA segments.

To measure agreement, the authors treated each human annotation round as an independent annotator, then used a leave-one-out setup: each human round was removed in turn and replaced by the LLM's annotation, and agreement was computed within the resulting group, averaging over all such groups. Scores and error counts were measured with Krippendorff's alpha using interval distance, chosen because it accounts for chance agreement, handles multiple annotations, and measures agreement on absolute scores. MQM scores followed the standard formula weighting major errors by 5, minor errors by 1, and punctuation errors by 0.1. Spans were evaluated with F1-based metrics: character overlap, span overlap (at least one character), category/severity-specific span overlap, and No error detection. Statistical significance used two-sided paired bootstrap resampling clustered by source segment with 10,000 iterations at p < 0.05, plus the Benjamini–Hochberg false discovery rate procedure at 0.05 for tables comparing many language pairs.

Why This Matters

Impact on research. The paper reframes the "can LLMs replace human annotators" question as a two-sided reliability problem. It points out that a low human baseline makes agreement comparisons unreliable, and it gives a concrete negative result for span-level annotation that lines up with the WMT25 metrics shared task, where the best autoraters reached only 9–13% character-level F1 versus 47% for human annotators. It also documents a practical engineering failure — LLM span indices that do not match their own output text in at least 80% of cases — that other researchers will need to handle.

Real-world applications.

• Machine translation quality assurance pipelines that need to decide when an LLM score can be trusted and when human review is required.
• Localization workflows, where the English-to-Egyptian Arabic finding shows LLMs can miss wrong-variant output that humans catch, a risk for locale-specific products.
• Rapid MT model iteration, where cheap LLM scoring is attractive but the paper shows agreement is not yet stable across domains or language pairs.
• Annotation vendor and platform quality control, where LLM agreement with individual human rounds can flag problematic annotators or rounds for review.

Industry relevance. The work comes from Apple and uses internal, professionally annotated long-context web data, reflecting production-grade evaluation concerns rather than only academic benchmarks. Its findings argue for careful validation before deploying LLM evaluators per domain and language pair, and it flags bias amplification — LLMs favored the Arabic variant with more training-data representation instead of penalizing the wrong variant as instructed. Cost and time are central: human evaluation is described as expensive and time-consuming, and the cited related work shows AI-assisted annotation improving recall (1.6 spans per segment versus 0.5 unassisted) and cutting annotation budget by 24%, or reducing annotation time by up to 12% when humans revise automatic MQM annotations.

Future Directions

1. Establish realistic reliability thresholds. The authors note the widely used 0.667 alpha cutoff is not well-established for fine-grained MT annotation and may be unrealistic given the task's difficulty and subjectivity; thresholds should be studied as a function of text length, domain, and number of annotations.
2. Diagnose the causes of low agreement. The paper calls for research into factors behind low agreement for both humans and LLMs, including annotator fatigue for humans and attention dispersion for LLMs, especially for long-context documents.
3. Test more models, prompts, and span-correction methods. The current results use four LLMs, a single zero-shot prompt, and a simple string-matching heuristic for span correction; the authors call for ablations across prompting strategies and better span localization.
4. Build human-LLM collaborative pipelines and refine the statistics. The paper argues a combined framework leveraging complementary strengths is the most promising path, and notes that significance testing examined only the null hypothesis of equal agreement — a more refined approach should also establish equivalence. Broader data coverage and resolving the data-contamination question (WMT23 MQM may be in training data; the internal Long-Context Web MQM set has never been publicly released) remain open.

Target Audience

MT researchers and practitioners working on evaluation, especially those comparing LLM annotators to human annotators; localization and language quality engineers deciding when automatic annotation is trustworthy; and NLP researchers studying LLM-as-judge reliability, annotation methodology, and human-AI collaborative workflows. Readers need basic familiarity with MT evaluation metrics and inter-annotator agreement statistics to follow the tables, though the paper explains its choices clearly enough for a motivated newcomer.

Authors’ abstract

Large Language Models (LLMs) are considered to be a more efficient and cost-effective alternative to human judgment for Machine Translation (MT) evaluation. With MT evaluation spanning a large number of language pairs, domains and levels of annotation granularity, LLMs must be thoroughly evaluated across these dimensions before being reliably used as alternatives to human evaluation. In this paper, we evaluate the performance of LLMs for two prominent MT quality evaluation schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA) by comparing their agreement with human annotators. We present results on a long-context test set of 70 language pairs and the publicly available WMT23 and WMT25 data, investigating both score and error span annotation agreement across a variety of language pairs and domains. Our results show that while LLM agreement with human annotators exceeds agreement between human annotators for some evaluation tasks, both vary substantially across annotation schemes, language pairs and domains and remain unreliable for most settings. Furthermore, we identify challenges facing both human and LLM annotators: humans are particularly challenged by fine-grained MQM annotations and low-resource language pairs, while LLMs struggle with minor errors, wrong language variants and error span annotation. Our results highlight the potential for improvement for both human and LLM annotation performance, possibly through human-LLM collaborative annotation pipelines that address the reliability issues identified in this work.