The Pulse
Apple Study Finds LLM Translation Reviews Still Unreliable
An Apple study compared four language models with human annotators on machine-translation quality tasks, including error labels and span detection. The models struggled with minor errors, error locations and language variants, though their
AI.info Team · Written with AI ·
LLM error spans trail human agreement
On a WMT25 test of translation-error spans, human annotators reached a span-overlap F1 score of 0.45; four large language models scored between 0.30 and 0.31 on the same measure. The figures come from an Apple study submitted to arXiv on October 7, 2026, that tests whether language models can replace people in judging machine translations. The researchers compared model annotations with human judgments across language pairs, translation domains and two evaluation schemes.
The study does not measure how well the models translate. It measures how consistently they identify translation problems and rate their severity, a distinction that matters when teams use automated judges to compare translation systems. The authors tested Gemini 2.5 Flash, GPT-OSS-120B, DeepSeek-V3.1 and Qwen3.5-397B against human annotations in Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA).
Four models, 70 language pairs
Apple’s experiments combine the public WMT23 MQM and WMT25 ESA datasets with an internal, long-context MQM set covering 70 language pairs. WMT25 covers 14 language pairs across five domains: dialogue, education, literary text, news and social media. The long-context set uses multi-paragraph web text, with human translations and outputs from neural and LLM-based translation systems.
The methods ask annotators to do more than assign a single quality score. MQM requires them to mark errors, classify each one and assign a severity; the score is then calculated from those annotations. ESA asks annotators to identify error spans and provide a score directly. The researchers measured agreement with Krippendorff’s alpha for scores and F1 for error spans, comparing model-human agreement with agreement between human annotators.
Minor mistakes and local variants trip up judges
Models had particular trouble finding minor errors, which can distinguish closely ranked translation systems once major mistakes are uncommon. In the WMT25 ESA character-overlap results, human annotators’ F1 was 0.43, compared with 0.16 to 0.18 for the four models. This character-overlap measure is distinct from span-overlap F1, which was 0.45 for humans and 0.30 to 0.31 for the models. Agreement also fell across less common error types, including omissions and missing spans.
A language-variant test exposed a different failure. In the English-to-Egyptian Arabic data, researchers estimated that 60% of translations used Modern Standard Arabic rather than the requested Egyptian variant. Human annotators marked those outputs as wrong-language translations and gave them zero scores; the LLMs often gave them high scores despite instructions to penalize the mismatch.
The models also struggled to return accurate locations for the errors they identified. At least 80% of spans extracted from the models’ submitted character indices failed to match the span text in the models’ own output, so the researchers had to realign the text spans before comparing annotations.
Better than people sometimes still means unreliable
Apple’s results are not uniformly unfavorable to LLMs: model-human agreement matched or exceeded human-human agreement on some tasks, language pairs and domains. But the results changed substantially with the model, evaluation scheme and language pair. The paper says agreement remained below the 0.667 level the researchers use as a threshold for tentative conclusions in nearly all ESA language-pair and domain combinations.
However, our findings suggest that LLMs are not yet a reliable substitute for human MT evaluation.
Hala Almaghout, Christian Federmann and Qin Gao, the paper’s authors
The authors argue that translation teams should test agreement for the specific languages and domains they plan to evaluate, rather than assuming a model’s score transfers across settings. Their findings also complicate the human baseline: people themselves disagree more on fine-grained MQM labels and low-resource language pairs. On the study’s WMT25 span-overlap measure, human agreement reached 0.45, while each tested model scored between 0.30 and 0.31.