Machine Translation, Alignment, and Meaning Preservation

In October 2017 a Palestinian construction worker in Beitar Illit posted a caption in Arabic. It said “good morning”. Facebook's automatic translation rendered it as “attack them” in Hebrew and “hurt them” in English. Israeli police read the translation. No Arabic-speaking officer read the original post before the arrest. The man was detained and released after a few hours of questioning. The Judea and Samaria District police confirmed that the mistranslation had caused the arrest.

Nothing in the Hebrew or the English output looked broken. It was ordinary, grammatical, confident text. It inverted a greeting into a threat. Facebook apologised on the record, in a statement to Gizmodo from Necip Fazil Ayan, an engineering manager in its language technologies group: “Unfortunately, our translation systems made an error last week that misinterpreted what this individual posted. Even though our translations are getting better each day, mistakes like these might happen from time to time and we've taken steps to address this particular issue. We apologize to him and his family for the mistake and the disruption this caused.”

Translation quality therefore begins with preserving meaning for the target audience and domain. Natural target-language wording matters. But it cannot compensate for an altered condition. It is exactly what makes an altered condition hard to notice.

The model and the product surround each other with data, constraints, and review.

Step 5 is not a matter of taste in every jurisdiction. In the US it is codified, at 45 CFR 92.201(c)(3): “If a covered entity uses machine translation when the underlying text is critical to the rights, benefits, or meaningful access of an individual with limited English proficiency, when accuracy is essential, or when the source documents or materials contain complex, non-literal or technical language, the translation must be reviewed by a qualified human translator.” That text comes from the HHS Section 1557 final rule “Nondiscrimination in Health Programs and Activities”, published on 6 May 2024 and effective on 5 July 2024.

The obligation is current. A Federal Register notice of 2 June 2026 records that a court vacated only the rule's gender-identity provisions, and that “[t]he other provisions of the Section 1557 Rule remain in force.” For US covered entities the review box in the diagram has a regulation number, an effective date, and a defined trigger: consequential text, essential accuracy, or complex, non-literal or technical language.

One overall score conceals errors with very different consequences. A real case separates the two dimensions better than a definition can.

A Kansas Highway Patrol trooper obtained consent to search a car through Google Translate. His question, “Can I search the car”, came out in Spanish as “¿Puedo buscar el auto?”. That sentence is fluent. Run back into English by the same system it becomes “Can I find the car” — a different question, and not the one consent was needed for. On 4 June 2018 Judge Carlos Murguia, of the US District Court for the District of Kansas, granted Omar Cruz-Zamora's motion to suppress. The court held that the government had not met its burden to show that consent was “unequivocal and specific and freely and intelligently given.” The order also records that “Gardner noticed several other instances in the video where Google Translate provided a literal but nonsensical translation.”

Adequacy failed while fluency held, and the evidence was suppressed. That is why the four dimensions below are rated separately rather than summed.

Create dedicated tests for phenomena whose consequences are not proportional to token count. Such a test set exists for one of the bullets below, and its numbers are public.

WinoMT is 3,888 sentences, balanced between male and female and between stereotypical and anti-stereotypical role assignments. Each one states the gender in English. Stanovsky and colleagues ran it in 2019 against four commercial systems and two state-of-the-art academic models, to see which of them carried that gender through. Google Translate reached 53.1% gender accuracy into Spanish, 39.6% into Italian and 37.7% into Russian. Every system except Microsoft Translator on German scored significantly better on male roles. Their conclusion is one sentence: “Ultimately, our analyses indicate that all tested MT systems are indeed gender biased.”

A separate group reached the same place from another direction. Prates and colleagues found the same masculine-default behaviour in Google Translate across 12 gender-neutral languages, using the US Bureau of Labor Statistics occupation list. A corpus-average metric on the same systems shows none of this.

BLEU aggregates modified n-gram precision with a brevity penalty across a corpus. It supports repeatable comparison when tokenization, references, and test sets are fixed.

It can underrate valid paraphrases and overlook meaning errors with high word overlap. Combine it with stronger learned metrics where validated, terminology tests, contrastive examples, human adequacy review, and post-edit evidence.

BLEU was proposed at ACL 2002, by Papineni and colleagues. Sixteen years later Matt Post showed at WMT 2018 that the number does not travel. “BLEU is in fact a parameterized metric whose values can vary wildly with changes to these parameters”. Those parameters “are often not reported or are hard to find, and consequently, BLEU scores between papers cannot be directly compared”. He quantified the spread: “differences as high as 1.8 between commonly used configurations”. He named the cause too. “The main culprit is different tokenization and normalization schemes applied to the reference.”

The plan should reflect language pairs, domains, and user consequences rather than one global score. Steps 1 and 4 once decided a widely reported claim, and the only variable that moved was the evaluation unit.

On 15 March 2018 a 24-author Microsoft paper claimed that its Chinese–English system had reached human parity on the WMT 2017 news task. Läubli and colleagues, at the University of Zurich and the University of Edinburgh, re-ran the comparison with professional translators and reported it at EMNLP 2018. Judged sentence by sentence, the parity result held: adequacy x=86, n=189, p=.244. Judged as whole documents, by the same raters, preference for the machine output fell from 50% to 37%. The preference for human translation became statistically significant — x=104, n=178, p<.05.

Their summary is the design instruction: “In a pairwise ranking experiment, human raters assessing adequacy and fluency show a stronger preference for human over machine translation when evaluating documents as compared to isolated sentences.” Freeze the test set at the unit your users actually read. Otherwise the release decision is being made on a different question.

Select twenty safety, medical, legal, or financial sentences with negation, modality, numbers, and named entities. Ask two qualified bilingual reviewers to annotate adequacy and severity independently. Compare automatic scores with the severe-error labels. Identify which metric and average would have failed to block a dangerous release.

Two published emergency-medicine audits show the shape of the answer. The first appeared in JAMA Internal Medicine in 2019. Khoong and colleagues scored 647 sentences drawn from 100 emergency-department discharge instruction sets: 594 (92%) were accurate in Spanish and 522 (81%) in Chinese. But 15 of the 647 Spanish sentences (2%) and 50 of the 647 Chinese sentences (8%) carried potential for clinically significant harm. A corpus average of 92% does not tell you which fifteen.

The second went wider, in the Journal of General Internal Medicine in 2021. Taira and colleagues had 20 native speakers rate 400 translations of 20 common discharge phrases across seven languages. Overall meaning was retained in 330 of 400 (82.5%). Per language, “Spanish had the highest accuracy rate (94%), followed by Tagalog (90%), Korean (82.5%), Chinese (81.7%), Farsi (67.5%), and Armenian (55%).” Mean fluency across the same set scored 4.2 of 5. Fluent at 4.2 out of 5, meaning-preserving 55% of the time in Armenian: that split is precisely what your twenty sentences are meant to expose.