Multilingual and Cross-Lingual NLP

A product can combine these strategies rather than selecting one architecture for every language.

The first column is not hypothetical, and it has a size. One shared model already covers 200 languages, evaluated over 40,000 translation directions on the FLORES-200 benchmark, with the XSTS human metric and a toxicity detector alongside it. It is a single Sparsely Gated Mixture of Experts, published in Nature in 2024 by the NLLB Team. The abstract states the result: “Compared with the previous state-of-the-art models, our model achieves an average of 44% improvement in translation quality as measured by BLEU.” Meta's own announcement of NLLB-200, in July 2022, adds a second number: for some African and Indian-based languages the translations were more than 70% more accurate.

Read those two numbers together and the column's third and fourth rows appear on their own. The 44% is an average over 40,000 directions. The more-than-70% figure exists precisely because per-language gains are not the average. One shared model can carry 200 languages and still owe every one of them a separate measurement. That is what capacity competition and unequal language quality mean, once they stop being adjectives.

More data in one language can influence every later layer. Two of these layers have been audited rather than argued about, and both audits came back with counts.

Collection and rights. Somebody sat down and read the corpora: 205 language-specific ones, drawn from CCAligned, ParaCrawl, WikiMatrix, OSCAR and mC4, hand-audited by Kreutzer and 50 co-authors and published in Transactions of the ACL in 2022. The abstract does not soften the finding: “Lower-resource corpora have systematic issues: At least 15 corpora have no usable text, and a significant fraction contains less than 50% sentences of acceptable quality.” Alongside the unusable corpora came widespread mislabelling and nonstandard language codes. Corpora, that is, whose language tag is itself wrong. The pattern survives into datasets built with that audit in mind. MADLAD-400 trained a document-level LangID model on 498 languages, then ran a manual audit of its own — “we discard 79 of the languages from our preliminary dataset”, its authors write — and released 419.

Tokenization. Vocabulary allocation has a measured price. “The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases.” That is Petrov and colleagues, at NeurIPS 2023. The spread does not disappear with a different vocabulary design: they measured over 4 times even for character- and byte-level models. A second group then priced it. Analysing OpenAI's API across 22 typologically diverse languages, Ahia and colleagues found some languages needing 5 times as many tokens as others, and for most mid-resourced Indic languages in non-Latin scripts “close to a 5x increase in cost compared to English”.

The same sentence, sent by two users, can occupy up to fifteen times the sequence and cost close to five times the money. Sequence length, context budget and price are one quantity seen from three sides. The tokenizer sets all three before the model has read a word.

Multilingual pretraining can align patterns across languages through shared parameters, subwords, parallel evidence, and similar structures; a classifier fine-tuned in one language may transfer to another without target labels.

Performance depends on language similarity, script, data amount, tokenizer efficiency, domain, task, and model capacity; a multilingual average should never stand in for evidence on a specific language.

The largest test of that opportunity arrived in 2020. XLM-R was a masked language model trained “on one hundred languages, using more than two terabytes of filtered CommonCrawl data”. It beat multilingual BERT by “+14.6% average accuracy on XNLI, +13% average F1 score on MLQA, and +2.4% F1 score on NER”. Over previous XLM models it improved “15.7% in XNLI accuracy for Swahili and 11.4% for Urdu”. Then the same paper names the ceiling. Past a certain number of languages at fixed capacity, “the overall performance on monolingual and cross-lingual benchmarks degrades”. Conneau and colleagues call that trade-off the curse of multilinguality.

Locale and community use can matter as much as the nominal language name, and where one of these phenomena has been given its own benchmark, the gap it hides becomes a number.

Code-switching got its first evaluation benchmark in 2020. GLUECoS spans six tasks in English-Hindi and English-Spanish. On its new code-switched English-Hindi natural language inference task — a two-way classification, so chance is 50% — multilingual BERT scored 61.09 accuracy, and a version fine-tuned on code-switched data reached 63.1. Khanuja and her co-authors do not dress that up in §5: “Modified mBERT performs only a little over chance, which shows that we are still in the early days of solving NLI for code-switched languages, and also indicates that our models are far from truly being able to understand code-switched language.”

Independently the same year, Aguilar and colleagues released LinCE, centralising code-switching corpora across Spanish-English, Nepali-English, Hindi-English and Modern Standard Arabic-Egyptian Arabic over four tasks, with baseline scores for LSTM, ELMo and multilingual BERT. A system reported as strong in English and strong in Hindi tells you nothing about the message that is both.

A product can launch languages under different scopes without pretending they have equal support. Step 1 asks for consequence and not only volume, and there is a documented case of what a translate-then-act path does when consequence has never been inventoried.

In October 2017 Israel Police arrested a Palestinian construction worker in the West Bank settlement of Beitar Illit. He had posted a photo of himself leaning on a bulldozer with the Arabic caption 'يصبحهم' — good morning. Facebook's machine translation rendered it as 'attack them' in Hebrew and 'hurt them' in English. The operative failure sat in the standfirst of the Haaretz report on 22 October 2017: “No Arabic-speaking police officer read the post before arresting the man, who works at a construction site in a West Bank settlement.” The mistake was discovered only during questioning. Facebook apologised in a public statement, carried in The Guardian: “We apologise to him and his family for the mistake and the disruption this caused.”

No aggregate score would have flagged that request. The system's average quality was irrelevant to the one person it was wrong about. The missing control was not a better model. It was a reader of the source language, before an irreversible action. That is what step 4 buys: where the failure severity is custody, the launch tier specifies abstention and human review, not a higher benchmark number.

Macro language averages can give small languages equal weight, while traffic weighting can hide them; both are useful if accompanied by per-language sample size, uncertainty, task, domain, and consequence.

Add dialect, locale, script, code-switching, length, and channel slices where ethically and legally supportable. Never infer language quality from one high-resource proxy.

Those slices already have an official vocabulary, so they need not be invented per dashboard. RFC 5646, or BCP 47, has defined language tags since September 2009: a primary language subtag, plus script, region and variant. It also warns that tags sharing a prefix are not guaranteed to be mutually intelligible. Its example is az, az-Latn and az-Cyrl: “A person fluent in one script might not be able to read the other, even though the linguistic content (e.g., what would be heard if both texts were read aloud) might be identical.”

The registry that RFC mandates is bigger than most dashboards imagine. At file-date 8 August 2026 the IANA Language Subtag Registry held 9,296 records: 8,276 language subtags, 305 region, 258 extlang, 225 script, 139 variant, 67 redundant and 26 grandfathered. A quality report with one row per language name throws away every one of those distinctions before the first metric is computed.

For five languages, list demand, native corpus, labeled data, tokenizer fertility, reviewer availability, legal constraints, baseline quality, and failure severity.

Assign each language to full support, limited scope, human-assist, translation fallback, or not yet supported; write the evidence needed to move it to a higher tier.

Three of those columns now have units, taken from published measurements rather than impressions. Tokenizer fertility is a cost multiplier: up to 15 times the sequence for the same text, and close to a 5x increase in cost compared to English for most mid-resourced Indic languages in non-Latin scripts. Native corpus is an audited quantity, not a crawl size: at least 15 of the 205 corpora in the 2022 hand audit had no usable text, and MADLAD-400 discarded 79 of its 498 languages after its own audit. Failure severity has a worked example: a mistranslated good morning and an arrest, with no Arabic-speaking reader between them. A language whose row is empty in all three columns is not a supported language. It is an untested one.