The State and Fate of Multilingual, Contextual Evaluation in the NLP World

AI summary

Overview

Research area: Multilingual evaluation of large language models — benchmark design, language coverage, cultural representativeness, and benchmark contamination.

Technical level: Intermediate. Readers should know what a benchmark, a task category, and data contamination mean, but no mathematics is required; the paper is an audit and a set of arguments rather than a new model or algorithm.

Scope (one sentence): A data-driven audit of 51 recent multilingual benchmarks covering 242 datasets and 219 languages, organized around coverage, representativeness, and rigor, with concrete evidence that the current evaluation ecosystem is wide but thin, heavily translated, and vulnerable to contamination.

What This Paper Is About

Multilingual benchmarks are the main instrument the NLP community uses to claim that a model "works" beyond English, yet nobody has systematically checked whether those benchmarks are broad enough, culturally valid, and uncontaminated. The authors take stock of the field in the spirit of Joshi et al. (2020)'s The State and Fate of Linguistic Diversity and Inclusion in the NLP World, asking what has actually been measured and what has been left out. The goal is to show where multilingual evaluation is performative rather than substantive, and to propose what a trustworthy evaluation framework would need to look like.

Key Contributions

1. A structured audit of 51 multilingual benchmarks spanning 242 datasets and 219 languages, with metadata extracted semi-automatically (using Claude Opus 4.6) and then manually verified against papers, dataset cards, and repositories — capturing languages, families, scripts, task types, translation status, native annotator involvement, cultural grounding, and licensing.

2. A three-pillar diagnostic framework — coverage, representativeness, and rigor — that turns vague concerns about multilingual evaluation into measurable quantities (e.g., share of translated dataset–language instances, task categories per resource level, contamination flag rates).

3. Fresh contamination measurements using two complementary methods, STAMP (Rastogi et al., 2025) across 9 datasets and 7 recent models, and the Data Contamination Quiz (DCQ, Golchin and Surdeanu, 2025) across 85 dataset–language partitions from 8 datasets, alongside a critical analysis of why neither method is language-neutral.

4. A transparency audit of 23 recent model releases (15 open-weight, 8 closed, 2024–2026) scored on five binary disclosure dimensions, plus a public dashboard (https://aka.ms/multilingual-evals) and a commitment to release metadata annotations and the analysis codebase.

Main Findings

• Coverage is wide but thin. 219 unique languages appear across 242 datasets, but the distribution follows a steep power law: Hindi, Chinese, Spanish, Swahili, and Arabic each appear in 19 or more datasets, while 48 of 219 languages (37%) appear in only a single dataset — the abstract and Figure 1 describe this as 36% appearing in only a single benchmark. For those languages there is no independent replication and no cross-dataset comparison.

• Typological and script concentration. Indo-European accounts for 83 of 219 languages (38%) across the 29 language families represented; Atlantic-Congo ranks second with 39 languages, almost entirely due to Africa-focused benchmarks such as AfroBench, IrokoBench, and Uhuru. Austroasiatic (3), Tai-Kadai (3), Japonic (1), and Quechuan (2) are poorly represented, and the entire Oceanic branch of Austronesian is absent. Latin script dominates, followed by Devanagari, Arabic, and CJK; scripts such as Ge'ez, Khmer, and Myanmar appear in only 1–3 benchmarks.

• Geographic holes. Sub-Saharan Africa leads with 62 languages (28%), followed by Europe (45, 21%) and South Asia (43, 20%); the Americas (14) and Central Asia (9) have modest representation, and Oceania is near-zero at 2 languages (1%) — a region home to over 1,400 living languages.

• Resource skew reverses on closer inspection. 65% of evaluated languages fall in the two lowest resource classes (Scraping-Bys and Left-Behinds, levels 0–1), while Winners (level 5) are only 5%. But the 10 Winner languages are each evaluated on a median of 14 task categories across multiple benchmarks, whereas the 160 low-resource languages (levels 0–2) are typically tested on only 1–3 tasks in a single benchmark.

• Task equity gap. For Left-Behinds, Translation accounts for 29% of all evaluation rows and Sentiment/Classification for 25%, while Coding, Safety/Toxicity, and Instruction Following each represent only 1%. Safety/Toxicity evaluation amounts to roughly 25 dataset–language rows across all datasets, concentrated almost entirely at higher resource levels — meaning safety coverage for low-resource languages is effectively nonexistent.

• Translation dominates construction. At the dataset–language level, 56% of instances are translated from English, 36% are natively authored, and 8% are partially adapted. At the dataset level, 128 of 242 datasets (53%) are originally authored, 85 (35%) are translated from English, 15 (6%) are adapted, and 14 (6%) use a mix.

• Translation dependence tracks resource level. Left-Behinds (n=178) have the lowest native share at roughly 20%, while Winners (n=360) sit at roughly 40% — the most under-served languages are the most likely to be evaluated with data not originating from their speech communities.

• A breadth–validity trade-off. Europe and East Asia are the most translation-dependent, while Sub-Saharan Africa has the highest share of natively authored content. Cultural grounding is highest for Sub-Saharan Africa and South Asia, driven by community efforts like CulturalBench, Samiksha, and TUMLU, while broad cross-regional suites score lower. The datasets that are methodologically strongest on representativeness have the narrowest language coverage.

• Annotator demographics are rarely disclosed. Only 8 of 51 benchmarks (16%) report annotator demographics such as geographic origin, ethnicity, or education, and only 17 (33%) disclose compensation. Researchers are the dominant creator type, with community volunteers and crowdworkers a small minority. Native-annotator share is highest in Central Asia and South Asia (roughly 50%), around 45% for Sub-Saharan Africa (n=543) with a large "Unknown" fraction, and lowest in the Americas (n=23).

• Metrics are English-centric. BLEU, ROUGE, exact match, and Word Error Rate can be poor proxies in morphologically rich languages, and LLM judges are reported elsewhere to align poorly with human judgments in multilingual and multicultural settings, especially for low-resource languages.

• Contamination persists, concentrated in older datasets. Using STAMP with 9 paraphrased versions per item generated by Qwen-3.5 397B across 7 recent models: PAWS-X 67.9% of model–language pairs flagged, MGSM 45.5%, AfriMGSM 21.0%, while newer suites are much lower — MILU 0.0%, INCLUDE 1.0%, Global MMLU 1.7%, IndicParam 4.3%, MMLU 2.0%. FLORES shows a sharp disagreement: prior surface-form methods flag 7 of 7 models, but STAMP detects contamination in only 1.2% of model–language pairs.

• DCQ confirms a similar ranking. With 20% as the random-chance floor, maximum recognition rates average PAWS-X 32.6% and MGSM 30.9%, with the remaining datasets ranging from 14.7% to 21.7%. Recognition is higher for English partitions: AfriMGSM 45.7% vs 15.3%, MGSM 44.0% vs 29.6%, PAWS-X 47.6% vs 30.1%.

• Neither detection method is language-neutral. STAMP's watermarked rephrasing is harder in typologically distant, low-resource languages, and its tokenizer-dependent hash partitioning can weaken statistical power. DCQ's decoys vary in perturbation distance from 5.8% (AfriMGSM, Amharic) to 48.1% (Global MMLU, English). A natural replicate holding 100 GSM8K items byte-identical and changing only the label from AfriMGSM to MGSM shifted recognition by 1 percentage point for Aya-Expanse-8B and 23 points for Gemma-3-12B — variation that cannot be attributed to contamination.

• Model releases disclose little. Across 23 releases, 35% report zero multilingual datasets and 87% provide no contamination analysis; only 3 models, all open-weight, report any contamination testing. The model–dataset matrix is 93% sparse, and natively authored datasets from the survey (AfroBench, IrokoBench, MILU) are almost entirely unused. Open-weight models score 2.47/5 on transparency versus 0.75/5 for closed models, with an overall mean of 1.87/5; no closed model discloses training language composition or reports contamination. Only 4 of 23 models report multilingual safety evaluation: Tiny Aya, Claude Opus 4.6, Claude Sonnet 4.6, and CommandA.

Methodology in Plain English

The authors built a catalog. They gathered 51 publicly released multilingual benchmarks through literature search, leaderboard tracking, and community repositories, deliberately mixing famous ones (XNLI, XQuAD, TyDiQA) with recent community-driven efforts for underrepresented languages (AfriSenti, MasakhaNER, IrokoBench).

For each benchmark they needed a consistent set of facts: which languages, which language families and scripts, which task types, whether content was translated or written natively, whether native speakers annotated it, whether it is culturally grounded, and its license. Rather than read 51 papers by hand, they had Claude Opus 4.6 extract these fields from each paper, with a confidence rating and a supporting quote, then pushed the results through a human review interface where reviewers checked values, corrected language lists, and adjudicated with dropdowns. Language family, macroarea, script, and resource level were filled in deterministically from Glottolog and Joshi et al. (2020)'s taxonomy.

Because 22 of the 51 benchmarks are aggregations that reuse other datasets, counting per benchmark would inflate coverage. The authors therefore deduplicate to the dataset–language level while keeping track of which benchmark each came from.

For contamination, they ran two tests that work differently. STAMP creates nine paraphrase variants of every test item and checks whether a model's score drops when the exact wording is no longer available — a big drop suggests memorization. DCQ instead asks a model to pick the real item out of four synonym-perturbed decoys plus a "none of these" option, with the decoys placed in positions the model initially under-selected to control for positional bias. The authors then stress-tested both instruments, showing where paraphrase quality and decoy difficulty vary by language and could masquerade as differences in contamination.

Finally, they scored 23 model releases on five yes/no transparency questions (training language composition disclosed, any multilingual benchmark reported, evaluation languages listed, contamination technique named, contamination results reported) and summed them into a 0–5 score.

Why This Matters

Impact on research. The paper converts a vague worry — "multilingual evaluation is not good enough" — into specific, checkable numbers. It shows that claims of multilingual generalization typically rest on dense evaluation of a handful of languages and a single data point for most others, and that the datasets with the widest language coverage are systematically the most translated and least culturally grounded. It also demonstrates that contamination detection tools themselves behave differently across languages, so apparent low contamination in low-resource languages may be an artifact of a weaker instrument rather than genuinely cleaner data.

Real-world applications:

• Model procurement and deployment decisions in governments and NGOs serving Global South languages, where a benchmark score for a language covered by one translated dataset is weaker evidence than it appears.
• Safety and harm reduction, since safety/toxicity evaluation for low-resource languages is effectively absent (roughly 25 dataset–language rows in total, concentrated at higher resource levels), leaving high-stakes deployment in those languages unmeasured.
• Benchmark and dataset construction, where the audit gives builders concrete design targets: native authorship, cultural grounding, documented annotator demographics and compensation, and contamination-aware release.
• Regulatory and policy work, such as commitments like "Strengthening Multilingual and Contextual Evaluations" from the New Delhi Frontier AI Commitments, which need measurable criteria rather than statements of intent.

Industry relevance. Model cards and technical reports are the primary evidence regulators and enterprise buyers use. The finding that 87% of audited releases provide no contamination analysis, that no closed model discloses training language composition, and that the model–dataset evaluation matrix is 93% sparse means buyers currently cannot verify multilingual claims. The proposed standardized multilingual evaluation card would give vendors a concrete reporting format and give customers a comparable basis for purchase decisions.

Future Directions

• Natively constructed, culturally grounded benchmarks designed with the communities whose languages they evaluate — going beyond translation or validation to include communities in defining tasks, rubrics, and failure modes, as Pariksha and Samiksha begin to do.

• Contamination methods that are validated per language. The authors argue that multilingual contamination claims require per-language instrument validation and convergence across complementary methods rather than direct comparison of raw scores, and call for contamination-aware design plus protection of test data (e.g., encryption, training-exclusion controls) rather than retroactive detection.

• Scalable but valid evaluation tooling: multilingual synthetic data, aligned multilingual LLM judges, and dynamic benchmarks that change over time to resist contamination — while resolving how LLM judges can be made to agree with human judgments in culturally grounded, low-resource settings.

• Evaluation beyond the model: moving to product-, user-, and real-world-impact evaluation, since model scores alone do not capture how systems behave in deployed, culturally situated settings, and resolving the acknowledged tension between scaling coverage and preserving local representativeness.

Target Audience

Benchmark and dataset builders who need a checklist of what current multilingual evaluation gets wrong; NLP researchers who cite multilingual benchmark numbers and should know how thin those numbers often are; evaluation and red-teaming teams at model labs who need to report multilingual results credibly; and policy or procurement staff who rely on model cards and technical reports to judge whether a system genuinely supports a given language. It is also a useful orientation read for graduate students entering multilingual NLP, since it maps the ecosystem and names the specific benchmarks that define it.

Authors’ abstract

Multilingual evaluation benchmarks are the primary instrument for assessing whether large language models generalize beyond English, yet the adequacy of these benchmarks has received little systematic scrutiny. We present a data-driven audit of 51 recent multilingual benchmarks spanning 242 datasets and 219 languages, organized around three pillars: coverage, representativeness, and rigor. Our analysis reveals that coverage is wide but thin with 36% of evaluated languages appearing in only a single benchmark, entire regions (Oceania, the Americas, Central Asia) are near-zero, and a stark task equity gap leaves low-resource languages evaluated on only 1-3 task categories versus 14 for high-resource languages. Representativeness is structurally compromised: translation from English remains the dominant construction strategy where 56% of all dataset-language instances are translated introducing artifacts and English-centric framing, while culturally grounded content is concentrated in a handful of community-driven benchmarks with narrow language scope. The ecosystem thus forces a trade-off between breadth and validity. Rigor is undermined by benchmark contamination, including translated benchmark leakage and parallel corpus overlap that evade surface-form detection. We synthesize these findings into concrete recommendations for building evaluation frameworks that are natively constructed, culturally grounded, contamination-aware, and designed to serve the communities whose languages they claim to evaluate. The dashboard with all the metadata collected can be found at https://aka.ms/multilingual-evals.