Gold vs.~Translation: Are We Measuring Language, Knowledge, or Locality in Translated Benchmarks?

AI summary

Overview

Research area: Natural Language Processing — multilingual benchmarking and evaluation validity of large language models.

Technical level: Intermediate. The paper is readable without specialist background, but it relies on regression modelling, Kendall's τ_b rank correlation, bootstrap confidence intervals, and inter-annotator agreement statistics (Fleiss' κ, Gwet's AC1), which readers should recognise.

Scope: A validity study testing whether DeepL-translated versions of MMLU can serve as proxies for natively created benchmarks, across seven native–proxy pairings, five Nordic and Baltic languages, and 29 instruction-tuned LLMs.

What This Paper Is About

Multilingual LLM evaluation routinely substitutes machine-translated versions of English benchmarks such as MMLU for natively built tests, without knowing whether the two measure the same thing. The authors reframe this as a proxy validity question: does a translated benchmark lead to the same evaluation conclusions as a native one, once general model strength is accounted for? Their goal is to move beyond simple rank-correlation comparisons and test whether translated scores add incremental predictive value for native-benchmark performance after controlling for model size and target-language proficiency.

Key Contributions

1. An incremental proxy validity framework that goes beyond rank correlation by controlling for confounds (log total parameters and a Belebele language-proficiency score) and testing whether a translated benchmark carries predictive signal beyond those background factors.
2. A multilingual cultural-sensitivity and required-knowledge annotation pipeline, validated against a three-annotator majority reference of 500 items across five languages (100 per language, 1,500 raw ratings from 15 pseudonymous annotator IDs).
3. Empirical evidence across seven native-vs-proxy pairings, showing that proxy validity varies systematically with content and locality overlap, with a fixed evidence rule producing four robust effects, two marginal effects and one null result.
4. Released artefacts: the annotations and merged analysis data, plus the annotation pipeline and analysis code, to support reproducibility and extension to other languages and translation systems.

Main Findings

• Ranking agreement spans a wide range. Kendall's τ_b between native benchmarks and their same-language MMLU proxies ranges from .89 [.78, .96] for Skolprov down to .51 [.25, .74] for SwedishFacts (all p < .001). Exam-ET (.88), INCLUDE-LT (.87) and NRK-Quiz (.85) follow closely; DanishCitizenTest (.82) and INCLUDE-ET (.72) are weaker.

• Top-5 overlap is perfect for three pairings. Skolprov, Exam-ET and INCLUDE-LT all have 5/5 top-5 model overlap between native and proxy; NRK-Quiz 4/5; DanishCitizenTest, INCLUDE-ET and SwedishFacts 3/5.

• High cultural sensitivity does not automatically break a proxy. NRK-Quiz (.85) and DanishCitizenTest (.82) correlate strongly with their proxies despite substantial culturally specific content, indicating that CS/CA status alone does not determine ranking agreement.

• Incremental validity is graded, not binary. Under one fixed evidence definition applied to all seven pairings: four are robust — Skolprov (ΔR² = .128, 95% CI [.038, .253], β = 1.06 [.80, 1.19]), INCLUDE-LT (ΔR² = .076, β = 0.94), NRK-Quiz (ΔR² = .041, β = 0.46) and Exam-ET (ΔR² = .024, β = 0.52); two are marginal — INCLUDE-ET (ΔR² = .059, β = 0.84, CI [−.02, 1.30]) and DanishCitizenTest (ΔR² = .009, β = 0.31); one is null — SwedishFacts (p = .242, β = 0.14, CI [−.10, .33]).

• Exam-ET's small increment reflects a strong baseline. Its baseline R² is .954, so there is little variance left for the proxy to explain.

• Leave-one-family-out (LOFO) refits sort robust from fragile. Across 17 model-family exclusions (leaving 25–28 of 29 models per refit), all four robust pairings stay positive and nominally significant. DanishCitizenTest loses significance when Cohere, Llama or Qwen are excluded (p = .110, .072, .096) and INCLUDE-ET when Qwen is excluded (p = .102); SwedishFacts stays non-significant in all 17 refits.

• Controlling for English MMLU preserves the ordering. Adding each model's English G-MMLUX score as a third control keeps six pairings significant by the nested F-test (Skolprov p_F < 10⁻¹², INCLUDE-LT < 10⁻⁹, INCLUDE-ET < 10⁻⁵, NRK-Quiz < 10⁻⁶, Exam-ET < 10⁻⁷, DanishCitizenTest < 10⁻³). SwedishFacts stays non-significant by the F-test (p = .092) though its bootstrap β CI is positive ([.06, .48]), i.e. weak mixed evidence rather than a clean null in that specification.

• Locality and response format are entangled. SwedishFacts is the weakest pairing and also the only open-ended (generative) benchmark in the study, so the design cannot separate Sweden-specific content from the generative format.

• The annotation pipeline is more reliable on native items than on English-origin items. Against the three-annotator native majority, GPT-5.2 reaches 90.6% binary CS/CA accuracy (macro-F1 89.5%); on the held-out GlobalMMLU-Lite test split (n = 217) it reaches 73.3%. Four-way required-knowledge accuracy on the 477 native items with a unique majority is 83.9% (macro-F1 82.4%), or 80.0% (400/500) if all 23 human ties are counted as errors.

• Human agreement varies by language. Unanimous 3-of-3 CS/CA agreement ranges from 65% in Norwegian Bokmål to 91% in Estonian; Fleiss' κ from .25 in Danish to .83 in Estonian; Gwet's AC1 from .60 in Norwegian Bokmål to .91 in Estonian. Danish shows a prevalence effect: κ = .25 (95% CI [−.05, .51]) despite 84% unanimous agreement, because 277/300 raw ratings are CS.

• Annotation errors lean toward over-predicting cultural sensitivity. Native CA recall is 80.7%, with 35 CA→CS versus 12 CS→CA errors, so estimated CS shares may be slightly upward biased.

• Content ranges widely across the native sets. The proportion of culturally sensitive items runs from 14.3% in Skolprov to 97.8% in SwedishFacts, compared with 27.8% in GlobalMMLU.

Methodology in Plain English

The authors assembled seven "native vs. proxy" pairings. Each native benchmark — written originally in Estonian, Swedish, Danish, Norwegian Bokmål or Lithuanian — is matched with a DeepL-translated MMLU in the same language (from EU20-MMLU for ET, LT, SV and DA; from m_mmlu for NB). A unified subset, G-MMLUX, maps the culturally annotated GlobalMMLU items to their translations and, after agreement filtering, contains 16,701 items across six languages.

To know which items need local knowledge, they adapted the GlobalMMLU cultural-sensitivity schema: annotators mark a required-knowledge type (none, cultural, regional, dialect), a time-sensitivity flag, an explicit culturally sensitive / culturally agnostic label, and a subject category. They built a human reference of 500 items (100 per language, three independent professional annotations each, recruited via ProZ.com) and separately built a GPT-5.2 annotation pipeline calibrated only on GlobalMMLU-Lite, then checked that pipeline against both the English held-out split and the native reference. The main regression analyses use total benchmark scores, so annotation uncertainty does not affect them.

They ran 29 instruction-tuned LLMs (Llama, Qwen, Gemma, Mistral, Falcon and several European-focused models) in four size bands: ≥70B (7 models), 15–32B (8), 7–9B (7) and ≤1.7B (7), using the LM Evaluation Harness with custom task implementations. Belebele in each target language served as the language-proficiency control.

Two analyses follow. First, ranking agreement via Kendall's τ_b with bootstrap confidence intervals and top-5 overlap. Second, the core test: two nested regressions per pairing, one predicting the native score from log total parameters plus Belebele, and one adding the proxy score — if the second model wins the nested F-test, the proxy carries information beyond general model strength. Because 29 models is small for regression, they add 5,000 model-level bootstrap resamples, LOFO stability checks, and an English-MMLU control. Evidence labels are fixed up front: robust requires a significant full-sample F-test, a bootstrap β CI excluding zero, and positive β with nominal p < .05 in all 17 LOFO refits; marginal is full-sample significant but fails a robustness criterion; null means non-significant F-test and a β CI containing zero. In-sample ΔR² is deliberately not used as a null test because it is non-negative by construction.

Why This Matters

Impact on research. The paper converts a commonly voiced worry ("translated benchmarks may not measure the same thing") into a testable statistical claim with controls. It shows that ranking correlation overstates validity, because strong models score well everywhere, and it offers a reusable framework and released annotation pipeline for other languages and translation systems.

Real-world applications:

• Model selection for multilingual deployment — teams can judge whether a translated MMLU score is defensible evidence for a specific target language use case or whether a native benchmark is required.
• Benchmark development and localisation — organisations deciding whether to translate an English benchmark or commission a native one get guidance tied to content overlap with MMLU.
• Regulatory and procurement evaluation — auditors assessing claims about non-English LLM capability need to know which evidence types overstate local competence.
• Cultural-sensitivity auditing at scale — the GPT-5.2 annotation pipeline (90.6% CS/CA accuracy against the three-annotator native majority) is a practical tool for labelling benchmark items for locality.

Industry relevance. Labs reporting multilingual results, and enterprises procuring models for Nordic or Baltic markets, both rely on translated benchmarks today; the paper's gradient — curriculum-overlap content transfers, locally anchored knowledge does not — gives a concrete rule of thumb for when those numbers can be trusted.

Future Directions

• Separate locality from response format. SwedishFacts is both the most Sweden-specific and the only generative benchmark. A controlled comparison would need generative benchmarks with low local content, or multiple-choice benchmarks with comparable Sweden-specific factual content.
• Vary the translation system. All proxies here are DeepL. Human translation or LLM-based translation could change absolute scores, rankings and effect magnitudes, so a multi-translator design is needed to separate content effects from translation-quality effects.
• Test typologically distant and lower-resource languages. The sample spans Germanic (SV, DA, NB), Finnic (ET) and Baltic (LT) languages — geographically and typologically narrow. Whether the gradient holds elsewhere is open.
• Scale up the model sample and the human reference. 29 models is small for regression and 500 human-annotated items leave 23 unresolved 1–1–1 required-knowledge ties, mostly in Lithuanian (13 of 100), so fine-grained cultural-versus-regional distinctions remain ambiguous.

Target Audience

Practitioners and researchers who build or consume multilingual LLM evaluations — benchmark creators, evaluation engineers at model labs, and procurement or policy teams assessing non-English model claims. It is also useful for methodologists interested in validity frameworks for AI evaluation, and for NLP researchers working on Nordic and Baltic languages who need to know which existing benchmarks carry independent signal. Readers wanting to reproduce or extend the work will find the annotations, pipeline and analysis code released alongside the paper.

Authors’ abstract

Translated versions of English-origin benchmarks such as MMLU are widely used to evaluate large language models in non-English languages, yet it remains unclear whether they measure the same constructs as natively created benchmarks. We frame this as a proxy validity question and compare DeepL-translated MMLU proxies against six native benchmarks in seven pairings across five Nordic and Baltic languages, evaluating 29 models on each pairing. Beyond ranking agreement, we test incremental proxy validity: does the translated score add predictive value for native performance after controlling for model size and language proficiency? Proxy validity varies substantially across benchmark pairings. Applying the same robustness criteria to all seven pairings, we find four robust effects, two marginal effects, and one null result. The strongest evidence occurs for curriculum-overlap benchmarks such as Skolprov ($ΔR^2$=.128, 95% CI [.038,.253]), whereas translated MMLU provides no significant incremental value for Sweden-specific factual knowledge in SwedishFacts (p=.242). Because SwedishFacts is also the only open-ended benchmark, we cannot fully separate the effects of local content and response format. Overall, the results indicate a clear pattern: translated benchmarks provide a better proxy when their source content overlaps with the native evaluation target, while locally anchored knowledge is less reliably captured.