Multilingual GSM-Symbolic: What determines capability transfer across languages?

AI summary

Overview

Research area: Multilingual large language model evaluation and cross-lingual capability transfer, using grade-school mathematical reasoning as the test bed.

Technical level: Intermediate. The paper is readable without deep statistics knowledge, but the central analysis is a generalized linear mixed model with interaction terms, and the results are reported as log-odds coefficients, so some familiarity with regression-style effect sizes and LLM benchmarking helps.

Scope in one sentence: The paper builds a verified, extensible, template-based multilingual math benchmark and uses it to estimate, jointly, which model and language properties determine how well capabilities transfer across languages.

What This Paper Is About

We know little about how a capability acquired in one language carries over to another, largely because multilingual evaluations rely on datasets that differ in formatting, difficulty and subdomain, or that are static and prone to saturation and leakage. This paper introduces Multilingual GSM-Symbolic, a dataset of item-matched word-problem templates across 15 languages, so that the same underlying reasoning problem can be tested in every language. It then uses that dataset to estimate which factors — model size, reasoning mode, language resource level, typological distance — predict performance, and how those factors interact.

Key Contributions

1. Multilingual GSM-Symbolic, described as the first verified, symbolic framework of its kind in this setting, covering 15 languages and including the code needed to extend and resample the dataset. It consists of 1,500 verified and localized templates (100 per language), with 20 samples drawn per template for 30,000 item-matched question–answer pairs; the templates can generate millions of variants.
2. A joint model of capability transfer across languages, using a crossed random-effects logistic model that separates language effects from problem difficulty, and which ranks the determinants of performance and identifies actionable interventions in order of effectiveness.
3. A demonstration that performance on an unseen language can be predicted to a reasonable degree from the remaining languages, and that this prediction improves sharply when just 10 templates in the target language are available.
4. A staged error analysis using the templates' <<lhs=rhs>> calculator annotations, showing that cross-lingual degradation accumulates across operand reading, intermediate computation and final answer rather than being a parsing problem alone.

Main Findings

• Ranked determinants of capability: Model size is the largest effect (β = 1.77), followed by language resource level (β = 0.77), reasoning (β = 0.67) and typological distance (β = −0.25). The paper expresses these in terms of one another: a 32B model evaluated in Marathi performs like a 10B model in English.
• Model-level effects: Larger models are more capable (β_size = 1.88, p < 0.001) and reasoning helps (β_reasoning = 0.67, p < 0.001), but the reasoning benefit shrinks with scale (β_size×reasoning = −0.34, p < 0.001).
• Language-level effects: Higher-resource languages are substantially easier. Structurally distant languages transfer worse (β_distance = −0.25, p = 0.001), but this is strongly correlated with fertility (r = 0.8); the uncorrelated relative fertility measure shows no meaningful effect (β_rel. fertility = −0.04, p = .19).
• Interventions differ by language type: Model size and reasoning reduce the low-to-high-resource gap (Finding 4 reports β_resource×size = −0.27 and β_resource×reasoning = −0.20, both p < 0.001; the conclusion restates these as −0.25 and −0.20). Reasoning has a small effect on distant languages (β_distance×reasoning = 0.11, p = .03) which does not reproduce under model ablations, and model size shows no effect on distance (β_distance×size = 0.02, p = 0.32).
• Prediction of unseen languages: The framework explains 92% of between-language variation but only 23% of model-by-language variation. Average language difficulty is predicted to 3.2pp (Pearson r = 0.93) and a specific model in a held-out language to 5.98pp (Pearson r = 0.96) — the abstract states this as within 6.0pp — a 63% and 39% reduction over the naïve average baseline. With 10 templates, these fall to 2.38pp and 4.19pp. Figure 6 covers 768 (model, language) pairs.
• Symbolic variants are harder, unsystematically: Across 705 (model, language) pairs, symbolic accuracy is 3.14pp lower than on the original GSM8K questions, and this gap is unrelated to the model's accuracy level (r = −0.02). Fitted jointly, the penalty is −0.30 in log-odds (p < 0.001) and varies by language (χ² = 21.6, p < 0.001), smallest in English (1.7pp, 95% CI [−0.6, 4.0]) and largest in Russian (5.5pp, 95% CI [4.6, 6.5]). The penalty is orthogonal to the features that predict transfer (resource level p = 0.68, typological distance p = 0.84).
• The gap accumulates through the solution: Resource-level effects grow from operand reads (β = 0.32, p < .001) to intermediate computations (β = 0.45, p < .001) to the final answer (β = 0.77, p < .001); typological distance behaves similarly (−0.08, p = .066; −0.15, p = .001; −0.25, p < .001). Prompt compliance (using boxed{}) also varies by language, from 94.5% in German to 77.1% in Marathi.
• Robustness checks: Adding a resource × distance interaction was non-significant (β = 0.02, p = .61) and adding script was non-significant (β = −0.03, p = .90), moving the distance effect only from −0.29 to −0.28. A template-by-language random effect has sd = 0.92, larger than the model-by-language term (0.61) and more than three times the language term (0.27), and improves AIC by roughly 10^5, but leaves every significant coefficient's sign and significance intact, inflates standard errors uniformly by 1.05, and leaves sd(language) at 0.27; the refit gives 91% of between-language and 16% of model-by-language variance explained versus 89% and 20% without it.
• Sample size: 50 samples reproduce the model ranking at 0.99, but at the full 100 templates a single (model, language) prediction still carries a standard error of 2.29pp, and it would take 400 templates to halve that. At 10 templates, rank correlation is 0.914 (original) and 0.964 (symbolic), with standard errors of 10.60 and 7.26 accuracy points respectively.
• Translation quality matters: A TOST equivalence test with a 1pp margin found machine-translated and human-verified Icelandic templates were not equivalent (p > 0.999), motivating the use of only human-validated languages. A matching test on English imperial versus metric units found equivalence (p < 0.01) within the margin.

Methodology in Plain English

The researchers started from GSM-Symbolic, which converts GSM8K word problems into templates where names and numbers can be substituted. They wrote their own parser (the original authors did not release one), extended its syntax for multilingual use, and then had native speakers with technical expertise translate and localize every template — including currencies, units, cultural references and numerals — so that the reasoning structure stays constant while surface form changes. Each template records the source language, the model used and the annotator's proficiency. They validated templates in English first to stop errors from propagating, then checked translations, and ran an error analysis by generating 20 instances per problem and manually reviewing any template that Claude Opus 4.8 (xhigh) answered incorrectly. An automated test suite checks that rendered default-value templates match the originals.

To model transfer, they treated each generated instance as a solved/unsolved outcome and fit a crossed random-effects logistic regression, with random effects for model family, model, reasoning variant, language, template and model-by-language. This design lets language-level features be judged against the right sources of variability rather than being inflated by model strength or problem difficulty. The features are: log-transformed Common Crawl page count as a resource proxy; typological distance from English as cosine distance over 103 URIEL morphosyntactic features; relative fertility as a tokenizer measure; and, on the model side, log-transformed size and a boolean for reasoning. They included interaction terms to test whether model properties narrow language gaps, and fit the model in R with lme4 and lmerTest.

Evaluation used Inspect AI and vLLM 0.22.1, zero-shot, with a human-translated version of the English prompt in each language. Only open-weight, instruction-tuned models were used, arranged in family × size grids so that scale could be estimated while holding architecture, tokenizer and pretraining mixture fixed. The set includes Olmo 3 Think, Qwen 3 and 3.5, Granite 3.2, EuroLLM, Apertus, OLMo 2 and 3 (which span 50%, 40% and 90% English pretraining data respectively), Qwen 2.5 and Gemma 3. Two models were dropped: EuroLLM-1.7B, which exceeded its own 4,096-token limit in 54% of cases, and Qwen 3.5 0.8B with reasoning on, which produced near-empty completions in 95% of cases.

Why This Matters

Impact on research. Prior work examined model scale or language features largely separately; this paper estimates them jointly and in a symbolic, item-matched setting, which is what allows the effects to be compared and converted into one another. It also reframes evaluation cost: if a model's performance in an unseen language can be estimated without running the full benchmark, exhaustive per-language-pair evaluation becomes avoidable.

Real-world applications.

• Model development for low-resource languages: The paper identifies size and reasoning as levers that narrow the low-to-high-resource gap, while noting that neither helps much for typologically distant languages, so effort can be targeted rather than spread evenly.
• Evaluation planning: With a budget of only 10 templates in a target language, a developer can get a plausible performance estimate (4.19pp error for a specific model, 2.38pp for overall language difficulty).
• Benchmark construction: The template design mitigates contamination by generating millions of unseen surface variants, and the released generation package allows fresh variants to be drawn at evaluation time.
• Dataset localization practice: Localizing units, currency and names, and validating translations, are shown to matter empirically — machine-translated Icelandic was not equivalent to human-verified Icelandic, while English imperial versus metric units were equivalent.

Industry relevance. The dataset is integrated with EuroEval and Inspect AI, and per-model, per-sample logs are released so results can be analyzed without re-running inference. The paper also shows that model-specific pretraining composition likely matters: in open-data families, Apertus shows a negative average English–non-English gap (−20.7), EuroLLM the smallest positive gap (7.0), and the English-dominant Olmo 2 and 3 the highest (41.7 and 25.0), though the authors caution these are conflated with family, post-training recipe and recency.

Future Directions

• Extending language and template coverage. The authors note that 100 symbolic templates across 15 languages likely underrepresents the value of broader coverage, and kept the dataset extensible so new languages can be added; they also observe that 400 templates would be needed to halve the 2.29pp standard error at the full 100-template budget.
• Generalizing beyond grade-school math. The determinants and their effect sizes may differ across domains, and the study is limited to instruction-aligned, non-base models with no Mixture of Experts architectures, despite their increasing prevalence.
• Better specification of the mixed model. The authors encourage including a template-by-language random effect — template difficulty is clearly not language-invariant (sd = 0.92) — while reporting their initial specification to avoid choosing after seeing results; they also suggest model-specific pretraining data numbers would likely improve fit over the Common Crawl proxy.
• Disentangling typological distance from fertility. Because the two are highly correlated (r = 0.8) and relative fertility is used instead, the authors explicitly say conclusions about typological distance should be interpreted with care.

Target Audience

This paper is most useful to researchers and engineers working on multilingual LLM evaluation, to model developers deciding where to invest for low-resource language performance, and to benchmark builders who need item-matched, contamination-resistant evaluation data. It also suits computational linguistics researchers interested in quantitative models of cross-lingual transfer. Readers looking for a single headline multilingual accuracy number will not find one here — the paper deliberately argues against reporting a single multilingual average and recommends per-language results instead.

Authors’ abstract

We understand little about how capabilities acquired in one language carry over to another, or what governs this transfer: evaluations rely on incomparable, saturation-prone datasets and rarely examine its determinants jointly. Identifying what predicts transfer would let us avoid exhaustive evaluation across all language pairs and let developers target the factors that limit performance in low-resource languages. To evaluate cross-lingual capability transfer, we introduce Multilingual GSM-Symbolic, an extensible multilingual mathematical dataset covering 30,000 item-matched question-answer pairs and spanning 15 languages. It utilises symbolic templates to prevent overfitting and ensure generalisation by allowing generation of millions of high-quality variations from a single sample. Using Multilingual GSM-Symbolic, we quantify the largest determinants of capability as model size ($β= 1.77$), language resource level ($β= 0.77$), reasoning ($β= 0.67$) and typological distance ($β= -0.25$). This joint estimation allows these determinants to be expressed in terms of one another: a 32B model evaluated in Marathi performs like a 10B model in English. Our findings have important implications for model developers, showing that model size and reasoning narrow the performance gap between low- and high-resource languages ($β= -0.27$ and $β= -0.20$, respectively), while similar levers have little or no effect on typologically distant languages. Overall, our analysis framework explains 92% of between-language variation, but only 23% of the model-by-language variation, and predicts a model's performance on an unseen language within 6.0pp (r=.96). Incorporating measurements from just 10 templates in the target language reduces this to 4.19pp, enabling reasonable estimates of performance with little or no downstream dataset.