Why Do Multilingual Reasoning Gaps Emerge in Reasoning Language Models?

AI summary

Overview

Research area: Multilingual natural language processing and reasoning language models (RLMs) — specifically, diagnosing why large reasoning models perform worse on low-resource-language inputs than high-resource ones.

Technical level: Intermediate. The paper assumes familiarity with chain-of-thought reasoning models, supervised classifiers, and standard evaluation metrics, but the core argument is conceptual rather than heavily mathematical.

Scope in one sentence: The paper decomposes the multilingual reasoning gap into understanding, reasoning, and generation stages, shows that understanding failures dominate, demonstrates that these failures are detectable, and proposes selective translation as an efficient mitigation.

What This Paper Is About

Reasoning language models generate long chains of thought before answering, and they perform better when the question is in a high-resource language such as English than in a low-resource language such as Swahili. Prior work has tried to narrow this gap with representation editing, prompting tricks, and prefix tuning, but nobody had systematically asked why the gap exists in the first place. This paper frames multilingual reasoning as three stages — understanding the input, reasoning in the model's dominant trace language, and generating the final answer in the input language — and then measures how much each stage contributes to the gap.

Key Contributions

1. First systematic stage-wise attribution analysis of the multilingual reasoning gap in RLMs, showing through targeted interventions and a Shapley decomposition that failures in the understanding stage are the dominant source of the gap, while generation contributes only marginally.
2. A systematic evaluation of understanding-failure detection methods, adapting techniques from hallucination and jailbreak detection — LLM-based monitoring, self-reflection, token-probability signals, and supervised detectors (mmBERT and a hidden-state prober) — and finding that supervised approaches perform best.
3. Selective Translation, a mitigation strategy that injects an English translation into the initial reasoning trace only when an understanding failure is detected. With Qwen3-4B it raises Polymath-Low average accuracy from 81.1 to 88.0 (versus 89.4 for full translation, i.e. translating 100% of inputs) while translating only about 20% of inputs.
4. Evidence linking understanding to translation ability, showing a strong positive Pearson correlation (r = 0.951) between translation quality on FLORES-200 (measured by GEMBA-DA, x→en) and reasoning performance ratio on Polymath-Low across ten languages and five models.

Main Findings

• Understanding failures dominate the gap. In the stage-wise attribution analysis across models and datasets, the understanding stage accounts for most of the multilingual reasoning gap, while the generation stage contributes only marginally. The reasoning share remains relatively small and shows no consistent trend across Polymath difficulty levels.

• Understanding intervention helps most in low-resource languages. On Qwen3-4B Polymath-Low, the Understanding Intervention lifts Swahili from 29.3 to 88.0. By contrast, the Answer Extraction from Reasoning Trace intervention (w/ T) shows little change from Base.

• The gap collapses once understanding is resolved. The average Reasoning Performance Ratio (the ratio of each language's accuracy to the best-performing language under Base) rises to nearly 1.0 across all Polymath splits after the Understanding Intervention — for example, Qwen3-4B on Polymath-Low goes from 0.82 ± 0.21 to 0.95 ± 0.03, and gpt-oss-20b on Polymath-High from 0.92 ± 0.05 to 0.98 ± 0.03.

• Reasoning ability correlates with translation quality. The scatter plot of reasoning performance ratio on Polymath-Low against GEMBA-DA translation quality on FLORES-200 across ten languages and five models yields r = 0.951.

• Supervised detectors win at detection. On Qwen3-4B Polymath-Low, the prober reaches 85.5 ± 1.3 balanced accuracy and the mmBERT detector 85.2 ± 1.2, against a random baseline of 66.2 ± 2.6; the LLM-based detector reaches 71.7 ± 1.0 and self-reflection only 61.6 ± 2.0. On Qwen3-4B MMLU-ProX-Lite the prober reaches 77.3 ± 1.0 balanced accuracy against a 55.4 ± 3.5 random baseline.

• Detection generalizes to unseen languages. Using French (fr), Marathi (mr), and Wolof (wo), both the mmBERT detector and the prober consistently outperform the average-confidence baseline and the LLM-based detector.

• Detectors work early. Both the mmBERT detector and the prober achieve performance comparable to the full-trace setting with only 4,096 tokens of reasoning trace, so reliable detection does not require the full reasoning process.

• Selective Translation is efficient. With the prober as detector, average Polymath-Low accuracy goes from 81.1 (Base) to 88.0, close to full translation's 89.4, translating 19.3% of inputs on average. On MMLU-ProX-Lite it goes from 72.7 to 74.3 versus 76.5 for full translation, translating 20.8% of inputs. Translator usage is highly skewed: 86.4% of Swahili Polymath-Low inputs required translation versus 1.6% of English inputs.

• English is the best translation target. Testing selective translation with Spanish, Arabic, Thai, and Telugu as targets, English performed best overall, and performance degraded as the target shifted toward lower-resource languages (e.g., Telugu), with the degradation more pronounced on low-resource evaluation languages.

Methodology in Plain English

The authors first define a three-stage model of how a reasoning model handles a foreign-language question: it understands (internally translating the question into its reasoning language, typically English), reasons in that language, and then generates the answer back in the original language.

To find out which stage is failing, they run two interventions. The Understanding Intervention prepends a fixed prefix to the reasoning trace that states an explicit, dataset-provided gold English translation of the question (e.g., "I understand the question as: '…'"). The Answer Extraction intervention pulls the final answer directly out of the reasoning trace rather than the final response, isolating generation-stage errors. Accuracy under Base, under each intervention, and under both is combined with a Shapley decomposition to assign credit to understanding and generation; whatever gap remains is attributed to reasoning. Languages whose Base performance is significantly below the best-performing language (Welch's t-test, p < 0.05) are the ones analyzed.

For detection, they define a binary task: given the Base-setting input and output signals, predict whether the model failed to understand. Labels come from the intervention results — a datapoint is an "understanding failure" if the model got it wrong under Base but right under the Understanding Intervention. They then compare an LLM-based detector, self-reflection, token-probability signals (average and minimum per-token confidence, input negative log-likelihood), a fine-tuned mmBERT detector, and a two-layer perceptron prober over the final-layer hidden state. Metrics are balanced accuracy, F1, and PR-AUC, chosen because over 86% of samples are labeled "understood."

Finally, Selective Translation wires the best detector to a translator: when a failure is flagged, an English translation (produced with GPT-4.1) is inserted into the opening of the reasoning trace.

Setup details: Models are Qwen3-4B and gpt-oss-20b (with Qwen3-1.7B, 8B, and 14B for scale generalizability). Benchmarks are Polymath (low, medium, high) and MMLU-ProX-Lite, with 125 test samples per language per Polymath difficulty level and 257 samples per language on MMLU-ProX-Lite. Accuracy is averaged over three runs with seeds, sampled at temperature 0.6, top-p 0.95, top-k 20, and maximum generation length 32,768 tokens. MGSM is the calibration data for Polymath-Low, and the MMLU-ProX-Lite validation split is used for MMLU-ProX-Lite.

Why This Matters

Impact on research. The paper reframes the multilingual reasoning gap from a diffuse "low-resource models are worse" problem into a specific, localized failure mode — the model cannot get the question into its reasoning language. That makes the problem detectable, measurable, and targetable, and it opens a clearer path toward equitable multilingual reasoning than approaches that force the reasoning language or require expensive target-language reasoning data.

Real-world applications:

• Multilingual assistants and customer support, where a user asks in Swahili or Telugu and the system must reason accurately without translating every query at full cost.
• Educational and tutoring tools serving students in mid- and low-resource languages, where a wrong answer from a misunderstanding is worse than a slow one.
• Cost-aware deployment pipelines, since Selective Translation translates only ~20% of inputs and the detector itself adds negligible latency.
• Confidence and abstention systems, since a detector trained on understanding failures could also flag inputs the model should decline rather than answer incorrectly.

Industry relevance. The efficiency argument is central for production: full translation is a 100% translator-usage strategy, whereas Selective Translation approaches its accuracy at roughly a fifth of the cost. The result that English is the best translation target, and that lower-resource targets reintroduce failures, is also a directly actionable design guideline.

Future Directions

1. Extending beyond math and STEM. The authors note their experiments cover mathematical and STEM reasoning tasks only, and that verifying generalization to domains such as commonsense reasoning remains open.
2. Testing on more and lower-resource languages. The study covers ten typologically and resource-wise diverse languages; the authors state this set does not cover all language families and that verification on additional, lower-resource languages would strengthen the claims.
3. Models that reason natively in non-English languages. The framework assumes the model internally translates into English. For a model that reasons natively in, say, Russian, the understanding stage may not be clearly separable — the authors flag this as valuable future work.
4. Integrating detection and mitigation into training. The conclusion proposes exploring whether detection and mitigation could be built directly into model training rather than applied as inference-time interventions.

Target Audience

This paper is most useful to researchers working on multilingual NLP, reasoning-model evaluation, and inference-time interventions; to engineers building multilingual products on top of reasoning models who need a cost-aware way to reduce language performance gaps; and to readers interested in model interpretability and failure detection, since the detection task is adapted from hallucination and jailbreak monitoring. Readers looking for a general introduction to reasoning models would find the paper assumes too much background.

Authors’ abstract

Reasoning language models (RLMs) achieve strong performance on complex reasoning tasks, yet they still exhibit a multilingual reasoning gap, performing better in high-resource languages than in low-resource ones. While recent efforts have been made to address this gap, its underlying causes remain largely unexplored. In this work, we show that this gap primarily stems from failures in language understanding-specifically, the model's inability to translate multilingual inputs into the language dominating its reasoning traces (typically English). As identifying understanding failures can enable targeted mitigation of the gap, we evaluate a range of detection methods and find that understanding failures are detectable to a meaningful extent, with supervised approaches performing best. Building on this, we propose Selective Translation, a strategy that incorporates an English translation into the initial reasoning trace only when an understanding failure is detected. Experimental results using Qwen3-4B show that Selective Translation substantially bridges the multilingual reasoning gap, achieving near full-translation performance while translating only about 20% of inputs. Together, our results show that failures in language understanding are the primary driver of the multilingual reasoning gap and can be detected and selectively mitigated, clarifying its origin and suggesting a path toward more equitable multilingual reasoning. Our code and data are publicly available at https://github.com/deokhk/RLM_analysis