Which Language Should a Skeleton Speak? Language Choices in Multilingual Reasoning

AI summary

Overview

Research area: Multilingual natural language processing, specifically structured prompting and reasoning in large language models (LLMs).

Technical level: Intermediate. The paper assumes familiarity with chain-of-thought prompting, greedy vs. sampled decoding, and standard accuracy metrics, but its core idea is conceptually simple.

Scope: A training-free empirical study of which language a reasoning "skeleton" (a short outline of a problem's structure) should be written in when an LLM solves math problems posed in many different languages.

What This Paper Is About

Skeleton-based prompting gives a model a brief structural outline of a problem before it reasons, and prior work has almost always written that outline in English. But a multilingual pipeline actually makes three separate language choices: the language of the user's question, the language of the skeleton, and the language of the reasoning and final answer. This paper builds a framework to systematically vary those choices and asks whether English skeletons are genuinely best, for which models, benchmarks and languages, and how stable the effects are across evaluation protocols.

Key Contributions

1. Formalizing language choice as a design variable. The authors define a language configuration triple (ℓ_q, ℓ_s, ℓ_a) — query, skeleton, answer — and propose the Language-Aware Skeleton Exploration Framework (LASEF), a training-free procedure that holds the skeleton's structure and generation method fixed while varying only language.
2. A multi-level evaluation design. They combine greedy decoding with multi-rollout sampling (T = 0.7, N = 5) across three math benchmarks of increasing difficulty (MGSM, MATH-500, PolyMath) and three Qwen2.5-Instruct model sizes (7B, 14B, 72B), plus a 19-language low-resource study.
3. Three identified patterns of skeleton-language effects. Directionally consistent, evaluation- and benchmark-dependent, and asymmetric negative — each supported by cross-benchmark checks (including MSVAMP) and statistical testing.
4. A translation ablation. By machine-translating English skeletons into five other languages with GPT-5-mini, they separate skeleton generation quality from cross-lingual alignment, showing the effects are not merely artifacts of how well a skeleton was written.

Main Findings

• English skeletons help, but only slightly on average. Applying English skeletons improved 13 of 19 low-resource MGSM languages with Qwen2.5-7B, with mean gains of +2.38 (target-language queries) and +2.02 (English-translated queries). The paper repeatedly describes this as a "small positive tendency."
• Most language-level gains are not statistically robust. After FDR correction, only Kazakh (kk) and Uyghur (ug) showed significant per-language improvements; the pooled item-level effect across the 19 languages remained significant.
• Translation, not skeletons, drives low-resource gains. Under English-translated queries the average rose from 33.99 to 58.06 (+24.07 percentage points), versus a further +2.02 from adding skeletons.
• Smaller models benefit more; scaling does not guarantee improvement. Qwen2.5-7B and 14B improved by +0.9 to +2.5 across all benchmarks after skeleton application, while Qwen2.5-72B showed an average drop of 0.2 in two of the settings examined.
• Easier benchmarks benefit more. With Qwen2.5-7B, skeleton gains were +2.2 on MGSM, +1.8 on MATH-500, and +1.2 on PolyMath.
• Low-resource languages gain the most from English skeletons. On MGSM with Qwen2.5-7B, Swahili went from 63.1 to 71.7 (+8.6) while Chinese gained +0.0 and Spanish +1.6; with Qwen2.5-72B on MATH-500, Swahili again showed the largest improvement (+3.7).
• English is not universally optimal. Five of the 19 low-resource languages (lo, eu, mt, ne, or) degraded with English skeletons, and non-English skeletons produced an average decrease of 0.63 to 2.60 points overall — but with notable exceptions.
• Pattern 1 — directionally consistent effects. Lao drops −2.03 with an English skeleton but gains +2.81 with a Thai skeleton (Tai–Kadai affinity); Basque and Armenian gain +6.32 and +5.17 with a Korean skeleton (agglutinative, SOV alignment). The signs hold under multi-rollout: lo–th +2.81 → +3.18, hy–ko +5.17 → +4.61, eu–ko +6.32 → +2.78.
• Pattern 2 — evaluation- and benchmark-dependent effects. Maltese shows large greedy gains with Russian (+8.62), Thai (+6.90) and Spanish (+6.61) skeletons, but multi-rollout attenuates them (mt–ru: −0.23) and the mt–ru gain reverses on MSVAMP.
• Pattern 3 — asymmetric negative effects. Mongolian–Russian degrades (−5.00 greedy, −2.65 multi-rollout) despite a shared Cyrillic script; Uyghur–Chinese degrades (−6.45 greedy, −1.12 multi-rollout) despite geographic proximity, because the languages differ in word order and morphology.
• The ablation attributes effects to surface form, not skeleton quality. Directionally consistent pairs stay positive with translated skeletons (eu–ko +6.32 → +4.42, hy–ko +5.17 → +1.61, lo–th +2.81 → +0.40), Maltese stays positive (mt–ru +5.15, mt–ko +5.06), and asymmetric negatives persist (mn–ru −3.43, ug–zh −6.94).

Methodology in Plain English

The researchers hold everything about the skeleton constant except its language. Every skeleton is kept short (about 120 tokens on average) and deliberately abstract: it names the problem's objects, variables, relationships and constraints, plus the relevant mathematical concepts, but never states solution steps, formulas or numbers. The prompt instruction is essentially "describe what the problem structure is, never how to solve it."

For each target language they compare four pipelines: ordinary chain-of-thought in the target language, chain-of-thought with an English skeleton, chain-of-thought on an English-translated query, and chain-of-thought with an English skeleton on an English-translated query. Accuracy is measured by Exact Match, and — importantly — only on samples where the answer was actually produced in the user's target language, so scores reflect genuine multilingual reasoning rather than answers that quietly reverted to English.

To check whether a result is a real effect or a decoding fluke, they rerun each condition with five sampled rollouts (temperature 0.7) alongside single greedy decoding, and they retest on a second benchmark (MSVAMP). To check whether a result is just a poorly written skeleton, they take the English skeleton, translate it with GPT-5-mini into five other languages, and rerun — if the effect survives with content held fixed, it is about language, not writing quality. Languages are also described along two axes: surface affinity (language family, script, geography) and structural alignment (typology and word order).

Why This Matters

For research, the paper pushes back on an implicit default: that English is the safe choice for structured reasoning prompts. It shows the question is empirical and context-dependent, that per-language gains often fail significance testing, and that single-protocol evaluations can produce misleading conclusions. It also introduces a reusable diagnostic protocol combining multi-rollout, translation ablation and cross-benchmark checks.

Real-world applications:

• Multilingual math and STEM tutoring, where systems must both reason correctly and return answers in the student's language.
• Localized customer support and technical QA, where a short English structural scaffold could precede a native-language explanation.
• Deployment planning for low-resource languages, where the paper suggests query translation buys far more than skeleton language tuning (+24.07 vs +2.02 points).
• Offline risk screening before deployment, flagging language pairings that are likely to hurt performance, such as Mongolian–Russian or Uyghur–Chinese skeletons.

Industry relevance: the study gives teams a justified default (English skeletons are a reasonable baseline) alongside a caution (gains are small, shrink with model scale, and can reverse). It also warns that a full configuration search is computationally expensive and should be treated as an offline diagnostic rather than something run per deployment.

Future Directions

• Extending beyond mathematics to domains such as commonsense reasoning and code generation, where the observed patterns may not transfer.
• Reconciling training-free analysis with fine-tuning, since the training-free constraint ensures broad applicability but forgoes gains from adaptation.
• Exploring the interaction between skeleton structure and skeleton language, which was deliberately held fixed here.
• Building efficient predictive methods that forecast which skeleton–target-language pairings are safe, replacing the expensive combinatorial search, plus revalidation whenever the benchmark, model or task changes.

Target Audience

Most useful to multilingual NLP researchers studying reasoning and prompting, and to practitioners deploying LLMs in non-English markets who need to decide what language their intermediate reasoning artifacts should use. It is also a good methodological read for anyone designing evaluations of prompting techniques, because its central lesson is that single-protocol, single-benchmark results can be fragile.

Authors’ abstract

Skeleton-based reasoning prompting is a promising training-free approach for structuring LLM reasoning, but prior work largely assumes an English-centric setting. We propose the Language-Aware Skeleton Exploration Framework (LASEF) to study skeleton-language choice in multilingual mathematical reasoning. Across math benchmarks, model scales, and languages, we show that English skeletons yield a small positive tendency on average, most visible for smaller models and low-resource languages. However, few language-level gains remain significant after correction, and English is not universally optimal. Combining greedy decoding, multi-rollout evaluation, translation ablation, and cross-benchmark validation, we further find three patterns of skeleton-language effects: directionally consistent, evaluation- and benchmark-dependent, and asymmetric negative. These effects cannot be fully explained by generation quality alone. Overall, skeleton language is a context-dependent design variable that requires multi-level exploration. All resources are released at https://github.com/lhsstn/LASEF.