Beg to Differ: Understanding Reasoning-Answer Misalignment Across Languages

AI summary

Overview

• Research area: Multilingual natural language processing, specifically chain-of-thought reasoning evaluation for large language models across languages and writing systems.
• Technical level: Intermediate. The framework is conceptually simple (does the stated reasoning actually justify the answer?), but the paper assumes familiarity with benchmarks, chain-of-thought prompting, Cohen's kappa, and human/automated annotation validation.
• Scope: The paper introduces a human-validated framework for measuring whether model-generated reasoning traces logically support their final answers across 6 languages and 6 frontier models on GlobalMMLU, and shows that non-Latin-script languages suffer at least twice the reasoning–answer misalignment of Latin-script languages.

What This Paper Is About

Multilingual model evaluation usually only checks whether the model's final answer is correct, which hides the possibility that a model reaches the right answer through reasoning that does not actually support it. The authors build a framework that strips the final answer out of a reasoning trace and asks evaluators—human and automated—to infer what answer the remaining reasoning implies. If that inferred answer differs from the model's stated answer (or is inconclusive), the trace is counted as misaligned, and the paper measures how often this happens across languages, scripts, and models.

Key Contributions

1. Evidence of systematic reasoning–answer misalignment in multilingual settings, with alignment dropping at least twice as much for non-Latin scripts as for Latin scripts, and with misalignment occurring independently of whether the final answer was correct.
2. A human-validated evaluation framework that isolates reasoning traces from conclusions and measures "Trace Inconsistency Rate" (TIR) at scale, validated against human annotation.
3. An inductively derived error taxonomy built through thematic analysis of human annotations, covering failure modes such as unsupported claims, ambiguous facts, logical contradictions, multiple answers, conflicting facts, illogical leaps, linguistic/translation errors, irrelevant content, and "other."
4. A distributional analysis of reasoning failures across language resource levels and scripts, showing that evidential errors (unsupported claims, ambiguous facts) dominate over logical contradictions.

Main Findings

• Scale of the study: The analysis covers 65k reasoning traces from GlobalMMLU questions across 6 languages (English, Spanish, Hindi, Arabic, Ukrainian, Korean) and 6 frontier models (Llama-4-Scout-Instruct, Llama-3.3-70B-Instruct, Qwen3-32B, Qwen3-32B-thinking, Qwen2.5-72B-Instruct, Qwen2.5-32B-Instruct). The automated evaluator was applied over 66,615 GlobalMMLU examples, keeping only instances with a valid letter prediction A–D. (One sentence in the introduction refers to "five languages," though the abstract and experimental setup list six.)
• Accuracy vs. reasoning alignment diverge: Accuracy dips from high- to low-resource languages (roughly 80% to 70%), but reasoning–answer alignment drops more than two-fold, meaning models can reach the right answer for the wrong reasons.
• Script matters more than resource level in this study: Latin-script languages (English, Spanish) outperform non-Latin-script languages (Hindi, Arabic, Ukrainian, Korean) by 4–6 percentage points in both accuracy and TIR—which the authors describe as twice the gap observed between high- and low-resource languages.
• Representative TIR values (lower is better):

• Accuracy ceiling vs. reasoning floor: Accuracy peaks at 87–89% in English and drops to 70–81% for Hindi and Korean, with an ordering of English > Spanish > Ukrainian/Arabic > Hindi/Korean, consistent with prior work on language resourcing (Costa-Jussà et al., 2022).
• Concrete misalignment examples: Qwen3-32B keeps 78.4% accuracy in Korean but its TIR jumps to 9.81%, over 4× higher than English (2.16%). Qwen2.5-32B-Instruct shows 13.3% Korean TIR despite 77% accuracy, meaning roughly 1-in-8 responses contain reasoning that contradicts its final answer.
• Extended reasoning helps consistency: Qwen3-32B-thinking reaches 0.91% English TIR versus 2.16% for the non-thinking Qwen3-32B.
• Misalignment is worst when the answer is wrong: When the final answer was incorrect, TIR rises substantially across all languages and models—Korean, Arabic, and Hindi are most severe, with some models exceeding 25% TIR (e.g., 28.10 for Qwen2.5-32B-Instruct on Korean, 27.55 for Llama-4-Scout-Instruct on Korean, 27.02 on Hindi), compared to the 5–13% rates in the overall table. English and Spanish still average about 11% TIR when models are incorrect.
• Misalignment also occurs when the answer is right: A separate table of TIR for correct answers shows lower but non-zero rates, ranging from 0.57 to 2.03 in English and from 6.28 to 8.91 in Korean across models.
• Failure modes are evidential, not just logical: Across models, unsupported claims and ambiguous facts make up the majority of detected errors (23% and 24% respectively), while illogical leaps account for 18%. The authors read this as models struggling more with grounding claims in verifiable information than with maintaining internal logical consistency.
• Thinking models show a resource gap in grounding: Qwen3-32B-thinking has 27% of responses flagged for unsupported claims in lower-resource-language settings versus 18% in higher-resource-language settings.

Methodology in Plain English

The authors start from a simple idea: separate a model's reasoning from its answer. For each question, the model produces a chain-of-thought trace ending in an answer marked with <answer></answer> tags. The researchers delete the final answer, leaving a "truncated trace," and hand that trace (plus the original question and options) to an evaluator. The evaluator must decide which answer option the reasoning alone implies, or mark it "inconclusive." If that inferred answer does not match the model's actual answer, the case is counted as misaligned, contributing to the Trace Inconsistency Rate (TIR). Non-English traces are back-translated into English using Command-R+ so that all evaluation happens in one language.

Human evaluation proceeded in two phases. A pilot with 2 expert annotators over approximately 100 reasoning traces from binary tasks used thematic analysis to build the error taxonomy iteratively. The scaled-up phase used stratified sampling across language, model, and cultural sensitivity, producing 72 annotations with 2 annotators per shard for reliability analysis. Annotators judged the inferred answer, logical coherence, information sufficiency, and flagged any problematic reasoning. The same protocol was then replicated automatically using Llama-4-Scout-Instruct as a judge, which was validated against human consensus before being applied at scale. Translation quality was checked by a native Spanish speaker who annotated 30 Spanish examples back-translated to English, finding only 2 of 30 marked with translation errors.

Why This Matters

• Impact on research: The paper argues that answer accuracy alone gives an incomplete picture of multilingual capabilities and that reasoning quality needs its own evaluation track. It also shows that reasoning failures can be predicted automatically once validated against human judgment, which opens the door to reasoning-aware benchmarks rather than accuracy-only leaderboards.
• Real-world applications:
  • Auditing deployed multilingual assistants for cases where they "sound right" but their justifications do not hold.
  • Monitoring safety-critical or regulated domains (medical, legal, financial, civic information) where an unsupported claim is a risk even if the answer happens to be correct.
  • Building better training or fine-tuning data by targeting the dominant failure mode—unsupported claims and ambiguous facts—rather than only incorrect answers.
  • Diagnosing calibration: high TIR on wrong answers may act as a signal of model uncertainty, which could inform abstention or escalation behavior.
• Industry relevance: Any organization shipping the same model to users in many languages needs to know whether quality transfers or only accuracy transfers. The finding that non-Latin scripts carry at least twice the misalignment of Latin scripts is a concrete deployment risk, and the automated judge offers a cheaper monitoring path than repeated human annotation.

Future Directions

• Broaden the judge setup: The authors used only one automated judge and suggest moving to majority voting over 3 judges for more robust outputs.
• Expand annotation and datasets: Human annotation covers a limited number of samples, and evaluation is restricted to GlobalMMLU plus a small sample of boolean comparative questions (Arcuschin et al., 2025); both could be scaled.
• Test even lower-resourced languages: The current language set is described as serving script diversity, but evaluation could extend to languages with fewer resources.
• Improve translation infrastructure: Translators could be updated to models such as gpt-4o and double-checked by additional native speakers for the selected languages.
• Study thinking models further: The authors explicitly encourage more evaluation of extended reasoning models, asking to what extent they can maintain factual grounding in resource-constrained language settings.

Target Audience

This paper is most useful for multilingual NLP researchers and benchmark designers, evaluation and trust-and-safety teams at organizations deploying the same model across many languages, and practitioners who build or audit chain-of-thought systems. Readers who care specifically about whether reasoning quality—not just answer accuracy—transfers across languages, or who need a validated method for detecting reasoning-answer misalignment at scale, will get the most from it.

Authors’ abstract

Large language models demonstrate strong reasoning capabilities through chain-of-thought prompting, but whether this reasoning quality transfers across languages remains underexplored. We introduce a human-validated framework to evaluate whether model-generated reasoning traces logically support their conclusions across languages. Analyzing 65k reasoning traces from GlobalMMLU questions across 6 languages and 6 frontier models, we uncover a critical blind spot: while models achieve high task accuracy, their reasoning can fail to support their conclusions. Reasoning traces in non-Latin scripts show at least twice as much misalignment between their reasoning and conclusions than those in Latin scripts. We develop an error taxonomy through human annotation to characterize these failures, finding they stem primarily from evidential errors (unsupported claims, ambiguous facts) followed by illogical reasoning steps. Our findings demonstrate that current multilingual evaluation practices provide an incomplete picture of model reasoning capabilities and highlight the need for reasoning-aware evaluation frameworks.