Ko-MuSR: A Multistep Soft Reasoning Benchmark for LLMs Capable of Understanding Korean

AI summary

Overview

Research Area: Natural Language Processing — multilingual and Korean-language large language model evaluation, long-context reasoning benchmarks.

Technical Level: Intermediate. Readers should be comfortable with LLM evaluation concepts (few-shot prompting, chain-of-thought, benchmark contamination), but the paper is empirically focused and does not require deep technical background to follow.

Scope: This paper introduces Ko-MuSR, the first contamination-resistant Korean benchmark for multistep "soft" reasoning over long narratives, and uses it to compare multilingual and Korean-specialized LLMs under various prompting strategies.

What This Paper Is About

Existing Korean NLP benchmarks largely test short-context knowledge, mathematics, or coding, and the few long-context Korean benchmarks that exist (e.g., CSAT-QA) are built from publicly available exam material, making them vulnerable to training-data contamination. Ko-MuSR addresses this gap by synthesizing entirely new, human-verified Korean narratives and questions that require multi-hop reasoning across long contexts, then using the benchmark to probe whether reasoning ability transfers across languages.

Key Contributions

1. A new benchmark: Ko-MuSR, the first Korean benchmark designed specifically for long-context, multistep soft reasoning, containing 250 questions across three subtasks (Murder Mysteries, Object Placements, Team Allocations) and built to minimize contamination.

2. Rigorous human validation: Every item was solved by the authors in a two-phase review, and a subset was additionally solved by five external participants with AI tools disabled, confirming answerability, logical consistency, and establishing human performance baselines.

3. Cross-lingual reasoning evidence: Systematic comparison of two multilingual LLMs (Qwen3 32B, LLaMA 4 Scout Instruct) against two Korean-specialized LLMs (Exaone 4.0 32B, HyperCLOVA X SEED Think 14B), showing that multilingual models outperform Korean-specialized ones even on Korean reasoning tasks.

4. Prompting analysis: Demonstration that carefully composed prompting (three few-shot examples + chain-of-thought + task-specific hints) substantially closes the gap to human accuracy for large models, while producing inconsistent or even harmful effects on small language models.

Main Findings

• Multilingual models beat Korean-specialized models on Korean reasoning. In zero-shot, direct, no-hint evaluation, Qwen3 32B achieved the highest accuracy on both English and Korean subtasks, while the Korean-focused Exaone 4.0 32B and HyperCLOVA X SEED Think 14B lagged notably — for example, HyperCLOVA scored 44.6% on Korean Murder Mysteries versus Qwen3's 67.6%. This suggests multistep reasoning ability transfers across languages and is driven more by general reasoning structure than by linguistic specialization.

• Model ranking tracks pretraining scale. Zero-shot accuracy ordering aligned with estimated pretraining tokens: Qwen3 32B (36T+ tokens) > LLaMA 4 Scout Instruct (~30T multimodal tokens) > Exaone 4.0 32B (14T tokens) > HyperCLOVA X SEED Think 14B (6T tokens).

• Structured prompting dramatically improves accuracy. The 3-Shot/CoT/Hint configuration raised results across all four large models. On Korean Team Allocations, three models exceeded human accuracy (87.5%) and one matched it. Hints alone were the single most influential component in ablations.

• Small language models behave differently. Applying the same 3-Shot/CoT/Hint configuration to five SLMs (Qwen3 0.6B/1.7B/4B, Exaone 4.0 1.2B, Exaone 3.5 2.4B) produced inconsistent results. The smallest model, Qwen3 0.6B, improved in only one subtask, and Qwen3 1.7B dropped 12.7 points on English Object Placements — echoing prior findings that in-context learning benefits scale with model size.

• Cross-lingual reasoning generalizes beyond culture-specific knowledge. Korean reasoning performance on Ko-MuSR mirrored results on HRMCR (a culturally grounded Korean benchmark), reinforcing that multilingual LLMs handle culturally situated reasoning, not just surface-level language tasks.

Methodology in Plain English

The researchers adapted the existing English MuSR benchmark synthesis pipeline rather than hand-writing questions. The process has three stages:

1. Sampling: Seed material (crime scenes, object types, Korean character names) is drawn or generated, with Korean-specific cultural adaptations produced via GPT-4o.
2. Reasoning tree expansion: An LLM (GPT-4o, or o1 for the harder Object Placements task) builds a structured tree of facts and deductions that logically determine the correct answer.
3. Narrative generation: An LLM writes a fluent Korean story consistent with the fact nodes, so that the answer can be derived through multi-hop reasoning over the text.

All prompts were translated into Korean to prevent English keywords from trivializing the reasoning. The synthesized items then went through two rounds of human review: first, each author independently solved every problem and flagged it as pass, minor issue, or fail; second, ambiguous items were independently re-checked by two annotators and kept only if both agreed.

For evaluation, four large LLMs and five small LLMs were tested on both the original MuSR (English) and Ko-MuSR (Korean), across multiple prompting configurations combining different numbers of few-shot examples, direct versus chain-of-thought responses, and presence or absence of task-specific hints. Each evaluation was run twice and averaged, using vLLM, transformers, and lm-evaluation-harness.

Why This Matters

Impact on research: Ko-MuSR fills a specific gap in Korean NLP evaluation — long-context, multistep reasoning with low contamination risk — and provides evidence that reasoning capability transfers across languages more readily than specialized training does. It also gives the community a controlled testbed for studying how prompting strategies interact with model scale and linguistic specialization.

Real-world applications:

• Korean-language AI assistants and chatbots: Enterprises deploying Korean LLMs for customer support, legal document review, or narrative analysis can use Ko-MuSR to check whether a model actually reasons over long Korean documents or merely pattern-matches.
• Model selection and procurement: Organizations choosing between multilingual and Korean-specialized models gain concrete evidence that multilingual models may be the safer default for reasoning-heavy Korean tasks.
• Educational technology: Evaluating AI tutors for Korean students on problems requiring multi-step inference over passages mirrors real reading-comprehension pedagogy.
• Government and public-sector AI: Korean public institutions investing in sovereign AI can use the benchmark to validate claims of competitive Korean reasoning performance.

Industry relevance: The finding that Korean-specialized models underperform multilingual ones — despite substantial national investment (e.g., HyperCLOVA X, Exaone) — is directly relevant to Korean AI companies, national AI strategies, and any vendor claiming language-specialized superiority.

Future Directions

• Extend beyond Korean. The authors explicitly note the benchmark covers only English and Korean; expanding the methodology to other under-resourced languages would test whether the cross-lingual reasoning findings generalize.
• Close the gap for Korean-specialized models. Improving specialized models to match multilingual performance — especially on the Object Placements subtask, where Korean models remained weak even with good prompting — is an open research target.
• Better prompting for small models. Since in-context learning degrades SLM performance, developing prompting or fine-tuning methods tailored to small models is a clear next step.
• Broaden task coverage and evaluation scale. The benchmark has three subtasks; adding diverse reasoning types (temporal, causal, spatial beyond current designs) and running more than two evaluation trials would strengthen reliability.

Target Audience

This paper is most valuable to NLP researchers working on Korean or multilingual LLM evaluation, benchmark designers interested in contamination-resistant synthetic data pipelines, and practitioners selecting or deploying Korean-capable models for long-context reasoning applications. It is also relevant to product and strategy teams at companies building Korean AI services, and to policymakers assessing the competitiveness of language-specialized national AI efforts.

Authors’ abstract

We present Ko-MuSR, the first benchmark to comprehensively evaluate multistep, soft reasoning in long Korean narratives while minimizing data contamination. Built following MuSR, Ko-MuSR features fully Korean narratives, reasoning chains, and multiple-choice questions verified by human annotators for logical consistency and answerability. Evaluations of four large language models -- two multilingual and two Korean-specialized -- show that multilingual models outperform Korean-focused ones even in Korean reasoning tasks, indicating cross-lingual generalization of reasoning ability. Carefully designed prompting strategies, which combine few-shot examples, reasoning traces, and task-specific hints, further boost accuracy, approaching human-level performance. Ko-MuSR offers a solid foundation for advancing Korean NLP by enabling systematic evaluation of long-context reasoning and prompting strategies.