KoSimpleQA: A Korean Factuality Benchmark with an Analysis of Reasoning LLMs

AI summary

Overview

Research area: Natural Language Processing — LLM factuality evaluation, multilingual and culturally grounded benchmarking.

Technical level: Intermediate. Readers should be comfortable with standard LLM evaluation terminology (accuracy, abstention/refusal rates, calibration error), but no specialized mathematical background is required.

Scope: This paper introduces KoSimpleQA, a 938-question human-curated Korean factuality benchmark built on the SimpleQA paradigm, benchmarks a range of open-source Korean-supporting LLMs on it, and analyzes how explicit reasoning affects factual accuracy, refusal behavior, and cross-lingual knowledge gaps.

What This Paper Is About

LLMs frequently produce factually wrong answers, and existing factuality benchmarks such as SimpleQA and Chinese SimpleQA cover English and Chinese but not Korean. Simply translating an English benchmark is insufficient, because many of its questions depend on Anglophone cultural context rather than knowledge that Korean-centric models would hold. The paper's goal is to build and release a Korean benchmark centered on Korean cultural knowledge, and to use it to measure model performance, calibration, and the effect of reasoning.

Key Contributions

1. Release of Korean SimpleQA (KoSimpleQA), described as the first benchmark targeting factuality in Korean cultural contexts — 938 human-authored, short, fact-seeking questions with unambiguous answers, available at https://github.com/naver-ai/KoSimpleQA.
2. A comprehensive evaluation across diverse open-source LLMs that support Korean, grouped into Korean Community LLMs (EXAONE 4.0, HyperCLOVA X SEED, Kanana 1.5, VARCO) and Multilingual LLMs (Gemma 3, Llama 3.1, Qwen3), establishing new baselines and showing that rankings differ from English SimpleQA.
3. An analysis of reasoning models ("-Think" variants), offering preliminary insight into the role of reasoning in factual question answering — specifically that it mitigates cross-lingual knowledge gaps and elicits latent knowledge rather than substituting for missing knowledge.
4. Refined evaluation practice: the authors adopt the SimpleQA metric suite but tighten the NOT_ATTEMPTED criteria so vague responses or false refusals are penalized as incorrect, and they add Expected Calibration Error (ECE) for quantitative calibration analysis.

Main Findings

• Difficulty: Even the strongest model evaluated generates a correct answer only 31.6% of the time. That model is HCX SEED-14B (CO = 31.6, F-score = 32.6).
• Scaling helps within families: Performance improves monotonically with model size inside each family. Scaling HCX SEED from 0.5B to 14B raises F-score from 3.6 to 32.6.
• Pre-training data distribution may matter more than parameter count: HCX SEED-14B achieves the highest F-score (32.6), roughly doubling Llama 3.1-70B (15.7) despite being five times smaller. Kanana 1.5-2.1B (15.3) exceeds Qwen 3-32B (10.6), and Gemma 3-27B (28.3), the strongest multilingual baseline, lags behind smaller Korean models.
• Persistent overconfidence: Better factual performance generally corresponds to better (lower) ECE. HCX SEED-14B reaches ECE 61.5 and Kanana 1.5-8B reaches 64.1, versus 71.4 for the larger Llama 3.1-70B. Across the dataset, models frequently assign confidence near 100% even when producing incorrect responses.
• Rank reversal versus English SimpleQA: Llama 3.1-70B drops from #1 on SimpleQA to #6 on KoSimpleQA, while HCX SEED-14B jumps from #8 to #1 — indicating performance on one culture's knowledge does not generalize to another's.
• Same knowledge, different behavior: When KoSimpleQA questions were translated into English and verified by humans, only the query language changed, yet behavior shifted sharply. HCX SEED-14B showed a large increase in Not Attempted (NA = 44.9 in English vs 6.1 in Korean) with decreased correct answers, while Llama 3.1-70B showed pronounced increases in both CO and NA.
• Reasoning improves results modestly and mainly where knowledge is latent: HCX SEED-14B-Think improves F-score from 32.6 (Base) to 34.3 (Think) and improves calibration to ECE 55.8, the highest accuracy on the benchmark. Qwen3 thinking variants also improve (for example, Qwen 3-8B from 5.3 to 7.9 F-score).
• Reasoning gains are larger in the model's weaker language: HCX SEED-14B gains 2.4 in Korean but 6.8 in English; Qwen 3-32B gains 0.4 in English but 2.0 in Korean.
• Reasoning reduces excessive refusal: HCX SEED-14B shows a reduction of 44.3 in NA on English data (NA 44.9 to 0.6).
• Reasoning is not a substitute for knowledge: In a qualitative case study, HCX's reasoning corrected an attribute hallucination by retrieving the canonical list of "1st-Gen New Towns," whereas Qwen 3-32B exhibited "ontological collapse," producing plausible-sounding but fabricated town names in both direct and reasoning modes.
• Not all reasoning training is equal: EXAONE 4.0-32B-Think did not show substantial improvement over its base version (F-score 16.2 versus 17.3, ECE 77.4 versus 68.4), which the authors hypothesize stems from differences in how reasoning capabilities were trained and integrated.

Methodology in Plain English

The benchmark was built by human annotators recruited through two Korean crowdsourcing platforms, Selectstar and Flitto, all of whom had to pass assessments from the data partners. Annotators wrote questions grounded in Korean cultural context — interpreted broadly to include both traditional heritage and modern socio-cultural trends — and each had to record up to four reliable source links.

Three constraints governed the questions. First, a difficulty requirement: a question was only kept if at least one of four strong closed-source models (HCX-005, gpt-4o-2024-05-13, gemini-2.0-flash, and claude-3-5-sonnet-20240620) failed to answer it correctly. Second, unambiguous answers: each question must have exactly one definitive short answer, and where multiple items form the answer (for example, the three symbolic meanings of the white background in the South Korean flag), all are required. Third, a temporal constraint: everything must be answerable using knowledge available up to December 31, 2023, to keep items stable.

Quality control was a two-stage process: cross-platform validation, where items written on one platform were checked by annotators from the other, followed by manual expert review by NLP researchers to fix typos, ambiguity, and time-sensitivity issues.

Evaluation follows SimpleQA's metrics judged by GPT-4o (from OpenAI's simple-evals implementation): Correct (CO), Not Attempted (NA), Incorrect (IN), Correct Given Attempted (CGA), and F-score, plus Expected Calibration Error (ECE) computed on attempted questions only with a bin size of 10. The NOT_ATTEMPTED criterion was tightened so vague answers or false claims that no record exists are counted as Incorrect, not NA. Generation used temperature 1.0, top_p 1.0, and a maximum generation length extended to 8192 tokens to accommodate reasoning models; every metric is the average of three runs with different random seeds. The authors also created a translated version of KoSimpleQA in English, passing through one round of human verification, to run the language-variation experiment.

Dataset statistics: questions average 29.22 characters (minimum 9, maximum 71) and answers average 5.24 characters (minimum 1, maximum 25).

Why This Matters

Impact on research: The paper shows that factuality benchmarks do not transfer cleanly across languages and cultures — the rank reversal between SimpleQA and KoSimpleQA is direct evidence that a high score on an English-centric benchmark is a poor proxy for culturally grounded Korean knowledge. It also supplies a reusable, human-curated, open-source resource and a stricter abstention/calibration protocol that other language communities can replicate.

Real-world applications:

• Deploying Korean-language assistants for information access, education, or cultural content delivery, where the benchmark indicates current models remain far from reliable.
• Evaluating and selecting models for Korean public-facing services, using culturally relevant questions rather than translated English ones.
• Informing data collection priorities for regional model developers, since the results suggest region-specific pre-training corpora outweigh generic scale.
• Guiding reliability engineering around abstention and confidence, since models assign near-100% confidence even to wrong answers.

Industry relevance: The paper comes from Naver Cloud and Seoul National University, and its findings bear directly on commercial decisions about which models to build or adopt for Korean markets, and on how much to invest in localized pre-training data versus simply scaling model size.

Future Directions

• Mitigating overconfidence in culturally specialized models. The authors explicitly frame their calibration findings as a foundation for applying existing mitigation techniques and as groundwork for future calibration research, rather than as a new algorithm.
• Understanding why reasoning training yields uneven benefits — EXAONE's lack of improvement suggests effectiveness depends on how reasoning is learned and integrated during training, an open question the paper raises but does not resolve.
• Extending the localized-SimpleQA paradigm to other languages and cultural communities, following the path from English SimpleQA to Chinese SimpleQA to KoSimpleQA. The paper notes KoLasSimpleQA covers nine languages but is not publicly available, has relatively small per-language sample sizes, and uses less rigorous difficulty filtering.
• Building on the cross-lingual knowledge gap analysis. The authors deliberately avoid claiming a single cause, noting disparities may arise from language-specific instruction tuning, refusal alignment, decoding behavior, or pre-training distributions — each a candidate direction for follow-up work.

Target Audience

Researchers and practitioners working on LLM evaluation, multilingual and low-resource language modeling, and factual reliability or hallucination reduction will benefit most. It is also useful for teams building or deploying Korean-language products who need to choose models or justify investment in localized training data. Readers interested in calibration and abstention behavior in LLMs will find the ECE analysis and tightened NOT_ATTEMPTED criteria applicable beyond Korean.

Note on unavailable content: the supplied paper text is truncated partway through Appendix D, so the full annotation guidelines, evaluation prompts (Appendices D, E, G) and complete SimpleQA comparison results (Appendix F) are not available in the source provided.

Authors’ abstract

We present $\textbf{Korean SimpleQA (KoSimpleQA)}$, a benchmark for evaluating factuality in large language models (LLMs) with a focus on Korean cultural knowledge. KoSimpleQA is designed to be challenging yet easy to grade, consisting of 938 short, fact-seeking questions with unambiguous answers. We conduct a comprehensive evaluation across a diverse set of open-source LLMs of varying sizes that support Korean, and find that even the strongest model generates correct answer only 31.6% of the time, underscoring the challenging nature of KoSimpleQA. Notably, performance rankings on KoSimpleQA differ substantially from those on the English SimpleQA, highlighting the unique value of our dataset. Furthermore, we observe that reasoning helps mitigate the cross-lingual knowledge gap in LLMs, which refers to disparities in their ability to manifest knowledge across languages. KoSimpleQA can be found at https://github.com/naver-ai/KoSimpleQA.