Thunder-KoNUBench: A Corpus-Aligned Benchmark for Korean Negation Understanding

AI summary

Overview

Research area: Natural Language Processing — evaluation benchmarks for negation understanding in Korean, with corpus-based linguistic analysis and large-scale LLM evaluation.

Technical level: Intermediate. The paper combines linguistic typology, corpus statistics (including KL divergence between distributions), multiple-choice benchmark design, and standard LLM evaluation/fine-tuning protocols.

Scope: The paper analyzes the statistical distribution of Korean negation in a real corpus, builds a 4,784-instance multiple-choice benchmark aligned to that distribution, and evaluates 47 LLMs on it.

What This Paper Is About

Negation is known to be difficult for large language models, and existing negation benchmarks are overwhelmingly English-focused, leaving Korean largely unexamined. The authors first show that both Korean and non-Korean LLMs degrade when negation is present in Korean tasks, then build Thunder-KoNUBench, a sentence-level multiple-choice benchmark whose negation-type distribution is matched to what actually occurs in Korean corpora. The goal is to measure whether models can identify the correct standard negation of a sentence rather than merely reacting to the presence of a negation marker.

Key Contributions

1. Demonstration of a Korean negation gap: The authors show that both Korean and non-Korean LLMs suffer measurable performance degradation when required to reason with Korean negation, including on simple sentence-level tasks.
2. Corpus study of Korean negation: They analyze the statistical distribution of Korean negation types and the clause structures in which negation appears, using a large-scale Korean corpus from the OpenAI Dataset Project.
3. Thunder-KoNUBench: A sentence-level multiple-choice benchmark of 4,784 instances that reflects the empirical distribution of Korean negation phenomena, with low KL divergence from the corpus distribution.
4. Large-scale evaluation and fine-tuning study: They evaluate 47 LLMs (18 Korean, 29 non-Korean) to analyze model size and instruction tuning effects, and they fine-tune 35 models with fewer than 20 billion parameters, finding that cloze-style supervision outperforms symbol-style supervision for learning negation.

Main Findings

• Negation hurts performance in existing Korean tasks: On 7,153 negation-containing KMMLU questions converted to binary-choice format, average accuracy was 62.8 for negative items versus 64.6 for their affirmative counterparts. Korean models scored 63.0 (negative) versus 66.0 (affirmative); non-Korean models scored 62.7 versus 63.6.
• Negation hurts context-sensitive reasoning more sharply: On 1,404 KoBest BoolQ questions converted to negated forms, performance dropped from 67.2 (original) to 53.7 (negated) across all models. Korean models fell from 73.2 to 58.6 and non-Korean models from 62.8 to 50.2, and all 43 models showed a decline.
• Roughly one in ten corpus sentences contains negation: From 29,476 sampled sentences, 3,160 negative sentences were identified, approximately 10.7% of the Korean corpus.
• Larger models are generally more robust: Within each model family, larger models performed better on negation in both cloze and symbol settings, which contrasts with earlier work suggesting larger models are less sensitive to negation. The trend was not strictly monotonic, with a slowdown or temporary decline especially among models with 8 to 12 billion parameters.
• Instruction tuning helps symbols but can hurt cloze: Instruction tuning improved performance in the symbol setting overall, but for non-Korean models it often degraded cloze performance, which the authors attribute to overemphasis on symbol-style multiple-choice formats and to the curse of multilinguality.
• Errors concentrate on local negation: In the cloze setting, more than 90% of incorrect choices were the local negation option, consistent across model families, instruction-tuning settings, model sizes, and performance levels. In the symbol setting, the concentration was reduced but local negation remained the dominant error type. The authors interpret this as models relying on surface-level negation markers rather than capturing negation's semantic effect.
• Cloze-style fine-tuning is the stronger supervision signal: Cloze-style fine-tuning improved Thunder-KoNUBench cloze performance by +34.2 (to 85.1) and improved symbol-format evaluation by +10.5 (to 69.6). Symbol-style fine-tuning improved symbol performance by +30.7 (to 89.8) but cloze performance by only +6.4 (to 57.3).
• Fine-tuning does not harm other tasks: Both fine-tuning styles left ARC Easy (71.5 and 71.4), ARC Challenge (49.2 and 49.2), HellaSwag (70.7 and 70.7), and Winogrande (67.4 and 67.3) essentially unchanged.
• Fine-tuning also helps broader Korean contextual understanding: Cloze-style fine-tuning raised KoBest BoolQ Original from 66.7 (+3.0) and Negated from 53.4 (+0.9), with larger gains than symbol-style fine-tuning.
• Benchmark validity is supported by human performance: Ten external participants each solved 50 randomly selected test questions with internet access restricted; scores ranged from 88.0 to 100.0 with a median of 98.0 and an average of 97.6.

Methodology in Plain English

The authors started with language data rather than building a benchmark from intuition. They took a large Korean corpus from the OpenAI Dataset Project ("Dataset and Large Language Models with Improved Korean Performance"), split it into sentences using the KSS library's split_sentences function, and randomly sampled 30,000 sentences with a fixed random seed of 2025. After removing sentences longer than 400 characters, 29,476 sentences remained. A rule-based detector flagged candidate negations, and the authors manually verified them, yielding 3,160 confirmed negative sentences, which they then categorized by negation type (안-type short and long form, 못-type short and long form, and 말다) and by clause type.

For the benchmark, they define standard negation as a recursive operation that negates the logical relations among main clauses and then negates each clause's predicate, and local negation as negation that applies only to a dependent clause or one of several main clauses. Each item presents an original sentence and four choices: the standard negation (the correct answer), a local negation, a contradiction, and a paraphrase. Original sentences were built by crawling Korean Wikipedia, splitting the text into two-sentence units, and merging each pair into one well-formed sentence with the OpenAI API, followed by manual correction. Standard and local negation options were written manually by the authors, because model-generated versions often negated the wrong part or introduced unintended changes; contradiction and paraphrase options were generated with the OpenAI API and then refined. A different author from the creator verified each item, and disagreements were resolved collectively.

Evaluation used the LM Evaluation Harness in two MCQA settings: cloze, where the model picks the option with the highest length-normalized log-likelihood (acc_norm, normalized by character count), and symbol, where the model picks a label such as A, B, C, or D (acc). Models were tested zero-shot and with 1, 2, 5, and 10 shots, with few-shot demonstrations sampled using three random seeds (1234, 308, 1028) and averaged. Supervised fine-tuning used the 2,500-instance training split on 35 models under 20 billion parameters with LoRA to limit catastrophic forgetting.

Why This Matters

Impact on research: The paper provides the first systematic sentence-level negation benchmark for Korean, a language previously absent from negation evaluation, and it shows that negation distributions in constructed benchmarks can be quantitatively matched to corpus distributions via KL divergence (0.007430 for negation types, 0.413696 for clause types). It also challenges the earlier claim that larger models are less sensitive to negation, and it introduces a benchmark that examines the negation operation itself (mapping P to ¬P) rather than only comprehension of naturally occurring negated sentences.

Real-world applications:

• Korean-language question answering and search systems that must correctly interpret negative queries and negative statements in documents.
• Sentiment analysis and opinion mining on Korean text, where negation reverses the polarity of a statement.
• Machine translation and multilingual assistants serving Korean users, where negation scope errors change meaning.
• Compliance, fact-verification, and safety filtering tools that must distinguish "the user did X" from "the user did not do X."

Industry relevance: The paper's findings on instruction tuning are directly actionable for teams building Korean-language models: instruction tuning for symbol-style multiple-choice formats can improve format proficiency while reducing robustness to negation in a low-resource language. The finding that cloze-style fine-tuning provides richer supervision than symbol selection points to a concrete training-format choice, and the small changes on ARC, HellaSwag, and Winogrande suggest negation-focused fine-tuning can be added without degrading general capabilities.

Future Directions

• Extending the corpus analysis to lexical negation: The authors note that because lexical negation is relational (for example, 있다 "exist" versus 없다 "not exist"), their corpus statistics focused on syntactic negation markers and did not count lexical negation, even though Thunder-KoNUBench includes lexical negation instances.
• Moving to naturalistic evaluation: The current benchmark asks models to identify which option correctly negates a given sentence; the authors state it does not evaluate negation understanding in natural contexts or how models comprehend and reason over negation in context.
• Investigating the non-monotonic scaling behavior: The observed slowdown or decline among models with 8 to 12 billion parameters is offered as a hypothesis about a mismatch between representational complexity and linguistic supervision, which remains to be tested.
• Improving local negation handling: Since over 90% of cloze errors and a majority of symbol errors select the local negation distractor, understanding and correcting this specific failure mode is an open problem.
• Exploring training signals further: The asymmetry between cloze and symbol fine-tuning (10.5% versus 6.4% transfer) suggests that generation-based supervision deserves further study for negation and possibly other low-resource linguistic phenomena.

Target Audience

This paper is most useful for NLP researchers working on negation, evaluation benchmark construction, or Korean-language modeling; for linguists interested in the empirical distribution of Korean negation types and clause structures; and for practitioners at organizations building or deploying LLMs for Korean-language applications, particularly those deciding how to instruction-tune or fine-tune models for low-resource languages. Readers with a background in linguistics but limited machine learning experience will still find the category definitions and corpus statistics accessible, while the benchmark design and error analysis sections assume familiarity with MCQA evaluation and fine-tuning.

Authors’ abstract

Although negation is known to challenge large language models (LLMs), benchmarks for evaluating negation understanding-especially in Korean-are scarce. We conduct a corpus-based analysis of Korean negation and show that LLM performance degrades under negation. We then introduce Thunder-KoNUBench, a sentence-level negation understanding benchmark that reflects the empirical distribution of Korean negation phenomena. Evaluating 47 LLMs on Thunder-KoNUBench, we analyze the effects of model size and instruction tuning, and perform error analysis to better understand model behavior. We further show that fine-tuning on Thunder-KoNUBench improves negation understanding and broader contextual comprehension in Korean.