From National Curricula to Cultural Awareness: Constructing Open-Ended Culture-Specific Question Answering Dataset

AI summary

Overview

• Research area: Natural language processing; cultural alignment of large language models, synthetic data construction, and supervised fine-tuning (SFT) for multilingual/culture-specific question answering.
• Technical level: Intermediate — the paper assumes familiarity with LLM post-training (SFT), LLM-as-a-judge evaluation, and standard cultural-knowledge benchmarks, but the pipeline itself is described in accessible terms.
• Scope in one sentence: The paper presents CuCu, a multi-agent LLM pipeline that converts national social studies curricula into open-ended, culture-specific QA pairs, and applies it to the Korean curriculum to build KCaQA, a 34.1k-pair dataset spanning Korean, English, Chinese, and Japanese, which is then validated through analyses and SFT experiments on cultural knowledge and alignment benchmarks.

What This Paper Is About

LLMs perform unevenly across languages and cultures because their training corpora are dominated by English-centric resources, which leads to culturally hollow or inappropriate outputs in underrepresented regions. High-quality supervision grounded in local civic norms, historical narratives, and social institutions is hard to obtain, and existing multilingual post-training data is mostly general-purpose or translated from English. The paper's goal is to use national curricula — expert-designed, publicly published documents that specify what a society prioritizes and how topics should be framed — as a structured prior for automatically generating open-ended, culture-grounded QA data for supervised fine-tuning.

Key Contributions

1. CuCu, a multi-agent LLM framework ("from national Curricula to Cultural Awareness") that transforms textbook learning outcomes into open-ended, culture-specific question–answer pairs. The pipeline covers initial query generation, culture-sensitive filtering, paraphrasing and augmentation, human validation and refinement, multilingual extension, user-tailored response generation, response evaluation, and response revision.
2. KCaQA, a Korean culture-specific QA dataset comprising 34.1k QA pairs — reported as 34,128 QA pairs (2,844 queries × 3 difficulty levels × 4 models) across four languages (Korean, English, Chinese, Japanese), generated from the 2022 Revised Elementary, Middle, and High School National Curriculum of Social Studies published by the Ministry of Education, Republic of Korea.
3. Quantitative and qualitative validation of the dataset, including topic-modeling comparison against Korean LIMA, readability statistics across languages and difficulty levels, LLM-as-a-judge scoring with GPT-5.2, human inspection, translation-quality scoring (COMET-Kiwi), and cross-lingual semantic similarity.
4. Controlled SFT experiments with Qwen3-14B on five datasets, showing that training on KCaQA gives the strongest gains on both cultural knowledge benchmarks (CLIcK, BLEnD, CulturalBench-Hard) and a cultural alignment benchmark (KorNAT) among the compared conditions.

Main Findings

• Culture-specific topic coverage: Topic modeling produces a Jensen–Shannon Divergence (JSD) of 0.807 between KCaQA and Korean LIMA, a general-domain alignment dataset; a 1k permutation test confirms the difference is significant (p < 0.01). For comparison, KoAlpaca v1.1 and KCaQA-filtered (culture-agnostic queries removed during filtering) yield JSDs of 0.201 and 0.343 respectively. Top-10 skewed topics center on Korean geography, economy, history, climate, and politics.
• Difficulty levels are distinguishable: Readability statistics increase with target level. For Korean, responses average 49.6 tokens / 3.3 sentences / 15.5 tokens per sentence / 0.14 rare-token ratio at the Basic level, 113.9 / 4.8 / 23.9 / 0.23 at Intermediate, and 178.2 / 6.0 / 30.2 / 0.30 at Advanced. The same increasing pattern in length and syntactic density holds for English, Chinese, and Japanese.
• LLM-as-a-judge quality: Using GPT-5.2, KCaQA achieves 0.91 accuracy on language selection, with scores of 8.56 for cultural appropriateness and 7.78 for language use. Per-setting scores in the accompanying table are highest for Korean settings (e.g., Korean Implicit Intermediate: 1.00 language selection, 9.40 cultural appropriateness, 8.70 language use) and lower for Chinese and Japanese (e.g., Japanese Advanced: 0.73, 8.41, 7.89), with the lowest language-selection values at 0.81 for Chinese Basic and Advanced.
• Human agreement with the LLM judge: Across 120 sampled QA instances (Korean and English), inter-rater agreement shows Cohen's Kappa of 0.98, 0.76, and 0.72 for the three evaluation dimensions.
• Multilingual extension quality: COMET-Kiwi scores of 0.941 (Ko–En), 0.924 (Ko–Jp), and 0.917 (Ko–Zh) for 1k sampled parallel queries; cross-lingual semantic similarity using text-embedding-3-large of 0.911 (Ko–En), 0.870 (Ko–Jp), and 0.857 (Ko–Zh) for 1k sampled parallel responses.
• Training gains with Qwen3-14B (CLIcK / BLEnD / CulturalBench / KorNAT): baseline 0.587 / 0.842 / 0.585 / 0.679; + CultureBank 0.611 / 0.859 / 0.607 / 0.655; + Korean LIMA 0.615 / 0.853 / 0.611 / 0.682; + National curriculum 0.633 / 0.866 / 0.615 / 0.721; + CuCu without source data 0.636 / 0.870 / 0.624 / 0.736; + KCaQA 0.662 / 0.891 / 0.634 / 0.744. KCaQA yields the strongest gains on both cultural knowledge and alignment.
• Curriculum alone is weaker than curriculum converted into QA: Training on the raw curriculum improves performance but trails training on KCaQA, which the authors read as evidence that converting curricula into structured, open-ended supervision provides non-trivial, transferable gains.
• The framework design itself contributes: CuCu without source grounding data still improves cultural knowledge and alignment.
• CultureBank does not substantially help: The authors attribute this to its sourcing from Reddit and TikTok and its focus on how people from other cultures perceive cultural differences and culture shock rather than grounded civic norms.
• Cross-cultural adaptability of KCaQA: Of 158 learning outcomes, 118 are classified as Simply Transferred, 35 as Target Modified, and 5 as Sample Removed, with almost perfect inter-annotator agreement (Fleiss' Kappa = 0.86). All Sample Removed outcomes relate to North Korea.

Methodology in Plain English

The authors treat a national curriculum as a ready-made outline of what a society considers important and how it wants topics framed, then build a pipeline that turns each learning outcome into realistic user questions and graded answers.

• Source material: The 2022 Revised Korean National Curriculum for Social Studies. The authors report extracting 357 atomic pairs of learning objectives and achievement criteria and describe filtering out 158 learning outcomes, which form the basis of the generated queries. Social Studies was chosen over other subjects because the others largely cover culture-agnostic general knowledge.
• Query generation: Solar Pro 2 generates and revises Korean queries from each learning outcome, while GPT-4o evaluates them, deliberately separating generator and evaluator to reduce self-style bias. Evaluation uses a 1–5 Likert scale with revision loops of up to five iterations. A culture-sensitive filtering step removes queries that rest on general, culture-agnostic knowledge (for example, a generic question about causes of colonialism is rejected; a question about daily life during the Japanese colonial period is accepted).
• Augmentation: Each query is paraphrased into two additional queries with different style and tone, then expanded into three variants — no country mentioned, implicit reference to "our/my" culture or country, and explicit naming of the culture or country.
• Human validation: One author, a Korean L1 speaker raised and educated under the Korean curriculum, manually validates and refines the generated queries before multilingual extension.
• Multilingual extension: Only the explicitly culture-marked variant is translated into English, Chinese, and Japanese, because implicit Korean assumptions would not carry over in other languages. This yields 2,844 queries (3 paraphrases × (3 Korean variants + 3 translations) × 158 learning outcomes).
• Response generation: Four open-source models — gpt-oss-120b, Qwen3-235B-A22B, Qwen3-Next-80B-A3B-Thinking, and DeepSeek-R1 — each produce Basic, Intermediate, and Advanced answers calibrated to different audiences, lengths, and assumed background knowledge. Answers are evaluated for language consistency, difficulty appropriateness, and alignment with the learning outcome, with revision up to five times. Four models × three difficulty levels × 2,844 queries gives 34,128 QA pairs.
• Validation: Topic modeling with Korean Sentence BERT and BERTopic, readability statistics using wordfreq (rare-token ratio computed over in-vocabulary tokens using the bottom 25% of Zipf-frequency values per language), GPT-5.2 as an LLM judge, human inspection, COMET-Kiwi for translation, and cross-lingual embedding similarity.
• Training setup: Qwen3-14B is fine-tuned for 3 epochs with batch size 16, learning rate 2e-4, warmup ratio 0.03, random seed 42, and gradient clipping disabled, using 4 NVIDIA H200 141GB GPUs with CUDA 12.8. Generation uses temperature 0.0 (greedy decoding), with gpt-4o-2024-08-06 and gpt-5.2-2025-12-11 as the specified proprietary model versions.

Why This Matters

The paper argues that culturally grounded supervision is a practical bottleneck for building culture-adapted or sovereign LLMs, and that national curricula offer an expert-structured, publicly available, and scalable alternative to ad hoc prompts or translated English datasets. It contributes both a reusable framework (CuCu) and a concrete artifact (KCaQA), with code and data released at https://github.com/haneul-yoo/cucu.

• Multilingual and sovereign LLM development: Teams building nationally aligned assistants can generate supervision for their own curricula, reflecting local civic norms and institutions rather than imported framing.
• Cultural-sensitivity evaluation and benchmarking: The dataset and analysis methods support testing whether models answer culture-specific questions in a way that matches local expectations across several languages.
• Educational and tutoring systems: The Basic/Intermediate/Advanced response levels, aligned with learning outcomes and achievement criteria, map directly onto differentiated instruction for different learner ages and backgrounds.
• Cross-cultural dataset adaptation: The Simply Transferred / Target Modified / Sample Removed classification offers a concrete procedure for judging which items can be ported to another culture and which cannot.
• Industry relevance: The authors include researchers from Samsung Electronics and Upstage, and the pipeline uses both proprietary APIs and open-source models with Apache or MIT licenses chosen specifically so that generated data can be used for further model training, which is a practical constraint for commercial deployment.

Future Directions

• Extending CuCu beyond Korea: The authors state the framework generalizes to other nation-states with publicly published curricula, naming the United Kingdom, India, Bangladesh, and Australia as examples, while cautioning about non-uniform curricula, framing shifts tied to the chosen output language, and inherited curriculum bias.
• Addressing the nation-state assumption: The limitations section notes that language, linguistic region, cultural region, ethnicity, and nation-state are distinct concepts, and that the approach is easier for a compact nation-state like South Korea than for nation-states spanning multiple geocultural regions.
• Broadening language coverage within Korea: The work covers Seoul Korean as used in the Republic of Korea and excludes Pyongyang Korean as used in the Democratic People's Republic of Korea and North Korea's educational standards; the authors note some content may not align with North Korean linguistic or educational norms.
• Moving beyond institutionalized knowledge: The authors note that curricula represent institutionalized knowledge rather than the full spectrum of cultural expression, and that the framework could accommodate alternative source materials should such resources become available.
• Handling curricular controversy: The paper calls for practitioners to examine adopted curricula carefully and to use grounded sources rather than author-defined filtering, leaving open how unresolved domestic or international controversies should be handled if they propagate into generated datasets and downstream models.

Target Audience

Researchers and engineers working on multilingual and culturally aligned LLMs, synthetic data generation, and post-training pipelines will find the framework and training comparisons most directly useful. The paper also suits NLP practitioners building datasets for underrepresented languages and cultures, evaluation researchers interested in topic divergence and LLM-as-a-judge validation for culture-specific content, and product or policy teams working on sovereign or nationally aligned AI systems who need a reproducible recipe for curriculum-grounded supervision.

Authors’ abstract

Large language models (LLMs) achieve strong performance on many tasks, but their progress remains uneven across languages and cultures, often reflecting values latent in English-centric training data. To enable practical cultural alignment, we propose a scalable approach that leverages national social studies curricula as a foundation for culture-aware supervision. We introduce CuCu, an automated multi-agent LLM framework that transforms national textbook curricula into open-ended, culture-specific question-answer pairs for supervised fine-tuning (SFT). Applying CuCu to the Korean national social studies curriculum, we construct KCaQA, comprising 34.1k open-ended QA pairs. Our analyses and training experiments suggest that KCaQA covers culture-specific topics and produces responses grounded in local sociocultural contexts.