CultureConverse: A Multilingual Multi-turn Simulation Harness for Culturally Grounded Assistance in East and Southeast Asia

AI summary

Overview

Research area: Natural Language Processing / cultural evaluation of large language models; multilingual, multi-turn dialogue simulation.

Technical level: Intermediate.

Scope: The paper introduces CultureConverse, a simulation and evaluation harness for measuring how well LLM assistants provide culturally grounded help in multi-turn conversations across 10 East and Southeast Asian regions, together with the CultureConverse-DS dataset and a benchmark of 18 models.

What This Paper Is About

Most cultural evaluations of large language models reduce "culture" to single-turn multiple-choice questions that test factual recall. That misses the common real-world case where a user asks for practical help over several turns — planning a wedding, handling funeral rites, negotiating with a landlord — and success depends on the assistant inferring unstated cultural constraints. CultureConverse builds a simulated, scored multi-turn environment to measure this kind of pragmatic, culturally situated assistance, and releases a large training corpus derived from it.

Key Contributions

1. A simulation harness. CultureConverse evaluates stateful, culturally constrained interactions across 10 East and Southeast Asian regions, 58 subgroup identities, and 7 domains (42 subdomains), grounded in curated Cultural and Taboo Knowledge Bases.
2. A large-scale benchmark. 18 frontier LLMs were benchmarked across 14,610 simulated dialogues, with automated metrics that align strongly with human consensus.
3. A released dataset. CultureConverse-DS contains 274,295 oracle-guided (gold-mode) training dialogues, from which the authors filter the 27,860 highest-quality samples (perfect 3H = 5 scores) for supervised fine-tuning.
4. Evidence of transfer. Fine-tuning two 8B models on those 27,860 samples improves in-domain assistance quality and transfers out-of-domain to cultural MCQ benchmarks and safety classification datasets.

Main Findings

• Automatic metrics track human judgement. Against 42 region- and language-matched annotators (three independent ratings per item, 1,250 gold-mode dialogues), the GPT-5 mini judge reached 90.1% ± 1 agreement and 0.581 MAE versus human consensus, compared with a human–human baseline of 87.5% and 0.727 MAE. Its bias of −0.126 indicates it scores slightly stricter than humans.
• GPT-5 mini gives the best assistance. Across 14,610 episodes, GPT-5 mini scored the highest 3H (assistance quality) at 4.28, while GPT-5.4 had the highest clean rate at 91.0%. GPT-5 mini and GPT-5.4 were highest on TDQ (realism) at 3.81 and 3.80.
• Assistance quality varies much more than realism. 3H ranged from 3.25 to 4.28 across the 18 models, whereas TDQ ranged narrowly (the reported average TDQ was 3.62, with components Naturalness 3.96, Plausibility 3.31, Typicality 3.60). Honesty (3.09–4.60) and Harmlessness (3.45–4.47) drove most of the differences.
• Weak assistants distort the interaction itself. Cross-model variation in TDQ was driven mainly by Plausibility and Typicality, meaning models that fail to track the user's situation or constraints also make the simulated dialogue less realistic.
• Fine-tuning helps, modestly, and transfers. Llama-3.1-8B-IT improved from 3.03 to 3.10 3H in English (+0.07) and 2.46 to 2.55 in native languages (+0.09); SEA-LION-v4-8B-IT improved 2.83 to 2.88 (+0.05) and 2.76 to 2.79 (+0.03). Out-of-domain, Llama-3.1-8B-IT gained +0.88 pp on 7 cultural MCQ datasets and +2.49 pp macro-F1 on 10 safety classification datasets; SEA-LION-v4-8B-IT gained +0.52 pp and +0.80 pp.
• Gold mode raises sample quality. Providing the assistant with oracle context (gold mode) improved 3H by 0.13 pp and clean rates by 5.5 pp relative to eval mode.
• Subgroup disparities remain. GPT-5 mini's 3H varied by only 0.011 across gender but showed a 0.165 spread across identities, scoring systematically lower for historically under-represented groups such as Tamil/Hindu or Indian/Muslim personas in Southeast Asia. Native-language performance was unstable: Gemini models gained +0.10 to +0.25 3H, while DeepSeek V3.1 degraded.
• Adversarial tiers are harder. Scenarios with social friction traps degraded assistance quality by 0.04–0.08 points across all 18 models.
• The dataset scale. The pipeline produced 288,905 episodes total: 14,610 benchmark (evaluation) episodes (7,305 English, 7,305 native) and 274,295 gold-mode training dialogues (137,144 English, 137,151 native). By challenge tier: 144,766 Standard, 72,171 Contextual, 71,968 Normative.

Methodology in Plain English

The harness treats each evaluation unit as an episode: a scored multi-turn dialogue between a simulated, culturally situated user and an assistant under test. Generation follows a five-stage chain:

1. Seed — a persona (region, identity, age cohort, gender) is paired with a parameterised, culture-agnostic template and a challenge tier (Standard, Contextual, or Normative). The paper pairs 348 personas with 420 templates for 146,160 seeds, allocated 50% / 25% / 25% across the three tiers.
2. Blueprint — a seed is expanded into a full scenario using retrieved evidence: a cultural knowledge base of 180,992 region-tagged Wikipedia passages and a taboo knowledge base of 962 curated entries. The blueprint is split into three visibility layers: public context, private user state, and an oracle state (success criteria, taboos, safety tripwires).
3. Shard — the blueprint is decomposed into an ordered, turn-by-turn plan of 3 to 5 phases, each with a user goal and explicit success/failure transition conditions. This enables progressive disclosure: constraints are revealed gradually rather than all at once.
4. Simulation — a user simulator driven by the shard plan converses with the assistant. Episodes run in English or the persona's native language, and in either Eval mode (assistant sees only public context and chat history, mirroring deployment) or Gold mode (assistant receives oracle guidance to produce higher-quality training targets).
5. Evaluation — an LLM judge with full oracle access scores the transcript on 1–5 Likert scales along two axes: Training Data Quality (Naturalness, Scenario Plausibility, Cultural Typicality) and Assistance Quality (Helpfulness, Honesty, Harmlessness, together called 3H), plus a binary "clean" flag for whether any hidden tripwire was violated.

GPT-5 mini serves as both the user simulator and default judge. Because GPT-5 models are also evaluated as target assistants, the authors checked for evaluation circularity by validating the judge against human annotators and by comparing it against a pool of 11 candidate judges and 165 three-judge ensembles. Validation retries failed API calls and regenerates malformed outputs. Fine-tuning used LoRA SFT on the 27,860 perfect-score episodes.

Why This Matters

Impact on research. The paper argues that cultural competence should be measured as situated behaviour rather than agreement with population-level statistics or MCQ recall. It provides an interactive benchmark, a large training corpus, and a human-validated judge recipe, giving the field a way to test pragmatic cultural assistance rather than trivia. It also shows a practical path from simulated cultural dialogue to improvements on existing static benchmarks.

Real-world applications:

• Wedding, funeral, and ritual planning assistance — the paper's own examples include a Malaysian Chinese father negotiating a hotel banquet for a morning tea ceremony and a Singaporean Chinese woman organizing 头七 rites around night shifts and neighbour concerns.
• Everyday consumer and service help in local languages where implicit etiquette norms (food, housing, money, mobility, health, hierarchy) determine whether advice is usable.
• Safety-aware assistant deployment in culturally sensitive contexts, where advice can be fluent but socially inappropriate or in violation of local norms.
• Equity auditing of deployed assistants, using the identity-level breakdown to detect which user groups receive systematically worse help.

Industry relevance. The dataset and harness are released openly (code and dataset at https://github.com/Social-AI-Studio/CultureConverse, CC BY 4.0). The finding that models differ far more on honesty and harmlessness than on conversational realism suggests that product teams should test culturally grounded assistance and safety tripwire avoidance directly, rather than assuming fluency implies competence. The 58-identity breakdown gives a template for fairness auditing across under-represented subgroups.

Future Directions

• Broader human calibration. The authors call for region-specific community review panels and a small human-authored gold dialogue subset to externally calibrate ecological validity scores.
• Stronger training recipes. The paper explicitly frames its fine-tuning gains as a proof of feasibility rather than evidence that CultureConverse-DS supports large or consistent improvements, and asks for joint improvement of the training recipe and human-informed data curation.
• Simulator realism. LLM-driven simulated users tend to collapse into predictable, formulaic prompting patterns; the paper uses stylistic directives to promote diversity, and frames rigorous simulator controls as a critical methodological requirement for future interactive benchmarks.
• Cultural essentialism and taxonomy. Whether equal weighting of 58 subgroups is preferable to proportional representation remains open, as does preventing diverse community practices from collapsing into monolithic stereotypes when using explicit persona labels.

Target Audience

Researchers and practitioners in multilingual NLP, cultural alignment, and LLM evaluation; benchmark and dataset builders interested in interactive, multi-turn assessment; fairness and safety teams auditing assistants across demographic subgroups; and engineers at organizations deploying assistants in East and Southeast Asian markets who need a repeatable way to test culturally grounded behaviour.

Authors’ abstract

Current cultural evaluations for large language models (LLMs) often reduce culture to single-turn factual recall via MCQs, failing to capture a common use case: users seeking practical help over multiple turns in culturally grounded scenarios. We introduce CultureConverse, a scalable, multilingual simulation and evaluation harness for culturally grounded assistant dialogue that covers 10 East and Southeast Asian regions, 58 subgroup identities, and 7 domains. Each simulated and evaluated episode produces a scored interaction where the assistant assists the user and infers cultural constraints from partial information. The resulting CultureConverse-DS dataset contains 14,610 benchmark (evaluation) episodes and 274,295 oracle-guided (gold-mode) dialogues. In our benchmark evaluation of 18 models, GPT-5 mini achieves the highest assistance quality. Human annotation experiments suggest that our evaluation framework is a sufficient proxy for human judgment. Performance gains from fine-tuning on 27,860 high-quality CultureConverse-DS samples improve in-domain assistance and transfer out-of-domain to cultural MCQ and safety classification benchmarks. We release the harness, both splits, and judge prompts to support interactive evaluation of cultural competency.