Do You Know About My Nation? Investigating Multilingual Language Models' Cultural Literacy Through Factual Knowledge

AI summary

Overview

• Research area: Multilingual Natural Language Processing; evaluation of cultural and factual knowledge in large language models.
• Technical level: Intermediate. The paper is accessible to readers familiar with basic LLM benchmarking concepts, though the two novel evaluation metrics (Total Coverage and Smooth Coverage) require some care to interpret.
• Scope: The paper introduces XNationQA, a 49,280-question parallel multilingual benchmark spanning nine countries and seven languages, and uses it to measure how well eight multilingual LLMs recall culturally specific factual knowledge consistently across languages.

What This Paper Is About

Most multilingual question-answering benchmarks are built by translating Western-centric English datasets, so they test language ability without fairly testing whether models actually know facts about different countries and cultures. The authors build a new benchmark (XNationQA) that pairs the same factual questions about nine nations with seven languages, then measure whether models answer consistently regardless of the language they are queried in. The goal is to find out where cultural knowledge lives inside multilingual models and whether it transfers across languages.

Key Contributions

1. XNationQA benchmark: A parallel multilingual dataset of 49,280 multiple-choice questions covering four domains (wars, leaders, monuments, national parks) for nine countries (Japan, India, China, Germany, Spain, Russia, Mexico, USA, UK), translated into seven languages (Japanese, Hindi, Chinese, German, Spanish, Russian, English) from 1,760 factual entities, with four prompt templates per entity to account for prompt sensitivity.
2. Translation quality validation: Three-way validation using semantic cosine similarity (average 0.89), back-translation BLEU (average 54.74), and human expert evaluation by native speakers (average 4.55 out of 5), establishing the corpus as a reliable parallel resource.
3. Two novel transference metrics: Total Coverage (TC), which measures the ratio of facts consistently known across all languages to facts known in any language, and Smooth Coverage (SC), a fuzzy-set extension that credits partial knowledge (e.g., correctly answering 2 of 4 questions about an entity).
4. Empirical characterization of "decoupling": The surprising finding that a model's linguistic competence in a language is largely independent of its cultural knowledge about the nation that speaks that language, meaning models often know more about China or India than Germany or Spain while performing poorly in Chinese or Hindi.

Main Findings

• English dominance across the board: Models perform best when queried in English, followed by other Western languages (German, Spanish, Russian), with Hindi, Japanese, and Chinese consistently trailing.
• Multilingual-specialized models underperform: Bloomz (46 languages) and Aya (101 languages) score near-random (roughly 30%) on date-recall tasks for wars and leaders, losing to narrower models like Mistral and LLaMA of comparable size. Overall ranking: GPT-4 (72%) > Mixtral-8x7B (60%) > LLaMA-3-8B-Instruct (59%) > Mistral-7B (42.6%) > LLaMA-2-7B (41%) > LLaMA-2-13B (40%) > Aya (32%) > Bloomz (30%).
• Topic matters more than nation for difficulty: Location-based recall (monuments, national parks) is easier than year-based recall (wars, leader birth years), where most models hover near chance.
• Poor cross-lingual transfer in open-source models: Total Coverage across all seven languages is often near 5% for open-source models. GPT-4 reaches 62.59% TC for monuments while most open-source models stay under 15%, revealing a severe gap in knowledge consistency, not just accuracy.
• Western-language bias in transfer: TC scores jump by 18x to 30x when restricted to Western languages (English, German, Spanish, Russian) versus all languages, showing that open-source models store cultural facts disproportionately in Western-language representations.
• Western language skill does not mean Western knowledge: Despite strong performance in Western languages, models sometimes know more about China, India, and Japan than about Germany or Spain, contradicting the intuition that Western-language fluency implies Western cultural literacy.
• Counterintuitive pre-training anomalies: Mixtral performs better in Russian than in Hindi, Japanese, or Chinese despite English/German/Spanish-heavy pre-training; LLaMA-3 similarly favors Russian over Hindi despite reportedly stronger Hindi optimization, indicating that pre-training data composition alone does not explain the results.

Methodology in Plain English

The authors mined English Wikipedia for lists of country-specific entities (national leaders, national parks, UNESCO heritage sites, and wars) for nine nations. They then enriched each entity with factual data from Wikidata, such as the birth year of a leader, the start year of a war, or the administrative location of a park or monument. Each entity was turned into four multiple-choice questions using manually written templates, with three distractor options generated either from random other administrative regions or from random offsets of 5–10 years from the correct answer. The templates and entity-option pairs were then translated into six additional languages using Google Translate and GPT-4, producing a parallel corpus where the same factual question exists identically in seven languages.

To validate translation quality, they measured cosine similarity of multilingual sentence embeddings between English and each translation, performed back-translation with BLEU scoring, and had native language experts rate a sample on grammar, fluency, and coherence. They then prompted eight instruction-tuned multilingual LLMs with each question in each language and recorded whether the model produced the correct option.

Two new metrics were introduced to go beyond per-language accuracy. Total Coverage asks: of all the facts a model knows in at least one language, what fraction does it know in all seven? Smooth Coverage softens this by giving partial credit per entity, computing the minimum membership score across languages divided by the maximum, averaged over entities. Both metrics were computed under four scenarios: all languages, pre-training languages only, Western languages only, and English-to-non-Western transfer.

Why This Matters

This paper is one of the first large-scale parallel benchmarks that explicitly balances factual coverage across culturally distinct nations rather than translating a Western dataset. It reframes how multilingual evaluation should be done: accuracy in a language is not the same as knowledge about the culture that speaks it, and the two can diverge sharply.

Impact on research:

• Provides a reproducible benchmark and metric suite (TC, SC) that future work can use to measure cross-lingual knowledge transfer, not just language ability.
• Challenges the common assumption that more languages in pre-training means better multilingual and multicultural performance, showing that Bloomz and Aya underperform narrower peers.
• Introduces "cultural literacy" as an operational concept grounded in educational theory (Hirsch, 1983) and applies it to LLM evaluation.

Real-world applications:

• Global customer service and virtual assistants: Companies deploying chatbots in Hindi, Japanese, or Chinese can use XNationQA to check whether the model actually knows local history, landmarks, and politics rather than just producing grammatical output.
• Educational technology and tutoring systems: A tutoring system in India, Mexico, or Spain needs factual knowledge about those countries; this benchmark exposes which models can reliably supply it.
• Fact-checking and content moderation: RAG pipelines and automated fact-checkers need consistent factual recall across languages; the low TC scores warn against assuming knowledge will transfer from an English source.
• Localized search and recommendation: Search engines localizing content for non-Western markets can measure whether their underlying LLM has the cultural coverage required to serve those markets equitably.

Industry relevance: The clear gap between proprietary (GPT-4) and open-source models on transference suggests that deploying open-source multilingual LLMs for global products risks uneven quality across regions. Concerns about inclusivity for a diverse global user base make this a product and trust issue, not just a research one.

Future Directions

1. Evaluate larger and newer multilingual models: The authors could not test very large models due to compute constraints, and they note that benchmarking the cultural literacy of colossal models (and newer open models like Qwen3, which they partially explored) remains an important next step.
2. Extend geographic coverage: XNationQA currently omits Africa and South America entirely; expanding to countries with less-resourced languages would test whether the Western-language bias is universal or specific to the current country mix.
3. Analyze pre-training data to explain the anomalies: The paper stops at benchmarking, leaving open the question of why linguistically inclusive models like Bloomz and Aya underperform. Investigating the topic distribution and biases of public corpora like BLOOM could reveal whether data mixture or data quality is the cause.
4. Investigate the decoupling of language and knowledge: Understanding why models possess facts about a nation without being able to express them in its language is a fundamental question about how LLMs store knowledge internally, and could inform better fine-tuning or retrieval strategies.

Target Audience

This paper is most useful to:

• Multilingual NLP researchers designing evaluation benchmarks or studying cross-lingual representation and transfer.
• LLM practitioners and product teams deploying multilingual models in non-Western markets who need to know whether open-source vs. proprietary models meet their cultural knowledge needs.
• AI ethics and fairness researchers working on linguistic and cultural inclusivity, who will find concrete evidence of Western-language and Western-content bias.
• Graduate students and advanced undergraduates in NLP looking for a well-documented example of benchmark construction, translation validation, and metric design with clear research questions and reproducible code.

Authors’ abstract

Most multilingual question-answering benchmarks, while covering a diverse pool of languages, do not factor in regional diversity in the information they capture and tend to be Western-centric. This introduces a significant gap in fairly evaluating multilingual models' comprehension of factual information from diverse geographical locations. To address this, we introduce XNationQA for investigating the cultural literacy of multilingual LLMs. XNationQA encompasses a total of 49,280 questions on the geography, culture, and history of nine countries, presented in seven languages. We benchmark eight standard multilingual LLMs on XNationQA and evaluate them using two novel transference metrics. Our analyses uncover a considerable discrepancy in the models' accessibility to culturally specific facts across languages. Notably, we often find that a model demonstrates greater knowledge of cultural information in English than in the dominant language of the respective culture. The models exhibit better performance in Western languages, although this does not necessarily translate to being more literate for Western countries, which is counterintuitive. Furthermore, we observe that models have a very limited ability to transfer knowledge across languages, particularly evident in open-source models.