Common to Whom? Regional Cultural Commonsense and LLM Bias in India

AI summary

Overview

• Research area: Natural Language Processing / cultural commonsense reasoning and bias evaluation in large language models, with a focus on sub-national cultural variation in India.
• Technical level: Intermediate. The paper uses standard benchmark construction and evaluation techniques (human annotation, LLM-as-judge scoring, multiple-choice bias testing, chi-square goodness-of-fit), described with enough procedural detail to be reproducible.
• Scope (one sentence): The paper introduces Indica, a human-annotated benchmark of 515 questions yielding 1,630 region-specific question-answer pairs across five Indian regions, and uses it to show that cultural commonsense in India is regional rather than national and that eight state-of-the-art LLMs both fail on region-specific knowledge and systematically default to Central and North Indian practices.

What This Paper Is About

Existing cultural commonsense benchmarks treat each country as culturally uniform, as if all citizens of a nation share the same everyday practices and norms. This assumption is questionable in a country like India, which has 28 states, 8 union territories, and 22 official languages. The paper asks whether Indian cultural commonsense is actually shared nationally or varies by region, builds a benchmark to test that question, and then measures whether LLMs know region-specific practices and which regions they implicitly treat as the default "Indian" culture.

Key Contributions

1. Indica, the first benchmark for sub-national cultural commonsense in India. It contains 515 questions spanning 8 domains of everyday life, with gold-standard answers from five regions (North, South, East, West, Central), producing 1,630 region-specific question-answer pairs.
2. Empirical evidence that Indian cultural commonsense is regional, not national. Only 39.4% of questions achieve agreement across all five regions, and pairwise agreement rates between regions range from 60.1% to 68.3%.
3. Two complementary evaluation tasks. Region-Anchored Short Answer (RASA), which tests whether models can generate region-specific knowledge when the region is given, and Region-Agnostic Multiple Choice Questions (RA-MCQ), which tests which regions' practices models pick when the region is not given.
4. A generalizable, transferable methodology. The pipeline runs from OCM-grounded question creation, to regional response collection, to consensus-based gold standard establishment, to bias measurement, and the paper illustrates how it transfers to another culturally heterogeneous nation, China.

Main Findings

• Regional variation dominates: Of 132 questions where all five regions provided valid answers, only 52 (39.4%) reached unanimous agreement. The paper states this confirms that cultural commonsense in India is predominantly regional, not national.
• Regional coverage is uneven: West India has intra-region consensus on 354 of 515 questions (68.7%), Central on 348 (67.6%), North and South on 326 each (63.3%), and East on 276 (53.6%). The lower East figure is attributed to greater internal diversity within that region.
• Closest and most distant region pairs: North-Central shows the highest pairwise agreement (68.3%), followed by West-Central (65.0%) and North-West (63.7%). South-East shows the lowest (60.1%).
• Domain matters for agreement: Universal agreement is highest for Traffic & Transport Behavior (22.6%), Education (13.8%), Clothing & Adornment (13.6%), and Communication (12.8%). It is lowest for Finance (7.5%), Food Processing & Consumption (6.0%), and Festivals & Rituals (1.8%).
• Models capture broad concepts but not regional specifics: On RASA, overall accuracy is tightly clustered between 49.5% and 52.6% (a range of 3.1 percentage points), but fully correct rates are only 13.4%–20.9%. The majority of responses (61.3%–75.3%) are only partially correct.
• Overexplaining is the dominant failure mode: An analysis of 100 partially correct Claude Sonnet 4.5 responses found overexplaining in 89% of cases, underspecifying in 1%, and both in 10%.
• Performance is uniform across regions and somewhat variable across domains: Fully correct rates differ by only 3–5 percentage points between regions, with North (14.3%–21.5%) and Central (13.4%–20.9%) marginally higher, and overall accuracy clustering at 49–54% across all regions. By domain, fully correct rates are highest for Traffic & Transport (20.3–32.4%) and Communication (18.6–29.7%), and lowest for Clothing & Adornment (5%–12.9%) and Finance (10.7%–16.9%); under overall accuracy these domain gaps compress from 17.7 to 8.5 percentage points, with most domains converging to 48–56%.
• All models show geographic bias on RA-MCQ: Against a uniform baseline of approximately 20% per region, all eight models deviate significantly (chi-square goodness-of-fit, p < 0.001). Central India is over-selected at 24.7%–28.8% (1.25–1.46× expected) and North India at 22.4%–26.1% (1.14–1.32× expected), with standardized residuals above +2.0.
• Strongest single over-selection: Central India is selected most by Gemini (28.8%, 1.46× expected), Qwen (27.8%, 1.41× expected), and GPT-5.2 (27.8%, 1.40× expected).
• West and East are under-represented: Most models under-select West India (12.9%–17.7%, 0.73× expected) and East India (13.3%–18.9%, 0.82× expected), with standardized residuals below -2.0. South India shows variable patterns (16.6%–19.9%, 0.88× expected).
• Proposed explanation for the default: The paper points to three reinforcing factors: Hindi (spoken primarily in North and Central states) dominating Indian-language web content, with South and East Indian languages such as Assamese, Odia, and Kannada having 10–100x less data in Common Crawl; multilingual tokenizers fragmenting underrepresented languages into less meaningful subword units; and Hindi-language media and Bollywood, centered in North/Central India, dominating cultural representation of India.
• Indica versus prior Indian benchmarks: In the paper's comparison table, Indica is the only benchmark marked as covering regional variation, commonsense, and everyday practices simultaneously; MILU, SANSKRITI, and IndicQuest are marked as not covering regional variation or commonsense, while DOSA, IndiBias, and FairI Tales cover only some of these dimensions.

Methodology in Plain English

Building the question set. The researchers anchored their topic selection in the Outline of Cultural Materials (OCM), an anthropological taxonomy with 90+ major categories and 700+ subcategories. They chose 8 domains relevant to everyday life: Interpersonal Relations, Education, Clothing and Adornment, Food Processing and Consumption, Communication, Finance, Festivals and Rituals, and Traffic and Transport Behavior. Within these they selected 18 OCM subcategories using three criteria (enough diversity to support multiple topics, non-overlapping practices, and everyday rather than institutional knowledge). GPT-4-0613 generated 8–10 candidate topics per subcategory, which the authors manually filtered down to 2–4 topics each, yielding 39 topics. They hand-wrote 3–8 seed questions per topic to demonstrate the desired open-ended, culturally grounded style, then used GPT-4-0613 to generate additional questions, targeting 15+ per topic. All questions were manually reviewed to remove ambiguity, eliminate redundancy, and exclude institutional facts. This produced 611 unique questions.

Collecting human answers. They recruited 5 participants per region through Prolific, requiring each to have lived in their region for the majority of their life, and each participant answered all 611 questions, giving 15,275 total responses (5 responses per question per region). The study was IRB-approved with fair-wage compensation.

Establishing gold answers. For each question within a region, at least 4 of 5 participants had to give semantically equivalent answers. GPT-4o provided an initial agreement assessment, and two independent annotators then manually reviewed every question using a custom annotation tool that displayed the model's preliminary classification alongside all raw participant responses. Inter-annotator agreement was perfect (Fleiss' κ = 1.0), and humans overrode GPT-4o in 7.6% of intra-regional, 28.9% of inter-regional, and 24.5% of universal cases. Of the 611 original questions, 515 (84.3%) achieved agreement in at least one region and were retained. The same process was applied to all 10 pairs of regions (inter-region agreement, requiring exactly the same cultural practice, not partial overlap) and to universal agreement across all five regions.

Evaluating models. Eight models were tested: Claude Sonnet 4.5, Gemini 3 Flash, GPT-5.2, Grok-4 Fast, DeepSeek-V3.2, Llama 3.3 70B, Mistral Large 3, and Qwen3-VL. Each question was run 30 times at temperature 1.0. In RASA, the region name is prepended to the question (for example, "In South India, what is the most common gift given to new parents?"), and free-form answers are scored by an LLM judge, Gemini 3.0 Flash, as Correct (1.0), Partially Correct (0.5), or Incorrect (0.0), with per-question averages taken over runs. The judge was validated on 200 responses (100 Qwen, 100 Gemini) by two independent human annotators, with inter-annotator agreement of 94%–100% (97% overall) and LLM-human agreement of 87%–88%. In RA-MCQ, 79 questions were built from cases where three or more regions gave distinct consensus answers; the model sees options such as Jewelry, Bangles, Clothes, Sweets without any region labels, while the hidden mapping records which region each option came from. Credit is split equally when an option represents multiple regions, and bias is tested with a chi-square goodness-of-fit test against the approximately 20% uniform expectation.

Why This Matters

Impact on research. The paper challenges a working assumption in cultural NLP: that a country is a valid unit of cultural analysis. It provides a reusable protocol, from OCM-grounded question design through regional annotation to statistical bias measurement, and shows that this protocol transfers to any culturally heterogeneous nation rather than being India-specific. It also separates two distinct failure modes that prior work often conflates: not knowing region-specific facts, and implicitly treating some regions as more representative than others.

Real-world applications:

• Auditing and localizing LLM-based assistants, chatbots, and search products deployed across India's regions, where a single national answer may be wrong for most users.
• Designing culturally aware recommendation or content-generation systems for festivals, food, clothing, and etiquette, where the paper shows regional answers genuinely diverge.
• Building evaluation suites for government, education, or public-service tools that must not default to one region's norms.
• Providing a template for bias audits in other culturally heterogeneous countries, as illustrated with China.

Industry relevance. Companies shipping multilingual or India-focused products need to know whether their models treat one region's practices as the national default. The paper's RA-MCQ design gives a concrete, low-cost diagnostic: hide the region labels and see which practices the model reaches for. The finding that all eight evaluated models, open and closed source alike, over-select Central and North India suggests this is a data and tokenization issue rather than a single vendor's problem, and the paper attributes it to Hindi-dominant web content, tokenizer fragmentation of underrepresented languages, and Hindi-language media dominance.

Future Directions

• Finer-grained regional and demographic analysis. The five-region division aggregates substantial internal diversity; the paper notes that South India alone encompasses Andhra Pradesh, Karnataka, Kerala, Tamil Nadu, Telangana, Puducherry, Lakshadweep, and the Andaman and Nicobar Islands, with distinct languages, cuisines, and festival traditions. Religion, caste, gender, and urban/rural residence within a region are named as unexplored dimensions.
• Broadening participant sampling. The pool came through Prolific and may skew toward English-speaking, digitally connected Indians; rural and non-English-speaking populations are identified as targets for expanded sampling.
• Adding domains. The authors note that OCM categories not used here, such as Marriage (OCM 580), Family (OCM 590), Law (OCM 670), and Living Standards and Routines (OCM 510), could reveal further regional variation.
• Replication in other countries. The paper sketches a China version (34 provincial-level divisions, 56 recognized ethnic groups, 130+ languages, using six statistical regions and 15 region pairs) and poses whether models would over-select eastern practices there, parallel to the Central/North India bias found here. Temporal re-collection is also implied, since Indica is a snapshot of practices reported by participants in 2025.

Target Audience

This paper is most useful to NLP researchers building or evaluating cultural commonsense and bias benchmarks; to practitioners deploying language models in India or other culturally heterogeneous markets; to evaluation and responsible-AI teams who need a concrete method for measuring geographic representation bias; and to social scientists and anthropologists interested in how taxonomic frameworks such as the OCM can be operationalized for computational evaluation.

Authors’ abstract

Existing cultural commonsense benchmarks treat nations as monolithic, assuming uniform practices within national boundaries. But does cultural commonsense hold uniformly within a nation, or does it vary at the sub-national level? We introduce Indica, the first benchmark designed to test LLMs' ability to address this question, focusing on India - a nation of 28 states, 8 union territories, and 22 official languages. We collect human-annotated answers from five Indian regions (North, South, East, West, and Central) across 515 questions spanning 8 domains of everyday life, yielding 1,630 region-specific question-answer pairs. Strikingly, only 39.4% of questions elicit agreement across all five regions, demonstrating that cultural commonsense in India is predominantly regional, not national. We evaluate eight state-of-the-art LLMs and find two critical gaps: models achieve only 13.4%-20.9% accuracy on region-specific questions, and they exhibit geographic bias, over-selecting Central and North India as the "default" (selected 30-40% more often than expected) while under-representing East and West. Beyond India, our methodology provides a generalizable framework for evaluating cultural commonsense in any culturally heterogeneous nation, from question design grounded in anthropological taxonomy, to regional data collection, to bias measurement.