Neuron-Level Analysis of Cultural Understanding in Large Language Models

AI summary

Overview

Research area: Mechanistic interpretability of large language models, applied to cultural understanding — natural language processing, model internals, and cultural evaluation benchmarks.

Technical level: Advanced. The paper assumes familiarity with transformer architecture, gated linear units (GLUs), gradient-based attribution, and neuron-level analysis methods.

Scope: This paper identifies individual neurons inside six open-source LLMs that drive culturally specific behavior, characterizes where those neurons live, and shows what happens to cultural and general language abilities when they are suppressed or unintentionally updated.

What This Paper Is About

LLMs are trained mostly on English-dominant data and tend to be culturally biased toward highly represented cultures, but nobody has looked inside the model to find out which parts produce culturally aware (or unaware) outputs. The authors build a pipeline called CULNIG (CULture Neuron Identification Pipeline with Gradient-based Scoring) that locates the individual neurons responsible for cultural understanding, separates neurons tied to all cultures from those tied to one culture, and tests whether those neurons actually matter by turning them off. The goal is both scientific (explain the mechanism) and practical (guide how models should be trained and edited).

Key Contributions

1. CULNIG, a gradient-based neuron identification pipeline with two filtering stages. It scores neurons by gradient-weighted attribution on cultural data, subtracts a control score computed on the same questions with the question content removed (to eliminate task-understanding neurons), and then removes top-scoring neurons on a purpose-built CountryRC dataset to eliminate superficial neurons that merely react to country names.

2. Identification and validation of culture-general neurons. These neurons contribute to cultural understanding regardless of the specific culture. The authors verify their causal role by masking them and measuring degradation across four cultural benchmarks and four general NLU benchmarks, using six LLMs.

3. Identification and analysis of culture-specific neurons. A z-score filter across the 16 BLEnD countries isolates neurons tied to one culture. The authors show these neurons affect not only their target culture but also historically or geographically related cultures.

4. An engineering application: role-aware module selection for fine-tuning. They show that fine-tuning only the modules containing the most culture-general neurons damages cultural ability, while fine-tuning modules containing none preserves it — a guide for choosing which parameters to update.

5. A new dataset, CountryRC (CRC). 50 problems per country (half for neuron identification, half for evaluation), generated with ChatGPT, where the correct answer is always a country name appearing in the context, requiring information propagation but not cultural knowledge.

Main Findings

• Culture-general neurons are extremely sparse. They account for less than 1% of all neurons. In gemma-3-12b-it the identified set is 8,087 neurons; gemma-3-27b-it 14,273; Qwen3-14B 7,340; Llama-3.1-8B-Instruct 4,268; phi-4 7,447; Falcon3-10B-Instruct 9,282.

• They are concentrated in shallow-to-middle MLP layers. This concentration holds in gemma-3-12b-it and is consistent across models.

• Masking them badly damages cultural ability but not general NLU. For example, gemma-3-12b-it drops from 64.22 to 37.93 on BLEnD_test, from 78.08 to 62.00 on CulturalBench, from 58.54 to 52.02 on NormAd, and from 64.08 to 58.46 on WorldValuesBench, while QNLI (75.37 to 72.77), MRPC (78.04 to 78.65), and CommonsenseQA (79.71 to 75.10) change little. Across models, BLEnD_test drops by up to 30%. Random neurons of the same count produce much smaller drops (e.g., 8,087 random neurons give 63.57 on BLEnD_test for gemma-3-12b-it, versus 37.93 for culture-general).

• The effect generalizes beyond the identification data. Neurons were identified using only three BLEnD categories (food, work-life, sport) but degrade the held-out categories (education, family, holidays/celebrations/leisure), a different benchmark (CulturalBench), different cultural attributes (NormAd's etiquette and WorldValuesBench's values), and multilingual short-answer questions.

• The identified neurons are not superficial country-name detectors. On the CountryRC evaluation set, models scored almost 100% both before and after masking.

• Module matters. Masking MLP neurons causes substantial degradation on cultural benchmarks while QNLI and MRPC are unaffected; masking attention neurons has a smaller overall impact. This leads the authors to set different thresholds for the two module types.

• Culture-specific neurons affect related cultures. Masking Mexico-specific neurons (gemma-3-12b-it) most strongly affects Mexico instances — mean rank of score reduction among the 16 cultures over six models is 1.17 — and the second most affected culture is Spain at a mean rank of 3.83. This pattern is weaker for NormAd, suggesting culture-specific neurons capture etiquette and values less well.

• Culture-general neurons encode concepts, not meta-level control. The top-scoring culture-general neuron in gemma-3-12b-it (the 1141st neuron in the attention query projection in the 15th layer) has positive scores on only 29% of instances in BLEnD_neur, with high-scoring instances spread across categories and target cultures. The top Iran-specific neuron has positive scores on only 38% of Iranian instances while also scoring high on Azerbaijan.

• Knowledge and values share neurons. When neurons are identified using a NormAd split (NormAd_neur) instead of BLEnD_neur, masking them still degrades all cultural benchmarks; and removing NormAd_neur-identified neurons from the BLEnD_neur set still harms NormAd_test and WorldValuesBench.

• Fine-tuning can erase cultural ability depending on which modules you update. Fine-tuning gemma-3-12b-it on QNLI or MRPC while updating only 10% of parameters (600 steps, learning rate 3e-5, evaluated every 200 steps): target scores improved in both conditions, but updating top-culture modules (all MLP modules from shallow to middle layers) decreased cultural benchmark scores, while updating bottom-culture modules (very shallow attention modules and very deep attention and MLP modules) barely affected them.

• A reported discrepancy with prior work. The authors could not reproduce CAPE's finding that culture neurons concentrate in upper layers; when replicated on gemma-3-12b-it, LAPE and CAPE neurons had negligible impact on evaluation scores.

Methodology in Plain English

The authors treat each dimension of a layer's hidden vector as a "neuron" and ask: if I nudged this neuron, how much would the model's probability of the correct answer change? This is measured with a gradient-based attribution score (activation value multiplied by the gradient of the output probability with respect to that activation), taken as the maximum over token positions, then aggregated over a dataset weighted by the model's confidence on each item. This is a first-order approximation of the neuron's causal effect, and it only requires a single forward/backward pass rather than masking millions of neurons one at a time (the paper notes LLMs contain millions of neurons, see Table 12).

To isolate cultural neurons specifically, they use three data splits. Cultural questions come from three BLEnD categories (12,701 instances), held-out evaluation from the other three (10,331 instances). A control set (BLEnD_ctrl) keeps the same answer choices and output-format instruction but strips the question, so subtracting its scores removes neurons that just handle task format. A second control set, CountryRC, contains questions whose answer is always a country name in the passage, so removing the top scorers on CRC_neur eliminates neurons that only fire on country-name tokens.

The selection rules: pick the top t% by the BLEnD_neur minus BLEnD_ctrl score, then discard the top r% by CRC_neur score. From a preliminary sweep they set t_MLP = 1%, t_attn = 0.2%, r = 1% (CULNIG-general). For culture-specific neurons (CULNIG-specific), they compute scores per country, convert them to z-scores across the 16 countries, and discard neurons with z below 0.5 so that only culture-distinctive neurons remain, using t = 0.3% and r = 1%.

Evaluation uses accuracy on BLEnD_test, CulturalBench (CultB), NormAd, CommonsenseQA (ComQA), QNLI, and MRPC, and the paper's own score_c metric for WorldValuesBench, which measures alignment with the majority survey response per country on a Likert scale. Four prompt formats per benchmark were prepared via ChatGPT. Statistical significance for the masking results uses bootstrapping with a sample size of 2,000 and a 0.05 threshold, with random-neuron baselines averaged over ten seeds.

Why This Matters

Culturally unaware LLMs produce skewed outputs that disadvantage underrepresented cultures, and prior fixes have been either data-level or evaluation-level without knowing where inside the model culture actually lives. This paper provides a location map and a causal test.

Real-world applications:

• Model editing for cultural knowledge. When using knowledge-editing methods to correct or insert cultural facts, one can target the modules that carry culture-general neurons rather than the whole model.
• Safer fine-tuning pipelines. Before fine-tuning on a general task, check which modules hold culture-general neurons and avoid updating them so cultural capability is not silently lost.
• Auditing for cultural competence. The neuron-level signature gives a diagnostic that is more direct than benchmark scores alone for detecting whether a model has genuine cultural representations versus surface cues.
• Fairer deployment planning. Knowing that cultural ability is concentrated in under 1% of neurons and in specific shallow-to-middle MLP layers tells engineering teams which parts of the model are fragile to compression, pruning, or parameter-efficient updates.

Industry relevance: Parameter-efficient fine-tuning and adapter-based training are standard practice. This work gives a concrete, cheap criterion (count of culture-general neurons per module) for deciding which modules to freeze and which to update, applicable to any team fine-tuning open-weight models for global products.

Future Directions

• Resolving the disagreement with CAPE. The authors could not reproduce CAPE's finding that culture neurons sit in the upper layers and report that LAPE and CAPE neurons had negligible impact in their evaluation. They attribute the gap to gradient-based versus activation-based attribution and to accuracy versus perplexity evaluation, and explicitly leave the discrepancy for future work.

• Extending the engineering application. The fine-tuning experiment covers only QNLI and MRPC. The authors suggest applying the module-selection idea to knowledge editing, to inserting new cultural knowledge, and to updating neurons outside the culture-general set so new knowledge is absorbed without overwriting existing knowledge — but state these applications are deferred to future work.

• Broadening the cultures and benchmarks covered. The culture-specific analysis is restricted to the eight countries appearing in all of BLEnD, CulturalBench, and NormAd (China, Indonesia, Iran, Mexico, South Korea, Spain, UK, USA).

• Understanding why shared neurons arise. The paper observes that culture-specific neurons bleed into historically or geographically related cultures, but the mechanism behind this sharing is not resolved.

Target Audience

Mechanistic interpretability researchers studying how LLMs represent knowledge and concepts; NLP researchers working on cultural bias, multicultural evaluation, and multilingual model behavior; and ML engineers who fine-tune or edit open-weight LLMs for global audiences and need to know which parameters they can safely change. Readers should be comfortable with transformer internals and attribution methods, since the core method is advanced even though the paper's framing is accessible.

Authors’ abstract

As large language models (LLMs) are increasingly deployed worldwide, ensuring their fair and comprehensive cultural understanding is important. However, LLMs exhibit cultural bias and limited awareness of underrepresented cultures, while the mechanisms underlying their cultural understanding remain underexplored. To fill this gap, we conduct a neuron-level analysis to identify neurons that drive cultural behavior, introducing a gradient-based scoring method with additional filtering for precise refinement. We identify culture-general neurons contributing to cultural understanding regardless of cultures, and culture-specific neurons tied to an individual culture. Culture-general and culture-specific neurons account for less than 1% of all neurons and are concentrated in shallow to middle MLP layers. We validate their role by showing that suppressing them substantially degrades performance on cultural benchmarks (by up to 30%), while performance on general natural language understanding (NLU) benchmarks remains largely unaffected. Moreover, we show that culture-specific neurons support knowledge of not only the target culture, but also related cultures. Finally, we demonstrate that training on NLU benchmarks can diminish models' cultural understanding when we update modules containing many culture-general neurons. These findings provide insights into the internal mechanisms of LLMs and offer practical guidance for model training and engineering. Our code is available at https://github.com/ynklab/CULNIG