Finding Culture-Sensitive Neurons in Vision-Language Models

AI summary

Overview

Research area: Mechanistic interpretability of vision-language models (VLMs), specifically how culturally grounded knowledge is represented inside a multimodal transformer, and how that connects to fairness in multimodal systems.

Technical level: Intermediate. Readers will get the most out of it with some familiarity with transformer decoder blocks, MLP/SwiGLU activations, and neuron-ablation methodology, but the high-level argument is accessible without that background.

Scope: The paper asks whether individual decoder MLP neurons in three 7B-parameter VLMs are preferentially sensitive to particular cultural contexts, introduces a new margin-based neuron selector (ConAct) to find them, and tests their causal importance by deactivating them at inference time on the CVQA benchmark across 25 cultural groups.

What This Paper Is About

Vision-language models handle culturally grounded images and questions unevenly: performance varies systematically across cultures, but it is not known how that cultural knowledge is stored inside the network. Prior interpretability work has found neurons specialized for languages, knowledge domains, modalities, and tasks, but multimodal work has mostly looked at vision-vs-text specialization rather than culture. This paper asks three questions: do VLMs contain neurons that preferentially activate on inputs tied to particular cultures, does ablating a small targeted subset selectively damage performance on that culture while leaving others intact, and where in the network do those neurons sit.

Key Contributions

1. Empirical evidence for culture-sensitive neurons in VLMs. The authors adapt activation-based neuron analysis from language models to a multimodal setting and show that a small subset of decoder MLP neurons, when deactivated, disproportionately degrades performance on questions tied to a specific culture while leaving other cultures largely unaffected.

2. A new identification method, Contrastive Activation Margin (ConAct). ConAct is a margin-based selector that scores each neuron by the gap between its highest activation probability across cultures and its nearest competing culture, rather than by deviation from the cross-culture mean. It assigns each neuron exclusively to its top culture. It is motivated by a preliminary observation that many neurons have high intrinsic variance across cultures (12.27% of neurons in Qwen2.5-VL-7B and 9.57% in Pangea-7B satisfy std_c(P) > mean_c(P)), which can inflate mean-based scores.

3. A causal ablation study across three VLMs and 25 cultural groups. Using CVQA, the authors compare five selectors — Random Selection (RND), Activation Probability (LAP), Activation Probability Entropy (LAPE), Mean Activation Difference (MAD), and ConAct — by masking the top r% = 1% of MLP neurons and measuring accuracy change and flip rate under self-deactivation versus cross-deactivation.

4. A layer-wise map of where culture-sensitive neurons live. The analysis shows these neurons are not uniformly distributed but cluster in specific decoder regions in a model-dependent and (for ConAct) partly culture-dependent way.

Main Findings

• ConAct wins on Qwen2.5-VL-7B and Pangea-7B. On Qwen2.5-VL-7B, ConAct produces the largest self-deactivation accuracy drop (−5.52 percentage points) with a small average cross-deactivation change (−0.64), giving a self–cross accuracy gap of −4.88. On Pangea-7B, ConAct again gives the largest self-deactivation drop (−4.33) with cross-deactivation of −0.72, a gap of −3.61.

• Flip-rate gaps agree. ConAct's self–cross flip-rate gap is +8.36 on Qwen2.5-VL-7B and +5.65 on Pangea-7B — the largest of the methods tested — meaning predictions shift substantially within the target culture but not across others.

• Baselines are less specific. LAP and MAD often produce broader off-diagonal interference on Qwen2.5-VL-7B and Pangea-7B, suggesting they capture shared or generic multimodal cues rather than culture-specific ones. LAP occasionally improves accuracy when masked (e.g., +0.96 on Qwen2.5-VL-7B self-deactivation, +1.00 on Pangea-7B), which the authors attribute to pruning of spurious or overly dominant features. LAPE shows little selectivity in these two models (self-deactivation +0.56 on Qwen2.5-VL-7B, −0.74 on Pangea-7B).

• LLaVA-v1.6-Mistral-7B behaves differently. Here LAPE produces the strongest self-deactivation drop (−4.43) but with essentially no specificity (self–cross gap +0.01, cross-deactivation −4.44). ConAct and MAD yield more selective neurons (−0.76 and −0.93 gaps respectively) but smaller effect magnitudes. The authors describe this model as showing "resistance to specific targeting."

• RND confirms the effect is not an artifact of arbitrary masking. Random masking produces only small changes: −0.19 accuracy on Qwen2.5-VL-7B, +1.02 on Pangea-7B, −0.50 on LLaVA-v1.6-Mistral-7B, with flip rates of 4.66, 6.45, and 7.01 respectively.

• Baseline accuracies are uneven across cultures. Qwen2.5-VL-7B achieves the highest median unablated accuracy (≈0.60), while Pangea-7B and LLaVA-v1.6-Mistral-7B reach around 0.50. Identification and evaluation splits yield similar performance, so the ablation results are not confounded by a train–test mismatch.

• Geographically linked cultures correlate. Deactivating EGY-identified neurons has a significant impact on ETO, another African culture group. Small accuracy gains appear for some cultures (e.g., BGR, ETA) when ablating neuron sets identified from other cultures, again consistent with a pruning interpretation.

• Neurons cluster in early-to-mid layers. In Qwen2.5-VL-7B (28-layer decoder), culture-sensitive neurons cluster in the first layer (layer 0) and early-mid layers (6–8), with sparse presence in deeper blocks. MAD tends to bypass central layers (15–18), while ConAct spreads selections more evenly across mid-to-late layers. ConAct also shows culture-specific deviations: for BGR and IDN, layers 6–8 contain a higher proportion of selected neurons than for other cultures.

• Task format survives ablation. Masked generations generally remain compatible with the multiple-choice response format at the sparsity levels studied, so the intervention does not collapse task framing even though it changes the selected answers.

Methodology in Plain English

The authors use a three-stage pipeline.

Stage 1 — Record. They instrument the decoder MLP nonlinearity branch (the SwiGLU gate, g) in each VLM and record per-neuron activations on CVQA questions the unablated model answers correctly. A valid-token mask excludes padding and special markers while keeping text and visual tokens consumed by the decoder. Only the decoder is instrumented, not the upstream vision encoder. They accumulate two counts per neuron and per culture: how often the neuron fires positively (K), and the total number of valid tokens for that culture (T), giving a token-level activation probability P = K / T. They restrict to correctly answered examples to reduce noise from failure-related activations.

Stage 2 — Identify. For each culture, each method ranks neurons and the top r% = 1% of all MLP neurons are selected as culture-sensitive. RND samples uniformly across all layers as a compute-efficient global baseline. LAP ranks by raw firing probability. LAPE ranks by Shannon entropy of the neuron's near-normalized culture profile (lower entropy = more selective), with an activity filter keeping neurons above the 95th percentile of firing probability and a candidate pool of the lowest-ρ fraction, where ρ = min(1, 5r). MAD ranks by absolute deviation from the cross-culture mean firing probability. ConAct instead assigns each neuron only to its top culture and scores it by P^(1) − P^(2), the gap to its runner-up culture, which makes it less sensitive to global variance.

Stage 3 — Intervene. They build a binary keep-mask that zeroes out the selected neurons and apply it to the SwiGLU output during inference, leaving everything else unchanged. They then compare masked and unmasked generations and report two metrics: accuracy change in percentage points and flip rate (the percentage of items whose predicted answer changes versus the full model). They test two conditions — self-deactivation (neurons identified from the same culture being evaluated) and cross-deactivation (neurons from a different culture) — and use the self–cross gap as the headline measure of cultural specificity: a larger negative accuracy gap means stronger culture-specific impact with less spillover.

Setup details. Data: CVQA, using its country–language pair taxonomy as the operationalization of culture (39 country–language pairs in the benchmark; the study covers 25 cultural groups). Three aggregated culture groups pool pairs sharing a country (India-all, Indonesia-all) or a language (all-Spanish). To avoid confounding culture with language proficiency, all experiments use the dataset's prepared English translations for questions and answer options. The dataset is split approximately 50/50 into identification (5,178 questions) and evaluation (5,196 questions) subsets. Models: Qwen2.5-VL-7B, LLaVA-v1.6-Mistral-7B, and Pangea-7B, chosen to differ in backbone, supervision, and cultural/linguistic coverage. Prompting uses a fixed multiple-choice template requiring the full option text rather than the label; decoding is deterministic (temperature 0) with a maximum generation length of 20, and answers are normalized with a heuristic that treats the last-mentioned option as the final decision.

Why This Matters

Impact on research. This is one of the first studies to extend neuron-level interpretability from language and modality specialization to culture in multimodal models, and to back it with causal ablation rather than correlational activation statistics. It also contributes a methodological lesson: mean-deviation selectors like MAD can be confounded by high intrinsic variance across cultures, and a contrastive margin can isolate more specific units. The finding that culture-sensitive neurons concentrate in early-to-mid decoder layers gives a concrete hypothesis for follow-up work on where multimodal cultural fusion happens.

Real-world applications:

• Targeted fairness interventions in multimodal systems. Knowing which neuron subsets drive culture-specific behavior creates a candidate handle for sparse fine-tuning or activation steering instead of full retraining.
• Diagnostic auditing of deployed VLMs. The ablation protocol can be reused to quantify culture-specific failure modes in commercial vision-language products used in global markets.
• Cultural bias mitigation in content pipelines. Systems that caption, parse, or describe culturally specific imagery (foods, clothing, landmarks) could use these tools to identify and correct systematic regional errors.
• Benchmark design. The self–cross gap offers a repeatable way to evaluate whether an intervention is genuinely culture-targeted or just degrading the model globally.

Industry relevance. Any company shipping a vision-language product to global users — photo description, visual search, document parsing, accessibility tooling, chart captioning — has a stake in whether cultural recognition is localized enough to be audited and corrected. The paper reports results across three different open 7B models, so the pattern is not tied to a single vendor's architecture.

Future Directions

1. Extend beyond English prompts. The study deliberately uses English translations to separate culture from language skill, which the authors acknowledge makes the construct closer to visual cultural knowledge than to culture-as-language-practice. Whether the same neurons emerge in multilingual settings is left open.

2. Look beyond decoder MLPs. The analysis covers only decoder MLP neurons, not attention heads, vision encoders, or alignment modules, all of which may also encode culture-sensitive behavior. The authors explicitly flag this as unfinished.

3. Pair identification with activation steering. The authors suggest the natural next step is not just finding and ablating these neurons but steering them, which would turn a diagnostic into an intervention.

4. Move past activation-frequency summaries. The current pipeline relies on aggregated firing statistics rather than finer-grained token-level or temporal dynamics, and hyperparameters for neuron selection were fixed for computational budget reasons — both offer room for refinement.

Target Audience

Researchers in mechanistic interpretability and multimodal model analysis, plus fairness and evaluation researchers working on cultural representation in AI. It is also relevant to practitioners building or auditing multilingual vision-language products, and to anyone interested in whether cultural knowledge in large models is locally encoded or distributed. Readers looking for a benchmark paper or a ready-made fairness mitigation method will find this is diagnostic rather than prescriptive: the authors state explicitly that their methods are for diagnostic use only and are not themselves fairness interventions.

Authors’ abstract

Despite their impressive performance, vision-language models (VLMs) still struggle on culturally situated inputs. To understand how VLMs process culturally grounded information, we study the presence of culture-sensitive neurons, i.e., neurons whose activations show preferential sensitivity to inputs associated with particular cultural contexts. We examine whether such neurons are important for culturally diverse visual question answering and where they are located. Using the CVQA benchmark, we identify neurons of culture selectivity and perform diagnostic tests by deactivating the neurons flagged by various identification methods. Experiments on three VLMs across 25 cultural groups demonstrate the existence of neurons whose ablation disproportionately harms performance on questions about the corresponding cultures, while having limited effects on others. Moreover, we introduce a new margin-based selector Contrastive Activation Margin (ConAct) and show that it outperforms probability- and entropy-based methods in identifying neurons associated with cultural selectivity. Finally, our layer-wise analyses reveal that such neurons are not uniformly distributed: they cluster in specific decoder layers in a model-dependent way.