Tears or Cheers? Benchmarking LLMs via Culturally Elicited Distinct Affective Responses

AI summary

Overview

Research area: Natural Language Processing / cross-cultural affective computing and LLM benchmarking. The paper sits at the intersection of multilingual evaluation, multimodal emotion understanding, and cultural alignment of large language models.

Technical level: Intermediate. The pipeline design and metrics are described clearly, but readers will need some familiarity with emotion taxonomies (discrete emotion categories, Russell's Circumplex Model of valence and arousal), benchmark construction practice, and standard LLM evaluation terminology.

Scope: The paper introduces Cedar, a multimodal benchmark of 10,962 instances across seven languages and 14 emotion categories, built exclusively from scenarios where the correct emotional label depends on cultural context, and uses it to evaluate 17 multilingual and multimodal LLMs.

What This Paper Is About

Existing cultural benchmarks for LLMs mostly test declarative cultural knowledge, such as geography facts, history, or established social customs. They largely miss the subjective side of culture: the same scene, image, or sentence can trigger different emotions in different cultural groups, so semantic equivalence across languages does not imply emotional equivalence. The paper builds a benchmark made entirely of such culturally divergent emotional scenarios and measures how well current models handle them.

Key Contributions

1. Cedar, a culture-specific multimodal benchmark. A benchmark of 10,962 instances spanning seven languages (Arabic, Chinese, English, Hindi, Japanese, Spanish, Swahili) and 14 fine-grained emotion categories, with 400 multimodal and 1,166 text-only samples per language. Every instance is curated so that emotional ground truth is contingent on cultural context, rather than assuming cultural universality.

2. A construction pipeline that uses LLM provisional labels for selection and human annotators for truth. The pipeline generates approx. 42K candidate Narrative-Question pairs, refines them, filters for cross-language emotional variation down to approx. 7K pairs, extends them to approx. 600 candidate Image-Narrative-Question triples, then selects 400 INQ triples and 1,166 NQ pairs based on maximal cultural variation. Ground-truth labels come exclusively from native-speaker human annotation, with at least five annotators per instance.

3. A multi-angle evaluation protocol. Beyond standard accuracy, the paper defines Emotion Prediction Propensity (EPP, reported as Global GEPP and Language-Specific LSEPP) and Russell's Quadrant Bias (RQB, reported as Global GRQB and Language-Specific LSRQB), mapping the 14 emotions onto valence-arousal quadrants.

4. An empirical demonstration that language consistency and cultural alignment dissociate. Evaluation of 17 representative multilingual and multimodal LLMs shows that matching prompt language to dataset language does not improve, and can degrade, culturally grounded emotion prediction.

Main Findings

• Multimodal is harder than text-only. Evaluated models consistently achieve lower standard accuracy on multimodal instances than on the text-only subset, which the authors attribute to the difficulty of visual-emotional grounding when symbolic imagery carries culturally grounded emotional weight.

• A resource and region gap in affective norms. Models perform better in high-resource languages such as English and Spanish, while performance degrades notably in Asian languages (Chinese, Japanese) and low-resource languages like Swahili. The authors read this as non-Western affective norms being inadequately encoded.

• Multilingual-optimized models are more stable. Models explicitly optimized for multilingual alignment, such as Aya-101 and Aya-Vision-8B/32B, show greater stability across languages and retain competitive performance with notably reduced cross-lingual variance despite smaller scale. Gemma3-27B-It reaches 39.82 average multimodal accuracy and 44.31 text-only accuracy; the best multimodal average reported is Gemini2.5-Flash at 41.10, and the best text-only average is Gemini2.5-Flash at 47.31, followed by GPT-4o at 46.57.

• Language-specific prediction quirks exist. In the Arabic group, Aya-Vision-8B shows a pronounced over-prediction of surprise in the multimodal subset, a pattern absent from other language groups.

• Systematic preference for high-arousal emotions. Models prioritize high-arousal emotions and marginalize deactivated states, under-predicting Quadrants III and IV. Valence tendencies differ by model: Claude4.5-Sonnet leans toward high-arousal, low-valence emotions (10.6%), while GPT-4o shows a contrasting inclination (14.2%) toward high-arousal, pleasant states. Mistral-Small-3.2 prefers low-valence, high-arousal emotions in Quadrant II (16.8%), and InternVL3.5-38B shows a mild bias toward Quadrant III (3.3%).

• Over-predicted emotions carry high variance. In the text-only set, models over-predict salient emotions such as surprise and amusement at the expense of subtle states like contentment and embarrassment, and the over-predicted categories show markedly higher variance and extreme outliers.

• Stability tracks accuracy. Models with higher overall accuracy show more balanced emotion distributions across languages. Gemini2.5-Flash is markedly more stable than Aya-Vision-8B, consistent with its higher multimodal accuracy of 41.10% versus 32.89%.

• English prompts win, and language matching backfires. English prompts consistently outperform prompts in other languages, and non-English prompts such as Arabic or Chinese degrade performance. Matching prompt language to dataset language does not yield reliable gains: Gemma3-27B-It with Japanese prompts on Japanese data reaches only 30.46% accuracy, lagging behind English (44.14%) and even unrelated Arabic prompts (39.30%). The authors hypothesize that language alignment may amplify latent cultural stereotypes, and conclude that models may rely on English as a latent reasoning pivot.

• English is the most stable language group by quadrant bias. LSRQB data points for English cluster tightly around 0%, while Arabic and Swahili show substantially larger dispersion.

• A Japanese-specific failure pattern. In the text-only setting, the Japanese group exhibits a pronounced negative clustering of approximately -20% in low-valence, low-arousal emotions, a pattern absent in other language groups, suggesting models struggle to internalize deactivated emotional nuances characteristic of the Japanese sociocultural context.

• Annotation quality measures. Inter-annotator agreement is reported via Krippendorff's alpha and average pairwise F1. For example, English text-only reaches alpha 0.884 and F1 0.810, while Swahili multimodal is lowest at alpha 0.693 and F1 0.577. Human evaluation of 100 randomly sampled instances confirmed that 93% of refined narratives successfully removed explicit emotion expressions while preserving situational meaning.

Methodology in Plain English

The researchers gathered culturally grounded seed material from existing resources (ArabCulture, Casa, Cultural Atlas, CulturalBench, CultureBank, JETHICS), translating non-English sources into English with GPT-4.5 and standardizing everything into one sentence format. GPT-4.5 then turned each sentence into a Narrative-Question pair describing a situation and asking about the protagonist's emotional state, producing roughly 42K candidates. Llama3.3-70B rewrote these into second person and stripped out explicit emotional wording, so models must infer feelings from implicit cues.

Filtering came next: sentences were kept only if they were 50 to 200 characters (about 2.7% of the candidate pool), non-social content was removed (approximately 15.88%), and PolyGuard was used to screen out toxic content including hate speech and harmful stereotypes. Then three strong models, Claude4.5-Sonnet, Gemini2.5-Flash, and GPT-4.5, labeled every instance in every language. Instances where the models completely disagreed were discarded as ambiguous; instances where predictions stayed uniform across languages were removed because they lacked cultural variation. This left approximately 7K pairs.

For the multimodal half, students retrieved public images via Google or Baidu, and Gemini2.5-Flash-Image generated synthetic images when none were available. GPT-4.5 then refined the Image-Narrative-Question triples so the narrative carried only information invisible in the image, and any instance the model labeled identically with and without the image was discarded as visually redundant, leaving approximately 600 triples.

For finalization, narratives were embedded with Qwen3-Embedding-8B to cluster semantically similar scenarios. Cultural variation inside each cluster was scored using Russell's Circumplex Model, mapping the 14 emotions into four valence-arousal quadrants and measuring cross-language quadrant disagreement. The highest-scoring instance per cluster was kept, yielding 400 INQ triples and 1,166 NQ pairs. English data was translated into the six other languages with GPT-4.5 and verified by native speakers, and ground-truth labels were collected from native speakers recruited via Prolific with at least five annotators per instance, adding two more if no majority was reached.

Why This Matters

Impact on research. The paper reframes cultural evaluation of LLMs away from static facts and toward affective interpretation, and provides evidence that language proficiency is not the same thing as cultural resonance. It also supplies a benchmark and a reusable pipeline design (LLM-provisional labels for selection, human labels for ground truth) that other researchers can adapt to new languages and modalities.

Real-world applications:

• Global customer service and chat systems that must read emotional tone correctly across markets rather than defaulting to English-centric assumptions.
• Content moderation and mental health or crisis-support tools where misreading culturally specific emotional cues (for example, mourning versus festivity) could cause harm.
• Cross-cultural marketing and media localization, where the same image or campaign can carry opposing emotional meanings in different regions.
• Assistive and educational technologies deployed in multilingual classrooms, where the emotional framing of scenarios affects judgments about learners.

Industry relevance. The findings suggest that adding more languages or multilingual training data does not automatically produce culturally aligned emotional behavior, and that prompt-language matching, a common practical shortcut, can even hurt. This affects how teams design evaluation suites, how they choose between multilingual-specialized models such as the Aya series and larger general models, and how they justify localization budgets. The reported dissociation between language consistency and cultural alignment is a direct caution for anyone shipping emotion-aware systems internationally.

Future Directions

• Broadening language coverage. The authors state that Cedar deliberately restricts itself to seven representative languages to prioritize high-fidelity native-speaker validation, and identify expanding to more cultural clusters as future work.

• Reconciling categorical and dimensional emotion frameworks. The paper acknowledges ongoing debate in affective science and uses 14 discrete categories alongside Russell's Model; how to unify these frameworks for evaluation remains open.

• Moving beyond high-consensus scenarios. Due to the cost of native-speaker annotation, Cedar targets high-consensus scenarios validated by strict majority voting. The paper points to analyzing minority annotations via multi-label analysis (Appendix B), leaving low-consensus and individual-level emotional variation largely unexplored.

• Understanding why language matching fails. The hypothesis that matched prompt and dataset languages amplify latent cultural stereotypes, and that models may use English as a latent reasoning pivot, is stated but not resolved, leaving a clear target for mechanistic and mitigation research.

Target Audience

Researchers and practitioners working on multilingual and multimodal LLM evaluation, cross-cultural NLP, affective computing, and AI fairness, as well as benchmark builders who want a template for combining LLM-assisted data selection with rigorous human ground-truth annotation. Product and localization teams deploying emotion-sensitive systems across multiple regions will also find the concrete model comparisons and the prompt-language findings directly actionable.

Authors’ abstract

Culture serves as a fundamental determinant of human affective processing and profoundly shapes how individuals perceive and interpret emotional stimuli. Despite this intrinsic link extant evaluations regarding cultural alignment within Large Language Models primarily prioritize declarative knowledge such as geographical facts or established societal customs. These benchmarks remain insufficient to capture the subjective interpretative variance inherent to diverse sociocultural lenses. To address this limitation, we introduce CEDAR, a multimodal benchmark constructed entirely from scenarios capturing Culturally \underline{\textsc{E}}licited \underline{\textsc{D}}istinct \underline{\textsc{A}}ffective \underline{\textsc{R}}esponses. To construct CEDAR, we implement a novel pipeline that leverages LLM-generated provisional labels to isolate instances yielding cross-cultural emotional distinctions, and subsequently derives reliable ground-truth annotations through rigorous human evaluation. The resulting benchmark comprises 10,962 instances across seven languages and 14 fine-grained emotion categories, with each language including 400 multimodal and 1,166 text-only samples. Comprehensive evaluations of 17 representative multilingual models reveal a dissociation between language consistency and cultural alignment, demonstrating that culturally grounded affective understanding remains a significant challenge for current models.