From Native Memes to Global Moderation: Cross-Cultural Evaluation of Vision-Language Models for Hateful Meme Detection

AI summary

Overview

Research area: Multimodal harm detection, specifically hateful meme detection with vision-language models (VLMs), evaluated through a cross-cultural and multilingual lens.

Technical level: Intermediate. The paper assumes familiarity with prompt-based evaluation of large multimodal models, zero-shot and one-shot inference, and classification metrics such as Macro-F1, but the experimental design itself is straightforward to follow.

Scope: A systematic evaluation of seven general-purpose VLMs and two task-specific meme-detection models across six natively sourced meme datasets (Arabic, Bengali, English, German, Italian, Spanish), varying learning paradigm, prompt language, and content translation to measure cross-cultural robustness.

What This Paper Is About

VLMs are trained largely on Western, English-centric data, so they may misread memes whose meaning depends on local humour, slang, or cultural reference. Existing multilingual benchmarks usually start from English content and translate it, which the authors argue cannot test how models behave on organically native material. The paper builds an evaluation framework rather than a new dataset, probing whether interventions such as native-language prompting or one-shot examples can close this cultural understanding gap, and whether the widespread "translate-then-detect" pipeline helps or hurts.

Key Contributions

1. A multi-dimensional evaluation framework for multilingual hateful meme detection that varies model type, learning paradigm (zero-shot vs. one-shot), prompt language (English vs. native), and content representation (native vs. machine-translated captions).

2. A benchmark comparison of seven general-purpose VLMs and two task-specific models (Pro-Cap, PromptHate) across six native meme datasets, producing systematic empirical evidence that "translate-then-detect" underperforms while native-language prompting and one-shot learning are effective.

3. Evidence of Western-centric bias in current VLMs, together with a three-tier characterisation of how sensitively each model reacts to prompt language.

4. A qualitative diagnostic analysis, including a human evaluation of Google Translate output and documented failure cases such as semantic flip, to explain why translation damages detection.

Main Findings

• A performance hierarchy tracks model scale. The paper reports a "strong positive correlation" between general model capability and average cross-lingual performance, with Gemini-2.5-Flash forming a top tier. It reports that no model achieves perfect stability, and that small models such as InstructBLIP-Vicuna-7B are not randomly erratic but "consistently poor," with tightly clustered low performance.

• Generalists beat specialists except in linguistically proximate transfers. General-purpose VLMs such as Gemini-2.5-Flash and GPT-4o-Mini establish a stronger average cross-lingual baseline than fine-tuned models. The PromptHate transfer matrix shows a "feast or famine" pattern, with strong results where the test language is linguistically close to the training language (the paper cites PromptHate trained on Spanish tested on Italian) and collapse on linguistically distant transfers.

• Prompt-language sensitivity forms three tiers. Tier 1 (robust): Gemini-2.5-Flash and Qwen-2.5-VL-7B-Instruct show low sensitivity to whether the prompt is English or native. Tier 2 (moderately sensitive): GPT-4o-Mini and LLaMA-4-Maverick. Tier 3 (brittle): InstructBLIP-Vicuna-7B and the CogVLM2 variants.

• Native prompts can hurt, not just help. For GPT-4o-Mini, zero-shot native prompting lowered average F1 to 0.553 from 0.621 with English prompts. The paper attributes this to English-centric safety guardrails and lexical over-sensitivity, where native slang or aggressive vocabulary triggers flags even in benign or sarcastic contexts. The same pattern appears in CogVLM2-Chinese-19B, whose zero-shot average fell to 0.287 with native prompts from 0.482 with English prompts, while InstructBLIP-Vicuna-7B improved from 0.314 to 0.386 with native prompts.

• One-shot learning helps most where baseline performance is weakest. InstructBLIP-Vicuna-7B gained 0.269 average F1 going from zero-shot English to one-shot English (0.314 to 0.583) and 0.207 from zero-shot native to one-shot native (0.386 to 0.593). The paper reports that top-tier models benefit only marginally; Gemini-2.5-Flash moved from 0.663 to 0.665 (English) and 0.671 to 0.683 (native). It notes that weaker models' final one-shot scores often still fall below larger models' zero-shot baselines.

• Translation carries a consistent penalty. Across all models and settings, the highest F1 scores appear on the heatmap diagonal, representing native-language content, while off-diagonal cells representing translated captions show a universal drop. The paper describes this as pervasive and independent of model scale.

• Human evaluation finds meaning loss in translation. Two native speakers per language pair rated 50 randomly sampled translated captions on a 4-point scale for correctness and meaning preservation. Spanish-to-English scored lowest at 2.89 on both dimensions, and Italian-to-English scored 3.22 (correctness) and 3.11 (meaning), while English-to-Arabic and English-to-German and German-to-English each scored 4.00 on both.

• Documented failure modes. An Arabic meme whose colloquial text was machine-translated to "You made me hate girls" was flagged as hate speech by the model's English safety pattern matching, though in context the paper describes the phrase as harmless exaggerated frustration. The paper also reports loss of sarcasm in Bengali-to-English translation and flattening of Arabic dialect into generic phrases.

Methodology in Plain English

The researchers did not build a new dataset. They selected six existing publicly available meme datasets, each organically created within its own language community: Prop2Hate (Arabic, 3,061 samples), BHM (Bengali, 6,852), HateMeme (English, 5,029), GerMemeHate (German, 179), DANKMEMES (Italian, 1,000), and DIMEMEX (Spanish, 2,263). Each dataset keeps its own original label set, so the task is multi-class classification rather than a single shared binary decision.

To test translation effects, they used Google Translate's v3 API to translate only the meme caption into every other target language, deliberately leaving the original image and any text embedded in that image untouched. This mimics a realistic "translate-then-detect" moderation pipeline without the noise that OCR-based image reconstruction would add.

Every model was then tested on a 2×2 grid: zero-shot vs. one-shot learning, crossed with English vs. native-language prompts. In the one-shot condition, each model received one gold-standard example per label; these examples were chosen by native speakers to be culturally typical, linguistically natural, and aligned with the dominant meme style, then fixed identically across all models. Results were scored with Accuracy and Macro-F1, computed only over outputs that matched the label schema; invalid outputs stayed below 0.5% across models and languages.

A qualitative review followed, plus the human translation-quality study described above, to explain the quantitative patterns.

Why This Matters

Impact on research. The paper argues the field should move from multilingual benchmarks to genuinely multicultural ones, and proposes that evaluation report robustness, fairness, and cross-cultural consistency alongside accuracy. It offers a reusable protocol for probing cultural alignment rather than a single leaderboard number, and it shifts the framing from diagnosing bias to testing concrete mitigation strategies.

Real-world applications:

• Social media and platform content moderation for non-English user bases, where translated pipelines may systematically over-moderate culturally nuanced but benign content.
• Deployment architecture design for global trust-and-safety systems, where the authors recommend cascading locally fine-tuned smaller models with powerful general-purpose VLMs rather than relying on a single translation-based model.
• Benchmarking and procurement of multimodal models in multilingual markets, where prompt-language sensitivity is a practical selection criterion documented per model.
• Machine translation quality assessment for culturally loaded content, since the paper shows correctness scores can remain high while meaning preservation degrades.

Industry relevance. The findings bear directly on moderation vendors and platform teams that operate translate-then-detect pipelines at scale, and on model developers whose safety guardrails behave inconsistently across languages.

Future Directions

• Building natively sourced, parallel multilingual and multicultural meme datasets, since the paper notes Multi3Hate's content originates from English sources and is subsequently translated.
• Developing evaluation metrics that capture robustness, fairness, and cross-cultural consistency, not just accuracy.
• Testing the proposed hybrid ensemble strategy, cascading locally fine-tuned models with general-purpose VLMs, and measuring whether it balances cultural sensitivity against general safety guardrails.
• Extending the six-language scope to additional linguistic and cultural ecosystems, and addressing the limitations the authors state are discussed in Appendix F of the paper.

Target Audience

Trust-and-safety and content-moderation practitioners evaluating global deployment pipelines; researchers in multimodal harm detection and multilingual NLP; VLM developers concerned with cross-cultural fairness and safety alignment; and evaluation specialists designing cultural robustness benchmarks.

Authors’ abstract

Cultural context profoundly shapes how people interpret online content, yet vision-language models (VLMs) remain predominantly trained through Western or English-centric lenses. This limits their fairness and cross-cultural robustness in tasks like hateful meme detection. We introduce a systematic evaluation framework designed to diagnose and quantify the cross-cultural robustness of state-of-the-art VLMs across multilingual meme datasets, analyzing three axes: (i) learning strategy (zero-shot vs. one-shot), (ii) prompting language (native vs. English), and (iii) translation effects on meaning and detection. Results show that the common ``translate-then-detect'' approach deteriorate performance, while culturally aligned interventions - native-language prompting and one-shot learning - significantly enhance detection. Our findings reveal systematic convergence toward Western safety norms and provide actionable strategies to mitigate such bias, guiding the design of globally robust multimodal moderation systems.