MemeArena: Automating Context-Aware Unbiased Evaluation of Harmfulness Understanding for Multimodal Large Language Models

AI summary

Overview

Research area: Evaluation and safety auditing of multimodal large language models (mLLMs), specifically their ability to interpret harmful meaning in memes; multi-agent / agent-as-judge evaluation frameworks.

Technical level: Advanced. The paper assumes familiarity with mLLM prompting, LLM-as-a-Judge pipelines, pairwise arena-style comparisons, Elo and Bradley-Terry ranking, and NDCG ranking metrics.

Scope (one sentence): The paper introduces MemeArena, a three-stage agent-based "arena" framework that simulates diverse socio-cultural interpretive contexts for harmful memes, fuses the opinions of a four-model judge panel into consensus guidelines, and produces bias-reduced rankings of 15 target mLLMs on context-aware harmfulness understanding.

What This Paper Is About

Existing evaluations of how well mLLMs understand harmful memes mostly reduce the problem to binary classification accuracy against predefined labels, which cannot capture how the same meme is read differently by audiences with different cultural, social, or political backgrounds. The authors argue that fairness also breaks down on the judging side, because evaluators themselves hold divergent conceptions of harmfulness, producing subjective scores. MemeArena is their proposed fix: an automated, agent-driven arena that generates perspective-specific harmfulness tasks, builds value-aligned reference guidelines through multi-round judge discussion, and ranks models by pairwise comparison.

Key Contributions

1. A context-centric, open-ended evaluation paradigm. The authors state this is the first work to automatically evaluate mLLMs' context-aware harmfulness understanding of multimodal memes through an analytical lens, rather than through binary detection accuracy against fixed labels.
2. The MemeArena framework itself, an agent-based, arena-style evaluation pipeline with three stages: Context Simulation & Task Formulation, Multi-view Fusion, and Judgment & Ranking. It automates both task design and value-aligned judgment.
3. Demonstrated bias reduction. Across four judge agents and four evaluation settings, MemeArena achieves the highest average NDCG (0.98) for inter-judge consistency, indicating judgments that align closely with collective/human preferences.
4. Public release. Code and data are available at https://github.com/Lbotirx/MemeArena.

Main Findings

• Gemini 2 dominates the leaderboard. It holds the highest overall Elo score (1448.16) across nearly all evaluation aspects, with an average win rate of 89.68% over 673 pairwise battles.
• Running order behind it: Qwen2.5-VL (32B) is second (overall Elo 1308.11, win rate 79.97%), followed by Gemini 1.5 (1303.03, 79.71%), Qwen-VL-Max (1099.89, 60.30%), GPT-4o mini (1093.74, 58.99%), GPT-4o (1054.36, 54.20%), Pixtral (124B) (1042.08, 53.25%), Step-1o (1023.59, 51.13%), Qwen2.5-VL (7B) (1005.71, 48.97%), Step-1v (914.45, 38.77%), Doubao-Pro (869.46, 34.41%), LLaVA-NeXT (34B) (818.73, 29.08%), Pixtral (12B) (747.07, 22.40%), Doubao-Lite (693.32, 17.96%), and LLaVA-NeXT (8B) (578.31, 10.65%).
• A Redundancy anomaly. Target models are otherwise fairly consistent across the five evaluation dimensions, but Gemini 2 (1079.73) and Qwen2.5-VL (32B) (1077.35) score comparatively low on Redundancy, meaning their background knowledge is accurate and comprehensive but less concise than that of Gemini 1.5 and GPT-4o.
• Model scale does not predict performance. Among open-source models, Qwen2.5-VL (32B) outperforms the larger Pixtral (124B) and LLaVA-NeXT (34B); among closed-source models, GPT-4o mini surpasses GPT-4o.
• MemeArena produces the least biased judge rankings. Average NDCG: MemeArena 0.98, human-written guidelines 0.96, w/o guideline 0.90, LLM-as-a-Judge 0.89. Under MemeArena, GPT-4o reaches NDCG 1.00, meaning its ranking is identical to the joint voting result of all judges.
• Self-curated guidelines are the most biased. In the LLM-as-a-Judge setting, Qwen2.5-VL as judge drops to NDCG 0.68, driven by ranking Gemini 2 lower and itself higher, i.e., self-evaluation bias. Under the w/o guideline setting, Step-1o falls to 0.68. When GPT-4o judges in the LLM-as-a-Judge setting, Gemini 2 receives an overwhelmingly high Elo far above all others.
• Human evaluation supports guideline quality. On a three-point Likert scale (1 poorest, 3 best), the generated guidelines scored 2.8293 for Readability, 2.5285 for Informativeness, 2.5041 for Soundness, 2.4797 for Persuasiveness, and 2.2846 for Conciseness. Agreement scores were 0.5478 (Psv.), 0.5157 (Inf.), 0.5057 (Snd.), 0.4862 (Rdb.), and 0.3747 (Cns.).
• Iterative discussion needs roughly eight rounds. Persuasiveness, Informativeness, and Soundness initially drop when non-judge mLLM answers are introduced, then steadily improve and plateau around the 8th round, indicating convergence.

Methodology in Plain English

MemeArena runs a harmful meme through three stages.

Stage 1 — Context Simulation & Task Formulation. Because memes are read differently by different audiences, the framework invents three interpretive contexts per meme: someone with a background highly relevant to the meme, someone moderately relevant, and someone completely unrelated who just happens to see it. Each context is turned into its own task asking the target model to interpret harmfulness from that specific standpoint. Target models answer in a chain-of-thought format with two explicit parts, [Background Knowledge] and [Reasoning], so perception and reasoning can be scored separately.

Stage 2 — Multi-view Fusion. Each meme therefore collects many perspective-specific analyses. A panel of judge agents (GPT-4o, Gemini 2, Step-1o, Qwen2.5-VL (32B) — deliberately drawn from different model families to diversify any inherent preferences) iteratively discusses and rewrites a shared guideline. In each round, one judge is randomly selected and shown the current guideline plus one randomly drawn analysis, and produces a refined version; judges never review their own analysis, and the total number of rounds exceeds the number of judges so every judge participates. The result is a single consensus guideline representing aligned judgments about that meme.

Stage 3 — Judgment & Ranking. Judges then perform pairwise comparisons of target-model responses against that guideline, scoring on five criteria: Instruction Following (overall adherence), Redundancy and Correctness (the background-knowledge part), and Relevance and Accuracy (the reasoning part). Pairwise outcomes feed an Elo system (scaling constant 400) and then a Bradley-Terry model, which reduces Elo's sensitivity to the order in which comparisons happen and handles ties by splitting them as 0.5 wins for each side.

Evaluation setup: 15 target mLLMs across 7 model families (GPT, Gemini, Step, Qwen, Doubao, LLaVA-NeXT, Pixtral) were evaluated on harmful memes collected from three public datasets — HarM, FHM, and MAMI. Bias was quantified with NDCG, treating the joint vote of all judges as the ideal ranking; lower NDCG for an individual judge means weaker agreement with the collective and therefore more bias.

Why This Matters

Impact on research: The paper argues that accuracy on binary harmful-meme classification is an impoverished proxy for harmfulness understanding, and that judge bias — not just model capability — shapes reported rankings. It contributes a reproducible protocol plus an explicitly bias-aware metric setup (NDCG against joint voting) for comparing agents and judges. It also reframes harmfulness as inherently multi-perspective, which has implications for how harmful-content benchmarks should be designed and interpreted.

Real-world applications:

• Content moderation: more nuanced, context-sensitive triage of meme-based abuse on social platforms, where literal keyword or label matching misses culturally coded harm.
• Model auditing and procurement: teams choosing between mLLMs for trust-and-safety workloads can use context-specific harmfulness interpretation as a selection criterion rather than generic capability scores.
• Red-teaming and policy review: simulating how different audiences (e.g., policy experts versus casual viewers) interpret a meme supports policy teams reasoning about disparate impact.
• Safety evaluation infrastructure: an automated alternative to expensive crowdsourced pairwise arenas for subjective judgment tasks.

Industry relevance: Platform trust-and-safety teams, model providers shipping vision-language models, and evaluators who currently rely on LLM-as-a-Judge pipelines all face the bias problem the paper documents — particularly the self-preference effect where a judge promotes itself or demotes rivals. The finding that model scale does not track harmfulness-understanding performance is directly relevant to model selection decisions.

Future Directions

• Remaining guideline bias. The authors acknowledge that some bias can still enter during guideline refinement, and note the growing need for human-in-the-loop processes given that many new mLLMs are trained on synthetic data distilled from GPT-series models.
• Better data and modalities. They plan to add newly established benchmarks or data collected from online communities to better capture shifting real-world harmful-content distributions, and to extend beyond static vision-language pairs to video-based media.
• More realistic context simulation. They want to explore role-play or persona-driven simulation to improve the realism and diversity of interpretive contexts, and to find ways to quantitatively assess how much contextual diversity contributes to evaluation comprehensiveness.
• Judge configuration and factual grounding. They plan systematic study of how judge panel size and model-family diversity affect reliability, and propose incorporating Retrieval-Augmented Generation so guidelines can be evidence-supported rather than resting solely on the internal knowledge of judge agents, which risks propagating shared misconceptions or missing recent cultural developments.

Target Audience

Researchers and practitioners working on multimodal model evaluation, AI safety auditing, and harmful-content detection, particularly those already using LLM-as-a-Judge or arena-style comparison pipelines and looking to reduce judge bias. It will also interest trust-and-safety engineers at social platforms, benchmark designers who need to justify abandoning binary labels for subjective harmfulness tasks, and NLP researchers studying multi-agent consensus and ranking methodologies. Readers without background in pairwise ranking metrics such as Elo, Bradley-Terry, and NDCG will need to read the methodology section carefully.

Authors’ abstract

The proliferation of memes on social media necessitates the capabilities of multimodal Large Language Models (mLLMs) to effectively understand multimodal harmfulness. Existing evaluation approaches predominantly focus on mLLMs' detection accuracy for binary classification tasks, which often fail to reflect the in-depth interpretive nuance of harmfulness across diverse contexts. In this paper, we propose MemeArena, an agent-based arena-style evaluation framework that provides a context-aware and unbiased assessment for mLLMs' understanding of multimodal harmfulness. Specifically, MemeArena simulates diverse interpretive contexts to formulate evaluation tasks that elicit perspective-specific analyses from mLLMs. By integrating varied viewpoints and reaching consensus among evaluators, it enables fair and unbiased comparisons of mLLMs' abilities to interpret multimodal harmfulness. Extensive experiments demonstrate that our framework effectively reduces the evaluation biases of judge agents, with judgment results closely aligning with human preferences, offering valuable insights into reliable and comprehensive mLLM evaluations in multimodal harmfulness understanding. Our code and data are publicly available at https://github.com/Lbotirx/MemeArena.