Multi-agent Undercover Gaming: Hallucination Removal via Counterfactual Test for Multimodal Reasoning

AI summary

Overview

• Research area: Multimodal reasoning and hallucination mitigation in large language / multimodal large language models, specifically the Multi-Agent Debate (MAD) paradigm.
• Technical level: Advanced. The paper assumes familiarity with MLLMs, chain-of-thought reasoning, multi-agent systems, counterfactual reasoning (structural causal models, do-calculus), CLIP/ViT embeddings, and FID scores.
• Scope: The paper proposes a Multi-agent Undercover Gaming (MUG) protocol that uses counterfactual image edits as ground truth to identify hallucinating agents within a multi-agent debate, and evaluates it on MMMU_VAL, MMStar, HallusionBench, and POPE.

What This Paper Is About

Multi-Agent Debate assumes that all debating agents are rational and will reflect on each other's arguments, but that assumption breaks down when the agents themselves hallucinate. The paper reframes debate as a social deduction game resembling "Who is Undercover?": one agent is secretly given a modified counterfactual image, and the others must detect it. Because the researchers introduced the modification themselves, they hold ground-truth labels for which agent is the hallucinating "undercover" player, letting a crowd of agents verify facts rather than merely agreeing by majority vote.

Key Contributions

1. The MUG protocol. A multi-agent framework that recasts MAD as an undercover-detection game, where one randomly chosen agent receives a counterfactual image I⁻ and the rest receive the factual image I⁺, followed by a summarization phase once the undercover agent is voted out.
2. Counterfactual image generation with explicit constraints. A pipeline that classifies the question type, builds a scene graph of the factual image to select edit targets, and prompts the Step1X-Edit model to make minimal edits, accepted only if a weighted score over visual similarity (ViT embeddings), semantic consistency (CLIP embeddings), and naturalness (FID) meets a confidence threshold.
3. Factual verification replacing statistical consensus. Counterfactual tests provide direct evidence about which agent is hallucinating, in place of majority voting or super-judge protocols.
4. A demonstrated performance gain over single-agent and MAD baselines. MUG improves accuracy on MMMU_VAL, MMStar, HallusionBench, and POPE relative to Self-Refine, MAD-Vote, and MAD-Judge, with improvements reported as statistically significant by a one-tailed paired t-test at p < 0.01.

Main Findings

• MMMU_VAL gains. Qwen2.5VL-7B with MUG reaches 50.3% accuracy versus a 45.0% baseline, a 5.3-point improvement. InternVL3-14B with MUG reaches 60.7% versus a 59.8% baseline, a 0.9-point improvement.
• MMStar results. Qwen2.5VL-7B (MUG) reaches 63.8% and InternVL3-14B (MUG) reaches 69.1% accuracy.
• Advantage over MAD variants (Qwen2.5VL-7B). MUG beats MAD-Vote by 5.6 points on MMMU (50.3% vs 44.7%) and by 16.0 points on HallusionBench average (53.8% vs 37.8%). It beats MAD-Judge by 2.9 points on MMMU (53.8% vs 50.2%) and by 3.6 points on HallusionBench average (53.8% vs 50.2%). With InternVL3-14B, MUG surpasses MAD-Vote by 5.5 points on MMMU.
• Hallucination detection. MUG achieves the highest HallusionBench average scores, 53.8% for Qwen2.5VL-7B and 58.0% for InternVL3-14B. Figure accuracy improves by 9.0 points over base Qwen (43.9% vs 34.9%) and 3.5 points over InternVL3 (51.2% vs 47.7%).
• Category-level HallusionBench strengths. Visual similarity 78.9% vs 69.7% baseline, OCR 86.2% vs 75.4%, and figure understanding 66.7% vs 46.3%; performance on mathematical reasoning and video analysis is described as competitive rather than improved.
• POPE results. MUG achieves 88.4% (Qwen2.5VL-7B) and 90.1% (InternVL3-14B) accuracy. Recall improves by 3.2 points (80.5% vs 77.3%) for Qwen and 1.7 points (88.2% vs 86.5%) for InternVL3. Category gains are largest in the random setting (88.5% vs 86.7%), with adversarial at 86.3% vs 85.2% and popular at 87.3% vs 86.2%.
• Narrowing the open-versus-closed model gap. Qwen2.5VL-7B with MUG reaches overall figures of 63.8%, 53.8%, and 88.4%, described as better than GPT-4v and Claude3.5-Sonnet in that comparison.
• Ablation. Removing counterfactual editing drops performance by 1.49 points on MMStar, 3.61 points on HallusionBench, and 1.08 points on MMMU. Removing the undercover agent mechanism drops it by 1.57 points on MMStar, 4.49 points on HallusionBench, and 2.67 points on MMMU.
• Round sensitivity. Accuracy peaks at round 1 across all three reported benchmarks: 69.40 on HallusionBench, 50.33 on MMMU, and 63.80 on MMStar, versus round-0 values of 67.31, 47.88, and 61.93 and later-round values that stabilize or decline.
• Game termination behavior. Undercover agents are eliminated fastest in the second round; in the third or fourth round survival percentages hover around 5%.
• Case study. A traditional agent misidentifies a logo reading "Red Deer" as Red Bull; MAD fails to correct this through majority voting; MUG surfaces the conflict between "my image says 'Red Bull'" and "Red Deer" and reaches the correct identification.

Methodology in Plain English

The system is defined as a state consisting of a question, a set of N agents, a collection of debating functions, and the agents' responses, evolving over rounds. Before the game, the researchers produce a counterfactual image by asking an LLM to classify the question type (for example, a "How Many" question maps to quantity editing) and to locate candidate objects using a scene graph of the original image, then prompting an image generator to make a minimal change. The edit is accepted only if it stays visually similar to the original, keeps the same overall semantics, and looks natural, measured by ViT similarity, CLIP similarity, and FID respectively; otherwise generation repeats.

In the game, one random agent receives the counterfactual image. In each detection round, every agent produces reasoning conditioned on its own image and the history of prior responses and votes. Regular agents aim for accuracy plus detection value; the undercover agent aims for plausibility minus suspicion. Agents then vote, weighing four factors: inconsistency with expected reasoning, deviation from the group consensus, accuracy of specific visual details, and behavioral suspicion such as vagueness. The agent with the most votes is eliminated. The game ends if the undercover agent is removed, if fewer than two regular agents remain, or if a maximum round count is exceeded. The survivors then enter a summarization phase and collaboratively produce the final answer using the factual image.

Experiments use Qwen2.5VL-7B and InternVL3-14B as backbones. Settings are reported as 8x A100 (40GB) GPUs, bfloat16, 224x224 image resolution, temperature 0.2 with top-p 0.001 and top-k 1 for Qwen2.5VL-7B (max 2048 output tokens), and temperature 1.0 with top-p 1.0 and top-k 50 for InternVL3-14B (max 4096 output tokens). Baselines include open-source models (DeepSeek-VL-7B, LLaVA-OneVision-7B, LLaVA-NEXT-Llama3-8B, LLaVA-v1.5-13B, InternVL2-8B, InternVL2-26B, InternVL3-14B, Qwen2.5VL-7B), closed-source models (Gemini-1.5-Pro, GPT4O_mini, GPT-4v, Claude-3.5-Sonnet), Self-Refine, and MAD with vote or judge aggregation.

Why This Matters

• Impact on research. The paper argues that MAD protocols inherited from human debate assume a rationality that LLM debaters may not possess. By supplying external ground truth through controlled edits, MUG replaces "the majority is probably right" with "we can check who is wrong," which is a different and more falsifiable basis for multi-agent reliability.
• Real-world applications.
  • Medical or clinical image review, where a model that misreads a detail could be caught by comparing against a deliberately altered reference.
  • Document and logo verification in e-commerce or brand-protection pipelines, as in the Red Deer / Red Bull case study.
  • Assistive tools for blind or low-vision users, where object hallucination directly affects what the user is told is present.
  • Automated quality control for visual question answering and captioning systems deployed at scale.
• Industry relevance. Because MUG is a protocol layered on top of existing models rather than a new trained model, it is a drop-in reliability wrapper for teams already running MLLMs in multi-agent or ensemble configurations. The reported narrowing of the gap between 7B open models and larger closed models is directly relevant to cost-sensitive deployments.

Future Directions

• Adaptive control of game rounds. Performance peaks at round 1 and declines afterward, so learning when to stop debating is an open problem.
• Explaining the round-3/4 plateau. Survival rates near 5% in later rounds suggest either that some hallucinations are hard to detect or that undercover agents blend in better in certain cases; the paper does not resolve which.
• Dependence on edit quality. The whole protocol rests on generating counterfactual images that are minimal, natural, and semantically consistent; the paper does not report how often generation fails the acceptance threshold or has to be repeated.
• Generalization beyond image edits. The protocol is demonstrated for multimodal counterfactual tests; extending the same ground-truth-by-construction idea to video, audio, or purely textual evidence is left open.

Target Audience

Researchers and engineers working on multimodal LLM reliability, hallucination mitigation, and multi-agent LLM orchestration. It is most useful to readers already comfortable with the MAD literature and with embedding-based image similarity metrics; readers new to multi-agent systems would need background reading first. Practitioners evaluating whether to replace majority-vote ensembling with a verification-based protocol are the most direct beneficiaries.

Note on reported figures: The paper gives two different sizes for two datasets. In the datasets section it states HallusionBench provides 346 samples and POPE contains 3,000 yes/no questions; Table 3 lists HallusionBench at 951 and POPE at 5,127, with MMMU_VAL at 1,050 and MMStar at 1,500. The paper does not reconcile these discrepancies.

Authors’ abstract

Hallucination continues to pose a major obstacle in the reasoning capabilities of large language models (LLMs). Although the Multi-Agent Debate (MAD) paradigm offers a promising solution by promoting consensus among multiple agents to enhance reliability, it relies on the unrealistic assumption that all debaters are rational and reflective, which is a condition that may not hold when agents themselves are prone to hallucinations. To address this gap, we introduce the Multi-agent Undercover Gaming (MUG) protocol, inspired by social deduction games like "Who is Undercover?". MUG reframes MAD as a process of detecting "undercover" agents (those suffering from hallucinations) by employing multimodal counterfactual tests. Specifically, we modify reference images to introduce counterfactual evidence and observe whether agents can accurately identify these changes, providing ground-truth for identifying hallucinating agents and enabling robust, crowd-powered multimodal reasoning. MUG advances MAD protocols along three key dimensions: (1) enabling factual verification beyond statistical consensus through counterfactual testing; (2) introducing cross-evidence reasoning via dynamically modified evidence sources instead of relying on static inputs; and (3) fostering active reasoning, where agents engage in probing discussions rather than passively answering questions. Collectively, these innovations offer a more reliable and effective framework for multimodal reasoning in LLMs. The source code can be accessed at https://github.com/YongLD/MUG.git.