DixitWorld: Evaluating Multimodal Abductive Reasoning in Vision-Language Models with Multi-Agent Dixit Gameplay

AI summary

Overview

Research area: Evaluation of multimodal abductive reasoning in vision-language models (VLMs), using a multi-agent game environment derived from the board game Dixit.

Technical level: Intermediate — the paper assumes familiarity with VLMs, benchmark design, and evaluation metrics, but the game framing is intuitive.

Scope: The paper introduces DixitWorld, an evaluation suite comprising a dynamic multi-agent game (DixitArena) and a static multiple-choice benchmark (DixitBench), to test how VLMs generate and select explanatory hypotheses from ambiguous visual observations.

What This Paper Is About

Abductive reasoning — inferring the best explanation from partial observations — is usually tested in vision-language models with static, single-agent tasks. The authors argue this misses the dynamic, communicative nature of abduction, where one agent must craft an ambiguous hypothesis and others must infer its correct source. DixitWorld reframes this as Dixit gameplay: a "storyteller" writes a cryptic clue for a private image, and "listeners" must pick that image from a set of decoys, allowing the paper to separate hypothesis generation from hypothesis selection.

Key Contributions

1. DixitWorld evaluation suite — a two-part framework (DixitArena and DixitBench) that decomposes multimodal abductive reasoning into generation and selection, released with a public GitHub repository (HKUST-KnowComp/DixitWorld).

2. DixitArena — an interactive four-agent environment where agents alternate between Storyteller and Listener roles under imperfect information, with a scoring rule that rewards only partial correctness (some, but not all, listeners identify the target image). It uses 84 custom Dixit-style illustrations and runs 21 matches of 24 rounds for 504 total rounds.

3. DixitBench — a static multiple-choice benchmark of 168 questions (84 images × 2 difficulty levels) that isolates the listener task, with distractor difficulty controlled by semantic similarity between pre-generated captions, producing distinct Easy and Hard subsets.

4. Empirical characterization of a storyteller–listener gap — six VLMs are evaluated in both settings, plus a human evaluation of clue Clarity and Creativity and several fairness/robustness checks.

Main Findings

• Larger and proprietary models lead overall. Normalized Overall scores in DixitArena (Table 1): Gemini-2.5-Flash 36.52, GPT-4o 34.58, Gemma3-27B 34.15, Gemma3-12B 31.68, Qwen2.5-VL-32B 30.63, Qwen2.5-VL-7B 17.39.

• Listener ability scales, storyteller ability does not. Listener scores rise with model size and proprietary status (Qwen2.5-VL-7B 30.87 up to Gemini-2.5-Flash 54.05), while Storyteller scores do not follow the same pattern (14.29, 15.48, 20.24, 32.14, 19.05, 30.95 for Qwen2.5-VL-7B, Qwen2.5-VL-32B, Gemma3-12B, Gemma3-27B, GPT-4o, and Gemini-2.5-Flash respectively). GPT-4o is the clearest example — strong as a listener (53.89) but mid-level as a storyteller (19.05).

• Most storyteller rounds score zero. Round outcomes were decomposed into Partial-Correct (the only scoring outcome), All-Correct (too obvious), and All-Wrong (too vague or misleading); over 70% of storyteller rounds fell into the two zero-point categories. Failure modes differ by scale: smaller models produce overly literal or vague clues, larger ones tend toward over-specificity.

• Opponent strength affects scores. Weaker models' scores drop when facing stronger opponents, which the authors attribute to clearer or more discriminative cues creating stronger abductive pressure; matches between weaker models show higher variance.

• DixitBench tracks DixitArena listener results. Direct-selection accuracy (Table 2): GPT-4o 75.60 total (78.57 Easy / 72.62 Hard), Gemini-2.5-Flash 64.89 (65.48 / 64.29), Qwen2.5-VL-32B 63.10 (70.24 / 55.95), Gemma3-27B 62.50 (63.10 / 61.90), Gemma3-12B 57.74 (58.33 / 57.14), Qwen2.5-VL-7B 36.67 (39.05 / 34.29).

• Semantic-distance difficulty labels are weak. The average Easy-vs-Hard gap is only 4.8%, so clues' semantic similarity is not a strong difficulty indicator for these abstract scenarios.

• Direct selection beats entailment scoring. When models rate all candidates independently (0–100) instead of choosing one, accuracy drops by roughly 5–20 percentage points (Table 4), e.g. GPT-4o falls from 75.60 to 59.55 and Gemma3-12B from 57.74 to 48.27, with relative rankings preserved. The authors conclude VLMs are better at comparative abductive choice than at calibrated entailment judgment.

• Human clues set an attainable bar. Human listeners reached 71.59% accuracy with AI-generated clues, closely matched by Gemini-2.5-Flash at 70.24%. Human-rated Clarity was highest for Gemma3-27B (0.387) and Creativity highest for Gemma3-12B (0.574), while Qwen2.5-VL-7B scored lowest on both (0.104 and 0.212).

• The environment is stable and unbiased. A leave-one-listener-out analysis showed small average deviations (from -0.08 to -0.29) with stability indices from 0.849 to 0.908; a chi-squared test on candidate image position gave χ² = 1.83, p = 0.61; hand-swap phases produced score differences under 0.3 points.

Methodology in Plain English

The authors turn Dixit into a test harness. In DixitArena, four agents are dealt four illustration cards each. The storyteller privately picks one target card and writes a metaphorical clue; the other three each submit a distractor card that could plausibly match the clue; the four cards are shuffled and the listeners pick which one they think is the target. Points are only awarded to the storyteller when the clue is partially correct — some listeners get it, some do not — which forces a balance between clarity and ambiguity. Each match runs 24 rounds across two phases of 12, and in phase two players swap hands (P1↔P3, P2↔P4) to remove hand-quality bias. Six VLMs (Qwen2.5-VL-7B/32B, Gemma3-12B/27B, Gemini-2.5-Flash, GPT-4o) played a full round-robin of 21 matches including self-play, with a fixed random seed of 42 for shuffling and candidate order, temperature 0.7, and JSON-formatted prompts. Models were accessed through the OpenRouter API at a total cost of approximately $100 USD.

DixitBench isolates the listener task. The authors generated one abstract caption per illustration, encoded all captions into 384-dimensional embeddings with sentence-transformers/all-MiniLM-L6-v2, and computed an 84×84 cosine similarity matrix. Hard items use the top 5 most similar captions as distractors, Easy items draw distractors randomly from ranks 30–80, yielding 168 items. Twenty sampled items were human-annotated for caption-image coherence, distractor plausibility, and difficulty consistency, with average agreement above 0.8 (Cohen's κ). Listeners were tested under two strategies: direct selection and entailment scoring. A separate human evaluation rated clue Clarity (via a three-step median-and-penalty formula) and Creativity, and measured human listener accuracy on AI-written clues.

Why This Matters

The paper argues that strong performance on static visual QA does not imply competence at generating useful, communicatively calibrated descriptions — a gap that matters as VLM agents are deployed in settings where they must be understood by others.

Real-world applications:

• Assistive and educational systems where an AI must describe an image so that a specific human can identify it, rather than describe it exhaustively.
• Human-robot or human-agent collaboration where an agent must communicate a referent under shared uncertainty.
• Creative tooling (narrative generation, game design, advertising) where controlled ambiguity rather than maximal specificity is the goal.
• Evaluation of multimodal assistants where over-literal descriptions waste users' time and over-vague ones cause errors.

Industry relevance: The results suggest that scaling alone does not fix hypothesis generation, so teams building instruction-following or agentic multimodal products should test generative communication separately from discriminative perception. The finding that entailment-based scoring (common in verification pipelines) underperforms direct selection by 5–20 points also warns that absolute plausibility rating is poorly calibrated and should not be trusted for fine-grained comparisons.

Future Directions

• Broaden task scope. The authors note Dixit represents only one class of multimodal abduction; temporal, causal, and interactive elements of real-world abduction are not captured by a static, turn-based setup.
• Expand model coverage. Evaluate models fine-tuned for visual entailment or narrative generation to test whether the role asymmetry and entailment-calibration findings hold.
• Study prompt and decoding sensitivity. The standardized configuration (temperature 0.7, JSON output) was chosen for comparability, but performance may shift with prompt wording, sampling parameters, or instruction tuning.
• Develop hybrid or adaptive evaluation protocols. Since results shift notably between direct selection and entailment scoring, better strategies for capturing pragmatic reasoning under uncertainty are needed.
• Address the asymmetry architecturally. The authors suggest progress requires explicitly modeling theory-of-mind, uncertainty, and ambiguity in agent design.

Target Audience

Researchers working on vision-language models, multimodal reasoning benchmarks, and multi-agent LLM evaluation; practitioners who need to assess generative communication rather than just perception in deployed multimodal systems; and cognitive-science-adjacent readers interested in operationalizing abductive reasoning and theory-of-mind through game environments.

Authors’ abstract

Multimodal abductive reasoning--the generation and selection of explanatory hypotheses from partial observations--is a cornerstone of intelligence. Current evaluations of this ability in vision-language models (VLMs) are largely confined to static, single-agent tasks. Inspired by Dixit, we introduce DixitWorld, a comprehensive evaluation suite designed to deconstruct this challenge. DIXITWORLD features two core components: DixitArena, a dynamic, multi-agent environment that evaluates both hypothesis generation (a "storyteller" crafting cryptic clues) and hypothesis selection ("listeners" choosing the target image from decoys) under imperfect information; and DixitBench, a static QA benchmark that isolates the listener's task for efficient, controlled evaluation. Results from DixitArena reveal distinct, role-dependent behaviors: smaller open-source models often excel as creative storytellers, producing imaginative yet less discriminative clues, whereas larger proprietary models demonstrate superior overall performance, particularly as listeners. Performance on DixitBench strongly correlates with listener results in DixitArena, validating it as a reliable proxy for hypothesis selection. Our findings reveal a key trade-off between generative creativity and discriminative understanding in multimodal abductive reasoning, a central challenge for developing more balanced and capable vision-language agents.