CLASH: A Benchmark for Cross-Modal Contradiction Detection

AI summary

Overview

Research area: Multimodal evaluation for vision-language models (computer vision / multimodal large language models). Technical level: Intermediate — the benchmark and evaluation protocol are accessible, but interpreting the modality-bias and fine-tuning results benefits from familiarity with VQA-style benchmarks. Scope: The paper introduces CLASH, a benchmark of MS COCO images paired with captions containing exactly one controlled contradiction, and uses it to measure whether multimodal models can spontaneously detect cross-modal conflicts.

What This Paper Is About

Existing multimodal benchmarks generally assume the image and text agree, declare one modality authoritative, or test abstention when information is missing. The authors argue that none of them measure a distinct capability: recognizing when two valid, authoritative sources provide contradictory information. CLASH fills that gap with image–text pairs built from MS COCO captions in which exactly one object or objective attribute has been changed, plus targeted questions in multiple-choice and open-ended formats.

Key Contributions

1. A diagnostic benchmark for multimodal conflict detection: CLASH provides approximately 15k filtered training samples and a human-verified test set of 1289 cases (655 object-level and 634 attribute-level contradictions), organized into five object and five attribute categories.
2. Extensive evaluation of state-of-the-art models: The authors evaluate closed-source models (GPT-5, GPT-4.1 Mini, Gemini 2.5 Pro, Gemini 2.5 Flash Lite) and a range of open-source models, exposing large performance gaps, systematic modality biases, and category-specific weaknesses.
3. Demonstration that targeted fine-tuning works: LoRA fine-tuning substantially improves conflict detection, with some models gaining over 75 accuracy points on contradiction samples.

Main Findings

• Closed-source models lead by a wide margin: On the multiple-choice task, Gemini 2.5 Pro detects conflicts 88.48% of the time and GPT-5 86.78%, while most open-source models score below 5%. InstructBLIP-T5-xxl is the strongest open-source model at 63.87%.
• Failures split into distinct modality biases: GPT-4.1 Mini (69.57%) and Gemini 2.5 Flash Lite (74.98%) answer with the image-grounded option, while InternVL 1.5 (52.40%) and LLaVa-1.5-7b (44.68%) pick the text-grounded option. On the open-ended task, mPLUG-Owl-2 shows extreme text bias at 97.03%, and LLaVa-1.5-7b and InternVL 1.5 also lean textual (62.10% and 55.51%).
• Larger models do not reliably detect more conflicts: Qwen2.5VL-32b (16.10%) beats Qwen2.5VL-7b (7.79%), but Qwen3VL-30b (3.81%) underperforms Qwen3VL-8b (7.13%).
• Instruction-following is a separate failure mode: Phi3-vision-128k produces unusable multiple-choice answers 59.63% of the time, MiniGPT4-7b 52.30%, and LRV-MiniGPT4-7b 51.74%. Newer models such as Qwen2.5VL-32b, Qwen3VL and LLaVa-OneVision produce no incorrect multiple-choice responses.
• Open-ended format changes the ranking: Gemini 2.5 Pro improves to 91.59% conflict detection (vs 88.48% multiple-choice) while GPT-5 drops to 81.16% (vs 86.78%), largely from empty responses. GPT-4.1 Mini improves sharply to 39.93% from 13.56%. The best open-source open-ended results are Qwen3VL-instruct-8b (20.49%), Qwen3VL-instruct-30b (19.80%), InternVL 1.5 (18.76%) and Qwen2.5VL-instruct-7b (17.13%).
• Fine-tuning is highly effective, and data quality beats data quantity: LLaVa-1.5-7b rises from 0.00% to 76.86% conflict detection when fine-tuned on the approximately 15k filtered samples, with overall accuracy of 84.08% and incorrect outputs falling to 0.47%. The same model trained on the approximately 30k unfiltered set reaches only 51.84% conflict detection. mPLUG-Owl-1 improves from 6.56% to 57.45%, with incorrect outputs dropping from 61.63% to 7.09%.
• Category-level weaknesses are systematic: Top-tier models are strongest on Animals (GPT-5: 97.32%, Gemini 2.5 Pro: 98.65%) and weakest on Household items (around 78%), while Environmental attributes are the hardest attribute category. Gemini 2.5 Pro outperforms GPT-5 on Numbers (85.83% vs 75.70%).

Methodology in Plain English

The benchmark is built in three automatic stages using Gemini 2.5 Flash. First, starting from a COCO caption, the model changes exactly one element — either an object (for example, snowboarder to skier) or an objective attribute such as color, number, shape, material or texture (for example, orange to blue) — while leaving the rest of the caption identical. Second, the model writes a subtle question about the changed element that does not signal that anything is wrong. Third, it produces answer options: one grounded in the image (the original caption), one grounded in the text (the conflicting caption), and one plausible distractor appearing in neither caption, plus a "Conflicting information – cannot answer" option as the correct multiple-choice answer.

Quality control combines rule-based word-level checks with LLM-based assessment using Gemini 2.5 Flash Lite. Automated checks verify that original words appear in the source caption, conflicting words appear in the modified caption, and distractors appear in neither. LLM checks confirm that attribute changes are truly objective rather than subjective descriptors, that objects are not synonyms or ambiguous, that questions are clear and answerable, and that answers are not near-duplicates or ambiguous terms. Every test sample then receives a binary accept/reject vote from a human annotator, and only accepted samples enter the final set. Categories were defined by extracting the changed word pairs, computing frequency distributions, and taking the 20 most common terms in each category; GPT-5 assigned words to the five object and five attribute categories, with manual verification. Samples required strict co-membership, meaning both the original and conflicting word had to fall in the same category, otherwise they were labelled "Other".

Evaluation covers a four-option multiple-choice task scored by strict string matching and an open-ended task scored by relaxed matching using tokenization, article removal, number normalization, word stripping and Porter stemming, with LLM-as-a-judge validation using Gemini 2.5 Pro and GPT-5. Accuracies are reported with bootstrap-estimated standard deviations over 1000 iterations. For fine-tuning, LLaVa-1.5-7b and mPLUG-Owl-1 were trained with LoRA for one epoch in bf16 with gradient checkpointing on a single A100-64GB, sampling conflicting and original captions with equal probability to avoid a bias toward flagging every input as a conflict.

Why This Matters

Impact on research. CLASH reframes evaluation around a dual ground truth paradigm, where both the image and the text are valid but contradictory, rather than treating the image as the sole truth. It provides a controlled setting to measure modality bias and conflict detection separately from hallucination, abstention, and conflict localization benchmarks, and shows those capabilities are not automatically acquired through scale or instruction tuning.

Real-world applications.

• Medical systems where a report states "no abnormalities" while imaging shows a fracture.
• Autonomous vehicles where a text or sensor summary indicates clear roads while camera feeds show barriers.
• Financial documents where typed amounts disagree with scanned figures.
• Any high-stakes workflow where inconsistent sources must be flagged for human review rather than silently resolved.

Industry relevance. The results give a concrete reason to test models for spontaneous conflict detection before deployment, and demonstrate that targeted, quality-filtered fine-tuning is a practical remedy rather than a costly architectural change.

Future Directions

• Extend the benchmark beyond object- and attribute-level conflicts to spatial, relational, temporal and causal contradictions.
• Move beyond everyday MS COCO scenes into specialized domains such as medical, scientific or technical content.
• Strengthen reliability of the synthetic pipeline through cross-model agreement checks or multi-annotator verification.
• Investigate why capability does not scale monotonically, since larger Qwen variants did not consistently detect more conflicts and scale alone did not guarantee improvement.

Target Audience

Researchers and engineers working on multimodal large language models, benchmark design and model reliability, as well as practitioners in safety-critical domains who need to decide whether a vision-language system can be trusted to flag inconsistent inputs for human review.

Authors’ abstract

Contradictory multimodal inputs are common in real-world settings, yet existing benchmarks typically assume input consistency and fail to evaluate cross-modal contradiction detection - a fundamental capability for preventing hallucinations and ensuring reliability. We introduce CLASH, a novel benchmark for multimodal contradiction detection, featuring COCO images paired with contradictory captions containing controlled object-level or attribute-level contradictions. The samples include targeted questions evaluated in both multiple-choice and open-ended formats. The benchmark provides an extensive fine-tuning set filtered through automated quality checks, alongside a smaller human-verified diagnostic set. Our analysis of state-of-the-art models reveals substantial limitations in recognizing cross-modal conflicts, exposing systematic modality biases and category-specific weaknesses. Furthermore, we empirically demonstrate that targeted fine-tuning on CLASH substantially enhances conflict detection capabilities.