CrossCheck-Bench: Diagnosing Compositional Failures in Multimodal Conflict Resolution

AI summary

Overview

Research area: Multimodal large language models (vision-language models), benchmark design, and diagnostic evaluation of compositional cross-modal reasoning.

Technical level: Intermediate — readers should be comfortable with VLM evaluation protocols, prompting strategies, and benchmark terminology, though the paper's framing (can a model detect that an image and its text contradict each other?) is intuitive.

Scope: The paper introduces CrossCheck-Bench, a 15k-item diagnostic benchmark built from real e-commerce artifacts with synthetically injected contradictions, and uses it to evaluate 13 vision-language models across three reasoning tiers and seven atomic capabilities.

What This Paper Is About

Vision-language models are usually trained and tested on image-text pairs that agree with each other, so it is unclear whether they can notice when the two modalities contradict one another. The authors build a benchmark of adversarial question-answer pairs drawn from real product listings, where the image and text conflict in ways that require multi-step reasoning to detect. The goal is not just to score models, but to diagnose where in a chain of perception-to-reasoning steps the breakdown happens.

Key Contributions

1. A hierarchical diagnostic framework. CrossCheck-Bench organizes evaluation into three cognitive tiers — L1 Perception, L2 Integration, and L3 Reasoning — decomposed into seven atomic capabilities (A1–A7) and 15 systematically constructed tasks, including six tasks that probe a single atomic capability and nine that require compositions.

2. A large, factually grounded adversarial dataset. The benchmark contains 14.69k QA pairs (described elsewhere as 15k, or 14,690 adversarial QA pairs) derived from 22.8k product listings and 22.8k multimodal clue graphs, spanning 30+ categories and 5 languages, built with more than 450 expert hours.

3. A systematic evaluation of 13 state-of-the-art VLMs, revealing a consistent accuracy decline from perceptual matching to logical contradiction detection, and identifying cross-frame alignment and rule-grounded inference as the major bottlenecks.

4. An intervention study. Conventional prompting (Chain-of-Thought, Set-of-Mark) yields only marginal gains, whereas Multimodal Interleaved CoT (MM-CoT), a two-stage protocol that interleaves grounding with symbolic reasoning, gives more stable improvements.

Main Findings

• Human–model gap is large. Seven expert annotators averaged 95.2% overall, surpassing the best proprietary model by over 18 points. Human performance on reasoning tasks stayed above 88%, while most models fell short of 76%.

• Proprietary models lead, but everyone degrades with complexity. GPT-4.1 and Gemini 2.5 Pro average above 76%, while the strongest open-source model (InternVL3-78B) peaks at 71.5%. The paper reports GPT-4.1 declining from 85.3% on L1 tasks to 75.7% on L3, InternVL3-78B falling from 71.5% at L1 to 64.0% at L3, and Qwen2.5-VL-72B dropping from 69.9% to 63.9%.

• Composition is where models break. In the controlled atomic-vs-compositional comparison, accuracy dropped by 12%–35% when tasks added numerical plausibility (A5), cross-frame reasoning (A4), or rule compliance (A7). Adding A7 to the BDC baseline (A1+A2+A6) cost GPT-4.1 22.4 points; adding A3+A4+A7 cost it 32.7 points; the pooled drop across the five listed models on that hardest composition was 30.6 points. When all capabilities A1–A7 were required, top models often failed to exceed 50% accuracy and smaller ones fell below 40%.

• Perceptual skills scale, symbolic skills do not. GPT-4.1 exceeds 85% on A2 and A3, but even top-tier models average below 75% across A4–A6. Ovis2-8B drops to 36.6% on A5, and most open models perform below 55% on A6. InternVL3-38B outperforms GPT-4.1 on A3, and Qwen2.5-VL-72B gets the best A6 result overall, suggesting architecture and supervision matter more than raw parameter count for some skills.

• Scaling gains are uneven across levels. InternVL improves 3.5 points from 38B to 78B on L1, and Qwen2.5-VL improves nearly 9 points from 7B to 72B on L1, but L2 gains are unstable and L3 accuracy stagnates or declines even at large scale.

• Prompting barely helps; interleaved reasoning does. Under intervention on A5/A6/A7, vanilla scores were 61.2 / 58.7 / 49.1; CoT gave 62.0 / 56.3 / 50.8 (+1.7% on A7 on average, and sometimes harmed perception); SoM gave 62.4 / 60.9 / 48.6; CoT+SoM gave 61.8 / 59.3 / 50.1 (no consistent gain); CSFT with 500 curated QA pairs gave 63.5 / 60.2 / 49.5 but failed to fix A7; MM-CoT gave 65.3 / 61.7 / 53.5. MM-CoT delivered +4.4% for GPT-4o over vanilla and +2.1% on average for open models.

• Failure is cascade-shaped, not random. A case study shows a model answering a Level-1 perception query correctly and then failing the dependent Level-2 integration and Level-3 conflict-reasoning tasks, indicating that early grounding errors propagate upward into confident but wrong high-level inferences.

• Data quality was explicitly audited. A 15% manual audit of cross-modal validation yielded 98.2% accuracy; the final clue graphs averaged 12.7 verifiable clues each with a 97.3% semantic consistency rate. Difficulty labels came from model consensus (76% agreement across GPT-4o, GPT-4.1, and Gemini 2.5 Pro) with experts overriding model votes in 18% of cases; 40% of samples underwent adversarial validation with 12% flagged for revision; inter-annotator agreement on a stratified 10% subset was 92.6% with Cohen's κ = 0.88.

Methodology in Plain English

The authors started from real e-commerce listings — 22.8k of them, each with at least 5 verified attributes and high-resolution images. From these they built multimodal clue graphs: structured quadruples of (entity, modality, attribute, value), so that every fact about a product is attached to a specific source, whether image or text. Entities were extracted with an ensemble of YOLOv8-L, GroundingDINO, and visual embeddings for images and a fine-tuned Qwen3-8B for text; attributes came from rule-based templates plus GPT-4o augmentation; GPT-4o then flagged cross-modal inconsistencies for correction.

Questions were generated to match the taxonomy in three ways: rule-based templates (45+ of them) for the atomic L1 tasks, GPT-4o generation with human refinement for L2 integration tasks, and expert-authored items for L3 reasoning tasks that require multi-step inference about deceptive intent or rule violations. Contradictions were injected synthetically so that the ground truth is known.

Quality control ran a three-step loop of expert review, model filtering, and difficulty balancing. Evaluation used a unified zero-shot QA protocol — single-choice items scored by exact match, open-ended answers semantically judged by GPT-4o — with open-source models run on NVIDIA H100 GPUs using temperature 0.7, top_p 0.9, and max tokens 512. For the intervention experiments, the authors compared CoT prompting, SoM bounding-box visual guidance, their combination, supervised fine-tuning on 500 curated QA pairs, and their proposed MM-CoT, in which a model first produces a rationale, that rationale is parsed to highlight relevant image regions, and the model is re-invoked with the marked-up image plus its own prior reasoning trace.

Why This Matters

Most multimodal evaluation assumes the image and the text agree. This benchmark targets the opposite case — evidence that contradicts itself — which is exactly the condition under which deployed systems can be confidently wrong. The paper's central claim is that this failure is structural: it is traceable to specific capability gaps (cross-frame alignment, numerical plausibility, rule-based logic) rather than being solved by scale or generic prompting.

Impact on research: It reframes multimodal evaluation from "can the model answer questions about aligned inputs" to "can the model verify whether its inputs are jointly consistent," and provides a capability-level diagnostic that attributes failures to perception, integration, or reasoning.

Real-world applications:

• E-commerce listing verification — catching mismatches between a product photo and its title or description.
• Counterfeit and brand-abuse detection, including brand circumvention and deliberate logo occlusion (the BC, BC*, BI, PI, and BI* tasks).
• Price and value plausibility screening, such as flagging a luxury item listed at an implausibly low price.
• Content moderation and IP-policy compliance checking, where rule-based logic must be applied to a mix of visual and textual evidence.

Industry relevance: The benchmark comes from ByteDance with collaborators at Zhejiang University and the Institute of Automation, Chinese Academy of Sciences, and it is aimed squarely at production concerns — fraud, misrepresentation, and platform policy enforcement on marketplace content. Its finding that lightweight supervised fine-tuning does not repair A7 logic gaps, while interleaved grounding-plus-reasoning prompting does, gives engineering teams a concrete signal about where to invest.

Future Directions

• Build training methods that target composition, not perception. Since scale yields smooth gains on A1–A3 but not on A4–A7, the open question is what supervision or architecture would raise performance on symbolic coordination.

• Refine iterative grounding-and-reasoning loops. MM-CoT is presented as a promising direction rather than a solution; its +4.4% gain for GPT-4o over vanilla leaves substantial headroom before the human 95.2% average.

• Understand why models "prioritize internal priors over conflicting external signals," one of the three key trends the authors report, and whether that tendency can be corrected without degrading perceptual accuracy.

• Extend beyond the e-commerce domain. The dataset is built from product listings in 5 languages and 30+ categories; whether the same capability gaps appear in social posts, advertisements, or news imagery is untested here.

Target Audience

Researchers and engineers working on vision-language models, multimodal evaluation, and content-integrity systems will get the most from this paper. It is also useful for benchmark designers interested in hierarchical, capability-level diagnostics, and for applied teams in e-commerce, trust-and-safety, and brand protection who need to know whether current VLMs can be relied upon to spot contradictory evidence.

Authors’ abstract

Multimodal Large Language Models are primarily trained and evaluated on aligned image-text pairs, which leaves their ability to detect and resolve real-world inconsistencies largely unexplored. In open-domain applications visual and textual cues often conflict, requiring models to perform structured reasoning beyond surface-level alignment. We introduce CrossCheck-Bench, a diagnostic benchmark for evaluating contradiction detection in multimodal inputs. The benchmark adopts a hierarchical task framework covering three levels of reasoning complexity and defines seven atomic capabilities essential for resolving cross-modal inconsistencies. CrossCheck-Bench includes 15k question-answer pairs sourced from real-world artifacts with synthetically injected contradictions. The dataset is constructed through a multi-stage annotation pipeline involving more than 450 expert hours to ensure semantic validity and calibrated difficulty across perception, integration, and reasoning. We evaluate 13 state-of-the-art vision-language models and observe a consistent performance drop as tasks shift from perceptual matching to logical contradiction detection. Most models perform well on isolated entity recognition but fail when multiple clues must be synthesized for conflict reasoning. Capability-level analysis further reveals uneven skill acquisition, especially in tasks requiring multi-step inference or rule-based validation. Additional probing shows that conventional prompting strategies such as Chain-of-Thought and Set-of-Mark yield only marginal gains. By contrast, methods that interleave symbolic reasoning with grounded visual processing achieve more stable improvements. These results highlight a persistent bottleneck in multimodal reasoning and suggest new directions for building models capable of robust cross-modal verification.