VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification

AI summary

Overview

Research area: Computer vision and multimodal large language model (MLLM) evaluation, specifically comparative visual perception across image pairs.

Technical level: Intermediate. The paper is readable without deep vision-model expertise, but it assumes familiarity with benchmark design, multiple-choice evaluation, and the distinction between semantic and low-level image changes.

Scope: The paper introduces and validates VDiff-Bench, a 1,756-question multiple-choice benchmark for fine-grained Image Difference Identification (IDI) across 10 change categories, and uses it to evaluate 11 contemporary MLLMs.

What This Paper Is About

MLLMs perform well on single-image tasks like visual question answering, but the authors find they often fail at a basic comparative skill: saying what changed between two similar images. Existing visual-difference benchmarks are limited either because their data is too easy (synthetic scenes, narrow change types) or because they score free-form captions with metrics that are sensitive to paraphrasing or dependent on unreliable LLM judges. VDiff-Bench addresses both gaps by pairing a real difference with two human-verified hard-negative descriptions and a "no difference" distractor, scored by exact answer-key matching.

Key Contributions

1. A new benchmark of 1,756 four-choice questions over 1,543 distinct image pairs, spanning 10 change categories: position (158 questions), motion (138), regional color (170), whole-image color (147), appearance/disappearance (201), noise/resolution (150), texture (177), substitution/size (199), OCR/text (221), and illumination (195).
2. A reformulation from Image Difference Captioning (IDC) to a multiple-choice IDI task, where each item contains the ground-truth difference, two hard-negative alternatives, and one fixed "no difference" option, enabling deterministic scoring without an LLM judge.
3. Ground-truth-conditioned hard-negative construction: false differences are generated with reference to the real changes using strategies such as swapping the target or source object, changing an attribute, or introducing a plausible alternative change, then filtered and rewritten by human experts.
4. A systematic evaluation of 11 MLLMs with category-level, pair-level (pair macro), group-level (semantic / low-level / OCR), and response-level (no-difference, alternative, invalid) analyses.

Main Findings

• No model dominates the taxonomy. Overall accuracy ranges from 35.8% to 89.6%, against a 25.0% uniform-guess baseline and a 33.3% informed-guess baseline. Gemini 3.1 Pro ranks first at 89.6% (pair-clustered 95% CI: 87.9–91.2), followed by Kimi K2.5 at 89.2% (87.4–91.0), Gemini 3.5 Flash at 88.8% (87.2–90.4), and Doubao Seed 1.6 Vision at 87.7% (85.7–89.7). Best category scores are distributed across Gemini, Kimi, and Doubao models.

• A large semantic-versus-low-level gap in smaller open models. Three 7–8B open-weight models score 52.5–70.6% on semantic changes but only 8.7–33.3% on low-level changes. In Table 4, LLaVA-OneVision-Qwen2-7B shows 52.5% semantic and 8.7% low-level; InternVL3.5-8B shows 69.7% and 16.0%; Qwen3-VL-8B Thinking shows 70.6% and 33.3%.

• Low-level failures are mostly missed changes, not confused descriptions. The abstract and introduction report that these models select the "no difference" distractor on 51.3–80.9% of low-level questions; Section 4.3 states that for the four models with the largest semantic–low-level gaps, 48.1–80.9% of low-level questions are answered with the no-difference option, substantially exceeding their selection of alternative descriptions.

• Scale helps but is not sufficient. Kimi K2.5 and Kimi K3 reach 88.8% and 82.8% low-level accuracy respectively, whereas the closed-source Grok 4.3 falls from 82.0% on semantic changes to 40.7% on low-level changes, scoring only 5.3% on noise/resolution and 15.3% on texture.

• Gaps of 37.2–53.8 percentage points between semantic and low-level performance are reported for Grok and the three 7–8B models.

• Appearance/disappearance is difficult even for the strongest systems. Its category maximum is 79.6%, compared with at least 90.8% in seven other categories.

• Text-only cue baseline. A text-only style baseline that uses only capitalization and punctuation cues without viewing images scores 57.6% overall, above the 25.0% and 33.3% chance baselines.

• Error decomposition varies by model. In Table 4, no-difference error rates range from 0.5% (Gemini 3.5 Flash) to 43.1% (LLaVA-OneVision-Qwen2-7B); alternative-selection rates range from 4.7% (Kimi K3) to 21.1% (LLaVA); invalid-response rates are 0.0% for most models, 5.2% for Kimi K3, 3.4% for Gemini 2.5 Flash, and 2.9% for Gemini 3.5 Flash.

• Grok 4.3 is a specific outlier on low-level perception. It reaches 96.4% on motion but only 5.3% on noise/resolution and 15.3% on texture, falling behind large open-source models such as Kimi K2.5 and K3.

Methodology in Plain English

The benchmark was built in three stages. First, image pairs came from several sources: existing annotated pairs (Spot-the-Diff, MotionEdit, OmniEdit, OmniDiff), 196 "find-the-difference" puzzle pairs collected from online sources, and unpaired images augmented into pairs. Second, the authors created harder negative answer choices by prompting strong MLLMs (Gemini 2.5 Pro and GPT-5.5) with the image pair and its true difference list, asking for statements that are factually wrong but visually plausible; human experts then inspected, rewrote, or discarded weak ones. Third, each question was assembled from the ground-truth difference, two false differences, and a fixed "no difference" option, then shuffled with a fixed random seed so correct-answer and distractor positions are approximately balanced across A–D.

To fill under-represented categories, the authors applied programmatic transformations to 150 FFHQ facial images: additive Gaussian noise sampled from a zero-mean distribution with standard deviation 15 in 8-bit pixel space (150 questions); skin- and edge-aware smoothing using a Gaussian kernel whose standard deviation is sampled from [2.4, 3.6] with residual detail mixed back at a random scale in [0.18, 0.36] (150 questions); global RGB shifts isolating chromatic change, scaled by 0.55 and capped at 12 pixel values (147 questions retained after excluding three flagged images); and gamma-plus-linear illumination changes using the first 50 sorted images from the Country211 test split, with brighter examples using γ ∼ U(0.70, 0.85) and b ∼ {5, …, 15}, and darker examples using γ ∼ U(1.15, 1.30) and b ∼ {−15, …, −5}. OCR/text and object-position categories were augmented with frontier image-editing models (Gemini-3-Pro-Image and GPT-image-2, with GPT-5.4-mini proposing position edits), and ground truth was derived directly from the editing prompts. Human annotators, cross-validated by a second expert, supplied descriptions for puzzle pairs without detailed source annotations.

Evaluation used choice accuracy as the main metric, with stratified breakdowns by category, semantic versus low-level group, OCR group, unique image pairs (pair macro), and error type. All models received both images plus four labeled options and the instruction to return one label only; open-source models were run at temperature 0.

Why This Matters

Impact on research: The paper argues that fine-grained visual difference identification is not an automatic byproduct of multimodal scaling but a distinct capability that must be explicitly developed and evaluated. It provides a judge-free, deterministic alternative to benchmark designs that rely on BLEU-4, ROUGE-l, or MLLM judges, both of which prior work has flagged as brittle or biased.

Real-world applications:

• Image and video editing pipelines, where an automatic evaluator must confirm the requested edit happened and no unintended changes (such as background alterations) occurred.
• Spot-the-difference and educational content, the authors' stated inspiration: puzzle games and children's playbooks containing many small, deliberate differences.
• Content moderation and media forensics, where detecting subtle tampering such as texture smoothing or noise inconsistency matters.
• Preservation checks in generative editing, motivated in the paper by user-identified failures of editing models to preserve human skin texture.

Industry relevance: The findings are directly relevant to builders of MLLMs and generative image-editing systems, since the paper positions a strong IDI model as a more grounded reward or verification signal for editing pipelines. The Grok 4.3 result specifically suggests that commercial scale and strong headline benchmark scores do not guarantee low-level comparative perception.

Future Directions

• Training objectives and data for low-level comparison. The paper proposes a two-factor account in which scale raises the ceiling while training data, objectives, and visual encoding determine whether capacity translates into precise cross-image comparison; testing that account is an open question.
• Closing the appearance/disappearance gap. This category tops out at 79.6% even for the strongest systems, well below the 90.8% or higher achieved in seven other categories.
• Extending beyond static image pairs. The benchmark covers 2D puzzle, real, edited, and rendered images; temporal or video difference identification is not addressed.
• Improving distractor construction. The paper notes that its model-generated false differences can be ungrammatical, unevenly distributed, or partially true, requiring human rewriting; whether generation and auditing can be made more reliable is unresolved.
• Reconciling the scale-versus-capability inconsistency, since bigger models did not uniformly perform better on the low-level subsets.

Target Audience

Researchers and engineers working on multimodal LLM evaluation, visual comparison, and image-editing evaluation; benchmark designers interested in multiple-choice formulations with hard negatives and judge-free scoring; and teams building generative image or video editing systems that need reliable automatic verification of what changed between two images.

Authors’ abstract

Multimodal Large Language Models (MLLMs) perform strongly on general visual understanding tasks such as visual question answering, yet they often struggle with a basic comparative skill: identifying what has changed between two similar images. We introduce VDiff-Bench, a challenging multiple-choice benchmark for fine-grained Image Difference Identification. VDiff-Bench contains 1,756 four-way questions over image pairs and covers 10 change categories: position, motion, regional image color, overall image color, appearance/disappearance, noise/resolution, texture, substitution/size, OCR/text, and illumination. Each question corresponds to two image inputs with 4 choices: the true difference, two hard negative descriptions, and a "no difference" distractor. To make the task challenging, we specifically curate ground-truth-conditioned negatives that require models to distinguish the actual change from nearby semantic alternatives. Experiments with 11 state-of-the-art open- and closed-source MLLMs show that fine-grained visual comparison remains brittle: models exhibit uneven performance across sources and change categories, with persistent failures on subtle low-level changes like noises and textures. For instance, three 7-8B-scale open-source MLLMs score 52.5-70.6% on semantic changes but only 8.7-33.3% on low-level changes like noise and texture, falsely assuming no changes between two image inputs. Surprisingly, despite strong performance of other closed-source commercial models, Grok 4.3 demonstrate remarkable performance drop on identifying noise and texture differences between images, falling significantly behind large open-source models like Kimi K2.5 and K3. Overall, VDiff-Bench provides a targeted diagnostic for evaluating comparative visual understanding in MLLMs, exposing failures that are not captured by standard single-image vision-language tasks.