Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks

AI summary

Overview

• Research area: Computer Vision / multimodal large language models (MLLMs), specifically Referring Expression Comprehension (REC) — grounding a natural-language expression to a specific region in an image.
• Technical level: Intermediate. The paper is readable without deep REC background, but familiarity with grounding metrics (IoU, Acc@0.5) and MLLM evaluation helps.
• Scope: The paper builds and releases Ref-Adv, a modern REC benchmark designed to require genuine textual and visual reasoning rather than "grounding shortcuts," and benchmarks 13 contemporary MLLMs on it.

What This Paper Is About

Classic REC benchmarks (RefCOCO, RefCOCO+, RefCOCOg) are close to saturated, with MLLMs reaching over 90% accuracy, yet the paper argues these benchmarks are weak tests of visual reasoning: expressions are often very short, images have few same-category distractors, and long redundant descriptions allow a model to succeed by matching only a subset of descriptors. The authors build Ref-Adv, a dataset that pairs linguistically non-trivial expressions with only the information needed to uniquely identify a target, surrounded by "hard distractors" that partially match the expression but do not satisfy it. The goal is to expose reliance on shortcuts and to measure real gaps in visual reasoning and grounding.

Key Contributions

1. A new REC benchmark, Ref-Adv, containing 5,000 instances over 2,833 images, with an average expression length of 11.5 words, an average of 4.01 same-category distractors per image, a 21.25% negation ratio, and a vocabulary of 5,308 — compared with average lengths of 3.6 (RefCOCO), 3.6 (RefCOCO+), and 8.4 (RefCOCOg).
2. A two-stage LLM-authored curation pipeline (with a human-authored subset) that first elicits group-level and intra-pair discriminators via GPT-4o, then composes minimally sufficient referring expressions, followed by a three-annotator verification protocol where a pair is kept only if all three annotators agree (18.7% keep rate for LLM-authored expressions).
3. Three diagnostic ablations — a model-bias "the one" prompt test, a bag-of-words word-order test, and a one-descriptor-deletion sufficiency test — showing Ref-Adv requires more textual and visual reasoning and contains fewer shortcuts than RefCOCO(+/g).
4. A broad evaluation of 13 contemporary MLLMs (closed and open source, including Qwen2.5-VL, Qwen3-VL, Qwen3.5, InternVL-3, Gemini 2.5-Flash, Gemini 2.5-Pro, GPT-4o, Claude-3.5 Sonnet, CogVLM-Grounding, and GLM-4.5V), plus the public release of Ref-Adv-s, a curated subset of 1,142 cases with evaluation code.

Main Findings

• Large performance drop versus classic benchmarks: Models that score highly on RefCOCO(+/g) fall markedly on Ref-Adv. Qwen2.5-VL-72B drops from 92.7% (RefCOCO), 88.9% (RefCOCO+), and 89.9% (RefCOCOg) to 58.3% on Ref-Adv; InternVL-3-14B drops from 92.0%, 87.6%, and 88.5% to 52.3%.
• Best result on Ref-Adv: With Set-of-Marks (SoM) and Chain-of-Thought (CoT), GPT-4o attains the best performance in the main results table at 63.7% Acc0.5, 38.4% Acc0.75, and 19.7% Acc0.9. On the Ref-Adv-s subset, Qwen3.5 397B-A17B-FP8 reaches 68.0% Acc0.5, 55.6% Acc0.75, and 34.2% Acc0.9.
• Less evaluatable through statistical bias: When the expression is replaced by the fixed prompt "the one," Qwen2.5-VL-72B scores 35.1% on RefCOCO (+13.7% vs Ref-Adv), 39.4% on RefCOCO+ (+18.0%), 38.3% on RefCOCOg (+16.9%), but only 21.4% on Ref-Adv; InternVL-3-14B shows the same pattern (35.9%, 38.0%, 38.2% versus 22.8%).
• Word order matters more on Ref-Adv: Under a bag-of-words prompt, Qwen2.5-VL-72B falls by 9.9% (RefCOCO), 10.7% (RefCOCO+), 14.6% (RefCOCOg), and 16.8% (Ref-Adv, 58.3% to 41.5%); InternVL-3-14B falls by 7.3%, 6.6%, 13.6%, and 13.7% (52.3% to 38.6%).
• Fewer grounding shortcuts: Deleting one descriptor costs Qwen2.5-VL-72B 4.7% (RefCOCO), 5.3% (RefCOCO+), 4.6% (RefCOCOg), and 6.4% (Ref-Adv, 58.3% to 51.9%); for InternVL-3-14B the drops are 4.9%, 5.2%, 4.7%, and 7.1% (52.3% to 45.2%).
• More distractors makes the task harder: Accuracy drops more in the ≥7 distractor group than in the 4–6 group (e.g., Qwen2.5-VL-72B with CoT: −0.2 in the 4–6 group and −2.7 in the ≥7 group). On Ref-Adv-s, accuracy degrades as distractor count increases across the evaluated model families.
• CoT helps on Ref-Adv but not on classic REC: CoT generally improves performance on Ref-Adv, whereas on RefCOCO(+/g) grounding often succeeds without extensive reasoning, so CoT can add verbosity or error. The paper notes that standard toolkits such as open compass and VLMEvalKit do not enable CoT for RefCOCO(+/g), consistent with this finding.
• Thinking variants outperform instruct counterparts: On Ref-Adv-s, thinking-mode variants substantially outperform instruct counterparts at the same model size (for example, Qwen3-VL 8B-Thinking at 59.5% Acc0.5 versus 8B-Instruct at 47.2%, and Qwen3-VL 32B-Thinking at 65.6% versus 32B-Instruct at 53.4%).
• Failure mode: In qualitative analysis, models with explicit reasoning often follow the intended chain but fail partway due to incorrect visual perception or misunderstanding of the expression, and frequently select the hard distractor as the answer.
• Long-tailed categories: The appendix reports that Ref-Adv follows a more long-tailed category distribution than RefCOCO, RefCOCO+, and RefCOCOg.

Methodology in Plain English

The authors sampled images from the validation and test splits of COCO and OpenImages v7, keeping only images with panoptic instance annotations. They filtered for images containing at least 3 candidate instances of the same category as the target, so that category alone cannot solve the task. Each candidate instance was given a number tag (similar to Set-of-Marks).

Because single-step prompting of GPT-4o produced overspecified descriptions with redundant descriptors, they used two stages: first, the LLM identifies the most similar pair of same-category objects (group A, containing the target and a hard distractor) versus the rest (group B) and proposes group-level and intra-pair discriminators; second, the LLM composes referring expressions from a minimal yet sufficient subset of those discriminators, either describing the target directly or negating the hard distractor's descriptors, while being instructed not to mention number tags. A subset of expressions was written by human annotators. Three annotators then verified every image–text pair, checking correctness, unambiguity, and the presence of hard distractors; a pair was kept only with unanimous agreement, yielding an 18.7% keep rate for LLM-authored expressions.

Evaluation uses Acc0.5, Acc0.75, Acc0.9, and mean accuracy (mAcc), where a prediction counts as correct when the IoU with the ground-truth box exceeds the threshold, and CoT is tested both on and off. GPT-4o and Claude-3.5 Sonnet were evaluated with Set-of-Marks using Semantic-SAM because of their limited native grounding ability; model-specific best-practice prompts were used for fairness.

Why This Matters

Impact on research: The paper argues that near-saturated scores on RefCOCO(+/g) overstate MLLM visual reasoning ability, and that evaluation should stress both understanding the full expression and accurate visual perception. Ref-Adv, together with the released Ref-Adv-s subset of 1,142 cases and evaluation code, is intended to guide future work on visual reasoning and grounding in MLLMs. The three ablations also offer a reusable template for auditing whether a benchmark actually requires reasoning.

Real-world applications (the paper states REC has important applications in real-world systems and downstream tasks but does not enumerate them):

• Language-driven image editing and region-level interaction, where a user refers to a specific object in a scene.
• Assistive and interface systems that require selecting one specific object among many similar ones from a spoken or typed description.
• Robotic or embodied perception pipelines that must resolve a reference such as "the one on the left" against visually similar candidates.
• Dataset and model auditing, where the bias, word-order, and descriptor-deletion tests can be adapted to check whether a model is genuinely following the instruction.

Industry relevance: The paper notes that MLLMs dominate the field and that proprietary systems (GPT-4o, Gemini, Claude-3.5 Sonnet) as well as open-source families (Qwen2.5-VL, InternVL-3) are all evaluated here. The consistently lower scores on Ref-Adv indicate that products built on grounding — from visual search to grounded assistants — may perform worse in scenes with many similar objects or with negated descriptions than headline classic-benchmark numbers suggest. The paper explicitly states the benchmark is not intended for surveillance or biometric identification.

Future Directions

• Closing the distractor gap: The paper flags handling larger numbers of similar distractors as a key open challenge, since accuracy falls most in the ≥7 distractor group.
• Improving perception, not just reasoning: Qualitative analysis shows failures often come from incorrect visual perception or misreading the expression, not from a broken reasoning chain, so better grounded perception is needed.
• Reconciling CoT behavior across benchmarks: The paper reports CoT gains on Ref-Adv but limited benefit on RefCOCO(+/g) with off-the-shelf checkpoints and no extra training, leaving open how reasoning training should be combined with REC evaluation.
• Extending the benchmark's diagnostics: The authors position Ref-Adv as a starting point, and the released Ref-Adv-s subset with evaluation code invites broader and reproducible follow-up evaluation (including model families spanning 2B to 397B parameters).

Target Audience

Researchers and engineers working on multimodal large language models, visual grounding, and referring expression comprehension; benchmark and evaluation designers interested in detecting shortcut learning; and practitioners who deploy language-to-region grounding in products and need realistic expectations of performance in cluttered or negation-heavy scenes. Graduate students entering the REC and MLLM evaluation area will find the ablations and dataset statistics a useful reference point.

Authors’ abstract

Referring Expression Comprehension (REC) links language to region level visual perception. Standard benchmarks (RefCOCO, RefCOCO+, RefCOCOg) have progressed rapidly with multimodal LLMs but remain weak tests of visual reasoning and grounding: (i) many expressions are very short, leaving little reasoning demand; (ii) images often contain few distractors, making the target easy to find; and (iii) redundant descriptors enable shortcut solutions that bypass genuine text understanding and visual reasoning. We introduce Ref-Adv, a modern REC benchmark that suppresses shortcuts by pairing linguistically nontrivial expressions with only the information necessary to uniquely identify the target. The dataset contains referring expressions on real images, curated with hard distractors and annotated with reasoning facets including negation. We conduct comprehensive ablations (word order perturbations and descriptor deletion sufficiency) to show that solving Ref-Adv requires reasoning beyond simple cues, and we evaluate a broad suite of contemporary multimodal LLMs on Ref-Adv. Despite strong results on RefCOCO, RefCOCO+, and RefCOCOg, models drop markedly on Ref-Adv, revealing reliance on shortcuts and gaps in visual reasoning and grounding. We provide an in depth failure analysis and aim for Ref-Adv to guide future work on visual reasoning and grounding in MLLMs.