Recognizing Dialogue Ambiguity via Candidate Set Visual Grounding

AI summary

Overview

Research area: Natural Language Processing, specifically visual grounding in dialogue — the problem of identifying which objects in an image an ambiguous user utterance might refer to as conversation develops.

Technical level: Intermediate. The paper assumes familiarity with bounding-box visual grounding, precision/recall evaluation, and zero-shot vision-language model (VLM) prompting, but its central idea is explained concretely.

Scope: The paper proposes a new task (Candidate Set Visual Grounding), builds a benchmark for it (CandVG), and evaluates 18 vision-language models across six families to test whether they can track how the set of plausible referents narrows across dialogue turns.

What This Paper Is About

When a user says something like "Can you pass me the drink?" while looking at a table with several drinks, no single object can be identified from the utterance alone. Prior work on interactive visual grounding assumes that once enough dialogue context accumulates, exactly one target object can be identified — and it evaluates models only at that final moment. This paper argues that what matters just as much is whether a system can correctly represent the set of still-plausible referents at every intermediate point, so it does not commit prematurely to the wrong object. The authors define this task, build a benchmark that scores candidate sets turn by turn, and test how well current VLMs do it.

Key Contributions

1. A new task formulation, Candidate Set Visual Grounding (CSVG). Given an image and the dialogue context available up to turn i, the goal is to predict the set of bounding boxes for all objects that remain plausible referents of the initial ambiguous utterance — not just one object. The paper distinguishes this from Context-Aware Visual Grounding (CVG), the standard single-object formulation.

2. CandVG, a benchmark for tracking candidate sets across turns. Built on an existing CVG dataset (InViG 21K), CandVG contains 1,249 dialogues and 4,833 evaluation samples, where each accumulated dialogue context is treated as a separate evaluation sample. It contains 2,549 CVG samples with a single object and 2,284 CSVG samples with multiple objects.

3. An evaluation protocol for set prediction. CVG is scored with Acc@0.5. CSVG uses the Hungarian algorithm for one-to-one matching between predicted and ground-truth boxes at IoU at least 0.5, then reports precision, recall, and F1.

4. A baseline study of 18 VLMs and a diagnosis of failure modes. The authors evaluate open-weight and proprietary models from six families (InternVL3.5, Qwen3 VL, Qwen3.5, GPT-5.6, plus the embodied-reasoning families RynnBrain and CosmosReason2) and identify candidate generation difficulty and premature commitment as the main sources of error.

Main Findings

• Scale helps CVG but not CSVG. On CandVG's CVG task, accuracy increases consistently with model scale within each open-weight generalist family, and Qwen3.5 27B achieves the highest accuracy among all evaluated models at 0.819. On the CSVG task, larger open-weight models do not consistently achieve higher F1.

• GPT-5.6 models lead on CSVG. Both GPT-5.6 Luna and GPT-5.6 Terra reach the highest CSVG F1 of 0.649 on CandVG. The best open-weight CSVG results come from the Qwen3 VL family (Qwen3 VL 32B at 0.543, Qwen3 VL 4B at 0.542).

• A persistent precision–recall gap. In CSVG, precision rises with model scale, but recall does not improve for larger models. Models identify precise subsets of the ground-truth candidate set but miss many plausible candidates.

• Candidate generation is one bottleneck. When ground-truth candidate sets are supplied as input (the "w/ GT" condition), every evaluated model improves its F1 over the "w/o GT" condition, with particularly large gains for Intern3.5 variants. Larger VLMs do not consistently score higher even with ground truth provided.

• Premature commitment is another bottleneck. Tracking the same 1,234 dialogues over the first three turns, Qwen3.5 27B predicts fewer candidates than ground truth from the first turn, Qwen3 VL 4B fails to shrink its predictions as context accumulates (and its prediction count increases from Turn 2 to Turn 3), and GPT-5.6 Luna narrows early while exceeding ground truth by Turn 3.

• Standard multi-object grounding is easier than CandVG. All models score higher F1 on gRefCOCO than on CandVG, and the precision–recall gap is consistently larger on CandVG. Among open-weight generalist VLMs, Qwen3 VL 4B shows the smallest gap between the two benchmarks.

• CVG on CandVG is harder than standard grounding benchmarks. All models score lower on CandVG's CVG setting than on RefCOCOg, and most also score lower than on Flickr30k Entities. Qwen3.5 27B obtains the highest score across all benchmarks in the CVG setting (0.819 on CandVG, 0.917 on RefCOCOg, 0.854 on F30k-Ent).

• Embodied reasoning post-training is not consistently beneficial. The 2B RynnBrain and CosmosReason2 models outperform Qwen3 VL 2B on both CVG and CSVG. At 8B, the pattern reverses in different directions: RynnBrain is better on CVG but worse on CSVG relative to Qwen3 VL 8B, while CosmosReason2 shows the reverse.

• Task-aware instructions partially help large models. Adding CSVG-specific instructions improves recall for all 27B+ models, with the stepwise instruction TAI-1 giving larger gains (for example, Qwen3 VL 27B+ recall +0.097 and Qwen3.5 27B+ recall +0.087). Effects on 4B models are inconsistent — recall improves only for Qwen3 VL under TAI-1 — and precision gains do not consistently follow.

• Extra candidates remain a separate problem. In CVG, Qwen3 VL 4B predicts multiple bounding boxes for 74.2% of samples, and GPT-5.6 models predict multiple boxes more frequently than Qwen3.5 27B, which may partly explain their lower CVG accuracy.

• Annotation reliability is high. Two annotator pairs each labeled 180 dialogues (360 total). Pair 1 achieved Acc 0.962, P 0.959, R 0.944, F1 0.943; Pair 2 achieved Acc 0.971, P 0.966, R 0.924, F1 0.935.

Methodology in Plain English

The authors start from InViG 21K, an existing dataset of natural human–human dialogues about images with bounding-box annotations. They preprocess its validation and test splits (2,998 dialogues), filtering out dialogues that lack initial ambiguity, have too many boxes, or have extreme size disparities, leaving 2,171 dialogues. They then sample 1,620 dialogues to balance candidate-set sizes, and manually screen out dialogues with annotation errors or no ambiguity in the first utterance, retaining 1,317 dialogues.

Three annotators each take an equal share of the dialogues. Working through each dialogue context turn by turn, they place points on every object they consider a plausible referent of the first utterance, adding or removing points as context narrows the field. Points are then mapped to the bounding boxes supplied by InViG; only dialogues with a complete one-to-one correspondence between points and unique boxes at every turn are kept, leaving 1,249 dialogues.

For evaluation, the authors feed models an image plus dialogue context and ask for bounding-box coordinates, capping output at 15 boxes to match CandVG's maximum candidate set size. All models share the same base prompt and differ only in the bounding-box output format. Models are evaluated zero-shot — no task-specific fine-tuning. To separate the two halves of the problem, one condition provides ground-truth candidate boxes so the model only has to select among them (w/ GT), while the other requires the model to generate candidates itself (w/o GT). The authors also run the same models on RefCOCOg and Flickr30k Entities (single-object grounding) and gRefCOCO (multi-object grounding) to compare against standard abilities. Open-weight models use greedy decoding with a maximum generation length of 512 tokens; GPT-5.6 uses a 2,048-token limit and medium reasoning effort. Experiments ran on four NVIDIA RTX 3090 GPUs, with the longest single-model inference run taking about six hours.

Why This Matters

Impact on research. The paper reframes dialogue ambiguity from a problem of picking the right object to a problem of correctly maintaining a set of objects over time. This gives the field a measurable target that sits between single-object grounding and full ambiguity resolution, and it provides evidence that better standard visual grounding does not automatically translate into better ambiguity tracking. It also introduces a diagnostic distinction — candidate generation versus candidate selection — that future work can address separately.

Real-world applications:

• Assistive and service robots that need to interpret underspecified requests such as "pass me the drink" and decide whether to act or ask a clarifying question.
• Human–robot collaboration in shared physical spaces, where a system must avoid acting on the wrong object when several remain plausible.
• Vision–language navigation, where an instruction may remain ambiguous until the agent has moved or gathered more context.
• Visual question answering systems that need to recognize when a question has no unique referent and respond accordingly rather than guessing.

Industry relevance. The work is directly relevant to companies building multimodal assistants and embodied AI systems, where premature commitment to a single interpretation causes real errors. The paper's funding acknowledgments reference JST PRESTO (Grant Number JPMJPR24TC) and FRONTia, a Japanese national program led by METI and NEDO aimed at developing a domestic multimodal foundation model for AI robots and physical AI — indicating explicit industrial and governmental interest in this capability. The benchmark and annotations are released under CC BY 4.0.

Future Directions

1. More diverse ambiguity dynamics. The authors note that CandVG is built from an existing dataset where candidate sets mostly shrink gradually toward a single object. Dialogues with sudden shifts that add candidates are underrepresented, and only 21 dialogues (1.68%) retain multiple plausible candidates at the final turn.

2. Moving beyond static boxes. The current setting uses static images and bounding boxes, which cannot capture environmental changes, viewpoint-dependent references, non-verbal cues, or referents such as object parts and regions.

3. Better methods for compact models. Task-aware instructions improved recall only for large-scale VLMs and were inconsistent at 4B. Since compact models are preferable for interactive deployment because of lower computational and latency demands, the paper calls for training and inference methods designed specifically to improve candidate set estimation in smaller models.

4. Broader model coverage and improvement strategies. Open-weight evaluation was capped at 38B parameters because of computational resources, and major proprietary families including Gemini were excluded, so the scaling trends may not generalize. The paper also notes that only inference-time prompting was tested, leaving strategies such as self-reflection unexplored.

Target Audience

This paper is most useful to researchers working on visual grounding, multimodal dialogue systems, and embodied AI who need a concrete way to evaluate whether a model recognizes unresolved ambiguity. It is also relevant to practitioners building robots or assistants that must decide between acting and asking clarifying questions, and to anyone evaluating vision-language models on tasks where a single correct answer is not the right framing. Readers should be comfortable with standard grounding metrics (IoU, Acc@0.5, precision/recall/F1) and with how VLMs are prompted for bounding-box output.

Authors’ abstract

Interpreting an ambiguous utterance in real-world interactions often requires context beyond a single utterance. When the utterance refers to objects in a visual scene, it is necessary to maintain and update the set of plausible referents as context accumulates, without assuming that the available information identifies a unique object. However, prior work on interactive visual grounding, which localizes objects referred to by linguistic expressions, has focused on identifying a unique object once sufficient dialogue context becomes available. To address this gap, we propose Candidate Set Visual Grounding (CSVG), a task for estimating the set of objects consistent with the given dialogue context. We further introduce CandVG, a benchmark that evaluates how the candidate set changes across dialogue turns. Our evaluation of vision-language models shows that stronger visual grounding capability does not necessarily yield better CSVG performance. In particular, models have difficulty predicting the complete candidate set, owing to challenges in candidate generation and premature commitment to a small set of candidates in early turns. Nevertheless, we find that this limitation can be partially mitigated in the evaluated large-scale models using instructions designed for CSVG.