Reference Games as a Testbed for the Alignment of Model Uncertainty and Clarification Requests

AI summary

Overview

• Research area: Natural Language Processing, specifically dialogue pragmatics, repair and clarification behavior, uncertainty quantification, and vision-language model evaluation.
• Technical level: Intermediate. The experimental design is straightforward, but the paper assumes familiarity with reference games, uncertainty quantification terminology, and vision-language model evaluation practice.
• Scope: The paper evaluates three vision-language models as listeners in a controlled color-grid reference game to test whether their internal uncertainty aligns with the clarification requests they produce.

What This Paper Is About

In human conversation, a listener who is unsure what a speaker means can ask a clarification question ("The light blue one?") rather than guessing. It is an open question whether language models can play this same listener role, recognizing their own uncertainty and expressing it through clarification. The authors argue that reference games are an ideal testbed for this question because they are controlled, self-contained, and make clarification needs explicit and measurable, and they use such a game to test three vision-language models.

Key Contributions

1. Proposing reference games as a testbed for clarification behavior. The authors argue that reference games, unlike open-ended dialogue settings, offer fixed alternatives and no need for external knowledge, so it is immediately clear when a description fails to single out the target and clarification is warranted.
2. A three-condition evaluation of clarification behavior in vision-language models. They compare a baseline reference resolution task against an experiment in which models are explicitly instructed to ask a clarification question when uncertain, across three difficulty conditions (far, split, close), for Qwen2.5-VL-7B, Qwen2.5-VL-72B, and GPT-5-mini.
3. An analysis linking clarification requests to measured model uncertainty. Using diversity sampling (five samples per round, majority vote), they derive a consistency-based confidence estimate and compare the baseline confidence of items that did versus did not elicit clarification requests.
4. A human-in-the-loop interaction experiment. They manually annotate 116 Qwen2.5-VL-72B clarification questions as task-relevant or not, have a human answer or reformulate in response, and re-measure accuracy and confidence to test whether clarification actually helps.

Main Findings

• Human performance sets the ceiling. Human accuracy on the color-grid reference game is 0.92 across all conditions on the full data (0.94 on the subset), rising from 0.91 (close) to 0.93 (split) to 0.97 (far) on the subset.
• Clarification rates diverge sharply across models. GPT-5-mini generated clarification questions in 13% of items and Qwen2.5-VL-72B in 24% (both subset and full dataset), while Qwen2.5-VL-7B almost never did (less than 0.1%).
• Only one model tracks its own uncertainty. GPT-5-mini's clarification rate rose systematically with difficulty, from 6% (far) to 17% (split and close). Qwen2.5-VL-72B's rates did not systematically track task difficulty.
• Baseline accuracy falls with difficulty, confidence stays high. Accuracy and confidence decreased from far to split to close for all models, mirroring human performance and previous findings. GPT-5-mini reached the highest baseline accuracy of 91% with 99% confidence; Qwen2.5-VL-72B reached 77% accuracy with 91% confidence (71% accuracy and 92% confidence reported for the full dataset); Qwen2.5-VL-7B reached 53% accuracy with 88% confidence (52% and 87% on the full dataset). The authors read the uniformly high confidence as a disposition toward overconfidence, particularly for the Qwen models.
• Clarification helps GPT-5-mini's accuracy but not the Qwen models'. GPT-5-mini's accuracy on non-clarification responses was 94% versus 91% baseline, matching human performance. Qwen2.5-VL-72B's accuracy on generated predictions was slightly lower on the subset (73% vs. 77% baseline) with no difference on the full dataset (71%). Under the relaxed accuracy measure, GPT-5-mini stayed at 94% while Qwen2.5-VL-72B improved to 80% (78% on the full dataset).
• Confidence barely drops on items that trigger clarification. Qwen2.5-VL-72B's mean confidence fell only from 91% to 87% for items where it generated a clarification request, which the authors identify as overconfidence.
• Clarification requests often target genuinely difficult items, but not consistently. For GPT-5-mini, items that elicited clarification requests had substantially lower accuracy than baseline (71% vs. 91%). For Qwen2.5-VL-72B this pattern was unstable: accuracy on those items was moderately higher than baseline on the subset (78% vs. 77%) but reversed on the full dataset (63% vs. 71%). For Qwen2.5-VL-7B, which generated almost no clarification requests, meaningful comparison was not possible, though on the few full-dataset items where it did, accuracy was much lower than baseline (20% vs. 52%).
• Most clarification questions are not useful. Only 42% of Qwen2.5-VL-72B's clarification questions were annotated as task-relevant. Providing human answers to these requests did not improve performance: on the 116-item clarification subset, accuracy fell from 0.776 to 0.741 (Δ −0.035) while confidence rose from 0.871 to 0.902 (Δ +0.031); over all items, accuracy fell from 0.767 to 0.759 (Δ −0.008) while confidence rose from 0.914 to 0.921 (Δ +0.007). The close condition was the exception, improving by +0.081 on the subset and +0.019 on the full dataset.
• Information-based uncertainty estimates do not fix the problem. Recomputing Qwen2.5-VL-72B confidence with maximum softmax probability (MSP) produced lower and more graded values than consistency-based confidence (0.84 versus the consistency value), but the model remained overconfident and its clarification behavior did not better align with uncertainty.

Methodology in Plain English

The researchers used the color-grid reference game dataset, which contains 197 games of 60 rounds each. In each round a speaker sees three 3×3 color grids (one target, two distractors) and describes the target; the listener must identify which grid is meant. The dataset has three difficulty conditions based on how similar the target is to the distractors: far (easiest), split (medium), and close (hardest). Only the initial speaker description was used, so the models always acted at the first turn. The three models were Qwen2.5-VL-7B-Instruct and quantized Qwen2.5-VL-72B-Instruct (both at temperature 0.7) and GPT-5-mini (no temperature control). The two Qwen models ran on the full dataset; GPT-5-mini was limited to a 500-round subset because of API costs, and 19 items were dropped because it produced null answers.

In the baseline experiment, models were shown a concatenated image of the three grids plus the speaker's utterance and had to answer first, second, or third. Each model was sampled five times per round and the majority vote was used as the predicted answer, giving a consistency-based confidence value equal to the proportion of samples matching that majority, yielding discrete levels of 0.4, 0.6, 0.8, or 1.0. In the clarification experiment, the prompt was changed to instruct the model to ask a clarifying question starting with "QUESTION:" when uncertain. Each model was sampled once, and the authors measured the clarification request rate, the accuracy of non-clarification responses, and a relaxed accuracy counting either a correct answer or a clarification request as acceptable. In the interaction experiment, a human (one of the authors) inspected all 116 clarification questions from Qwen2.5-VL-72B, labeled each task-relevant or not, and responded: answering directly when the question was relevant, or reformulating the original utterance when it was not. The model was then re-run on those clarified dialogues to recompute accuracy and confidence.

Why This Matters

• Impact on research. The paper positions reference games as a reusable, controlled testbed not just for reference resolution but for the interactional qualities of vision-language models, connecting the uncertainty quantification literature to the study of clarification and repair. It also adds evidence that high linguistic confidence in models does not reflect reliable internal calibration, and that clarification behavior can be decoupled from actual uncertainty even in a task where uncertainty is quantifiable.
• Real-world applications:
  • Conversational assistants and chatbots that need to ask the user a question instead of confidently guessing when a request is ambiguous.
  • Multimodal and vision-grounded interfaces, where a user's instruction may refer to one of several similar objects on screen.
  • Human-robot interaction and instruction-following, where an incorrect guess is costly and a well-targeted clarification is cheap.
  • Evaluation and benchmarking of dialogue systems, using reference games as a measurable proxy for pragmatic competence.
• Industry relevance. The finding that a commercial model (GPT-5-mini) aligns clarifications with uncertainty better than the open-weight Qwen models, but still only asks in 13% of items and still misses cases, is directly relevant to teams deploying models in interactive products where silent failure is dangerous. It also argues for testing clarification behavior with human-in-the-loop evaluation rather than accuracy alone.

Future Directions

• Testing whether iterative grounding, such as letting models interact over many rounds and build common ground and conceptual pacts as human dyads do, changes clarification behavior.
• Developing or applying better-calibrated uncertainty quantification for commercial models, since consistency-based sampling was the only signal available uniformly across the evaluated models and information-based estimates were only explored for the open-weight Qwen2.5-VL-72B.
• Improving the quality of generated clarification questions, given that only 42% of Qwen2.5-VL-72B's questions were task-relevant and human answers to them did not improve end-to-end accuracy.
• Extending the testbed to other ambiguity types, other datasets, and larger model sets, and separating listener errors from flawed speaker descriptions in the underlying data.

Target Audience

Researchers and graduate students in computational linguistics, dialogue systems, and multimodal NLP who study grounding, repair, and clarification; practitioners building interactive vision-language applications who need to know when a model should ask rather than answer; and evaluation researchers looking for controlled, measurable benchmarks of pragmatic and interactional model behavior.

Authors’ abstract

In human conversation, both interlocutors play an active role in maintaining mutual understanding. When listeners are uncertain about what speakers mean, for example, they can request clarification. It is an open question for language models whether they can assume a similar listener role, recognizing and expressing their own uncertainty through clarification. We argue that reference games are a suitable testbed to approach this question as they are controlled, self-contained, and make clarification needs explicit and measurable. To test this, we evaluate three vision-language models comparing a baseline reference resolution task to an experiment where the models are instructed to request clarification when uncertain. The results suggest that even in such simple tasks, models often struggle to recognize internal uncertainty and translate it into adequate clarification behavior. This demonstrates the value of reference games as testbeds for interaction qualities of (vision and) language models.