Context informs pragmatic interpretation in vision-language models

AI summary

Overview

• Research area: Computational pragmatics and the cognitive science of language models, specifically multi-turn reference resolution in vision–language models (VLMs) evaluated against human behaviour.
• Technical level: Intermediate.
• Scope: The paper tests four open-weights VLMs on human iterated reference game transcripts under eight context conditions that vary the amount, order, and relevance of prior conversation, comparing model accuracy to naïve human matchers.

What This Paper Is About

Understanding language in conversation requires more than processing a single utterance: a listener must use the prior conversational history to figure out what a speaker means by an ambiguous or novel description. This paper asks whether state-of-the-art open vision–language models do this kind of context-sensitive pragmatic reasoning, using iterated reference games with abstract tangram images as a minimal test case. The goal is to characterise not just how accurate models are, but how their sensitivity to the amount, order, and relevance of context compares with that of humans.

Key Contributions

1. A systematic set of eight context conditions applied to the same iterated reference game data, manipulating whether in-context trials come from the same original game as the test trial, whether they all come from one game, their order (original, permuted, reversed, or none), and whether the test trial's tangram has been seen before (yoked: 720 trials; shuffled: 7056; backward: 720; ablated: 720; other-within: 6480; other-across: 6480; random: 7704; no context: 720).
2. Evaluation of four instruction-tuned open-weights VLMs of different sizes (Qwen 2.5 VL 32B, Gemma 3 27B, Llama 3.2 11B, Kimi VL A3B) on a paradigm previously used with humans, using a shared system prompt and a single image showing all 12 tangram shapes labelled A through L.
3. New human comparison data: the authors collected data on Prolific for the backward (N = 89) and random (N = 107) conditions using the same procedure as the previously collected yoked (N = 99) and shuffled (N = 97) conditions.
4. Diagnostic analyses beyond overall accuracy: trial-wise model–human correlations, item-wise variation, analysis of learning from previous repetitions, confusion-matrix correlations across conditions, a lemma-level error analysis, and a text-only control condition with no tangram images.

Main Findings

• Above chance but not human-like without context: All models performed above the chance level of 0.083 across all conditions, but in the no context condition they were only slightly above chance. Naïve humans were relatively high even in the first repetition and stayed steady over time.
• More in-context examples help models: In the random condition, model performance improved as the number of in-context examples increased, suggesting some adaptation to the task.
• Order matters, and models differ from humans on it: Performance in the shuffled condition was generally poorer than in the yoked condition, matching the trend in naïve humans. However, models were more accurate in the backward condition than in the shuffled condition, unlike humans, suggesting that working backwards from conventions to earlier, less conventionalised expressions may be easier for models than for people.
• Rapid gains from relevant history: While humans outperformed models on the first trials seen, models were able to rapidly exploit conversational history to reach approximately 0.8 accuracy.
• Relevance is critical: When context was drawn from different games (other-within and other-across), model accuracy dropped to 0.3–0.5. The boost from context therefore is not simply task adaptation; it depends on the context and the test trial coming from the same original game.
• Target-specific experience matters: Performance was also low in the ablated condition, implying that prior experience with the test trial's own tangram is crucial, since conventions are not systematically inferrable from other tangram conventions, even within the same game.
• Weak trial-wise calibration with humans: Yoked model correlations with human performance were r = .10–.27 against a human split-half r = .42 [.32, .50]; backward model correlations were r = .23–.40 against a human split-half r = .48 [.40, .56]. Per-model values in the yoked condition were Qwen 2.5 VL = .10, Gemma 3 = .20, Llama 3.2 = .25, Kimi VL = .27; in the backward condition .23, .31, .40, and .35 respectively; in the shuffled condition .28, .34, .38, .37; and in the random condition .61, .58, .55, .57.
• Best and most divergent models: Gemma 3 was the best performing model across most conditions. Qwen 2.5 diverged from the other VLMs in the yoked, shuffled, and backward conditions, performing much worse in later rounds, which the authors speculate is due to its handling of longer context lengths.
• Performance worsens over original rounds in reordered conditions: In shuffled, backward, random, and no context, performance generally worsened across rounds, and humans (and some models) showed highest performance in the second repetition rather than the first.
• Large item-wise variation: Both naïve humans and models display large item-wise variation, sometimes exceeding variation due to repetition number, matcher, or condition, with the largest variation in the other-within and other-across conditions. Gemma 3 appears especially accurate for tangrams A and E compared to other models.
• Learning from previous repetitions: In the yoked, shuffled, and backward conditions, when matchers answered a previous repetition correctly they tended to answer the next repetition correctly, with accuracy > 0.7 (except Qwen 2.5 in the yoked condition). When they had answered incorrectly, models were generally able to learn from mistakes in the yoked and shuffled conditions (again except Qwen 2.5), whereas naïve humans were at about 0.5 accuracy.
• Condition structure in confusion matrices: Correlations across conditions were relatively high for all models (all r > .83), except for the ablated condition (r = .16–.87) and the no context condition (r = .04–.80), with yoked/shuffled/backward clustering together and other-within/other-across/random clustering together.
• Errors concentrate on non-literal language: In the random condition error analysis, the grand mean model–human discrepancy was −0.186. The only lemma with a positive centred discrepancy occurring in at least 4 games was "bunny"; 39 lemmas had negative centred discrepancy and occurred in at least 4 games. Notable groups were body parts (e.g., "hand", "leg", "foot", "arm") and prepositions and direction words (e.g., "behind", "up", "right"). Models were not worse than average for person-related words (e.g., "person", "man", "guy"), shape words (e.g., "shape", "triangle", "square", "diamond"), or some spatial terms (e.g., "left", "down", "side", "top").
• Images are used, but unevenly: In the text-only paradigm, all models showed near- or at-chance performance in the first repetition across all conditions and in the no context condition when no images were shown, indicating that they were using visual information. Models generally performed worse without images, although Llama 3.2 in fact improved in most cases, suggesting it primarily used text information to select matches.

Methodology in Plain English

The researchers reused an existing dataset of iterated reference games. In each game, a grid of 12 tangram images was shown; on each trial one player (the describer) saw one image highlighted and typed a description, and the other players (matchers) picked which of the 12 images they thought was meant. Each game had 2–6 players who played 6 rounds of 12 trials each, so every image appeared once per round. Ten of these games were used.

For the model experiments, the setup mimicked a later study in which naïve human participants read transcripts of those conversations and guessed the target image, receiving feedback about whether they were correct but not being told the right answer when wrong. Each VLM received a system prompt describing the task plus a single image of all 12 tangrams labelled A through L, then the preceding trials formatted as a chat history with participant text as the user turn and the model providing the target tangram as the assistant turn. Unlike humans, models were given the correct target for the in-context trials. The description for the test trial was then provided, and the model's log probabilities for the tokens A through L were renormalised with a softmax; the probability assigned to the correct target served as accuracy.

The eight conditions manipulated context: yoked (all trials of the same original game in original order), shuffled (same trials, permuted order), backward (same trials, reversed order), ablated (original order but with in-context trials sharing the test trial's tangram removed), other-within (all in-context trials from a single different original game), other-across (in-context trials drawn randomly from all other games), random (fully shuffled across games and trial orders), and no context (no in-context trials). Comparisons to humans used trial-level correlations, item-level accuracy, learning analyses conditioned on the previous repetition of the same tangram, correlations among confusion matrices, and a lemma-level error analysis. A text-only control run without tangram images probed how much the models relied on vision.

Why This Matters

Impact on research. The work shows that VLMs can perform ad hoc pragmatic reasoning across multiple turns, and that they are sensitive to the amount, order, and relevance of context — but that they diverge from humans in the shape of that sensitivity and are weakly calibrated trial by trial. This reframes the question from "can models do the task" to "do models use context in the same way people do", and it introduces context-relevance manipulations as a useful diagnostic. It also connects to broader in-context learning findings, where more examples help most on difficult tasks.

Real-world applications

• Multi-turn conversational assistants, where correctly interpreting a new user message depends on retaining and using earlier turns.
• Dialogue systems that must resolve referring expressions that shift meaning over the course of a conversation, rather than matching fixed labels.
• Evaluation pipelines for VLMs that need tests going beyond single-turn accuracy, including calibration against human behaviour.
• Assistive or collaborative interfaces where a user coins an ad hoc term ("the bunny one") and expects the system to follow the convention later.

Industry relevance. The finding that performance collapses when context comes from a different source conversation (0.3–0.5 accuracy) matters for deployed systems that mix information from multiple sessions or documents. The observation that one model (Llama 3.2) improved without images, and the evidence that models rely on text more than expected, is directly relevant to teams designing multimodal products. The Qwen 2.5 divergence in later rounds points to long-context handling as a practical engineering constraint.

Future Directions

• Characterising the role of prompt engineering and of the causal and attentional factors underlying context use in VLMs.
• Disentangling the effect of feedback: models received the correct target for every in-context trial, whereas naïve humans only learned whether their own selection was right; future work could equalise this or test models with correctness-only feedback.
• Evaluating models in a true multi-turn setting, since humans experienced trials sequentially while models did not.
• Testing description generation, which is a harder task than interpretation and remains difficult for machine learning models, particularly in few-shot settings with abstract referents.
• Extending beyond the limited set of abstract tangram stimuli to naturalistic stimulus sets to test generalisability.
• Further study of humans' own sensitivity to context, and of how item-level properties such as nameability of particular tangrams affect accuracy.

Target Audience

Researchers working on pragmatics and language model evaluation, cognitive scientists comparing humans and machines on communication tasks, and machine learning practitioners building or evaluating multi-turn and multimodal conversational systems. Readers with some familiarity with vision–language models and in-context learning will get the most from it, though the experimental design is described in enough detail to be accessible to newcomers to the paradigm.

Authors’ abstract

Iterated reference games - in which players repeatedly pick out novel referents using language - present a test case for agents' ability to perform context-sensitive pragmatic reasoning in multi-turn linguistic environments. We tested humans and vision-language models on trials from iterated reference games, varying the given context in terms of amount, order, and relevance. Without relevant context, models were above chance but substantially worse than humans. However, with relevant context, model performance increased dramatically over trials. Few-shot reference games with abstract referents remain a difficult task for machine learning models.