Diagnosing the Reference Gap: What Visual Scaffolds Can and Cannot Fix in Multimodal Reasoning

AI summary

Overview

Research area: Multimodal large language models (MLLMs) — specifically, why these models fail at multi-step visual reasoning and whether simple visual annotations added to the input image can fix those failures.

Technical level: Intermediate. The core ideas are explained in plain terms and require no mathematical background, but familiarity with multimodal models and standard benchmark evaluation helps.

Scope (one sentence): The paper introduces ReGap-Bench, a paired benchmark that holds a scene, question, and answer fixed while varying only the visual reference marks in the image, in order to test whether a “reference gap” — rather than a perception failure — is what limits multi-step visual reasoning.

What This Paper Is About

Multimodal models can often recognize individual objects in an image yet still miscount them or lose track of which items they have already processed. The authors argue this is not a perception problem but a reference gap: the model fails to keep a stable link between its language-based reasoning and the specific visual entities, regions, or locations it is talking about. The paper asks whether external visual marks — numbers printed on objects, shaded regions, or grid lines — can close that gap, and, crucially, whether such marks only help when they match the visual operation a task actually requires.

Key Contributions

1. A paired benchmark (ReGap-Bench) spanning five visual reasoning tasks — shape counting, angle ranking, size ranking, clock ranking, and transformation prediction — with 200 scenes per task. The 1,000 scenes render to 3,800 images, because each underlying scene is drawn with and without different reference scaffolds while the question and gold answer stay fixed.

2. Evidence that task-matched references help and mismatched ones do not. The authors show that the gains depend on aligning the reference with the visual routine the task demands, not simply on adding more marks to the image.

3. Three supporting analyses — of mismatched references, incomplete references, and model scale (six Qwen3-VL Thinking models from 2B to 235B parameters) — establishing that the effect is not just extra visual information, that more complete references help more, and that the gap does not disappear with scale.

4. A training-free “visual skill” that lets an agent infer which visual routine a task needs and draw the corresponding reference scaffold at inference time, evaluated on CartoMapQA (100 tasks across eight subtasks) with the Codex GPT-5.5 agent.

Main Findings

• Task-matched references improve accuracy by up to 25%. Across all five tasks and all evaluated models, the matched scaffold produced gains, whereas mismatched scaffolds gave little benefit or actively hurt.

• Numbering helps counting and clock reading. Numbering improved shape counting by 6.0% to 17.5% and clock ranking by 8.5% to 25.0%. For clock reading specifically (a stated time within 5 minutes of ground truth), numbering raised accuracy from 10.6% to 28.6% for Claude, 58.2% to 82.5% for Gemini, 67.5% to 75.6% for Qwen3-VL, and 2.9% to 17.4% for InternVL. GPT behaved differently: its stated times were already accurate (90.3% plain vs. 90.8% with numbering), but it gave no time at all for 34% of clocks on plain images, dropping to 12% with numbering.

• Models differ in whether they invent their own references. On plain shape-counting images, 52.0% of Gemini’s responses assigned numbers to shapes and 31.5% explicitly scanned by rows or columns, while only 15.5% of GPT’s responses used a systematic referencing strategy and 84.5% used ambiguous references. With numbering, 100% of Gemini’s and 86.0% of GPT’s responses referred to the printed numbers.

• Shading helps most on hard, close comparisons. Shading improved size ranking by 4.0% to 11.0%. On plain images, pairwise accuracy already exceeded 91% for every model; errors concentrated on shapes with similar sizes, where pairs differing by less than 0.10 in scale factor were ordered correctly only 67.5% to 82.2% of the time, versus at least 97.5% for pairs differing by 0.20 or more. Shading helped most on exactly those close pairs.

• Shading reduces a specific angular error. On angle ranking, shading nearly halved the rate of reading the wrong side of the rays (i.e., confusing an angle θ with its conjugate 360° − θ): Claude 3.3% to 1.7%, Gemini 18.4% to 8.5%, InternVL 23.9% to 13.1%. GPT showed no reduction (13.3% vs. 13.8%), consistent with its nearly unchanged pairwise accuracy. Pairwise accuracy on angles rose for four of five models, including 76.4% to 85.2% for Gemini and 59.7% to 68.0% for Qwen3-VL.

• Grids help, but less and less reliably. Grid lines improved size ranking by 2.0% to 8.0% and transformation prediction by 2.0% to 5.5%, generally smaller than shading on size ranking. Gemini used grid coordinates systematically in 51% of its transformation-prediction responses, while GPT never did, matching the larger grid gain for Gemini (5.5%) than GPT (2.0%).

• Mismatched references confirm the mechanism. Applied to shape counting, numbering improved accuracy by 11.3% on average, shading by only 1.7%, and grid lines reduced accuracy by 4.1%. The same separation appeared on clock ranking.

• Completeness of the reference matters. Varying the number of dial numerals on clock ranking, mean accuracy rose from 23.9% with none, to 25.3% with two, 31.8% with four, and 38.0% with all twelve — a non-decreasing trend for every evaluated model.

• Scale does not remove the gap. Across six Qwen3-VL Thinking models from 2B to 235B, overall accuracy rose with scale, but the numbering advantage stayed between 9.0% and 23.5% and did not shrink monotonically.

• Perception checks support the framing. In a preliminary probe with Claude-Opus-4.7 on plain shape-counting scenes, the model counted all shapes correctly in 193 of 200 scenes (96.5%) and judged a single cropped shape correctly 95.9% of the time, yet counted the target class correctly only 72.0% of the time.

• The training-free skill transfers. On CartoMapQA, the agent drew an overlay on 57 of 100 tasks and improved mean accuracy in every overlay category: numbering 63.7 to 70.1 (+6.4), grid lines 92.5 to 96.7 (+4.2), shading 33.3 to 66.7 (+33.3), and no change (86.0 to 86.0) on the 43 tasks where no overlay was chosen.

Methodology in Plain English

The central design is a controlled comparison in which only one thing changes. For each problem, the researchers generate a scene (shapes, angles, clocks, or transformation panels) once, then render it several times: once plain, and once for each reference scaffold. The objects keep the same shapes, sizes, and positions; only numbers, light-gray interior fills, or faint grid lines are added. The question text and the correct answer are identical across all versions, and the references are assigned in a way that does not reveal the answer.

This lets them measure a “reference effect” — the accuracy with a scaffold minus the accuracy on the plain version of the same problem. A positive effect suggests the model could already perceive what it needed but could not keep hold of it while reasoning.

The choice of which scaffold goes with which task is motivated by Ullman’s theory of visual routines, which treats complex visual reasoning as a sequence of elementary operations like marking, bounded activation, and boundary tracing. Numbering supplies stable addresses, supporting marking and indexing. Shading makes a bounded region explicit, supporting bounded activation. Grid lines supply a common spatial frame, supporting focus shifts and boundary tracing.

The authors evaluate five models — Claude-Opus-4.7, Gemini-3.1-Flash-Lite, GPT-5.4-mini, Qwen3-VL-30B-A3B-Thinking, and InternVL3.5-38B — using exact match, plus pairwise accuracy for ranking tasks to give partial credit. They then run ablations that swap scaffolds onto tasks they were not designed for, vary how complete the clock dial numbering is, and sweep model scale from 2B to 235B. Finally, they package the three references as an agent skill with written rules, example before-and-after images, and a drawing script, and test it on CartoMapQA.

Why This Matters

Impact on research. The paper reframes a class of multimodal failures that the field has largely treated as a perception problem — addressed with higher-resolution encoders and finer visual features — as a binding problem between language and visual entities. The paired intervention design is a reusable diagnostic template: it separates “the model could not see it” from “the model could not keep track of it,” which most existing benchmarks conflate. The scale result is also a useful corrective, showing that a 235B model still benefits from numbering, so the fix is not simply more parameters.

Real-world applications.

• Document and chart question answering, where a model must repeatedly locate the same row, column, or data point while reasoning across it.
• Map and navigation assistants, the setting of the CartoMapQA evaluation, where counting junctions, measuring distances, and isolating regions are routine operations.
• Visual inspection and quality control, where an operator or agent must enumerate defects or parts without double-counting.
• Educational and tutoring tools, where ranking angles, sizes, or times is a common exercise and errors are often traceability errors rather than recognition errors.

Industry relevance. The proposed fix requires no training and no architecture change — it is an image-preprocessing step plus a prompt rule — which makes it cheap to deploy on top of existing proprietary or open models. The finding that a scaffold can hurt when mismatched is equally practical: teams adding annotations or highlights to inputs need task-specific selection, not blanket markup.

Future Directions

• Automating reference selection more reliably. The agent chosen an overlay on 57 of 100 CartoMapQA tasks and matched references to operations broadly, but the paper does not report how often the selection itself was wrong.
• Scaffolds for tasks beyond the five tested. ReGap-Bench covers counting, three ranking tasks, and transformation prediction; whether the same routine-matching logic extends to dense text-in-image reading, multi-panel diagrams, or video is untested.
• Combining external scaffolds with learned grounding. The paper positions external references as an alternative to training models to emit coordinates; whether the two are additive is an open question.
• Making models use references they are given. GPT never adopted grid coordinates even when grid lines were drawn, and gained far less than Gemini as a result — understanding what determines whether a model incorporates an available scaffold is unresolved.

Target Audience

Researchers and practitioners working on multimodal large language models who care about visual reasoning evaluation and failure analysis; benchmark designers looking for controlled-intervention methodology; and applied teams building vision-language agents for documents, maps, or diagram-heavy domains who want a low-cost, training-free intervention. Readers interested in cognitive-science-inspired accounts of machine visual reasoning — the paper builds directly on Ullman’s theory of visual routines — will also find it relevant.

Authors’ abstract

Multimodal large language models can recognize visual content yet struggle to reason reliably over it across multiple steps. In counting, a model may see all objects but lose track of which ones it has counted. We argue that such failures, commonly attributed to perception, reveal a reference gap: the failure to bind language to the relevant visual entities, regions, or locations. Motivated by the theory of visual routines, we test whether external references such as numbering can narrow this gap. We introduce ReGap-Bench, a paired benchmark of five visual reasoning tasks in which the scene, question, and answer are fixed and only the reference varies. Results show that task-matched references improve accuracy by up to 25\%, while mismatched references provide little benefit or even hurt performance. These findings identify the reference gap as a bottleneck distinct from visual perception and show that constructing task-appropriate visual references can serve as a training-free visual skill for improving multimodal reasoning.