Give the Model a Finger: Developmentally Aligned Evaluation of Counting in Vision-Language Models

AI summary

Overview

Research area: Multimodal AI evaluation — specifically, how vision–language models (VLMs) enumerate objects, and how to diagnose why they fail, using counting principles from developmental psychology as a measurement framework.

Technical level: Intermediate. No new model training is involved, but readers should be comfortable with benchmark terminology (exact-match accuracy, MAE, bootstrap confidence intervals) and with the idea of an agent loop that calls image tools.

Scope: A training-free harness of simple image-manipulation tools ("a finger, a magnifier, and an eraser") combined with a point-level audit protocol, evaluated across 9 VLMs, 682 images, and 7 harness conditions, to test whether giving models the epistemic actions children use makes them count better — and to measure the serial counting procedure when it is externalised as marks on the image.

What This Paper Is About

VLMs answer correctly on very small sets of objects but degrade into rough estimation as sets grow, much like children before they learn to count. Children cross that barrier by doing something physical: pointing at each item while saying number words, sliding counted items aside, and splitting a large array into smaller groups. The paper asks whether giving a VLM the same kinds of actions — drawing a numbered tag at a coordinate, zooming in, greying out counted items, overlaying a grid — lets it count better, and, more importantly, whether the marks it leaves behind can be audited against counting principles to reveal exactly which sub-skill is missing.

Key Contributions

1. A developmentally aligned evaluation protocol. The counting principles from Gelman and Gallistel — one-to-one correspondence, cardinality, abstraction, order irrelevance, and the regularity/arrangement effect — are turned into concrete, measurable predictions. Every tag a model draws is scored against point-level ground truth as on-target, duplicate, phantom, or (for objects) omitted, so a wrong count decomposes into specific principle violations rather than a single accuracy number.

2. A training-free harness of "dumb" epistemic tools. Four tools (numbered tag, crop/zoom, erase/grey-out, coordinate grid) plus a scripted 2×2 tiling routine are implemented as plain PIL operations with no detector, segmenter, or OCR behind them. All perception and all decisions remain the model's own; the harness only draws and crops what it is told.

3. An empirical study of 9 models, 682 images, and 7 harness conditions. Headline contrasts carry 95% paired bootstrap intervals, and the whole study cost under 153 US dollars. All code, stimuli, and trajectories are slated for release.

4. A dissociation result. The constituent capacities — pointing precision, one-to-one tracking, cardinal read-out, checking, order invariance — do not form a single "counting ability"; they dissociate across models, which is precisely what a developmentally aligned evaluation is meant to expose.

Main Findings

• The exact-counting range is a model-specific signature spanning nearly two orders of magnitude. Using R₈₀ (the largest n at which an isotonic fit of exact accuracy stays at or above 80%), the range runs from 3 for Grok 4.20, to 6–7 for the Qwen models, 7 for Gemini 3.1 Flash-Lite and GPT-5 mini, and 150 for both Gemini 3.7 Flash (95% CI [100,200]) and Gemini 3.1 Pro (CI [50,200], widened by one dipping mid-battery level). Claude Sonnet 5 sits at 20 (CI [6,50]). Every model is at ceiling for n ≤ 4.

• Photographs move every cliff to the left. On FSC-147 (8–80 objects), direct exact accuracy on the common subset ranges from 11.0% (Qwen3-VL-8B) to 52.0% (Gemini 3.7 Flash), and even the strongest models fall to about 20% in the 61–80 bin. PixMo-Count (2–10 objects) is much easier, at 62.2%–92.2%.

• As an intervention, the finger mostly fails. Only one gain in the small-count photograph regime is CI-solid: Qwen3-VL-32B rises from 65.6% to 78.9% on PixMo-Count (+13.3 pp, CI [+4,+22]). On dense FSC-147 photographs no tool arm moves any model reliably, and verbal chain of thought is the strongest single intervention for Flash-Lite (41.0%, +18.0 pp [+7,+29]). On dot arrays, Flash-Lite moves from 36.0% to 58.0% with tag+crop (tag alone +14.0 pp [+4,+26]) and its range from R₈₀ = 7 to 20, but every other model is flat or loses — CI-solidly so for GPT-5 mini (-30.0 pp), Grok 4.20, Grok 4.5, and Qwen3-VL-8B.

• Pointing precision is the prerequisite. The median on-target tag lands 1.6 thousandths of the image side from its object for Gemini 3.7 Flash and 5.1 for Flash-Lite, versus 34.1 for Grok 4.20 and 40.4 for GPT-5 mini. The imprecise pointers add 31.9–52.7 phantom tags per 100 objects and are exactly the models that lose accuracy when given the tag loop.

• Dense scenes fail by omission, not double counting. For models that can point, the dominant residual on FSC-147 is skipping: 38.1 omissions per 100 objects for Flash-Lite and 28.7 for Gemini 3.1 Pro, against only 4.4 and 1.8 duplicates — the same asymmetry seen in children's keeping-track errors.

• Models rarely check their work. Only 17.2% of Flash-Lite trajectories issue a second tag call (46.7% for Gemini 3.1 Pro): one-shot tagging without the verification habit children acquire. Paradoxically, removing the verification prompt raises Flash-Lite's accuracy from 39.9% to 45.5%, because the prompt makes it re-estimate verbally rather than re-inspect.

• The answer is read off the tags, not the tags off the answer. When the harness silently deletes one on-target tag after the first tag call, Gemini 3.7 Flash re-tags the missing object in 84% of trajectories and Flash-Lite in 57%, and the final answer equals the number of visible tags in 84% and 87% of cases. GPT-5 mini, the decoupled pointer, repairs in only 33%. For precise pointers, the trace is the computation.

• Cardinality is mostly respected but decouples for imprecise pointers. 93.0% of Flash-Lite and 100.0% of Gemini 3.1 Pro trajectories on DotArrays announce exactly the number of tags, versus 66.9% for GPT-5 mini — models that tag, distrust the tags, and estimate anyway.

• Order irrelevance is often violated. Forced to tag the same 68 arrays in opposite traversal orders, models follow the instructed scan but end with the same count on only 44% of arrays (Flash-Lite), 24% (Qwen3-VL-32B), and 86% (Gemini 3.7 Flash). Against same-order repeat controls, reversal adds +21.7 pp (CI [+14,+31]) and +15.4 pp ([+8,+23]) of excess disagreement for the mid-tier models and none (+0.7 pp) for Gemini 3.7 Flash. A 180°-rotation control, which visits dots in reverse at undiminished accuracy (63%), still disagrees +9.6 pp more often than the repeat control for Flash-Lite, so part of the raw effect was difficulty but an order effect remains.

• The children's regular-over-scattered advantage appears only under the routine. With tags, Flash-Lite scores 90.3% / 79.2% / 70.8% on regular / random / clustered arrays; under direct estimation the ordering reverses (51.4% / 58.3% / 66.7%). The advantage survives hexagonal and holed lattices for Flash-Lite (85% and 90% against 55% on random arrays of the same n) but not Qwen3-VL-8B (25/25 vs 25%), and a lattice rotated by 20–40° defeats every model and arm (30%).

• The verbal routine collapses before the visual one at scale. Gemini 3.7 Flash's CoT range is 80 versus 150 under direct. Of 30 CoT replies at n ≥ 80, the 6 that tally row by row are all exact; the other 24 partition the image into regions, estimate each, and add — exact 8 times, under-counting by -7.2 on average.

• Seeing marks barely beats listing them. Blind pointing (list) matches or beats tag in 11 of 15 cells where both ran. What the rendered trace buys is auditability: on FSC-147, Flash-Lite's blind lists carry 61.8 phantom points per 100 objects against 24.8 for tag.

• The scripted tiles routine is brittle. It is the best arm on sparse dot arrays for Flash-Lite (MAE 0.8) but collapses to 3.0% on FSC-147, and on PixMo-Count it double counts by construction (mean answer 9.9 for a truth of 6.0).

Methodology in Plain English

The researchers built an agent loop in which a VLM is shown an image and can take a small number of actions (budget T = 6 tool calls by default), each emitted as a single JSON line. The tools are deliberately unintelligent: one draws a numbered dot wherever the model asks, one returns an enlarged crop, one greys out points the model names, and one overlays a coordinate grid with 0–1000 tick labels. The harness re-renders the working image after each action and sends it back with a short verification prompt asking whether every tag sits on a distinct object and whether anything is still untagged. Numbering is supplied by the harness across calls, so stable order is scaffolded rather than tested.

Seven harness conditions were compared: direct (answer only), chain of thought, blind pointing (emit one coordinate per object without seeing them rendered), tag, tag+crop, full toolbox, and the scripted tiling routine. An additional strict-seriality arm (tag-1) allows one point per call with a budget of 80 calls, so every mark is placed only after the previous one is seen.

The evaluation rests on three stimulus sets: DotArrays, a 272-image controlled battery of black discs with n from 1 to 60 in regular, random, and clustered arrangements plus factor sets for heterogeneous shapes, distractors, touching discs, and very small discs; FSC-147, 200 real photographs with 8–80 dot-annotated instances across 29 classes; and PixMo-Count, 180 human-verified photographs with 2–10 instances. An XL extension of 30 images (n from 80 to 200) locates the cliff for the two strongest models, and 60 lattice-control images test the arrangement effect.

For the audit, tags are matched to ground-truth points by minimum-cost assignment with a per-image distance threshold of half the median nearest-neighbour distance (floor 15 px). Unmatched tags near an already-matched object count as duplicates; unmatched tags far from every object are phantoms; unmatched objects are omissions. All are reported per 100 ground-truth objects. Two further probes are distinctive: a tag-perturbation test that secretly adds or removes a tag to see whether the answer is read off the marks, and the forced opposite-traversal experiment with repeat, rotation, and mirror controls to separate genuine order dependence from ordinary inference instability.

Nine models were accessed through a single API (OpenRouter) at temperature 0 with low reasoning effort. Three inexpensive tier-1 models ran every arm on every image; two tier-2 models skipped tag+crop and tiles; four tier-3 models ran four arms on a stratified subset. Cross-model comparisons use a 240-image common subset (50 DotArrays, 100 FSC-147, 90 PixMo-Count); ablations use a 143-image subset. A contrast is called CI-solid when its 95% paired bootstrap interval over images excludes zero.

Why This Matters

Impact on research. The paper reframes counting evaluation: instead of asking whether a model counts, it asks which constituent capacities it has. Because the audit localises the missing prerequisite — imprecise pointing, lost track, an unchecked answer, an order-dependent count — it produces diagnoses rather than leaderboard deltas, and it offers a template for auditing other serial visual procedures that are normally hidden inside a model's activations.

Real-world applications:

• Quality inspection and inventory counting — counting parts, cells, or produce in photographs, where knowing whether a failure is a pointing error or a tracking error determines whether a fix is worth pursuing.
• Medical and scientific imaging — counting cells or colonies, where an auditable mark-by-mark trace is more defensible than an opaque number.
• Accessibility and assistive tools — interfaces that expose a model's counting steps to a human reviewer who can see exactly which items were counted and which were skipped.
• Evaluation tooling for model developers — a cheap, training-free diagnostic harness (the full study cost under 153 US dollars) for characterising a model's enumeration limits before deployment.

Industry relevance. The result that a finger helps only models that can already point is directly actionable: adding tool use to a model with poor coordinate prediction can make it worse. The ablation finding that a coordinate grid is the most useful single addition suggests a low-cost lever for teams building agentic vision pipelines, and the finding that a verbal chain of thought collapses past about 80 items while an externalised record degrades more slowly argues for combining verbal and visual routines rather than picking one.

Future Directions

• Whether the routine can be learned rather than borrowed. The authors stress that their models neither learn from the tags nor carry the routine to the next image — it is a prosthesis, not a curriculum. Running the battery on models trained on child-scale data, and testing whether they can acquire the procedure and the habit of checking it, is named as the natural next step.

• Fixing the verification step. The current verification prompt makes models re-estimate verbally instead of re-inspecting, and removing it actually improves accuracy. Training or prompting a genuine re-inspection step is an open engineering question.

• Extending the stimulus range and object geometry. The photographic sets currently cover only 8–80 and 2–10 objects, and FSC-147 dots mark approximate centres, so tags on elongated objects can be scored as phantoms. Broader count ranges and better-suited annotations would test whether the omission-dominated error profile holds.

• Calibrating pointing as a first-class target. Since pointing precision predicts which models benefit, improving coordinate prediction (the grid helped) and supplying something like feedback for the model's coordinate-prediction "motor" error is a concrete route to making the loop therapeutic rather than merely diagnostic.

Target Audience

Researchers working on multimodal evaluation and VLM capability analysis; practitioners building agentic vision pipelines who are considering tool-augmented counting; and cognitive scientists interested in computational models of numerical cognition, particularly those who want counting principles operationalised as measurable tests rather than interpreted as claims that models resemble children. The paper is explicit that the developmental alignment lies in the evaluation — the correspondences are qualitative, not psychometric fits — so readers seeking a cognitive model of the child will not find one here, while readers seeking a reusable diagnostic protocol will.

Authors’ abstract

Vision-language models (VLMs) are exact on small sets and drift into estimation on larger ones, a profile reminiscent of children before they master counting. Children cross the barrier with epistemic actions: pointing at each item while reciting number words, moving counted items aside, partitioning the array. We give VLMs the same actions as dumb image tools with no visual intelligence of their own in a training-free agent loop. Our central claim is about measurement, not intervention: a model that counts by marking leaves a normally hidden serial procedure on the image, and the developmental principles of counting turn those marks into an interpretable audit of serial visual reasoning: every tag is classified against point-level ground truth as on-target, duplicate, phantom, or omitted, the answer is checked against the number of tags, and order irrelevance is tested by forcing opposite traversal orders. Across 9 models, 682 images, and 7 harness conditions we find that (i) the exact-counting range is a model-specific signature spanning nearly two orders of magnitude (R80 from 3 to 150 dots); (ii) as an intervention the finger mostly fails: tags help 4 of 9 models on small-count photographs, hurt imprecise pointers, and rarely beat a verbal routine or a blind point list; what the loop buys is an auditable trace, not accuracy; (iii) as measurement it succeeds: a tag-perturbation probe shows that precise pointers read their answers off the tags, pointing precision tracks which models benefit, dense scenes fail by omission, and models rarely revise after seeing their own tags; and (iv) the capacities dissociate: models follow an instructed traversal order yet often change their answer when it reverses, and children's regular-over-scattered advantage appears only under the routine and survives lattices that defeat a rows-times-columns shortcut.