Matched Outcomes, Divergent Gaze: How Foveated MLLMs Search Compared to Humans

AI summary

Overview

• Research area: Computer vision and human-vision modelling, specifically the comparison of multimodal large language models (MLLMs) against human eye-movement behaviour during goal-directed visual search.
• Technical level: Advanced. The paper assumes familiarity with foveation models, scanpath analysis, signal-detection theory, and mixed-effects statistics.
• Scope: The paper tests whether three general-purpose MLLMs, driven fixation-by-fixation through a foveation renderer calibrated to human acuity, search the way humans do across three separate axes: the present/absent decision, the efficiency of finding the target, and the gaze process itself.

What This Paper Is About

Human visual search is serial: because the fovea only resolves fine detail within roughly 1–2 degrees of gaze, a target spotted coarsely in the periphery must be fixated before it can be confirmed, and those fixations form a scanpath. The paper asks whether MLLMs given the same foveated, acuity-limited view of a scene will search the way people do — a question that matters both for using MLLMs as stand-ins for human observers and for interpreting attention-alignment scores as evidence that a model "sees like us." The authors decompose "human-like search" into decision, finding, and gaze, and test each axis separately on COCO-Search18.

Key Contributions

1. A three-axis dissociation. The authors show that decision, finding, and gaze come apart: models match or exceed humans on the first two axes while diverging sharply on the third, under the same human-matched foveated input.
2. A shared, non-human gaze signature across three unrelated models. Qwen3.5-35B-A3B, GLM-4.6V-Flash, and Gemma-4-E4B all produce low-entropy, large-amplitude, highly self-consistent scanpaths, and a principal-component analysis of the five-statistic gaze signature places all three in a region disjoint from the human reference.
3. Evidence that no degradation regime recovers human-like search. Sweeping the synthetic degradation factor k does not produce a level where first-saccade targeting matches the human rate while eventual success is preserved; convergence happens only where models are failing to resolve the scene.
4. A methodological critique and a null model. Because answer-alignment and saliency metrics are computed on outcomes or a time-collapsed map, the authors argue they cannot certify human-like vision on the process axis; a non-serial searcher that reaches human-or-better outcomes is offered as a null model for isolating the behavioural cost of the human serial bottleneck.

Main Findings

• Decision: models match or exceed humans. One-shot present-target detection is near ceiling and comparable across models (target-present/target-absent existence accuracy 0.99/0.81, 0.99/0.81, and 0.97/0.80), with a false-positive bias on absent scenes of 0.19/0.19/0.20. In the agentic task the present/absent decision is highly sensitive for every model (d' 3.84/4.23/3.14 against the human 2.91), and absent scenes are declared absent at 0.93/0.91/0.96.
• Finding: models reach the target faster than people. Every model fixates the target on the first saccade far more often than humans (TFP@1 0.97/0.97/0.80 against the human 0.49), attains comparable eventual success (TFP-end 0.98/0.98/0.93 against 0.93), and issues no more fixations (median 2/2/3 against 3).
• One target-absent behaviour is already non-human. Qwen3.5-35B-A3B declares absence after a single fixation (NFix-TA 1) versus the human median of 5 — an efficiency win that is itself not human-like.
• Gaze: a shared divergence. Gaze entropy lies below the human value (Cliff's δ −.67/−.65/−.27) and saccade amplitudes exceed it (+.50/+.61/+.23), indicating spatially concentrated sampling and direct movements to the target.
• Scanpaths are far more self-consistent than human scanpaths are with each other. Cross-seed (agent-to-agent) ScanMatch is 0.84/0.91/0.71, well above the human-to-human agreement ceiling of 0.53.
• Center bias is human-like. The models look in broadly human-relevant places (sub-threshold δ), so the divergence is in the order and dynamics of fixations rather than their locations; density correlation is 0.58/0.63/0.50 at the human-matched condition.
• The gap is not an acuity effect. The human-matched Geisler–Perry condition is behaviourally indistinguishable from the sharp condition for every model (e.g., TFP@1 0.97 against 0.97), even though it measurably degrades the periphery.
• No k yields human-like search at human-like success. For Qwen3.5-35B-A3B, first-saccade targeting falls to the human rate at k=32 (TFP@1/TFP-end 0.42/0.71); for Gemma-4-E4B at k=16 (0.42/0.75). By that point eventual success has already collapsed.
• Where the gaze signature converges, the models are failing rather than searching. At k24, eventual success is 0.81/0.65/0.65 against the human 0.93, and refixation rises to δ +.52/+.45/+.38 versus +.04/−.13/+.13 at GP — the revisiting of cells the model cannot resolve.
• Determinism is what persists. Across the legible range, cross-seed ScanMatch remains 0.78/0.80 at k16 and 0.70/0.71 at k24 for the two reasoning-tuned models against the 0.53 ceiling; Gemma-4-E4B falls to 0.48 at k24, below the ceiling.
• Gemma-4-E4B is the most human-like model by magnitude. It shows the smallest effect on five of seven statistics (entropy, saccade amplitude, scanpath length, center bias, coverage; gaze-entropy δ −.27 vs. Qwen −.67; cross-seed ScanMatch 0.71 vs. 0.84), and its fixation count is the only fixed effect whose mixed-effects 95% interval contains zero. The authors read this descriptively as weaker single-pass targeting, not a more human-like strategy.

Methodology in Plain English

The researchers took a standard human visual-search dataset, COCO-Search18, in which ten observers per scene searched 1680×1050-pixel images (subtending roughly 54°×35°, about 30 px/deg) for a cued object and reported present or absent. They used a frozen, category-stratified validation subset of 141 target-present and 144 target-absent scenes.

They then built a deterministic foveation renderer that mimics the eye: the image is sharp at the current gaze point and progressively degraded with eccentricity, following the Geisler–Perry acuity falloff equation. Four condition families were used — sharp (no foveation), Geisler–Perry (the only condition calibrated to human acuity), a synthetic "gist" family that scales the peripheral cutoff by k ∈ {8, 16, 24, 32, 48, 128} with Geisler–Perry as the k=1 member, and crop (a fovea-only disc of about 2.5°).

Each search episode starts at a forced central fixation. At each step the model sees the scene rendered at its current gaze point, with earlier glimpses retained in context, and returns exactly one directive line: LOOK(x,y), FOUND(x,y), or ABSENT. Only the final directive line is parsed, so any preceding reasoning text is discarded. A LOOK directive maps normalized coordinates to display pixels and becomes the next gaze point. No search policy is imposed; a 50-glimpse cap bounds runaway episodes but never forces termination, so every episode ends on the model's own decision. Decisions use a single free-form generation per step at temperature 0.6 under 5 seeds, and the harness, prompt, and renderer are identical across models. The full run comprises 3 models × 285 scenes × 5 seeds × nine conditions = 38,475 episodes, all completed.

Three models were tested: Qwen3.5-35B-A3B (a 35B mixture-of-experts with 3B active parameters, reasoning-tuned), GLM-4.6V-Flash (about 9B, reasoning-tuned), and Gemma-4-E4B (about 4B, instruction-tuned, run with thinking disabled).

For analysis, the authors applied a one-shot existence test first, restricting later analysis to trials a model answered correctly, and then measured three things: decision (existence accuracy, d' and criterion), finding (target-fixation probability by saccade, with a hit defined as a fixation within 1° of the target box, plus fixation counts), and gaze (saccade amplitude, gaze entropy on a 14×9 grid, refixation, scanpath length, center bias, turning angle, coverage). Effect sizes against the human distribution use Cliff's δ, with |δ| > 0.33 treated as non-trivial, and are confirmed by linear mixed-effects models with crossed random intercepts for scene and rater. Scanpath similarity uses ScanMatch in three modes: agent-to-human, agent-to-agent (cross-seed determinism), and human-to-human (all pairs per scene, giving the 0.53 ceiling).

Why This Matters

The paper's central claim is that matching retinal input reproduces where humans look but not how the looking unfolds in time, and that the gap sits on a process axis that answer-alignment and saliency metrics simply do not measure. If correct, this means such metrics cannot certify human-like vision, and that zero-shot MLLMs are suitable stand-ins for outcome and spatial questions but not for temporal, process-level ones.

Real-world applications:

• Benchmark design and evaluation. Teams using MLLMs as proxies for human observers in perception studies need to know which axes their metrics actually cover, and this work identifies a blind spot.
• Human-factors and interface testing. Where gaze behaviour matters (cockpit displays, medical imaging review, driving interfaces), a model that reaches the right answer by a non-human search process will not predict human error patterns or inspection times.
• Medical image reading and industrial inspection. Search-heavy expert tasks depend on serial inspection; a surrogate that skips it will misestimate both workload and missed-target rates.
• Saliency and scanpath prediction. The human-to-human ceiling of 0.53 and the cross-seed determinism numbers provide reference points for how much agreement is realistically attainable, and the finding that models exceed that ceiling warns against treating high similarity scores as evidence of human likeness.

Industry relevance: the paper is directly relevant to anyone deploying MLLMs in interactive visual-search or agentic perception pipelines, and to groups building attention-alignment metrics for model evaluation or for advertising and content-layout optimization, where a time-collapsed saliency map is often assumed to capture the whole of visual behaviour.

Future Directions

• A causal test of serial sampling. The authors note that they do not manipulate architecture directly, so the attribution of the gap to missing serial sampling is by elimination (acuity and spatial prior ruled out). Imposing serial sampling on a model would test this causally.
• Search-trained and agentic models. Search-trained agentic, pointing-native, and frontier closed-source models are named as the most pertinent next comparison but were not evaluated; the paper leaves them out of scope.
• Disentangling architecture, training recipe, and sparsity. The cross-model trend is confounded because these factors covary across the three models, and the authors report it only descriptively. A controlled study could separate them.
• Prompt sensitivity and untested constructs. The gaze signature was measured under a single prompt whose memory clause may partly shape refixation, and fixation durations are human-only; the authors also note that cross-seed agreement and human inter-observer agreement are not identical constructs, since COCO-Search18 has no matched human intra-observer baseline.

Target Audience

Researchers working on human vision modelling, foveated and active-perception architectures, and scanpath prediction; MLLM evaluation and benchmarking practitioners who use attention-alignment or saliency metrics as evidence of human likeness; and human-factors or applied-perception teams considering MLLMs as surrogates for human observers in search-heavy tasks.

Authors’ abstract

Human visual search is serial: the fovea must land on a candidate to confirm it, and those landings form a scanpath. Whether multimodal large language models (MLLMs), given the same foveated input, search as humans do bears on their use as models of human vision and on attention-alignment scores. We compare three general-purpose MLLMs with human eye-movement scanpaths on goal-directed search (COCO-Search18), driving each model fixation by fixation through an identical, human-matched foveated view and assessing it along three axes: the decision of target presence, the efficiency of reaching the target, and the gaze process itself. The axes dissociate. On the decision and on target acquisition the models match or exceed humans, detecting present targets near ceiling and reaching them on the first saccade more often than people do. The gaze process is not human. Under the human-matched condition, all three share one signature: low-entropy, large-amplitude, self-consistent scanpaths that agree with themselves far more closely than two humans agree with each other. That is consistent with a single-pass, non-serial architecture rather than a limit of acuity. Matched retinal input reproduces where humans look but not how the looking unfolds in time, and no degradation regime recovers human-like search at human-like success. The gap sits on a process axis that answer-alignment and saliency metrics do not measure. Because they miss it, such metrics cannot certify human-like vision, and zero-shot models suit outcome and spatial questions but not temporal, process-level ones.