AgentVQA: A Unified Benchmark for Agentic Visual Understanding

AI summary

Overview

Research area: Computer vision and multimodal evaluation — specifically benchmarks for vision-language models (VLMs) operating as agents (GUI navigation, robotics, egocentric video, games, 3D spatial planning).

Technical level: Intermediate. Readers should know what a VLM and a VQA benchmark are; no specialized simulator or robotics background is required.

Scope: The paper introduces and validates AgentVQA, a 1,000-question offline multiple-choice benchmark assembled from 23 source datasets across five agentic domains, and reports results for 22 VLMs.

What This Paper Is About

Vision-language models are increasingly used as the decision-making core of agents, but the standard way to test them — running them inside simulators or real environments — is expensive, slow, and hard to reproduce. Existing offline question-answering benchmarks are a cheaper alternative, yet they are fragmented: each covers one domain, uses its own answer format, and varies in difficulty, so results cannot be pooled or compared. The paper's goal is a single curated benchmark that preserves realistic agentic scenarios from many domains while being cheap and deterministic to run.

Key Contributions

1. A unified benchmark. AgentVQA consolidates 23 source datasets into 1,000 manually verified multiple-choice questions spanning five domains — Web Agents, Egocentric Videos, Robotics, Games, and Spatial Planning — balanced at 200 questions per domain (40 or 50 per source dataset).

2. A human-centered, model-assisted curation pipeline. A five-stage process in which Gemini 2.5 Pro proposes three hard negatives for non-multiple-choice tasks, Gemini and Claude screen candidates for ambiguity and annotation errors, and human annotators repair or reject the rest. Only about 22% of initial candidates survive to the final benchmark.

3. A standardized interface for heterogeneous targets. Coordinates, actions, spatial relations, and predicted states from the original datasets are all expressed as a common multiple-choice format while keeping the semantics of each original task.

4. An empirical validation of the benchmark as an offline proxy. Model rankings on AgentVQA correlate strongly with those from SpatialWorld, an online closed-loop benchmark, while diverging from general-purpose visual and reasoning benchmarks.

Main Findings

• The best model reaches only 58.5% accuracy. Across 22 evaluated models, Gemini 3.7 Flash is the strongest at 58.5% average accuracy; GPT 5.6-sol think-high and Qwen3.7 Flash both average 55.0%, Gemini 3.1 Pro 55.7%, and the weakest, Llama-4-Maverick, 28.0%.

• Difficulty varies sharply by domain. Averaged over models, performance is highest on Egocentric Videos (59%) and Games (51.5%), then Spatial Planning (46%), Robotics (38%), and lowest on Web Agents (29%). Only three models exceed 40% on Web Agents. Games shows the widest spread, from 23% to 70%.

• No single model dominates all domains. Gemini 3.7 Flash leads Games (70%), Spatial Planning (62.5%), and Robotics (52.5%), but GPT-5.6-sol thinking-high leads Web Agents (40.5% versus 47%) and Qwen3.7 Flash leads Egocentric Videos (72.5%).

• The open-source gap is narrow. Gemini 3.7 Flash (58.5%) leads the best open-source model, Qwen3.8 27B (50.9%), by 7.6 points. Kimi K3 (48.3%), Qwen3.5 35B (47.9%), and Qwen3.5 397B (47.8%) all outperform Claude Opus 4.8 (44.0%) and Claude Sonnet 5 (43.4%).

• Reasoning helps unevenly. GPT-5.6-sol gains 16.8 points with thinking-high relative to thinking-none (55.0% versus 38.2%), while GLM 4.1V performs worse with thinking than without it; the authors report GLM-4.1V-Thinking gets stuck in "thinking loops."

• Action history matters. Removing action history lowers performance by 9.5% on AitW and 10% on MONDAY, averaged over five models (Gemini 3.1-pro, GPT-4o, Qwen3.5 27B, Kimi K3, Claude Sonnet 5).

• Rankings diverge from general benchmarks, but track online evaluation. Spearman correlation with AgentVQA is 0.67 for MMMU-Pro, 0.59 for GPQA-Diamond, and 0.44 for ChartQA-Pro, whereas AgentVQA's Spatial Planning domain correlates 0.895 (p = 8×10⁻⁵) with the SpatialWorld Physical track over 12 shared models.

• Errors come from perception and state construction, not reasoning. In an analysis of Gemini 3.1 Pro's incorrect predictions (100 error-mode assignments manually reviewed), 78% of Web Agents errors were grounding errors, 71% of Games errors were visual miscomprehension, 65% of Robotics errors were spatial confusion, and 62% of Spatial Planning errors were viewpoint gaps. Egocentric errors were spread across motion misperception (39%), perception gaps (29%), and spatial confusion (15%). Task misunderstanding and reasoning-answer inconsistency were rare.

Methodology in Plain English

The authors first picked 23 existing public datasets, chosen because their tasks carry a genuine "agentic signal" (choosing actions, predicting next steps, reasoning about consequences), cover distinct domains, can be converted to a single-answer question, are openly licensed, and are large enough to sample from. Candidates were sampled across datasets, task types, and input modalities while keeping their original questions and annotations.

Non-multiple-choice tasks were converted by prompting Gemini 2.5 Pro with the image, task prompt, correct answer, action history, and metadata to write three difficult wrong options — some near-misses (coordinates a few pixels off, wrong but adjacent UI elements) and some semantically similar (a tablet instead of a phone). Grounding distractors were checked by script to confirm they fall outside the ground-truth box.

Two rounds of filtering followed. A multi-turn Gemini-and-Claude pipeline flagged ambiguous instructions, inconsistent annotations, multiple plausible answers, and weak distractors, removing roughly 45% of candidates. Human annotators then reviewed what remained and rejected about 60% as ambiguous, incorrect, or too easy, repairing promising examples where possible. A final human pass confirmed exactly one correct answer and plausible but clearly wrong alternatives. Figure 4 illustrates the problem this solves: a bounding box around scissors that includes background lets a click missing the object be scored correct, and a goal like "delete a Lyft account" can have several valid next actions when the question does not require the shortest route.

For evaluation, models answer with a single letter; outputs are parsed programmatically, with questions skipped as unevaluated if no unambiguous answer is found. Models without native video support get 32 evenly spaced frames. Temperature is 0.8 for reasoning models and 0.2 for non-reasoning models. Spatial coordinates are presented in each model's native convention (normalized [0, 1000] in (y, x) order for Gemini, absolute pixels for Qwen), and all images are given at native resolution. A separate robustness check permuted option order across 5 models, 6 datasets, and 5 seeded permutations, finding a standard deviation of approximately 1.

Why This Matters

Impact on research: The paper argues that agentic competence is better viewed as a collection of domain-specific capabilities than a single general ability. Its evidence that AgentVQA rankings barely correlate with MMMU-Pro, GPQA-Diamond, and ChartQA-Pro (0.67, 0.59, 0.44) while correlating 0.895 with an online simulator benchmark suggests general VQA scores are a poor proxy for agentic readiness — and that cheap offline benchmarks can substitute for expensive rollouts when the domain is matched.

Real-world applications:

• Selecting which VLM to deploy as the perception-and-decision layer in a UI automation or mobile-assistant product, without standing up a device farm.
• Pre-screening robot manipulation models on grounding and grasp-target questions before committing to physical or simulated rollouts.
• Evaluating assistant models for first-person video understanding, such as smart-glasses or AR guidance systems, where tracking state over time matters.
• Diagnosing where a model stack breaks — localization, motion, or 3D relations — so engineering effort targets the actual bottleneck.

Industry relevance: The narrow 7.6-point gap between the best closed and best open model, with several open-weight models beating Claude Opus 4.8 and Sonnet 5, is directly relevant to deployment cost decisions. The finding that extended reasoning helps some models and causes "thinking loops" in others is a practical caution for anyone paying for reasoning tokens.

Future Directions

• Link offline error patterns to online failures. The authors propose connecting specific error categories (grounding, motion, viewpoint) to failures observed during interactive execution, and testing whether gains on controlled questions translate into more reliable online behavior.

• Explain the cross-domain disagreement. The SpatialWorld correlation is strong only in the domain-matched pair (0.895); the lower correlations for other domains are left as evidence that rankings depend on the capability assessed, raising the question of how many domains a benchmark needs before aggregate scores are meaningful.

• Fix the grounding bottleneck. Since 78% of Web Agents errors and 65% of Robotics errors are grounding or spatial confusion rather than reasoning failures, it is open whether better visual encoders, tool use for localization, or higher-resolution inputs close the gap.

• Address reasoning-mode instability. The "thinking loops" in GLM-4.1V-Thinking and the uneven benefit of extended reasoning point to unresolved questions about when chain-of-thought helps agentic visual tasks.

Target Audience

Researchers and engineers who build or select vision-language models for agentic use — robotics, GUI automation, gaming agents, and embodied AI — and who need a cheap, reproducible way to compare models across those settings. It is also useful for benchmark designers interested in the human-in-the-loop curation and hard-negative generation methodology, and for practitioners deciding between open-weight and closed models for deployment. Readers looking for new model architectures or training methods will not find them here; the contribution is evaluation.

Authors’ abstract

Vision-language models (VLMs) can perform a broad range of tasks across diverse settings, yet their performance in agentic contexts remains poorly understood. Although online evaluations in simulators and real-world environments are essential for measuring interactive agent performance, they are costly and challenging to scale. Complementary offline VQA benchmarks can enable faster and more reliable comparisons across agentic domains, yet existing benchmarks remain limited and fragmented. To address this gap, we introduce AgentVQA, an offline benchmark for evaluating VLMs on realistic agentic scenarios across five agent domains: Web Agents, Robotics, Egocentric Videos, Games, and Spatial Planning. We build AgentVQA by manually selecting 1000 challenging, real-world agent examples from 23 source datasets and adapting them for reliable offline evaluation. The resulting benchmark preserves realistic scenarios from interactive environments across diverse domains, and its model rankings track those from corresponding online evaluations. Across the 22 evaluated models, the best achieves only 58.5% accuracy. Our cross-domain analysis shows that errors are driven primarily by failures to ground and maintain accurate representations of the current and evolving visual state, including localizing relevant UI elements, interpreting motion, tracking state changes over time, and inferring 3D spatial relationships, rather than by task misinterpretation or downstream logical reasoning.