PointerBench: A Diagnostic Benchmark for Fine-Grained GUI Grounding

AI summary

Overview

Research area: Multimodal AI agents for computer use — specifically GUI grounding, the sub-task of translating a natural-language instruction into precise screen coordinates.

Technical level: Intermediate. The paper is readable without deep technical background, but assumes familiarity with multimodal models, benchmarks, and evaluation metrics such as success rate and intersection-over-union-style coverage.

Scope: The paper introduces and evaluates PointerBench, a 1,500-example diagnostic benchmark with three 500-example subsets that isolate fine-grained GUI grounding in spreadsheets, long-text documents, and 100 professional applications, testing both frontier multimodal models and 39 specialist GUI grounding configurations.

What This Paper Is About

Computer-use agents must not only plan a sequence of actions but also point at exactly the right pixel — a cell boundary, a single character, a small icon. Existing end-to-end benchmarks measure whether a long task succeeds, which mixes up planning errors, grounding errors, and environment failures, while existing static grounding benchmarks mostly cover conventional GUI controls. PointerBench isolates the grounding step alone, deliberately removing temporal planning, and probes geometries that prior benchmarks largely skip: 1–3 px spreadsheet dividers, inter-character carets, thin text spans, and tightly bounded regions across 100 professional applications.

Key Contributions

1. A three-part diagnostic benchmark of 1,500 screenshot–instruction pairs, split evenly into PointerBench-Sheets (structural and geometric spreadsheet grounding), PointerBench-Text (character-, caret-, and region-level textual grounding in six languages), and PointerBench-Pro (icon, text, and mixed controls across 100 professional applications).
2. Pixel-exact target geometry with executable scorers, all rendered or assembled deterministically at a fixed 1024×768 frame, with point targets scored by containment and bounding-box targets scored by a coverage threshold of 0.90 and a precision threshold of 0.70.
3. A dual evaluation protocol: 37 general-purpose systems run under one unified absolute-pixel JSON prompt, and 39 open-weight specialist GUI grounding configurations (35 checkpoints, 2B–32B parameters) run under each model's native prompt, coordinate convention, or inference-time wrapper.
4. A diagnostic analysis that surfaces failures hidden by aggregate scores, including per-target-type breakdowns, a caret-placement decomposition, a bounding-box failure taxonomy, a language comparison, a reasoning-budget sensitivity study, and a coordinate-frame compliance analysis.

Main Findings

• Aggregate scores hide a text bottleneck. Claude Fable 5 averages 83.2% across PointerBench but only 69.6% on PointerBench-Text; GPT-5.6 Sol drops from 77.7% average to 53.4% on Text; the leading native specialist UI-Venus-2-9B drops from 54.5% to 25.8%. Text is the hardest suite for every family tested.
• Claude Fable 5 leads the frontier leaderboard at 83.2% average (97.6% Sheets, 69.6% Text, 82.4% Pro), ahead of Claude Opus 5 at 81.1% (98.8%, 63.2%, 81.2%) and GPT-5.6 Sol at 77.7% (97.8%, 53.4%, 82.0%). GPT-5.6 Luna (72.7%), GPT-5.6 Terra (72.3%), and Claude Sonnet 4.6 (72.2%) follow.
• Rank order flips between suites. Opus wins Sheets at 98.8%, the three leaders are within 1.2 points of each other on Pro (Fable 82.4%, Sol 82.0%, Opus 81.2%), but the ranking separates on Text where Fable reaches 69.6% versus Opus at 63.2% and Sol at 53.4%.
• MVP-32B is the strongest specialist overall at 59.3% average (68.4% Sheets, 34.0% Text, 75.4% Pro), best on every track among the 39 specialist configurations. Behind it, a rank reversal appears: UI-Venus-2-9B is second on Pro (73.6%), GTA1-32B is second on Text (29.6%) and overall (54.9%), and OpenCUA-32B is second on Sheets (66.0%).
• No specialist exceeds 34.0% on PointerBench-Text. The best is MVP-32B at 34.0%, followed by GTA1-32B at 29.6%, UI-Venus-2-9B at 25.8%, and MAI-UI-8B at 23.2%.
• Accuracy falls sharply as targets get finer. Word localization is easiest (Fable 90.2%, Opus 83.9%, Sol 76.8%, GTA1-32B 70.5%), but top specialists fall to 11.1%–18.5% on individual characters and 22.5%–34.2% on carets. In the caret-placement decomposition, no frontier model exceeds 50% accuracy on placements before, after, or between words, and UI specialists remain below 24%.
• Region (bounding-box) grounding is weak even for frontier models, where Fable reaches 56.0%, Opus 50.7%, and Sol 30.6%. Point-only specialists receive 0.0% on all 134 region tasks by construction; GUI-G2-7B reaches 12.7% and UI-Venus-Ground-7B 11.2%.
• Bounding-box failures have different shapes across models. Fable and Opus most often predict boxes that are too large (26.1% of the 134 box tasks for both), while Sol and Luna most often clip the target (39.6% and 41.8%).
• Thin spreadsheet geometry remains error-prone. Fable reaches 78.1% on row-resize handles, and falls to 91.4% on the edge group despite 100.0% on cells, headers, and colors. UI-Venus-2-9B reaches 89.0%–100.0% on spreadsheet cells and headers but only 31.6%–55.8% on borders and resize handles and 3.0% on relative offsets.
• Icons dominate professional-interface failures. Icons make up only 38.0% of PointerBench-Pro examples but 57.3% of pooled failures (215 of 375); icon accuracy for Fable, Opus, Sol, and Luna ranges from 70.0% to 75.3%, versus 89.0%–92.9% for text and 81.4%–85.9% for other controls.
• Multilinguality does not explain the text bottleneck. Fable scores 69.6% on both English and non-English examples, Sol 53.2% versus 53.6%, and Luna 45.6% versus 47.6%; Opus is the largest gap at 59.6% English versus 66.8% non-English (250 examples per group).
• Coordinate-frame compliance can dominate raw scores. Gemini 3.7 Flash and Qwen3.8 Max emit coordinates in a 0–1000 frame despite the prompt requesting absolute 1024×768; their official scores are 3.1% and 2.8%, but a uniform frame conversion raises them to 78.1% and 55.4%. Qwen3.6 Flash rises from 1.5% to 31.9% under the same transformation.
• Reasoning budget helps unevenly. GPT-5.6 Luna rises from 51.6% (disabled) to 70.8% (low) and 72.7% (medium); Sol from 67.5% to 77.3% and 77.7%; Sonnet is non-monotonic (68.7% disabled, 66.9% low, 72.2% medium); Fable's medium setting beats low by only 1.4 percentage points; Grok 4.20 improves from 1.8% to 8.7% and Qwen does not improve. Paired McNemar tests on the low-versus-disabled comparison give 337 corrected and 49 regressed for Luna (p = 5.59×10⁻⁵⁴), 199 and 52 for Sol (p = 1.90×10⁻²¹), and 22 and 49 for Sonnet (p = 0.00182).
• A trivial baseline fails. Clicking the screen center achieves 0.3% average (0.4% Sheets, 0.2% Text, 0.2% Pro).
• Target size matters but does not explain everything. Median target area is 0.215% of the screen in Sheets, 0.0286% in Text, and 0.129% in Pro; the median Text target is roughly 7.5× smaller than the median Sheets target and 4.5× smaller than the median Pro target. The largest-area quartile is on average 10.7 percentage points more accurate than the smallest.

Methodology in Plain English

The researchers built a benchmark where every example is a 1024×768 screenshot paired with one natural-language instruction and a target recorded as exact pixel coordinates — either a point or a box. Sheets and Text examples are generated programmatically from controlled interface descriptions, so the exact glyph and cell geometry is known rather than hand-annotated; Pro examples are drawn from a larger collection of generated professional interfaces, balanced so each of the 100 applications contributes exactly five examples. Spreadsheet layouts are varied with row and column offsets, styling changes, and optional application chrome so that a semantic location does not map to a fixed screen position. Text examples are kept only when they identify a unique target.

Scoring is deliberately asymmetric. A point counts as correct if it lands inside the annotated target region. A predicted box counts as correct only if it covers at least 90% of the true target and at least 70% of the predicted box lies inside the target, which punishes clipping the target while tolerating a small margin around it. Malformed outputs count as failures, and the reported metric is binary success rate per suite plus its unweighted mean.

For the frontier comparison, all models receive an identical system prompt demanding absolute pixel coordinates with the origin at the top-left and JSON output ({"point": [x, y]} or {"bbox": [x0, y0, x1, y1]}), with the original images submitted unmodified through OpenRouter providers and the response format enforced by a JSON schema. Because dedicated GUI agents use their own conventions, specialists are instead run with their native prompt, coordinate normalization, or inference-time wrapper, so the two leaderboards are not directly interchangeable. A separate controlled study re-runs six models at their available reasoning settings on the same 1,500 examples.

Why This Matters

PointerBench makes the case that a single task-completion number is a poor guide to agent reliability. By holding the action fixed and varying only the geometry of the target, it turns "the agent failed" into "the agent cannot place a caret between two characters" or "the agent cannot find the right icon" — distinctions that matter when an agent is clicking inside a real user's document.

Real-world applications:

• Spreadsheet automation: resizing columns, selecting color-coded cells, and resolving relative references like "paste cell B14 to cell E15" require thin-boundary and grid-structure grounding that current models still miss.
• Document editing assistants: placing a cursor after a specific word, or selecting a precise invoice field, is the primitive behind any text-editing agent; character- and caret-level accuracy of 11.1%–18.5% for specialists is far from usable.
• Professional software support: agents operating CAD, creative, and engineering tools must identify small icons and controls in dense, high-resolution interfaces, where icons account for 57.3% of failures.
• Evaluating model upgrades: teams can use the three suites as separate diagnostic signals rather than a single leaderboard rank, and the coordinate-frame analysis shows when a low score reflects formatting non-compliance rather than poor perception.

Industry relevance: the benchmark is released with metadata and evaluation code, and the finding that Gemini 3.7 Flash jumps from 3.1% to 78.1% under a uniform coordinate conversion is directly actionable for anyone deploying models behind an API with mismatched coordinate conventions. The result that reasoning budgets help some models substantially and others barely at all also informs inference-cost decisions for agent products.

Future Directions

• Closing the character- and region-level gap, which remains the largest headroom in the study: no specialist exceeds 34.0% on PointerBench-Text and the best region score among frontier models is 56.0%.
• Better icon understanding and UI semantics, since icons are 38.0% of PointerBench-Pro examples but 57.3% of pooled failures and only reach 70.0%–75.3% accuracy among the leading models.
• Standardizing coordinate output conventions, given that frame conversion alone moves Gemini 3.7 Flash from 3.1% to 78.1% and Qwen3.8 Max from 2.8% to 55.4%; the authors explicitly call for evaluation under normalized output representations.
• Extending beyond single-step grounding to resolution robustness and to the interaction between precise grounding and temporal planning, which the authors list as limitations — the benchmark uses only a fixed 1024×768 frame and explicitly does not measure behavior under resizing or arbitrary display resolutions.

Target Audience

Researchers and engineers building or evaluating computer-use agents and GUI grounding models, particularly those working on multimodal model post-training, reinforcement learning for grounding, or inference-time prediction aggregation. It is also useful for benchmark designers interested in a model of how to isolate a single capability with deterministic, pixel-exact generation, and for product teams who need to know which interface types remain unreliable before shipping an agent. Readers looking for an end-to-end measure of agent task success should note the authors' own caveat that PointerBench scores are not a complete measure of computer-use capability.

Authors’ abstract

Multimodal Large Language Models have shown promising performance in computer-use tasks, from planning user workflows to grounding natural-language actions to precise UI targets. Existing end-to-end benchmarks primarily emphasize long-horizon task completion, while GUI grounding benchmarks remain focused on conventional controls and a limited set of professional interfaces. We introduce POINTERBENCH, a diagnostic suite of 1,500 screenshot-instruction pairs probing three complementary capabilities: thin sub-element identification and geometric grounding in spreadsheets (POINTERBENCH-Sheets), character-level grounding in long-text documents (POINTERBENCH-Text), and mixed GUI grounding across 100 professional applications (POINTERBENCH-Pro). We carefully curate each task with pixel-exact target geometry and evaluate both general-purpose multimodal models and 39 specialist GUI grounding configurations. We find text grounding to be the most challenging setting, with Claude Fable 5, GPT-5.6 Sol, and the leading native specialist UI-Venus-2-9B achieving 69.6%, 53.4%, and 25.8% success on POINTERBENCH-Text despite respective 83.2%, 77.7%, and 54.5% average success across POINTERBENCH. These results reveal substantial headroom in fine-grained GUI grounding across frontier and specialist models. We publicly release the benchmark and evaluation code at https://github.com/warmwindOS/pointerbench.