IGG: A Benchmark for Interactive GUI Grounding under Visibility Constraints

AI summary

Overview

Research area: GUI agents / vision-language model grounding and benchmark design (Artificial Intelligence; human-computer interaction evaluation).

Technical level: Intermediate. The paper is readable without deep RL background, but familiarity with vision-language models, supervised fine-tuning, and GRPO-style reinforcement learning helps.

Scope: The paper introduces IGG, a 601-task benchmark across 16 controllable mock GUI environments that measures whether an agent can first reveal a hidden target through interaction and then click it, and it diagnoses and partially closes the gap between humans and current VLMs on this capability.

What This Paper Is About

Existing GUI grounding benchmarks (such as ScreenSpot) assume the target element is already visible in a single static screenshot, but real interfaces often hide targets off-screen, behind popups, inside collapsed panels, behind hover cues, or behind a timer. IGG asks the missing question: can an agent perform the right interaction to make the target visible before attempting to ground and click it? The goal is to isolate this "visibility recovery" step, measure where current models fail, and test whether the capability can be trained.

Key Contributions

1. A task formulation. Interactive GUI grounding is formalized as visibility recovery (selecting a reveal action and executing it at the correct location or with correct parameters) followed by target recognition and final-click localization, deliberately separated from both static grounding and end-to-end navigation. Success is scored at a single click, exactly as in static grounding, but only after the target must be revealed.

2. The IGG benchmark. 601 tasks across 16 controllable GUI environments (modeled on Gmail, Google Calendar, VS Code, Figma, Spotify, YouTube, and others), organized into seven sub-types under three constraint types (Viewport, Multi-State, Temporal/Manipulation), solved with a five-action space of Scroll, Move, Drag, Wait, and Click. Environments are deterministic, use stable element identifiers, and support click-time programmatic scoring; each task ships with a human-verified ground-truth trace used only for efficiency metrics.

3. Controlled diagnostic interventions. Experiments that tell the model which interaction is needed versus performing the reveal interaction for it, isolating execution of the reveal action as the bottleneck rather than target recognition.

4. A training study. Frequency-balanced supervised fine-tuning on successful trajectories followed by online GRPO, producing consistent gains across Qwen3-VL, InternVL3.5, and GUI-Owl-1.5.

Main Findings

• Large human–model gap. In a human study with 15 participants on a 21-task suite (315 trials, three tasks per sub-type), participants reached 94.6% accuracy with a mean completion time of 19.8 seconds. GPT-5.5 reached 79.4% overall, and the strongest evaluated open-source VLM (GUI-Owl-1.5-8B) reached only 31.6%. Qwen3-VL-8B reached 30.8% and Qwen2.5-VL-7B 22.3%.

• The bottleneck is executing the reveal, not recognizing the target. Telling Qwen3-VL which interaction to perform raised accuracy only from 30.8% to 33.9%, whereas performing the reveal trajectory for the model raised it to 81.9%. For GPT-5.4 Mini (Low), the full-task accuracy was 72.4% and the reveal-performed condition reached 90.4%.

• Difficulty is uneven across sub-types. GPT-5.5 scored 86.2% on Viewport constraints, 89.4% on Multi-State constraints, and 60.1% on Temporal/Manipulation constraints. The weakest sub-type for every open-source model was Movable Occlusion (CT3-3), where all three open-source models scored 1.7% while GPT-5.5 reached 40.0%. Delayed State Activation (CT3-1) was nearly saturated, with Qwen3-VL at 95.7% and GPT-5.5 at 95.7%.

• Simple heuristics separate easy from hard reveal patterns. Target-agnostic fixed scrolling, hovering, and waiting revealed 68.8%, 61.4%, and 87.1% of their matched tasks, but popup dismissal and occluder dragging reached only 17.6% and 16.9%. The hard cases require identifying which element controls visibility and how to manipulate it.

• Failures are omissions of the needed action. When the ground-truth trajectory required Drag, Scroll, or Move, failed Qwen3-VL runs omitted that action in 59/59, 207/247, and 71/96 cases respectively. Base VLMs emitted no drag actions at all on held-out tasks.

• The capability is learnable. Two-stage training (frequency-balanced SFT then online GRPO) raised Qwen3-VL from 40.0% to 59.2% GA on the 120-task validation split, InternVL3.5 from 28.3% to 47.5%, and GUI-Owl-1.5 from 35.0% to 52.5%. On Movable Occlusion, frequency-balanced SFT emitted Drag in 7/8 required-drag tasks and solved 4/8, which GRPO raised to 5/8.

• Initialization matters for RL. GRPO reached 60.0% on Dynamic Target Interaction from the frequency-balanced seed but only 30.0% from the standard SFT seed, and did not improve over its seed model when initialized from standard SFT.

• Procedural hints help but do not substitute for learned manipulation. Instructions that state the procedure raised Qwen3-VL from 30.8% to 56.9% over all 601 tasks, yet hinted Qwen3-VL solved 0 of 8 held-out movable-occlusion tasks, whereas SFT+GRPO solved 5 of 8 without any test-time hint.

• Efficiency and cost. On the validation comparison, GPT-5.5 averaged 66.5 seconds per task versus 14.2 seconds for the trained Qwen3-VL, and evaluating GPT-5.5 over the complete benchmark cost approximately $74.9.

• Robustness to layout. Regenerating the tasks at five viewport sizes yielded 127 tasks valid in every layout (103 training, 24 held-out); the trained Qwen3-VL beat its base model by 11.1–17.3 points at every viewport, including 43.3% to 60.6% at 390×844 mobile portrait.

• Environment fidelity checks. Matched mock–real pairs scored higher DINO similarity (0.7724 vs. 0.5442), higher SigLIP similarity (0.8094 vs. 0.6211), and lower DreamSim distance (0.2529 vs. 0.5229) than non-matched pairs. On 16 paired mock–real tasks (one per environment) model ordering was preserved, with Qwen3-VL SFT+GRPO leading at 56.3% in both settings (Pearson r = 0.98, Spearman ρ = 1.00). Human ratings on a five-point scale averaged 4.50 for visual realism, 4.64 for interaction realism, and 4.75 for task executability.

Methodology in Plain English

The researchers built their own miniature web applications rather than testing on live sites. Each of the 16 mockups imitates a familiar tool and is fully deterministic: the same starting screen, the same click behavior, and a fixed identifier on every element, so a scripted evaluator can check whether a click landed on the intended element without fuzzy matching. Tasks are written as goal-only instructions ("Click 'Cart' for 'Wireless Mechanical Keyboard'") that name the target but never hint at the procedure; annotators rewrote any instruction that leaked the steps and replayed every trace to confirm the target is unique and reachable. A Playwright-driven Chromium evaluator runs at 1440×900, allows an eight-action budget with greedy decoding, and scores a task as successful only when the clicked element's ID matches the target ID.

To find out where models fail, the authors ran controlled interventions on a four-stage breakdown of each episode: choosing a reveal action, executing it, recognizing the target once exposed, and localizing the final click. They first told the model which interaction type was needed, then separately performed the ground-truth reveal themselves and asked only for the click. They also applied simple target-agnostic heuristics (always scroll, always hover, always wait) to see which constraint patterns are mechanically easy.

For training, they fine-tuned on human-verified successful trajectories (479 tasks for training, 120 held out after excluding two tasks whose traces could not be replayed), upweighting rare reveal actions such as Move, Wait, and Drag, then applied online GRPO with a reward combining click-time success, bounded credit for visibility-changing progress, a per-step penalty, and penalties for wrong, empty, invalid, or repeated ineffective actions. Finally, they validated the benchmark with human studies on solvability and on how faithfully the mockups resemble the real applications they imitate.

Why This Matters

Impact on research. Static GUI grounding has been a standard evaluation axis since ScreenSpot, and navigation benchmarks collapse planning, exploration, and grounding into one end-to-end success signal. IGG carves out a third axis — visibility recovery — that is neither. Its intervention results reframe the problem: the gap is not perceptual but procedural, which points future work toward action execution priors rather than better screenshot encoders.

Real-world applications:

• Accessibility tooling: agents that operate interfaces for users who cannot scroll, hover, or drag themselves, where the hidden-target problem is most acute.
• Automated UI testing: deterministic, replayable checks for whether a control is reachable only after hover, expansion, dismissal, or delay.
• Robotic process automation: scripts that must dismiss popups, expand collapsed panels, and wait for asynchronous loads rather than clicking a fixed coordinate.
• Agent training data: human-verified reveal trajectories as supervision for broader GUI agents.

Industry relevance. The cost and latency findings matter for deployment: GPT-5.5's accuracy advantage came with roughly 66.5 seconds per task and about $74.9 for a full benchmark pass, while a trained Qwen3-VL ran at 14.2 seconds per task without paid API inference — a practical argument for training smaller open models on interaction priors instead of relying on frontier APIs for every reveal step.

Future Directions

• Extending beyond controlled mockups. The authors state as a limitation that deterministic mockups do not replace evaluation on live interfaces; broader real-interface evaluation is the obvious next step, building on the 16-task paired mock–real study.
• Handling the hardest manipulation cases. Movable Occlusion remains far from solved even after training (GPT-5.5 at 40.0%; trained Qwen3-VL at 5/8 on the held-out drag subset), and hinted models solved 0 of 8 of these tasks — precise spatial manipulation is the clearest open problem.
• Better action-space coverage and RL initialization. The ablation showing that GRPO fails to improve from a standard SFT seed raises the question of how to make rare actions appear in the policy's support more reliably, and whether other reward-shaping or exploration schemes can substitute for frequency-balanced SFT.
• Generalization across layouts and longer horizons. The viewport study measured only 127 tasks valid in all five layouts; whether learned reveal behavior transfers to responsive reflow at scale, and how visibility recovery composes with multi-step navigation, remain open.

Target Audience

Researchers and engineers working on GUI agents, computer-use agents, and vision-language model grounding who want a benchmark that isolates the interaction step preceding a click. It is also useful for practitioners building RPA or accessibility automation who need evidence on where current models break and whether fine-tuning on interaction trajectories is worth the investment. Readers interested in benchmark construction will find the human-solvability protocol, fidelity checks, and deterministic evaluator design directly reusable.

Authors’ abstract

GUI grounding benchmarks ask whether an agent can localize a target that is already visible in a static screenshot. Real interfaces often violate this assumption: a target may be off-screen, appear only after a state change or hover, remain occluded, or become actionable after a delay. This leaves the ability to recover target visibility unevaluated. To measure this reveal step, we introduce Interactive GUI Grounding (IGG), a benchmark in which an agent must first make the target visible through interaction and then click it. IGG contains 601 tasks across 16 controllable GUI environments and seven visibility-constraint sub-types, with a compact action space of scrolling, moving, dragging, waiting, and clicking. In our human study, participants achieve 94.6% accuracy, while GPT-5.5 reaches 79.4% and the strongest evaluated open-source VLM reaches only 31.6%. To identify the source of this gap, we conduct two controlled interventions. Telling Qwen3-VL which interaction to perform raises its accuracy only from 30.8% to 33.9%, whereas performing the interaction for the model raises it to 81.9%. The bottleneck is executing the reveal interaction, not recognizing the target once it is exposed. We further show that this capability is learnable, demonstrating that visibility recovery can be strengthened consistently across model families.