GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning

AI summary

Overview

Research area: Multimodal GUI (graphical user interface) visual grounding for computer-use agents, combining large-scale dataset construction with a two-stage post-training recipe (distillation followed by reinforcement learning).

Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, on-policy self-distillation, reverse KL divergence, and policy-gradient methods such as RLOO, plus GUI grounding benchmarks and coordinate-normalized action formats.

Scope: The paper introduces a 586K-screenshot, 633K-instruction GUI grounding dataset (ScreenRef), a two-stage post-training method (D2RL), and an 8B-parameter grounding model (GroundGUI-8B) evaluated on five benchmarks.

What This Paper Is About

Building GUI agents that reliably click the right on-screen element requires high-quality visual grounding data, but high-fidelity, cross-platform grounding datasets are scarce: the best-performing models use proprietary data that is not released, while public datasets have limited diversity or noisy annotations. The paper's goal is to close that gap by releasing a large, multi-stage-verified grounding dataset spanning desktop, web, and mobile, and by training an 8B model on it with a training procedure that avoids the generalization problems of plain supervised fine-tuning and the sparse-reward failure of standard reinforcement learning.

Key Contributions

1. ScreenRef, a GUI grounding dataset of 586K screenshots and 633K referring expression instances (585,981 screenshots and 633,369 instructions as reported in Section 3.3), spanning desktop, web, and mobile platforms with image resolutions from 0.1 to 8.3M pixels, and covering four task types: text recognition, visual matching, spatial understanding, and refusal.

2. Distillation-to-Reinforcement Learning (D2RL), a two-stage post-training framework that applies on-policy self-distillation (OPSD) first to shape a stronger starting policy, then REINFORCE Leave-One-Out (RLOO) to optimize task-level grounding success.

3. GroundGUI-8B, a model built by training Qwen3-VL-8B-Instruct with sequential SFT and D2RL, reported at 95.2% on ScreenSpot-V2, 63.9% on ScreenSpot-Pro, 65.4% on OSWorld-G, 86.8% on MMBench-GUI L2, and 38.3% on UI-Vision.

4. Open release: data, code, and model are released (dataset under a gated non-commercial research-only license; code and checkpoints under Apache 2.0), with claims of surpassing several 32B-scale models such as ScaleCUA-32B, OpenCUA-32B, and GTA1-32B.

Main Findings

• Benchmark results: GroundGUI-8B scores 95.2% on ScreenSpot-V2, 63.9% on ScreenSpot-Pro, 65.4% on OSWorld-G, 86.8% on MMBench-GUI L2, and 38.3% on UI-Vision.

• Gains over the base model: Relative to Qwen3-VL-8B, GroundGUI-8B gains 3.5 points on ScreenSpot-V2, 5.5 on MMBench-GUI L2, 14.0 on ScreenSpot-Pro, 10.6 on OSWorld-G, and 20.8 on UI-Vision.

• Comparison against closed-data 8B models: GroundGUI-8B surpasses Step-GUI-8B by 1.3 points on ScreenSpot-Pro and 1.2 points on MMBench-GUI L2, and outperforms Holo2-8B and MolmoPoint-8B on OSWorld-G, ScreenSpot-Pro, and ScreenSpot-V2. Step-GUI and Holo2 do not release their training data.

• Data quality comparison: Under SFT only, Qwen3-VL-8B trained on ScreenRef averages 76.4 across ScreenSpot-Pro, OSWorld-G, MMBench-GUI L2, and ScreenSpot-V2, versus 72.9 for ScaleCUA SFT and 73.7 for GroundCUA SFT — a 3.5-point and 2.7-point gain respectively. On OSWorld-G specifically, the ScreenRef SFT model beats the ScaleCUA and GroundCUA models by 3.2 and 5.3 points; the paper attributes this partly to ScreenRef containing refusal samples, which the other datasets lack.

• Ablation results: Starting from the same SFT checkpoint, SFT along averages 68.0, adding OPSD gives 69.1 (a 1.1-point gain), adding RLOO gives 68.9 (a 0.9-point gain), and full D2RL gives 69.9. D2RL also outperforms an RLOO variant trained on the same 12K post-training data (RLOO†, 69.1) by 0.8 points, which the authors use to argue the gain comes from the stronger initial policy rather than extra data exposure.

• Dataset composition: Text recognition is the largest category at 50.4% of instances, followed by spatial understanding (24.4%), visual matching (21.6%), and refusal (3.5%). Desktop contributes 267,172 instances, web 303,790, and mobile 62,407. Web is text-heavy (55.0% text recognition), desktop has the highest share of visual matching (23.2%) and refusal (5.2%), and mobile is more balanced between text recognition (38.7%) and spatial understanding (38.0%).

• Dataset comparison: ScreenRef and JEDI are the only datasets in the comparison table that include refusal samples. ScreenRef reports the widest resolution range (0.1 to 8.3M pixels) among the compared datasets.

Methodology in Plain English

The authors built the dataset through two pipelines. The first pipeline crawls real interfaces and extracts UI element metadata: accessibility trees for desktop, DOM structures for web, and parsed XML layout files for Android. Noisy metadata boxes are filtered by comparing them against OmniParser V2 detections and discarding boxes with IoU below 0.5. Instruction–box pairs are then generated by Qwen3-VL-235B-A22B-Instruct and verified by Qwen3.8-27B, which checks whether the answer matches the instruction given the image and the cropped box. This yields three instruction types: text recognition, visual matching, and spatial understanding.

The second pipeline converts public tutorial videos into training tuples through three stages: keyframe extraction (using block-level frame differences, down-weighting full-frame changes, re-ranking candidates with transcript action verbs and YOLOv8 cursor detections, capped at 60 candidates per video, then searching forward and backward for the nearest static frames), joint instruction and action annotation (a GPT-5.5 API call that segments tasks and annotates actions with element-level referring expressions grounded only in visible screen evidence), and post-hoc visual grounding (Qwen3.7-Plus predicts target presence, a box/coordinate, and a confidence score; below a threshold of 0.95 the proposal escalates to Claude Opus 5, and samples still below threshold are discarded). Refusal samples are created by random-pairing instructions with frames and verified as genuinely infeasible by Qwen3.8-27B.

For training, SFT runs on 4 B300 GPUs with global batch size 32, learning rate 1×10⁻⁶, AdamW, and 2 epochs; in the second epoch the loss weights for text recognition and refusal samples are reduced to 0.2. D2RL then starts from the SFT model using 12K grounding samples derived from GroundCUA (excluding screenshots identical to those in UI-Vision), running 2 epochs on 8 B300 GPUs with learning rate 1×10⁻⁶, a cosine schedule, batch size 42, and no KL penalty. Stage I (OPSD) matches a frozen teacher's token distribution on the student's own rollouts under reverse KL divergence, where the teacher sees the ground-truth box highlighted plus a textual localization hint and the student sees only the original screenshot and instruction; coordinate-digit tokens are weighted by β·k_t where k_t reflects digit significance. Stage II (RLOO) samples 8 rollouts per prompt and uses a binary point-in-box reward: 1 if the predicted click point falls inside the target box, 0 otherwise, with unparseable responses gated to 0. Each rollout's reward is compared to the average reward of the other rollouts in the group.

Why This Matters

Impact on research. The paper tests whether the field's reliance on proprietary data is necessary: it argues that a carefully filtered, verified, open dataset plus a staged post-training recipe can let an 8B model match or beat the closed-data 8B baselines (Holo2-8B, Step-GUI-8B, MolmoPoint-8B) and reported numbers from several 32B-scale models. It also isolates a specific RL pathology — zero-reward rollouts on hard samples like small icons or spatial-relation queries — and proposes pre-RL distillation as the remedy, with an ablation showing the gain is not merely from extra data exposure.

Real-world applications:

• Desktop automation and accessibility tools that need to locate small, dense UI controls in professional software such as CAD, creative suites, scientific tools, and office applications.
• Cross-platform mobile and web agents that must resolve icon-only buttons, which the paper notes are underrepresented in HTML/DOM-derived datasets.
• Refusal-aware agents that recognize when a requested target is not present on screen, reducing spurious clicks during infeasible tasks.
• Training infrastructure for teams that cannot afford proprietary grounding datasets and need a public alternative.

Industry relevance. Grounding accuracy is the bottleneck capability for computer-use agents. The paper's headline comparisons — open 8B weights overtaking closed-data 8B models and competing with 32B models — speak directly to cost and deployment tradeoffs. The dual licensing (Apache 2.0 for code and checkpoints, gated non-commercial research-only for ScreenRef) also signals the release posture enterprises should expect.

Future Directions

• Privacy auditing. The ethics statement says the authors have not yet performed a systematic human privacy audit of screenshots of real interfaces, which may incidentally contain personal or account information; the dataset documentation asks users to report such content and removal requests are actioned with a corrected revision.

• Improving hard-sample grounding. Sparse rewards remain an issue on small icons and spatial-relation queries. The ablation reports the largest D2RL gains on UI-Vision (38.3 with D2RL vs 34.3 for SFT) and OSWorld-G (65.4 vs 64.9), suggesting spatial and fine-grained reasoning is where headroom remains.

• Reducing reliance on proprietary teacher models. The data pipeline depends on GPT-5.5, Claude Opus 5, Qwen3.7-Plus, Qwen3.8-27B, Qwen3.6-27B, and Qwen3-VL-235B-A22B-Instruct; the reproducibility statement notes prompts, filtering rules, per-stage filtering statistics, and the position-aware weighting strength β will be added to the code repository.

• Broadening benchmarks and platform coverage. Mobile accounts for only 62,407 of 633,369 instances, and the paper does not report results on every platform-specific benchmark; expanding the mobile subset and testing on additional professional-software benchmarks are natural extensions.

Target Audience

Researchers and engineers working on GUI agents, computer-use agents, and multimodal visual grounding; teams building or fine-tuning vision-language models for screen understanding; dataset builders interested in multi-stage verification, refusal supervision, and synthetic-data quality control; and RL practitioners studying how distillation can be combined with policy-gradient methods to address sparse-reward tasks. Readers without a grounding or RL background will find the dataset-construction sections accessible but the D2RL formulation and benchmark tables demanding.

Authors’ abstract

Constructing robust GUI agents relies on multimodal visual grounding: the ability to precisely map natural language intents to actionable on-screen components. Despite abundant desktop, web, and mobile benchmarks, high-fidelity resources tailored for diverse environments are surprisingly scarce. To bridge this gap, we introduce the Screen Referring Grounding Dataset (ScreenRef), a comprehensive GUI grounding dataset consisting of GUI screenshots paired with natural language referring expressions and box or point annotations. Spanning desktop, web, and mobile domains, ScreenRef comprises 586K screenshots and over 633K referring expression instances, covering tasks such as text recognition, visual matching, spatial understanding, and refusal scenarios. To boost the model's grounding performance, we propose the Distillation-to-Reinforcement Learning (D2RL) method to address the limited generalization of supervised fine-tuning and the sparse reward problem in reinforcement learning. Utilizing ScreenRef, we train Qwen3-VL-8B-Instruct via sequential SFT and D2RL training, yielding GroundGUI-8B. GroundGUI-8B achieves 95.2% on ScreenSpot-V2, 63.9% on ScreenSpot-Pro, 65.4% on OSWorld-G, 86.8% on MMBench-GUI L2, and 38.3% on UI-Vision. Among 8B-scale models, it produces substantial gains over models trained on open-source datasets, with performance comparable or even superior to models trained on closed-source data. Code, models, and datasets are available at https://github.com/PrentisAI/GroundGUI.