Grounding Computer Use Agents on Human Demonstrations

AI summary

Overview

• Research area: Multimodal vision-language models for GUI/computer-use agents, specifically UI element grounding on desktop software.
• Technical level: Intermediate. The paper assumes familiarity with supervised fine-tuning, reinforcement learning from verifiable rewards, and GUI grounding benchmarks, but the core idea (better annotated data beats more data) is accessible.
• Scope: This paper introduces GroundCUA, a human-annotated desktop grounding dataset, and GroundNext, a family of 3B and 7B grounding models trained on it, evaluated across five grounding benchmarks and one multi-step agentic benchmark.

What This Paper Is About

Computer-use agents must turn a natural-language instruction into a precise click on the correct on-screen element, a step called grounding. Existing grounding datasets are large but mostly built automatically from web and mobile interfaces, and they underrepresent the dense, icon-heavy, high-resolution layouts typical of desktop software. The authors address this gap by collecting expert human demonstrations on desktop applications, annotating nearly every visible element, and training models on that data.

Key Contributions

1. GroundCUA, described as the largest expert-annotated desktop grounding dataset: 87 applications across 12 categories, 56K screenshots, and over 3.56M human-verified element annotations, built from over 10,000 task demonstrations performed by trained annotators.
2. A 700K image-instruction training set derived from those dense annotations, containing three instruction types (Direct, Functional, and Spatial) generated by prompting a multimodal LLM with annotated boxes, application names, element labels, and surrounding context.
3. The GroundNext family of vision-language models at 3B and 7B scales, built on Qwen2.5-VL-Instruct and trained in two stages: supervised fine-tuning on 700K curated datapoints, then reinforcement learning on 10K new elements not in the SFT set.
4. A comprehensive analysis of SFT versus RL roles, cross-domain generalization beyond desktop, and the effect of using open-source applications on grounding performance.

Main Findings

• SFT results lead across scales. GroundNext-3B (SFT) scores 66.4 average with UI-Vision included (48.6 ScreenSpotPro, 62.2 OSWorld-G, 75.5 MMBench-GUI, 87.3 ScreenSpot-v2, 58.2 UI-Vision), versus 54.3 for the next best SFT-only 3B model, GUI-Actor-3B. GroundNext-7B (SFT) scores 69.2 average, versus 56.1 for JEDI-7B and 55.3 for GUI-Actor-7B.
• Data efficiency is the central claim. GroundNext is trained on 700K instructions, described as less than one-tenth the training data of prior work (JEDI is cited at 9M datapoints), yet outperforms larger SFT baselines.
• Controlled dataset comparison. Training the same base model (Qwen2.5-VL-3B-Instruct) on 100K samples from each of Aguvis, UGround, OS-Atlas (Desktop), JEDI, and GroundCUA shows GroundCUA yields significantly higher average SFT scores than all other sources.
• RL adds consistent but modest gains. GroundNext-3B (RL) reaches 68.4 average with UI-Vision and 70.0 without it; GroundNext-7B (RL) reaches 70.5. Models already fine-tuned on GroundCUA gain the least from RL, which the authors attribute to GroundCUA's supervision leaving fewer remaining errors.
• Strong agentic performance. On OSWorld-Verified with OpenAI o3 as planner, GroundNext-3B (RL) attains 50.6 overall (OS 62.5, Office 47.0, Daily 55.0, Pro 73.5, Workflow 36.5), comparable to JEDI-7B at 51.0 despite being less than half the size, and better in 3 of 5 categories. It surpasses OpenCUA-72B (46.1), Qwen3-VL-Flash (41.6), and Claude-4-Sonnet (41.4), while Claude-4.5-Sonnet scores 62.9.
• Cross-platform generalization holds partially. GroundNext-7B (RL) achieves 89.2% on MMBench-GUI mobile and 81.9% on web, versus InfiGUI-G1-7B at 90.9% and 85.3%. On ScreenSpot-v2 the model is comparable on mobile but falls behind on web.
• Icons are the biggest win. On ScreenSpotPro, GroundNext outperforms most models by an average of 10.7% in icon recognition, with 15.9% gains in Development and 8.4% in Creative over InfiGUI-G1-7B. The authors link this to the density and diversity of icon annotations in GroundCUA.
• Dense, fine-grained annotations. GroundCUA averages 64.1 elements per screenshot (up to 542), with a resolution range of 0.4 to 7.0 megapixels and average element area of 0.13% of the screenshot, compared with 7.8 average elements and 0.53% element area for OS-ATLAS.

Methodology in Plain English

The authors partnered with a data labeling company and had trained annotators design and perform everyday computer-use tasks, such as drafting a document or editing a spreadsheet, in 87 open-source desktop applications. Rather than random clicking, these real task trajectories produce screenshots that resemble actual usage. From the recordings, they extracted keyframes capturing the interface state immediately before each action, then annotated every visible element with a bounding box and a text label; PaddleOCR was used to extract raw text for long passages like source code. About 50% of elements were assigned to one of eight high-level categories.

From these annotations, a multimodal LLM was prompted with boxes, labels, categories, and context to write instructions in three flavors: Direct (describing an element's attributes or position), Functional (describing the intended action), and Spatial (describing relative position to other elements).

For modeling, they fine-tuned Qwen2.5-VL-Instruct at 3B and 7B, updating both the vision encoder and the language model, on 8 H100 GPUs with a global batch size of 128. They then applied RL using the RLOO method with a hand-designed discrete reward that buckets the normalized distance between the predicted point and the ground-truth box into six score levels from -1.0 to 1.0, rewarding predictions inside the box more as they approach its center. They used a group size of 8, batch size 64, and one epoch on a single 8-GPU H100 node.

Why This Matters

The paper argues that for desktop grounding, annotation quality and density matter more than raw data volume. If that holds, teams can build capable agents without multi-million-sample training corpora or large compute budgets, which shifts effort from scraping to curation.

Real-world applications:

• Desktop automation and RPA: Reliably clicking the right toolbar icon or menu item is the precondition for automating routine work in office, CAD, creative, and scientific software.
• Accessibility: Accurate grounding of natural-language commands to on-screen controls supports agents that operate software on behalf of users who cannot navigate interfaces directly.
• Resource-constrained deployment: A 3B model competitive with far larger models and proprietary APIs matters where inference speed and memory are limited.
• Cross-platform assistants: Demonstrated generalization to mobile and web interfaces indicates one grounding model can serve multiple platforms.

Industry relevance: the dataset and models are intended to be released under a permissive license from open-source applications, and the results give enterprises a concrete signal that expert-annotated data, not scale alone, drives dependable computer-use agents.

Future Directions

• Scaling training. The authors note they trained with limited scale and compute, and that GroundCUA can support variable-sized fine-tuning sets to push performance further.
• Better reward signals. The dense annotations could support precise, expressive rewards for RL beyond the simple discrete scheme used here, enabling systematic study of reward design.
• Cross-domain generalization. Desktop workflows involve complex multi-window tasks while mobile and web tasks are lighter; mixing data across domains could improve transfer, though balancing domains and addressing transfer bottlenecks need careful study.
• Continual learning and adaptation. GroundCUA includes platform- and category-level metadata, which could support research on how agents adapt to unseen applications.
• Application similarity analysis. The authors suggest future work could test whether applications more similar to those in the dataset yield higher performance.

Target Audience

Researchers and engineers working on GUI agents, multimodal grounding, and UI automation; practitioners choosing training data for computer-use models; and teams building agentic systems under compute or latency constraints. Readers interested in dataset construction methodology, reinforcement learning from verifiable rewards, or human-in-the-loop annotation pipelines will also find it useful.

Authors’ abstract

Building reliable computer-use agents requires grounding: accurately connecting natural language instructions to the correct on-screen elements. While large datasets exist for web and mobile interactions, high-quality resources for desktop environments are limited. To address this gap, we introduce GroundCUA, a large-scale desktop grounding dataset built from expert human demonstrations. It covers 87 applications across 12 categories and includes 56K screenshots, with every on-screen element carefully annotated for a total of over 3.56M human-verified annotations. From these demonstrations, we generate diverse instructions that capture a wide range of real-world tasks, providing high-quality data for model training. Using GroundCUA, we develop the GroundNext family of models that map instructions to their target UI elements. At both 3B and 7B scales, GroundNext achieves state-of-the-art results across five benchmarks using supervised fine-tuning, while requiring less than one-tenth the training data of prior work. Reinforcement learning post-training further improves performance, and when evaluated in an agentic setting on the OSWorld benchmark using o3 as planner, GroundNext attains comparable or superior results to models trained with substantially more data,. These results demonstrate the critical role of high-quality, expert-driven datasets in advancing general-purpose computer-use agents.