DeskForge: Dense Supervision from Desktop Environments for Computer-Use Agents

AI summary

Overview

Research area: Computer vision and GUI/computer-use agents — specifically, GUI grounding (locating interface elements from natural-language instructions) and the generation of large-scale annotated desktop training data.

Technical level: Intermediate. The motivating problem and results are accessible, but the paper sits at the intersection of dataset construction, Linux desktop systems engineering, vision-language model fine-tuning, and agent evaluation.

Scope: The paper introduces DeskForge, a controllable Linux desktop environment that composes and explores real applications to produce dense annotations, and DeskForge-1M, the resulting corpus of 1.21M annotated observations, 159.7M element instances, and 917K click transitions, then measures the effect of fine-tuning four vision-language models on a 200K grounding view of that corpus.

What This Paper Is About

Computer-use agents must click the right control in dense desktop scenes where multiple applications, overlapping windows, and visually similar widgets compete for attention, but existing training resources rarely pair such multi-application scenes with dense annotations or vary those scenes in a controlled way. DeskForge addresses this by generating the desktop itself rather than merely recording it: real applications run under scene specifications that control application selection and initial states, content, window layout and stacking, appearance, and resolution, while an annotation pipeline fuses screenshots, accessibility trees, and window geometry into dense element labels and records the outcome of each executed click. The goal is to show that this controllable composition is a scalable source of supervision that improves both GUI grounding and long-horizon task completion.

Key Contributions

1. DeskForge, a controllable desktop environment. It composes and explores real applications under configurable desktop conditions, collecting dense structured observations and recorded interactions at scale. It varies application states, content, window layout, appearance, and resolution, and supports extension through configuration entries (appearance presets, resolutions) and application manifests.

2. DeskForge-1M, a large-scale densely annotated desktop corpus. The corpus contains 1.21M annotated observations (1,207,368), 159.7M element instances (159,723,780), and 917K recorded click transitions, spanning 19 applications, seven appearance presets, and seven display resolutions, combining dense per-screen annotations with action-linked before/after observations.

3. Grounding gains across held-out and external conditions. Fine-tuning four vision-language models on 200K grounding examples improves accuracy in every held-out condition and on all five external GUI grounding benchmarks, with mean external gains ranging from 1.5 to 24.9 points. Dense annotations also train a whole-screen element detector.

4. Downstream task-completion benefits under a fixed planner. Changing only the action-model weights within a fixed Qwen3.6-27B planner, all four fine-tuned action models solve more WebArena-Infinity and OpenApps tasks, connecting single-target grounding supervision to improved multi-step task completion.

Main Findings

Held-out desktop grounding improves for all four models. Qwen3.5-4B's mean accuracy over the four held-out conditions rises from 76.26% to 87.54%; UI-R1-3B rises from 38.01% to 84.59%; Gemma4-E4B rises from 12.27% to 85.12%; InternVL3.5-8B rises from 58.35% to 81.42%. Every model improves in every held-out condition (New Scenes, Theme, App, Resolution).

Gains are largest where desktop scenes are hardest. As the number of applications on screen grows from one to four or more, fine-tuned Qwen loses only a few points while its base model and the strongest reference model, UI-Venus-2-9B, degrade steadily; the gap to the base widens from 9.7 to 14.3 points. The gap likewise grows from 10.9 to 13.7 points as the target's visible-area loss rises to 35%.

Transfer to five external benchmarks is uniform. All four models improve on ScreenSpot-Pro, ScreenSpot-v2, OSWorld-G, UI-Vision, and MMBench-GUI. Qwen gains 11.51 percentage points on ScreenSpot-Pro (30.17 to 41.68) and 10.11 points on OSWorld-G (51.24 to 61.35). Gemma4-E4B's mean over the five benchmarks increases by 24.9 points; UI-R1-3B moves from 14.48% to 27.20% on ScreenSpot-Pro and from 39.87% to 47.14% in mean accuracy; InternVL3.5-8B shows a more modest mean gain of 1.51 points.

Transfer is not confined to desktop screens. On the ScreenSpot-v2 breakdown, Qwen and UI-R1 gain most on desktop interfaces (+14.37 and +5.69 points) but also improve on mobile (+10.77 and +0.60) and web (+6.41 and +0.91), even though DeskForge-1M contains no mobile screens.

Long-horizon task completion increases under a fixed planner. With a fixed Qwen3.6-27B planner, Qwen3.5-4B rises from 31 to 50 of 119 tasks on a custom WebArena-Infinity panel and from 3 to 15 of 100 tasks on OpenApps. Gemma4-E4B shows the largest Infinity increase, from 5 to 40 tasks, and rises from 2 to 12 on OpenApps. InternVL3.5-8B gains seven tasks in each environment (37 to 44, and 4 to 11), and UI-R1-3B gains seven Infinity tasks and five OpenApps tasks (30 to 37, and 7 to 12).

Dense annotations support whole-screen element detection. An RT-DETRv4-L detector fine-tuned on the dense annotations outperforms the OmniParser v2 detector and ScreenParse YOLO11L on GroundCUA across three localization metrics, reaching 62.17% group F1 compared with 58.55% and 57.59%.

Annotation quality was audited. A human audit found 99.8% of sampled element annotations correct, and 97.8% of uniformly sampled instructions judged sound; automated audits excluded the 4.7% of captures with inconsistent or implausibly sparse annotations.

The corpus is dense and cluttered. Observations contain 132 annotated elements on average (median 121; truncated at 400 in the figure, maximum 1,161), 90.9% record at least two applications, and 97.7% contain occluded elements, with more than 30% of elements affected in nearly half. Element counts are about 7.6 times that of the largest resource in the paper's comparison table.

Methodology in Plain English

The researchers built a Linux desktop environment in which real applications run in isolated sessions (via Xvfb, a private D-Bus session, xfwm4, a MATE panel, and Caja desktop components). Each scene is defined by a specification that picks applications and their initial states, prepares their content, and sets layout and stacking, appearance, and resolution. Because accessibility APIs alone are application-dependent and often incomplete, the annotation pipeline reconciles three sources: screenshots, accessibility observations, and measured window geometry. It produces element records with geometry, text, type, window ownership, and interaction properties, and — importantly — stores both a full element extent and its visible region as a union of rectangles, so a partly covered control keeps only the exposed portion as a valid click area. Accessibility names are kept separate from displayed text through character geometry and pixel checks, and selected window controls missing from accessibility trees are recovered from window geometry.

The environment then explores scenes by sampling actionable elements and clicking them within their estimated visible regions, recording whether the interface changed or not. Each transition links a before and after observation with the executed action and an aggregate effect summary. To build grounding examples, a vision-language model is prompted with the before and after observations plus target context to write a single-step instruction; instructions are filtered to describe a goal identifiable from the before screenshot alone, without coordinates or annotation markers. The successor observation informs instruction construction but is not given to the grounding model.

For evaluation, splits are assigned at the scene level so interaction frames stay together. Four held-out conditions test new scenes with represented attributes, held-out applications (GNOME System Monitor, Pluma, Xarchiver), a reserved appearance preset (Quartz Night Nord), and a reserved resolution (2880×1800). Four vision-language models — Qwen3.5-4B, Gemma4-E4B IT, InternVL3.5-8B, and UI-R1-3B — are fine-tuned with LoRA on 200,000 single-target grounding examples (sampled from 551,651 training-split instructions), using eight NVIDIA H100 GPUs for one epoch with an effective global batch size of 64, holding prompts, preprocessing, decoding, and scoring fixed within each base/fine-tuned pair.

Why This Matters

Impact on research. The paper reframes desktop data collection from recording to generating: because the environment composes applications in controlled scenes, the factors that make grounding hard — number of applications, occlusion, theme, resolution — become deliberate variables rather than accidents of a crawl. Its comparison table argues that among the resources it lists, only DeskForge combines dense labels, multi-application scenes, a configurable environment, visibility geometry, and recorded transitions. It also shows that a GUI-specialized checkpoint (UI-R1-3B) benefits, not just general-purpose VLMs. DeskForge-1M is usable without executing the environment, since screenshots, annotations, and transitions are stored, so training and held-out evaluation require no virtual machines, application installation, or environment resets.

Real-world applications:

• Training the visual action-execution component of computer-use agents that operate real desktop software (browsers, editors, file managers, and desktop utilities).
• Screen parsing: a detector trained on the dense annotations localizes every visible element on a screen, a component that complements instruction-conditioned grounding.
• Accessibility-aware interface understanding: visible-region geometry, occlusion state, and separated accessible names versus rendered text support click targeting that avoids covered or non-visible controls.
• Evaluation of agent reliability across desktop configurations: the held-out theme, resolution, and application splits provide a way to test generalization rather than memorization.

Industry relevance. The acquisition pipeline is described as CPU-only and scales out without coordination across independent sessions; one campaign ran 300 jobs with 24 CPU cores each (7,200 cores) and five desktop sessions per job (1,500 concurrent sessions), measured at 84,807 observations per hour, or about 9.4 hours for 795,725 observations. That makes controllable desktop data generation a practical alternative to large-scale manual annotation for teams building desktop agents. The downstream result — better task completion with only the action model changed, under a fixed planner — maps onto production architectures that separate planning from grounding.

Future Directions

1. Extending beyond the Linux backend and accessibility-rich applications. Current acquisition uses a Linux backend and relies on applications whose accessibility trees expose the main interface content; broader application integration, native-platform backends, and complementary visual annotation would extend coverage to interfaces that expose less structure.

2. Learning from structure and state change, not just single targets. VLM training currently focuses on single-target grounding; learning jointly from interface structure and recorded state changes remains open.

3. Forward and inverse dynamics objectives. The retained transitions (917K recorded clicks) could support forward and inverse dynamics objectives for learning about action outcomes.

4. Goal-conditioned multi-application tasks with verification. Developing goal-conditioned multi-application tasks with state-based verification would extend DeskForge toward online training of long-horizon computer-use agents.

Target Audience

Researchers and engineers working on GUI grounding, screen parsing, and computer-use or browser agents; dataset builders interested in controllable, synthetic-to-real supervision pipelines; practitioners fine-tuning vision-language models for interface understanding; and accessibility or HCI researchers who care about visible-region geometry, occlusion, and reconciling accessibility metadata with what is actually rendered on screen.

Authors’ abstract

Computer-use agents need to reliably ground action targets in complex desktop scenes, where multiple applications, overlapping windows, and visually similar controls compete for attention. Existing training data rarely pair such scenes with dense annotations or vary them in a controlled way. We introduce DeskForge, a controllable desktop environment that composes and explores real applications to generate large-scale supervision for computer-use agents. It varies application states, content, window layout, appearance, and resolution, and fuses screenshots, accessibility trees, and window geometry into dense element annotations while recording the outcome of each executed action. Using this environment, we construct DeskForge-1M, a corpus of 1.2M annotated desktop observations containing 159.7M element instances. We fine-tune four vision-language models on 200K grounding examples drawn from DeskForge-1M. All four improve across held-out desktop conditions and on all five external GUI grounding benchmarks; for Qwen3.5-4B, accuracy increases by 11.51 percentage points on ScreenSpot-Pro and 10.11 points on OSWorld-G. The gains also translate to long-horizon task completion: under a fixed planner, the fine-tuned action models solve more WebArena-Infinity and OpenApps tasks, with Qwen3.5-4B increasing from 31 to 50 of 119 tasks and from 3 to 15 of 100 tasks, respectively. These results show that controllable composition of real desktop environments provides a scalable source of supervision for improving both GUI grounding and long-horizon computer use. The framework code, the dataset, and the fine-tuned model are available from the project page: https://saidgurbuz.github.io/deskforge/