OpenGCUA: Toward a Fully Open Stack for Generalist Computer Use

AI summary

Overview

Research area: Computer-use agents (CUAs) / agent systems — spanning GUI control, code execution, and real-device deployment (submitted under Computer Vision, arXiv:2610.22200v1, 15 Oct 2026).

Technical level: Intermediate. Readers should be comfortable with agent loops, tool-calling harnesses, and benchmark metrics such as success rate and steps per task; no deep model-training background is required, since this release contains no trained model.

One-sentence scope: An open-source project report that defines "generalist computer-use agents" (GCUAs), proposes a code-execution-plus-harness-to-model-decision-transfer design, and evaluates it across eight frontier models on an 82-task OSWorld-v2 offline subset and four real macOS tasks.

What This Paper Is About

Frontier computer-use systems such as GPT-6 Astra's are closed: their weights, prompts, real-device harnesses, SDKs, and trajectories are unavailable, so the community cannot reproduce or compare design choices. The authors argue that computer use has shifted from "operating a GUI" to "completing work on a real computer by whatever means works" — screenshots, accessibility trees, shell commands, and APIs — and that the model should decide which interface to use at each step. OpenGCUA is an ongoing project to open that full stack, and this report is its first release: an agent design, an evaluation, execution trajectories from eight frontier models, and preliminary real-device harness and cross-device SDK code.

Key Contributions

1. A decision-based definition of GCUA. Rather than defining an agent by its components, the paper defines a GCUA by three decisions the model makes at each step — what to observe, how to act, and how to organize its work — and shows that GUI agents, CLI agents, multi-agent hybrids, and the authors' earlier Qwen-UI-Agent (GUI+CLI, batched) are each special cases that fix some of these decisions in advance.
2. Code execution over GUI and CLI, with evidence it beats batching. The agent writes one Python program per step combining PyAutoGUI desktop actions, a bash() function for shell commands, and (in the real-device harness) Playwright — allowing conditions, loops, and verification inside a single model call. The authors identify five recurring program patterns in GPT-6 Astra trajectories.
3. "Harness-to-model decision transfer." Selected workflow rules fixed by the harness — when to take a screenshot, how to track task state, stating a program's purpose — are handed to the model, while the harness retains execution, record-keeping, and context retention (rolling screenshot buffer, micro-compaction).
4. An open release: implementation, prompts, trajectories, harness, and SDK. The release includes code and prompts, evaluation trajectories from all eight models, and preliminary versions of a real-device harness (for users' own computers) and a cross-device SDK giving one programming interface across computers, mobiles, and browsers.

Main Findings

• Every tested model improved. On the 82-task OSWorld-v2 (v2.1) offline subset, the full OpenGCUA configuration raised the cross-model mean partial score from 45.3 to 64.9 (+19.6 points) and cut steps per task by an average of 53.0% relative to a GUI-only baseline that already supported batched actions. Full-task success increased for every model.
• Largest gains came from weaker baselines, but the strongest model gained too. GLM-5.3-Flash gained 35.2 points and DeepSeek-V4.1-Flash 29.8. GPT-6 Astra, the highest baseline scorer, went from 67.1 to 75.7 while dropping from 102.1 to 33.5 steps per task (−67.2%).
• The model spread narrowed. The paper states the score gap between the strongest and weakest models was narrowed by more than half.
• Fewer steps does not imply fewer output tokens. Output tokens per task fell for six of eight models, averaging a 27.2% relative reduction (Claude Opus 5: 378.3K → 89.3K; Kimi K3: 302.5K → 139.6K), but GPT-6 Astra used roughly the same output-token budget despite its large step reduction.
• Code execution adds value beyond shell access. With the same GUI and shell access, writing GUI operations as code and combining them with programmatic operations added 6.1 points to mean partial score and cut steps per task by 14.8% against the GUI+CLI baseline. For Qwen3.8-Max specifically, the reported improvement over the batched GUI+CLI hybrid was 5.2 points and 17.0% fewer steps.
• On-demand observation improves caching at no score cost. Letting the model decide when to request a screenshot raised cache hit rates by 10.5 and 14.4 percentage points for two models without lowering task scores.
• On real devices, the gap lies in the model, not only the harness. With the harness held fixed across four real macOS tasks, Qwen3.8-Max used 19.4 times as many reasoning tokens as GPT-6 Astra and 5.7 times as many output tokens in total — yet completed fewer tasks. Case studies attribute this to heavy reasoning on simple operations, difficulty adapting to real-device conditions, and weaker code-execution ability.
• Five recurring code-execution patterns were identified: cross-interface composition, loop-based execution, result-dependent execution, observation preparation (e.g., assembling 25 views into a contact sheet), and executable verification (e.g., checking that 64 untouched package entries remain unchanged).

Methodology in Plain English

The authors did not train a model for this release. Instead they built a harness — prompts, tool interfaces, and an execution runtime — that lets existing frontier models behave as GCUAs, then measured what changed.

The model is asked to emit one Python program per step inside a <code> block. That program can click with PyAutoGUI, run shell commands through bash(), compute in Python, and (on real devices) drive a browser with Playwright. The harness syntax-checks and executes the program and returns standard output, standard error, and exit status. Because the program itself can loop, branch, and assert, intermediate results can steer the next operation without another model call — the key contrast with batched actions, described in the paper as "a program without control flow."

Two additional mechanisms move decisions from harness rules to the model: a screenshot() function the model calls only when it wants a new image, and a todo interface (todo.update() / todo.list()) where the model records progress with statuses distinguishing pending, attempted, unverified, and claimed-verified work. The model also writes an <intent> line before each program stating what it will change and preserve. The harness keeps fixed policies only for context housekeeping — a rolling screenshot buffer and "micro-compaction" of older text outputs.

Evaluation followed the GPT-6 Astra release's protocol on the 82-task offline OSWorld-v2 subset, which excludes 26 tasks needing internet access. Eight models were run: Claude Opus 5, GPT-6 Astra, Gemini 3.8 Flash, Qwen3.8-Max, Qwen3.8-Flash, Kimi K3, DeepSeek-V4.1-Flash, and GLM-5.3-Flash. Metrics were binary success and partial-credit score, plus efficiency measured as steps per task, output tokens, reasoning tokens, and cost computed from published prices including cache discounts.

Why This Matters

Impact on research. The paper reframes computer-use evaluation away from "can the model click correctly" toward "can the system choose the right interface and organize its own work," and supplies a formal notation (Eq. 1) that lets competing agent families be compared on exactly which decisions they leave to the model. Releasing trajectories from eight frontier models — including GPT-6 Astra and Claude Opus 5 — gives the community data it otherwise cannot obtain from closed systems, and the finding that on-device cost differences are 19.4x between models makes execution efficiency a first-class research target rather than an afterthought.

Real-world applications:

• Delegating multi-step office work (spreadsheets, email drafts, slide decks) where the agent must mix clicking with file and command-line operations.
• Cross-device workflows, via the SDK's common interface across computers, browsers, and phones.
• Assistive automation for users who cannot perform GUI operations themselves, using accessibility-tree actions that need no pointer movement.
• Enterprise automation on users' own machines, where the harness must coexist with a human user and act on background windows.

Industry relevance. The paper documents a vendor-level shift already underway: OpenAI's computer-use guidance now recommends writing and running short programs with PyAutoGUI or Playwright rather than relying on the earlier mouse-and-keyboard tool, and OpenAI reports GPT-6 Astra finishing OSWorld 2.0 tasks in about 47% less time than GPT-5.6 Sol and being 1.9x faster on Mind2Web. OpenGCUA gives organizations outside those vendors a reproducible reference design, with the explicit efficiency metrics (steps, tokens, cost) that determine whether delegation actually saves time versus doing the task manually.

Future Directions

• Train the model. This release contains no trained model; the stated roadmap item is a model that unifies computer and browser use with code execution and harness-to-model decision transfer built in.
• Push more harness decisions into the model. The authors frame decision transfer as progressive — the current implementation moves screenshot requests and task-state tracking, and they envision shifting further workflow control as models improve.
• Close the real-device gap. The 19.4x reasoning-token difference and the failures traced to adaptation and code-execution weakness define concrete targets, as does the observation that screenshot-only feedback makes result-dependent control harder (motivating accessibility trees as structured, program-readable state).
• Mature the harness and SDK, and build real-device evaluation. Both are explicitly preliminary; the paper calls for evaluation environments and benchmarks on real devices, particularly macOS, where shared control, background windows, and evolving requests are not captured by existing benchmarks.

Note: the provided paper text is truncated partway through Table 3 (the full GCUA-versus-GUI+CLI ablation), so the per-model values for that table beyond Qwen3.8-Max, and any cost figures, are not reported here.

Target Audience

Agent researchers and engineers building or evaluating computer-use systems; teams deciding between GUI-only, batched GUI+CLI, and code-execution agent architectures; and practitioners deploying agents on real user machines who need to reason about token cost, caching, and safe shared control. Readers new to the area will find the Section 2 taxonomy (GUI vs. CLI vs. multi-agent hybrid vs. GCUA) a useful orientation, while benchmark-level results sit in Section 3.3.

Authors’ abstract

Computer use has reached a turning point: the focus is expanding from GUI-only interaction to systems that complete tasks on users' computers through visual and programmatic interfaces. We define a generalist computer-use agent (GCUA) as a system in which the model chooses how to observe, act, and organize its work across visual and programmatic interfaces. Yet frontier implementations, such as GPT-6 Astra's computer-use system, remain largely closed, limiting systematic study of their design. We introduce Open Generalist CUA (OpenGCUA), an ongoing project to develop and open the full stack for GCUAs with the community. This report presents the first release of OpenGCUA. Code execution is a key feature of our agent design: it lets intermediate results determine subsequent operations and enables models to apply their existing programming capabilities directly to computer interaction. We also propose harness-to-model decision transfer, moving selected observations and task-state decisions from fixed harness rules to the model. On OSWorld-v2, our full configuration improves task scores across all eight evaluated models, including GPT-6 Astra and Claude Opus 5, raising mean partial score by +19.6 points and reducing reported steps per task by an average of 53.0% relative to batched GUI-only execution. We release our implementation and prompts, evaluation trajectories from all eight models, and preliminary versions of a real-device harness and a cross-device SDK. The harness addresses real-device challenges not fully captured by benchmarks, while the SDK provides a common programming interface across computers, mobiles and browsers. Across four real macOS tasks under our real-device harness, selected Qwen3.8-Max runs use 19.4 times as many reported reasoning tokens as GPT-6 Astra, highlighting substantial differences between models in real-device execution efficiency.