Hybrid Computer Use: A Unified Programmatic, Semantic, and Visual Interface for General Agents

AI summary

Overview

Research area: Human-Computer Interaction / computer-use agents (LLM agents that operate desktop software), with connections to accessibility APIs, agent-computer interface design, and benchmark evaluation.

Technical level: Intermediate. Readers should know roughly what an LLM agent loop, a benchmark score, and a screenshot-based GUI agent are; no deep systems knowledge is required to follow the argument.

Scope (one sentence): The paper proposes Hybrid Computer Use (HCU), an interface that gives a general agent three channels — code/shell, accessibility (AX), and visual GUI — under model-directed control, and evaluates it on OSWorld 2.0, OSWorld 1.0, WeaveBench, MacAgentBench, and Windows Agent Arena.

What This Paper Is About

Most computer-use agents see the screen as an image and act with a mouse and keyboard, which forces an LLM to imitate human interaction even when the operating system already exposes structured accessibility data or lets it run code directly. HCU instead exposes all three interaction channels to the same general agent and lets the model decide which to use, or how to mix them, on a per-step basis. The goal is to raise both task success and interaction efficiency (steps, tokens, time, and cost) without retraining the underlying model.

Key Contributions

1. A unified three-channel interface. HCU combines the agent's own programmatic tools (code, shell, files) with semantic AX tools and visual GUI tools, the latter two exposed via the Model Context Protocol (MCP). It adds no planner, subagents, or extra loop of its own, so it attaches to existing agent harnesses.

2. A redesigned AX channel built for execution, not just observation. AX tools return compact views with target references bound before view compression, execute several actions plus optional wait steps in one call, and report per-target effect summaries and AX diffs as post-action feedback — so fewer round trips are needed.

3. Cross-channel action guards plus a usage skill. Since all channels act on one desktop, an action in one channel can invalidate evidence held by another. An AX target guard re-resolves each reference against live elements before input, and a GUI layout guard compares the current desktop layout to the screenshot the coordinates came from; a supplementary HCU skill explains channel choice, efficient composition, and recovery.

4. Evidence that the interface, not just the model, moves the frontier. HCU achieves the highest Partial and Binary scores on OSWorld 2.0, and beats the CUA Driver baseline across four backbone models on OSWorld 1.0 and across Linux, macOS, and Windows in the appendix.

Main Findings

• Best reported OSWorld 2.0 scores. HCU reaches 72.37% Partial with GPT-6 Astra and 35.19% Binary with Claude Opus 5 — the highest Partial and Binary among the full-set results compared. Only StateAct is cheaper than HCU with GPT-6 Astra (about $7.80 vs. $10.03 per task) but at 61.60% Partial; the two form the performance–efficiency frontier. The 72.60% Partial listed for the GPT-6 Astra computer-use loop is OpenAI's offline-subset result and is marked unranked.

• Gains over the matched baseline on OSWorld 2.0. Versus CUA Driver with GPT-6 Astra, HCU is +3.55 points Partial and +0.90 Binary with 29.81% fewer tool calls at nearly identical Cost ($10.03 vs. $10.05). With Claude Opus 5 it improves Partial by 3.75 points and Binary by 3.76 points over that model's official leaderboard result, at 20.26% lower Cost.

• Better on every measure in the GPT-5.4 ablation (OSWorld 1.0). HCU scores 65.59% vs. 62.30% for CUA Driver — 3.29 points higher — while using 29.80% fewer Steps, 56.85% fewer input tokens, 29.85% less Time, and 55.36% lower Cost.

• AX is the largest single contributor. Adding AX to the programmatic tools and GUI channel raises Score by 2.79 points and cuts Steps by 14.08%, input tokens by 10.44%, Time by 12.58%, and Cost by 10.64%. Removing GUI leaves Score within run-to-run variation (66.08%) but increases Steps by 12.45%, so GUI mainly buys efficiency when pixels are needed. Removing the skill drops Score to 64.00 and raises Steps to 33.47; removing AX drops Score to 62.79 and raises Steps to 33.28.

• The skill sharply reduces invocation errors. Rejected AX and GUI calls for malformed arguments fall from 17.37% to 1.36%, and task-runs that repeat such rejections on the same tool while varying arguments fall from 44.69% to 0.37%. With the skill, 87% of AX action calls omit the observe field (selecting the default AX diff) versus 0.2% without it, and GUI input actions drop from 4.96 to 1.97 per task.

• Gains transfer across models, with different channel habits. On OSWorld 1.0, HCU improves GPT-5.4 by 3.29 points (62.30 → 65.59), Claude Opus 5.5 by 2.94 (68.68 → 71.62), Gemini 3.8 Flash by 12.90 (54.67 → 67.57), and Grok 4.7 by 6.81 (67.90 → 74.71), using 18–57% fewer input tokens at 8–55% lower Cost. Within HCU's AX/GUI calls, AX accounts for 80% for GPT, 68% for Grok, and 51% for Gemini, while Claude uses GUI for 94% and batches more (58% of its acting calls contain two or more actions, vs. 11–21% for the others).

• Observation and execution split across channels. For GPT, Gemini, and Grok, on 29–31% of the tasks where each used GUI, all GUI calls were screenshots while actions went through AX — the interface lets a screenshot supply context for a subsequent semantic action rather than forcing one channel for both.

• Gains concentrate on tasks that use the desktop tools. Nearly all of the Score gain over CUA Driver comes from tasks where HCU makes at least one AX or GUI call; the group without such calls contributes only −0.7 to +0.1 points. On tasks where both configurations invoke desktop tools, HCU uses 19–40% fewer Steps.

• Overall Step counts can hide channel effects. Claude's overall Steps are nearly unchanged even though its Score rises: on tasks where only HCU invokes desktop tools it gains 7.6 points in Score with 53% more Steps, which largely offsets savings elsewhere.

• Cross-platform appendix results (released HCU, GPT-5.4). OSWorld 1.0 (360 tasks): 70.13% Score vs. 62.47% for CUA Driver (+7.66 points), with 38.6% fewer Steps, 40.7% less Time, 67.0% fewer input tokens, and 59.1% lower Cost. WeaveBench (114 tasks): 53.25% vs. 46.37% for CUA Driver and 43.84% for official OpenClaw, with 24 vs. 12 tasks scoring at least 0.8 and $1.889 vs. $3.518 per task. MacAgentBench UI-dependent subset (28 tasks): 25/28 (89.3%) vs. 24/28 (85.7%), at 51.4% lower Cost. Windows Agent Arena (111 tasks): 80.86% vs. 71.89% (+8.97 points, 88 vs. 78 full-score tasks), at 50.6% lower total Cost.

Methodology in Plain English

The researchers started from an observation about how agents are usually wired: they get a screenshot, guess coordinates, and click. That design throws away two things a real computer offers — a structured description of every control (accessibility trees) and the ability to just run code. HCU hands all three to the model at once.

The engineering work is mostly about making the semantic channel fast and safe to use. Instead of dumping a huge accessibility tree, HCU strips structural wrappers and condenses repeated content into a compact view, but binds each target reference to a real element before compressing, so shortening the view never changes what a reference points to. The agent can then send a batch of actions — plus wait steps that hold until an expected state appears — in a single call, and gets back a summary of what changed plus an AX diff, so it often doesn't need a separate observation round.

Because all channels share one desktop, stale evidence is the main hazard: an AX action might open a dialog over coordinates from an old screenshot. Two guards run immediately before any input. The AX guard re-resolves each target against live state and rejects the action (rather than redirecting it to a similarly named element) if identity or action-specific requirements don't match. The GUI guard compares current desktop bounds, visible windows, stacking order, active window, and target-point layout against those recorded with the screenshot the coordinates came from, blocking input and returning a fresh screenshot if anything moved.

Finally, a written "skill" — distilled from recurring mistakes in development trajectories — tells the model what each channel is good for, how to batch and continue from feedback, and how to recover. Evaluation is comparative: the same harness, the same programmatic tools, and the same model, varying only the interface (HCU vs. CUA Driver), measured not just on score but on steps, tokens, wall-clock time, and dollar cost.

Why This Matters

Impact on research. The paper makes an interface-design argument rather than a model-capability one: a human-like screenshot-and-mouse action space may itself cap what a strong model can do. The finding that three of four models route most desktop calls through AX once it is available supports this, and the authors connect it to the "bitter lesson" — the capabilities HCU exposes already exist on standard desktops and need no retraining. The tool-call-level analysis also gives a template for comparing interfaces rather than just leaderboard numbers.

Real-world applications:

• Long-horizon professional workflows (the OSWorld 2.0 target), where sustained progress across many operations makes per-step efficiency the binding constraint.
• Enterprise desktop automation on managed machines, where AX-addressed actions are more reliable than pixel coordinates and where token and dollar cost decide whether a deployment is viable.
• Cross-platform agent products (Windows, macOS, Linux), since the same public AX and GUI contract is backed by native backends (UIA3 and Win32 on Windows, native macOS backends) and the appendix reports gains on all three.
• Assistive-technology-adjacent tooling: HCU builds on accessibility APIs originally created for assistive technologies, so improvements in programmatic AX control also serve automation that targets the same control trees.

Industry relevance. Cost figures in the paper are per task and often roughly halved versus the baseline (for example $0.325 vs. $0.664 on Windows Agent Arena, and $1.889 vs. $3.518 on WeaveBench). For anyone shipping agents at volume, that is the difference between a demo and a product, and the paper is co-authored by Microsoft Research with contributors from Fudan University and Tsinghua University.

Future Directions

• Treating the desktop as a programmable environment. The authors note that AX exposes element states code can read and act on, and that batched actions with wait steps provide a substrate for folding desktop work into code execution and programmatic tool calling — an open design direction rather than a finished result.
• Why models divide the channels so differently. Claude routes 94% of its desktop calls through GUI while GPT sends 80% through AX. Understanding whether this reflects training, harness conventions, or task mix — and whether it can be steered — is unresolved.
• Interface design as a first-class variable. The paper shows interface choice changes score and efficiency at fixed model and harness; extending that comparison to more interfaces and more benchmarks would test how general the finding is.
• Closing the gap to the unranked upper reference. The 72.60% Partial listed for the GPT-6 Astra computer-use loop is an offline-subset result, so how much of the remaining headroom is interface versus model is not established.

Target Audience

Researchers and engineers building computer-use or desktop-automation agents, especially those designing agent-computer interfaces, tool schemas, or MCP servers rather than training models. It is also useful for accessibility-API practitioners interested in non-assistive uses of AX trees, and for product teams evaluating whether screenshot-based agents are cost-competitive with structured automation. Readers focused on model training, or seeking a from-scratch introduction to agent benchmarks, will find less here.

Authors’ abstract

Computer-use agents increasingly tackle tasks ranging from everyday desktop activities to long-horizon professional workflows. Yet interfaces centered on screenshots and mouse-and-keyboard actions can impose a human-interaction prior, requiring repeated screen interpretation while leaving the computer's semantic and programmatic capabilities underexposed. We argue for exposing these capabilities and letting models choose and combine them. We introduce Hybrid Computer Use (HCU), a unified interface for general agents that combines three complementary channels: programmatic interaction through code and command-line tools, semantic interaction through accessibility (AX), and visual interaction through graphical user interfaces (GUI). Our HCU supports efficient AX interaction and guards against stale-state actions during channel switching. A supplementary HCU skill guides agents in selecting and coordinating channels to use the unified interface effectively. Our designed AX tools return compact observations, execute actions in batches, and report their effects as AX diffs, so the model needs fewer round trips. On OSWorld 2.0, HCU achieves the highest Partial and Binary scores among the compared systems. Analyzing agent trajectories, we find that all evaluated models improve with HCU yet allocate work across the channels differently and consistently outperform compared baselines in terms of both efficiency and task success, highlighting the value of unified computer-use interfaces that allow flexible channel selection and combination.