CUAWright: A Minimal Unified Interface for Digital Agents

AI summary

Overview

Research area: AI agents / computer-use agents (web, desktop, and CAD automation), with a focus on agent harness design rather than model training.

Technical level: Intermediate. Readers should be comfortable with the idea of an agent loop (observe → act → observe), command-line tooling, and benchmark scores, but no deep machine-learning background is required.

Scope: The paper introduces CUAWright, a roughly 3K-line terminal-only harness that gives a computer-use agent bash as its sole action interface and a persistent file system as its workspace, and evaluates it against GUI-native and hybrid CLI baselines on six benchmarks spanning web, desktop, and computer-aided design tasks.

What This Paper Is About

Most computer-use agents pair a model with a domain-specific "harness" — a browser or desktop environment exposing a fixed set of human-designed tools and screenshot observations decided before the task starts. The authors argue this prevents capable coding models from operating on system state programmatically and from building their own tools mid-task. Their goal is to show that a minimal interface — terminal commands only, plus a file system the agent can write to — works better and costs less than GUI-native or hybrid harnesses across many digital task domains.

Key Contributions

1. A minimal terminal-native harness (CUAWright). Every model action is a terminal command; the same action space is held fixed across web, desktop, and CAD domains. Web tasks are driven by agent-written code against the browser debugging protocol, desktop tasks through CLI tools or direct file editing, and GUI input (mouse/keyboard via xdotool/pyautogui) is used only as a fallback when a surface genuinely blocks programmatic control.

2. Vision on demand instead of by default. Rather than appending a screenshot at every step, the default observation is terminal output; the agent must call a screenshot CLI tool with a save path and then an image-loading tool to bring visual information into context.

3. Persistent file system as external memory and tool-construction substrate. There is no separate memory interface — remembering, logging, and building reusable programs are all ordinary terminal actions on the workspace, and files survive context compaction.

4. Broad empirical evaluation plus ablations. Six benchmarks (Online-Mind2Web, Odysseys, OSWorld 2.0, WeaveBench, BenchCAD, CadGenBench), single- versus multi-agent ablations, a file-system ablation, and a qualitative analysis of agent-created tools and failure modes.

Main Findings

• Web, live sites (Online-Mind2Web, 300 tasks across 136 websites): With GPT-5.4, CUAWright reaches 88.1% overall versus 83.4% for the screenshot-based GUI baseline (+4.7), with the largest gain on hard tasks (88.8 versus 76.6). With GPT-5.5 it reaches 89.4% (+6.0 over the GUI baseline).

• Long-horizon web (Odysseys, 200 tasks): CUAWright with GPT-5.4 achieves 77.5% success with an average of 76.1 steps, versus 33.5% for the GUI baseline (+44.0). With GPT-5.5 it reaches 88.0% (+54.5).

• Desktop (OSWorld 2.0, 105 of 108 public tasks): The authors exclude tasks 026 and 041 (unavailable GitLab server) and 082 (nested Docker). With GPT-5.5 xhigh, CUAWright scores 63.2% partial reward versus 47.5% for the official baseline (+15.7), with 25.7% versus 13.9% strict completion, at $10.1 versus $16.1 per task. With GPT-5.6 Sol Max it scores 67.9% versus 62.7% (+5.2) at $4.4 versus $13.9 per task.

• Headline framing: The abstract reports a 33.2% relative improvement in partial reward and a 37.5% estimated cost reduction on OSWorld 2.0 against the published GPT-5.5 baseline. The paper also states the design costs roughly one-third of conventional interactive paradigms on OSWorld 2.

• Multi-agent does not help: Adding planner and self-verification sub-agents as CLI tools produced 60.4% (GPT-5.5) and 67.3% (GPT-5.6 Sol) versus 63.2% and 67.9% for the single agent. The paper attributes this to planners struggling to write accurate success criteria and frontier models already self-verifying; multi-agent also raised median turns from 73 to 137 (GPT-5.5) and from 101 to 151 (GPT-5.6 Sol).

• The file system is essential: Blocking programmatic file access dropped GPT-5.6 Sol from 67.9% to 55.5% partial reward — the paper cites a 12.1-point decrease.

• Long-horizon desktop/apps (WeaveBench, 114 tasks, GPT-5.5 high, 200-step limit): CUAWright passes 61 of 114 tasks, improving PassRate from 35.1% (Codex CLI) to 53.5% and overall score from 49.9 to 66.4; OpenClaw scores 33.3/46.6 and Hermes 31.6/46.6.

• CAD from images (BenchCAD, 212-sample subset of 17,900: 72 easy, 72 medium, 68 hard): With tool access, CUAWright reaches 79.0% mean IoU versus 55.8% reported officially for GPT-5.5 (+23.2). The reproduced no-tool score is 43.8% versus 44.4% officially reported, which the authors use to argue their subset is comparable or slightly harder.

• CAD with a tool server (CadGenBench, all 81 public fixtures): Using the same Build123d kernel as ordinary shell commands instead of registered MCP tools gives 49.4% versus 45.7% overall with GPT-5.5 (+3.7) and 55.7% versus 53.2% with GPT-5.6 Sol (+2.5). The abstract summarizes CAD gains as 8.1%–41.6% relative improvement over other CLI-based harnesses.

• Agents spontaneously build tools: In OSWorld 2.0 runs, GPT-5.6 Sol explicitly created at least one saved code/executable file in 89 tasks (84.8%), producing 776 executable files of which 192 (24.7%) were reused across at least two command events. Created tools clustered around inspection/extraction, interface automation, artifact construction, and verification.

• Terminal-only tasks outperform mixed ones: On OSWorld 2.0 with GPT-5.6 Sol Max, the 30 terminal-only tasks (28.6%) averaged 71.89% partial reward and 33.3% exact completion (10/30), versus 66.32% and 25.3% (19/75) for the 75 mixed GUI+terminal tasks (71.4%).

• Failure profiles differ by model: GPT-5.5 most often misread the task (19.4%), while GPT-5.6 Sol more often produced outputs that looked correct but lacked precision (19.0%), placed correct content in the wrong location (10.5% versus 3.9%), or left part of the task incomplete (7.6% versus 5.8%); unchecked assumptions occurred at about 5.8% for both.

Methodology in Plain English

The authors formalize the task as a partially observable decision process where the state splits into application state and a persistent file-system state. At each step the agent sees terminal output (optionally an image it explicitly loaded) and emits one executable shell command, which changes the environment and produces the next observation. Three auxiliary Python tools are provided in the workspace (image_show.py, ask_user.py, submit.py), and the agent runs them like any other command.

For web benchmarks, the workspace runs on the host with a per-command timeout, observations truncated at 24,000 characters, transcript compaction every 20 steps, and a 100-step cap. For desktop tasks, each task runs in a container-hosted Ubuntu VM from the OSWorld 2.0 environment, with commands capped at 300 seconds, outputs capped at 16,000 characters (kept as head and tail around a truncation notice), and a 300-step total cap across roles. Browser control uses the Chrome DevTools Protocol at 127.0.0.1:1337, with accessibility-tree inspection preferred over screenshots.

To test whether the file system matters, they ran a version prompted to avoid filesystem writes through run_command, forcing changes through GUI controls. To test multi-agent value, they installed a planner agent and a self-verification agent as callable CLI tools inside the container. They then manually reviewed traces and ran a failure-mode classification, and they illustrate behavior with detailed case studies, including OSWorld task 008 (a reimbursement claim reconciled across mail and banking apps using two agent-written scripts called 13 times), task 070 (an AWS billing fix using a 37-line cdp_click.py on top of a 115-line CDP helper, invoked 15 more times, scoring 1.0), and a WeaveBench Jaeger latency-diagnosis task that scored 0.93875.

Why This Matters

The paper argues that digital environments are far more programmable than their graphical interfaces suggest, so the interface design — not just the model — is a major lever on agent performance and cost. If a single bash action space transfers across live web, desktop software, and CAD kernels, harness builders can stop hand-crafting domain-specific tool sets for every new environment.

Real-world applications:

• Enterprise back-office automation: reconciling expenses, cross-checking receipts against bank records, and filing claims across multiple business web apps (demonstrated in OSWorld task 008).
• IT and cloud operations: investigating billing anomalies, removing runaway cloud resources, revoking access keys, and posting incident updates in team chat (task 070).
• Observability and incident response: exporting distributed traces, computing latency percentiles, capturing packets, and writing a root-cause report with a reproducible evidence bundle (WeaveBench).
• Engineering design workflows: reconstructing 3D solids from orthographic views and iteratively generating CAD programs, where the harness reaches 79.0% mean IoU on BenchCAD.

Industry relevance: the OSWorld 2.0 results pair higher scores with lower per-task API cost ($10.1 vs $16.1 for GPT-5.5 xhigh; $4.4 vs $13.9 for GPT-5.6 Sol Max), which matters directly for anyone pricing agent deployments at scale. The finding that frontier models already write and reuse their own tools suggests harness complexity — and the engineering cost of maintaining it — can be reduced rather than increased.

Future Directions

• Safe state modification. The authors flag that direct writes to system state risk unintended or irreversible side effects, and explicitly leave transactional execution, validation, sandboxing, permission control, and automatic rollback to future work.
• Rethinking multi-agent design. Since planner/verifier sub-agents did not help here but helped in GUI-native settings, an open question is what division of labor (if any) suits terminal-native harnesses.
• Closing residual failure modes. Precision failures (correct-looking but imprecise output), wrong-location artifacts, and incomplete tasks remain the dominant errors for the strongest models evaluated; how much can be fixed by verification prompting versus interface design is unresolved.
• Extending the single-action-space claim. The paper tests web, desktop, and CAD; whether one terminal interface generalizes to settings not reducible to programmatic APIs — and how far agents get without any OS-level input primitives — is left open.

Target Audience

Agent researchers and harness/infrastructure engineers building computer-use systems will get the most from this paper, particularly those deciding between GUI-centric, hybrid, and code-first designs. It is also useful for practitioners deploying agents on desktop or web workflows who care about cost per task, and for benchmark designers interested in how harness choice confounds reported model scores. Readers wanting implementation specifics for a new domain should note the paper focuses on interaction and execution under controlled settings and does not provide safety mechanisms for production state changes.

Authors’ abstract

The prevailing approach to computer-use agents couples a model with a domain-specific harness: a browser or desktop environment equipped with human engineered tools that are fixed before task execution. As models' coding capabilities improve, the GUI native and static harness prevents them from direct programmatic operation on system state, as well as flexible construction of tools. To this end, we introduce CUAWright, a minimal terminal harness of roughly 3K lines of code that uses bash commands as its sole action interface, and a file system as its evolvable space for dynamically creating tools and managing the context. We conduct comprehensive experiments across a wide range of digital tasks, and demonstrate that by giving the agent a minimal, programmable interface, it achieves substantially stronger results compared to their GUI or hybrid CLI interface across a wide range of tasks. On OSWorld 2.0, CUAWright delivers a 33.2% relative improvement in partial reward while reducing estimated cost by 37.5% compared with the published GPT-5.5 baseline. On Online-Mind2Web and the long horizon Odysseys benchmark, CUAWright substantially outperforms GUI native harness by 4.7% and 44.0% in success rate, respectively. Furthermore, we found the gains extend to CAD applications that require accurate visual understanding and CLI interaction: on CADGenBench and BenchCAD, our unified harness yields 8.1%-41.6% relative improvements over other CLI based harnesses with GPT-5.5. Together, these results suggest digital environments are far more programmable than their GUI interfaces imply, and a minimal terminal-focused harness is the key for better performance and efficiency.