Skip to content
AI.info

AI agents

Computer Use and GUI Control

Design multimodal agents that operate graphical interfaces while managing grounding errors and application boundaries.

By the end you can

Screenshots are the least structured integration an agent has

Computer-use agents act through screenshots, accessibility APIs, keyboard, pointer, clipboard, and application state. What they meet there is noisy visual grounding, hidden windows, focus changes, modal dialogs, and cross-application side effects.

How noisy is the grounding? ScreenSpot-Pro isolates that one step and measures nothing else. Not planning. Not a multi-step task. Only whether a model can put the pointer on the element a real screen actually contains. The screenshots are authentic, high-resolution and professional, spanning 23 applications, five industries and three operating systems. The 2025 paper reports the outcome plainly: “Existing GUI grounding models perform poorly on this dataset, with the best model achieving only 18.9%.”

A cascaded visual search lifted that to 48.1 percent, with no additional training. Read the number from the other side. The pixels were never sufficient on their own. Most of the recovered ground came from searching for the target rather than from looking harder at it.

GUI action is often the least structured integration an agent has. So before a sensitive action the runtime should preview it, identify the target semantically, check the application against an allowlist, and verify the result afterwards.

A pointer event carries no record of what it hit — and on authentic professional screens the best existing grounding model hit the intended target 18.9 percent of the time.

Visual

Ground the screen semantically before any input lands

Screen state has to be grounded in something semantic before an input action is safe. The application boundary decides where that action lands. Boundary and verification need separate owners and separate tests.

The middle station of that chain is not an aspiration. It is specified. Roles and properties have a published mapping onto the desktop accessibility APIs: MSAA with IAccessible2 1.3, User Interface Automation, Linux/GNOME ATK and AT-SPI, and the macOS Accessibility Protocol. W3C made those Core Accessibility API Mappings 1.1 a Recommendation in 2017. Microsoft describes the Windows layer directly: “Microsoft UI Automation is an accessibility framework that enables Windows applications to provide and consume programmatic information about user interfaces (UIs). It provides programmatic access to most UI elements on the desktop.”

Read that sentence for the word “most”. Programmatic identity exists for most elements, not all. That is exactly why grounding, boundary and verification stay three separate stations rather than one. OSWorld frames the same split as accessibility-tree versus screenshot-only agent input. The screenshot-only path carries no identity at all.

FigureProcess · 5 steps
  1. 1

    Screen state

    Pixels, windows, focus, dialogs, and visible application context.

  2. 2

    Semantic grounding

    Roles, labels, paths, and object identity behind the visual target.

  3. 3

    Input action

    Pointer, keyboard, clipboard, drag, or system command.

  4. 4

    Application boundary

    Allowed programs, destinations, and data flows.

  5. 5

    Verification

    Read back the resulting object, state, or receipt.

Example

A shipped desktop agent whose malicious payload was stopped by a syntax error

A desktop coding agent that millions of developers run inside their editor shipped them malicious code automatically. The agent was the Amazon Q Developer extension for Visual Studio Code. An inappropriately scoped GitHub token in its CodeBuild configuration let a threat actor commit code, and that code went out in release 1.84.0. AWS pulled 1.84.0 from distribution and released 1.85.0. Its security bulletin is dated 23 July 2025 and was updated two days later.

Then the sentence to sit with, the one about why nothing burned down: “AWS Security has inspected the code and determined the malicious code was distributed with the extension but was unsuccessful in executing due to a syntax error.” No allowlist stopped it. No commit preview stopped it. No verification step read back what had changed. A defect in the attacker’s own code stopped it. A defect in an attacker’s code is not a control you can put in a design document.

  • Decision at stake: whether the agent’s own distribution path counts as part of its blast radius. Release 1.84.0 reached desktops automatically, carrying instructions no user on those desktops had asked for.
  • Hidden assumption: that the code arriving at the desktop is the code the vendor wrote. What broke that was, in AWS’s description, an inappropriately scoped GitHub token in a build configuration.
  • Primary control question: what actually halted the action, and would it halt the next one? Here the answer was a syntax error, not an allowlist, a commit preview or a read-back.
  • Evidence to collect: the dated, versioned trail that lets an outsider check the story — bulletin AWS-2025-015 of 23 July 2025 updated 25 July, the withdrawn 1.84.0 and its SHA256, the 1.85.0 replacement, CVE-2025-8217 and GHSA-7g7f-ff96-5gcw of 26 July 2025.

Case

Anthropic’s own October 2024 computer-use numbers, and the benchmark under them

The public numbers are modest, and the vendor published them itself. Announcing computer use on 22 October 2024, Anthropic reported that “Claude 3.5 Sonnet scored 14.9% in the screenshot-only category—notably better than the next-best AI system’s score of 7.8%”. The same announcement called the capability “at times cumbersome and error-prone” and told developers to “begin exploration with low-risk tasks”.

What makes 14.9 legible is knowing what it was scored on. The benchmark is OSWorld, published in April 2024. It is an executable computer environment supporting Ubuntu, Windows and macOS. On top of it sit 369 real computer tasks, defined and executed on Ubuntu. A separate set of 43 Windows tasks has to be activated by the user, because of copyright. The abstract states the headline result: “While humans can accomplish over 72.36% of the tasks, the best model achieves only 12.24% success, primarily struggling with GUI grounding and operational knowledge.”

That final clause is the whole subject of this lesson. It was written by the people who built the measurement, not by anyone selling a product. The gap between 12.24 percent and over 72.36 percent is not a gap in ambition or in speed. It is grounding and operational knowledge: knowing what the thing on the screen is, and knowing what the application will do next.

Position

The vendor’s own number is the one a recording leaves out

Watching an agent drive a desktop is the most persuasive material this field produces and among the least informative. A recording is one run, on one machine, in one state. It carries no denominator. The announcement of 22 October 2024 is worth putting next to it, because Anthropic published both halves: 14.9 percent in the screenshot-only category of OSWorld against 7.8 percent for the next best system, alongside a warning that the capability is “at times cumbersome and error-prone”.

One vendor’s number invites the reply that the vendor chose the benchmark. So take a second one, on a different operating system, from the same season. Windows Agent Arena, built at Microsoft, adapted the OSWorld framework to a real Windows desktop in September 2024. The project page notes that “Our initial release consists of 154 diverse tasks”, spanning Office, browser, Windows settings and VS Code. Its purpose-built multimodal agent Navi did better than the OSWorld baseline and nowhere near a person: “Our agent achieves a success rate of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted human.” Two operating systems, two teams, the same shape of gap.

Then there is a fifth question, and it is the one nobody asks. Benchmarks are not fixed objects. OSWorld has been rewritten by its own maintainers. The OSWorld-Verified release of 28 July 2025 reports that XLANG Lab “collected, verified, validated, and fixed 300+ pieces of feedback, involving approximately two months of dedicated effort from a ~10-person team”. Epoch AI looked again three months later: “A major release in July, 2025 included updates to most task instructions and evaluation functions. Even since then, an additional 10% of task instructions have been changed.” Epoch calls that practice “highly atypical” for a benchmark that is not live. Two scores reported on the same benchmark on different dates may not be scores on the same tasks.

Ask for the benchmark, the category, the model, the date — and which revision of the benchmark; a recording carries none of the five.

Example

Operational signals for computer-use agents

Desktops interrupt constantly, so the first signal to collect is whether the agent survives window movement, focus changes, and modal interruptions. Whether sensitive files can move to unapproved destinations is a separate check, not something folded into that first signal. Two further signals only mean something after repeated or adversarial runs. And “adversarial” has a published shape now, not just a recommendation attached to it.

One vendor has reported the denominator with the result. Announcing the Claude for Chrome pilot on 25 August 2025, Anthropic wrote: “We conducted extensive adversarial prompt injection testing, evaluating 123 test cases representing 29 different attack scenarios. Browser use without our safety mitigations showed a 23.6% attack success rate when deliberately targeted by malicious actors.” With mitigations in autonomous mode, that fell to 11.2 percent. Note what 11.2 is. Not zero, published anyway, by the vendor, next to the number it improved on. That is the format to ask for: a count of cases, a count of scenarios, a before and an after. It is also the format a demonstration can never produce.

  • Signal 1: The agent survives window movement, focus changes, and modal interruptions.
  • Signal 2: Sensitive files cannot move to unapproved destinations — the check has to run on the destination, because the operator may never see the transfer.
  • Signal 3: Each action records application, target identity, and resulting state; a denominator like 123 test cases across 29 attack scenarios exists only because every run was recorded.
  • Signal 4: A failed visual grounding attempt stops before an irreversible operation — in the Amazon Q extension incident the irreversible operation was stopped by a syntax error instead, which is luck rather than a control.

Analogy

Remote Hands in a Data Center

Guiding remote hands through a video feed means confirming every rack, cable, and label before anything is unplugged. On a desktop the confirmation matters more, not less. A file can be revealed or duplicated with nothing on screen appearing to move.

GUI breadth increases the need for semantic identity and transfer controls.

Key idea

Cross-application workflows can create invisible data movement

Clipboard content, drag-and-drop, recent files, and autofill can cross trust zones without an obvious tool call. A model may not recognize that the destination is public or belongs to another account.

This hazard has a registered, graded instance. An AI command injection in Microsoft 365 Copilot “allows an unauthorized attacker to disclose information over a network”, in Microsoft’s own description. That is CVE-2025-32711, published 11 June 2025 under the title M365 Copilot Information Disclosure Vulnerability, and Microsoft rates it Critical with a base score of 9.3. Two fields of that score are the lesson. UI:N, meaning no user interaction was required. S:C, a scope change — the impact left the component that was attacked. NIST’s National Vulnerability Database attaches its own, lower primary score of 7.5 HIGH with S:U, so the Critical rating is the vendor’s own, not a scold from outside. Microsoft’s advisory adds: “This vulnerability has already been fully mitigated by Microsoft. There is no action for users of this service to take.”

The channels are as unremarkable as that suggests. The browser-specific attack types Anthropic red-teamed for Claude in Chrome included hidden malicious form fields in a webpage’s DOM, invisible to humans, and injections carried in URL text and tab titles that only an agent would ever read. On that challenge set the new mitigations moved the attack success rate from 35.7 percent to 0 percent.

Constrain applications and origins. Label sensitive data. Display exact source-to-destination transfers before commitment.

The transfers that leak are the ones nobody logged — and CVE-2025-32711 carries UI:N, meaning nobody had to touch anything at all.

Steps

Design a safe desktop task

Choose one desktop task that crosses two applications and design it to be safe end to end. A good design makes the data movement between those two applications visible instead of invisible. It also leaves a trace showing how the agent behaved when windows moved, focus changed, or a modal interruption arrived.

Read the five steps below against the Amazon Q Developer extension incident rather than against principle. Declaring allowed applications and origins treats a build pipeline and a distribution channel as part of the agent’s reach, which is where the inappropriately scoped token sat. Identifying objects semantically is the step a screenshot cannot supply and that User Interface Automation and the accessibility mappings can. Requiring commit previews is the step that nothing in release 1.84.0 performed. Verifying completion is the step AWS finally carried out by hand, after the fact, by inspecting the shipped code. Same work, done in the only order that leaves users exposed.

FigureProcess · 5 steps
  1. 1

    Declare allowed applications

    List programs, accounts, directories, and websites the run may access.

  2. 2

    Identify objects semantically

    Use paths, document IDs, owners, and labels rather than visual position.

  3. 3

    Control transfer channels

    Restrict clipboard, upload, download, and drag-and-drop.

  4. 4

    Require commit previews

    Show destination, payload, and consequence for sensitive actions.

  5. 5

    Verify completion

    Read back the saved file, sent message, or changed setting.

Reach for the desktop only when no typed interface exists

Computer use should be the fallback integration when a safer typed interface is unavailable. Broad compatibility does not make it the preferred execution path. The measured versions of that trade-off are already on record. Over 72.36 percent for humans against 12.24 percent for the best model in the OSWorld paper. 74.5 percent for an unassisted human against 19.5 percent for a purpose-built agent on Windows Agent Arena. And 18.9 percent for the best grounding model asked to do nothing but point at the right element.

Whoever operates these agents carries two standing duties. Watch for data that moves invisibly between applications. Report how the agent held up when the desktop interrupted it.

Choosing pixels over a typed interface buys reach and pays for it in every guarantee you would otherwise have about what the agent can touch.

Key takeaways