The Pulse
Vercel Labs Opens Agent Desktop for Accessibility-Tree Control
Agent Desktop gives software agents a command-line and C-library route to operate desktop apps through operating-system accessibility APIs. Its 1.0 release added Linux support alongside macOS and Windows, while the project’s documentation s
AI.info Team · Written with AI · fact-checked ·
Agent Desktop reads the interface, not its pixels
Vercel Labs’ agent-desktop reached its 1.0 release on October 28, giving software agents a command-line route into desktop apps through operating-system accessibility APIs. Instead of interpreting a screenshot and estimating where a button sits, an agent receives a structured representation of the interface and acts on an identified element. The project is open source under the Apache-2.0 license.
The tool is a Rust command-line interface and C library for macOS, Windows and Linux. Its basic loop is to take a snapshot of an app, act on an element reference, then take another snapshot to check the result. The agent supplies the reasoning; agent-desktop provides the desktop controls and structured responses.
Snapshot references bind actions to what the agent saw
Each interactive element in a snapshot receives a reference, and actions use that reference rather than screen coordinates. References belong to a specific snapshot: if the interface changes, the tool can report a stale or ambiguous target instead of silently redirecting an action to a different element. Its responses include structured error codes and, in many cases, suggestions for what the agent should do next.
Ref-based actions run headless by default: they use accessibility APIs without moving the pointer or taking foreground focus. A separate --headed option allows physical input when needed. That distinction gives developers a choice between interacting with interface elements through their accessibility representation and using visible mouse or keyboard input.
One package spans three operating systems
Developers can install the tool with npm install -g agent-desktop; the package includes native binaries and platform helpers for x64 and ARM64. The 1.0 release brought Linux support to the integrated CLI, C library and npm package, using AT-SPI on Linux systems with glibc 2.35 or later. The project’s latest listed release, version 1.0.2, followed on October 30.
Setup differs by platform. macOS requires Accessibility permission, while Linux use depends on the user’s graphical session and at-spi2-core; on Windows, the tool works with apps running at the terminal’s integrity level without an additional grant. The documentation says Agent Desktop does not automate web pages and directs users to agent-browser for those.
Accessibility trees set the boundary—and expose private data
Agent Desktop cannot operate every interface simply because it runs on a supported operating system. The project documentation says it cannot see a user interface that publishes no accessibility tree, and its platform table lists differences in mouse, window and screenshot support. A screenshot command exists, but the normal observe-and-act workflow does not depend on screenshots or pixel matching.
The same access that makes desktop automation possible can expose sensitive material. Vercel Labs warns that snapshots, clipboard reads, screenshots, command output and debug artifacts can contain private desktop content, and advises against putting them in public logs or uploads. An optional JEV helper can send screen descriptions to a configured model provider; the repository describes it as separate from the native CLI.