Are We There Yet? Assessing Computer-Use Agents for Blind Users' Accessible Interaction with Desktop Applications

AI summary

Overview

Research area: Human-Computer Interaction, specifically accessibility research at the intersection of computer-use agents (CUAs) and blind screen-reader users' desktop workflows.

Technical level: Intermediate. The paper combines a qualitative longitudinal field study with a quantitative cross-model evaluation of agent execution traces, so readers benefit from familiarity with both accessibility research methods and how GUI agents perceive and act on interfaces.

Scope: A three-week diary study with 8 blind screen-reader users and a cross-model replay evaluation of 1,258 participant-issued commands across 12 desktop applications, assessing how well five contemporary computer-use agents support nonvisual desktop interaction.

What This Paper Is About

Computer-use agents — systems that perceive graphical interfaces through screenshots and UI structure, reason over natural-language instructions, and click, type, and navigate on a user's behalf — are being positioned as a general-purpose paradigm for human-AI interaction. The open question is whether they actually work for blind screen-reader users, who interact with desktop software sequentially and by keyboard, in ways that conflict with GUIs built around spatial layouts. The authors built a screen-reader-accessible CUA prototype called OLLA, deployed it with blind participants during real desktop work, and then re-ran every command participants issued through four additional models to see where current agents succeed and where they break down.

Key Contributions

1. A human-centered dataset. 1,258 blind user-issued desktop commands collected during a three-week longitudinal deployment, paired with detailed CUA execution traces including screenshots, UI trees, model responses, generated actions, and interaction histories.

2. A human-centered empirical evaluation. An assessment of contemporary CUAs grounded in blind users' real-world nonvisual desktop workflows, identifying systematic failures in grounding, planning, constraint tracking, and interaction management across multiple large language models.

3. A characterization of beyond-automation needs. An analysis of how blind users envision CUAs as collaborative support — for adaptive guidance, interface learning, troubleshooting, and productivity assistance — rather than as autonomous task replacements.

4. An accessible CUA prototype. OLLA, a screen-reader-friendly interaction layer over existing CUAs that lets blind users issue commands, monitor execution progress, and review agent actions nonvisually, without modifying the underlying agent architecture or reasoning process.

Main Findings

• CUAs complete a meaningful but unreliable subset of tasks. Across five models evaluated on the same 1,258 participant-issued commands, success rates were: GPT-5 at 52.5% (95% CI: 49.8–55.3%), Claude Sonnet at 48.5% (45.7–51.3%), Gemini 2.5 CU at 43.9% (41.2–46.6%), UI-TARS at 39.8% (37.2–42.6%), and Qwen3-VL at 37.9% (35.3–40.6%).

• The top model's lead was not statistically significant. GPT-5's 4.0-percentage-point advantage over Claude Sonnet did not reach significance in a paired McNemar test (χ² = 3.25, p = .071).

• Partial completion was common and often substantial. Partial completion ranged from 33.3% to 34.9% across models. Among partial traces, GPT-5 completed 68.3% of required reference steps on average before breaking down, Claude 65.6%, Gemini 60.6%, UI-TARS 56.2%, and Qwen3-VL 53.0%.

• UI grounding and hidden-path discovery were the most frequent breakdowns. Ungrounded action generation accounted for 24.6% of GPT-5 failures (147/597, study only) and 22.6% of failures across all models (789/3,489). Hidden-path failures accounted for 21.4% of GPT-5 failures (128/597) and 20.7% across all models (722/3,489) — agents handled visible options but missed deeper variants, such as custom margins or a 15-by-15 table that required opening a dialog.

• Prior knowledge sometimes overrode what the UI actually showed. Learned procedure reliance accounted for 17.1% of GPT-5 failures (102/597) and 9.6% across all models (335/3,489); default collapse accounted for 11.4% (68/597) and 6.5% (227/3,489); structured intent failures (formulas, ranges, operators, grouping, ordering) accounted for 5.9% (35/597) and 5.7% (199/3,489).

• State maintenance broke down mid-task. Constraint binding failures accounted for 20.6% of GPT-5 partial cases (89/431) and 22.4% across all models (481/2,148); contextual state tracking for 10.2% of GPT-5 failures (61/597) and 8.7% across models (304/3,489); cross-application coordination for 7.4% (44/597) and 6.0% (209/3,489).

• Termination and final commit actions failed. Termination recognition — stopping too early, looping, or repeating actions without state change — accounted for 17.6% of GPT-5 partial cases (76/431) and 19.1% across all models (410/2,148). Other failures reached the right path but missed the commit point, such as selecting PDF export in Word without clicking the final Save button.

• Application context mattered. Performance varied by application, suggesting effectiveness depends not only on the model but on application structure, control visibility, and task type. The largest command volumes came from Word, Excel, PowerPoint, and OneNote.

• Reported success rates are conditional on what participants chose to attempt. Interviews indicated some participants avoided sensitive tasks such as banking, passwords, or personal documents, and stopped attempting certain task types after repeated failures — for example, avoiding editing tasks in OneNote after Word formatting failures.

• Participants wanted collaboration, not full autonomy. Envisioned roles clustered into understanding interfaces before acting, situated help and troubleshooting mid-task, user-controlled execution with confirmations before consequential actions, and learning and productivity support. One participant said, "I want it to ask me before clicking something important. I do not want it making decisions on its own."

Methodology in Plain English

The team ran an IRB-approved three-week diary study with 8 blind screen-reader users. Because existing CUAs typically require visual monitoring of screenshots, the researchers built OLLA, a screen-reader-accessible layer that sits on top of existing agents and lets users issue commands, follow execution progress, and review agent actions through audio — without changing the agent's reasoning. Participants installed it with a .exe installer and used it during their normal desktop activities. The deployed agent was GPT-5, held constant so participants had a consistent experience.

For each command, the system logged the user's command, UI tree, screenshot, model response, generated action, and interaction history. Participants issued 1,258 commands across 12 applications. The researchers manually grouped commands that represented the same underlying task, producing 304 normalized task intents, which they grouped into eight broader task categories. To compare agents under controlled conditions, they re-executed each participant-issued command in a fresh live application instance with Claude Sonnet, Gemini 2.5 CU, UI-TARS, and Qwen3-VL, holding the pipeline, system prompt, action schema, execution environment, and action executor constant.

Four human annotators independently performed each task to establish the minimum sequence of required reference steps, then labeled each model trace as success, partial completion, or failure, annotating applicable failure modes. Inter-annotator agreement was Krippendorff's α = 0.84, with disagreements resolved by consensus. Post-study interview data were analyzed using hybrid reflexive thematic analysis combining deductive codes from the research questions with inductive codes emerging from responses.

Why This Matters

This work shifts CUA evaluation away from well-formed benchmark prompts with controlled initial states and toward commands that blind users actually issue during everyday desktop work — a setting the paper argues existing benchmarks such as OSWorld, Windows Agent Arena, and WebArena do not capture. It provides the first human-centered, trace-level evidence of where assistive CUAs fail, and it reframes what "success" should mean by showing that blind users want clarification, explanation, recovery support, and confirmations alongside automation.

Real-world applications:

• Screen-reader productivity tools. Agents that explain unfamiliar dialogs, menus, and layout structure could reduce the effort blind users spend locating controls and understanding what changed after an action.
• Safe agentic delegation. Confirmation and approval mechanisms before hard-to-undo actions — settings changes, file modification, deletion, sensitive data — map directly to the concerns participants raised.
• Workflow learning. Agents that turn action plans into screen-reader-friendly step guidance could help users learn applications well enough to repeat tasks independently, rather than creating dependence.
• Spreadsheet and structured-content assistance. Formula, range, and operator errors point to a need for verification support in tools like Excel, where the paper notes blind users already face effortful verification when using generative AI.

Industry relevance is direct: the paper discusses OpenAI's Operator, Anthropic's Computer Use Tool, Microsoft Copilot, Google DeepMind's Project Astra, and Google's Gemini Spark as the emerging CUA ecosystem, and it evaluates models from OpenAI, Anthropic, Google, and open-weight families. Accessibility-focused prior work cited in the paper already shows CUA performance drops under assistive-technology interaction conditions, so the failure taxonomy here is relevant to any vendor shipping agentic desktop automation.

Future Directions

• Richer training and evaluation data. The authors propose extending their dataset with annotations for uncertainty, clarification opportunities, rejected actions, recovery attempts, and decision changes, and building semi-automatic tools to convert interaction logs into supervision for clarification, backtracking, and recovery.

• Reward-guided backtracking. Reward models for CUAs could combine task completion with signals such as goal progress, relevant-control discovery, repeated-action avoidance, constraint preservation, and recovery — plus nonvisual-specific signals like clarification, state explanation, and caution around actions that are hard to verify or undo.

• Assistive CUAs designed for learning. Systems could describe the screen, identify controls, suggest screen-reader-friendly steps, ask clarifying questions, and offer recovery options, turning action plans into guidance that keeps users in control.

• Broader generalization. The paper's stated limitations open questions about low-vision users and participants under 18, non-Windows platforms with different accessibility APIs and permission models, non-English and mixed-language use, dynamic web applications where DOM, accessibility-tree, and visual state may diverge, and evaluation of additional models beyond the five tested.

Target Audience

Accessibility and HCI researchers studying assistive technology and agentic interfaces; CUA and GUI-agent developers who need to know how their systems behave under nonvisual, real-world usage; screen-reader users and the organizations that support them; and product teams at companies building or integrating computer-use agents into desktop environments. Readers interested in benchmark design will also find value in the paper's argument that existing CUA benchmarks underrepresent everyday nonvisual workflows.

Authors’ abstract

Computer-use agents are emerging as a paradigm for agentic human-AI interaction, combining language reasoning with multi-modal interface grounding to operate GUIs. Yet their effectiveness for blind screen-reader users in real-world desktop workflows remains unclear. We present a three-week diary study with 8 blind users using OLLA, a screen-reader-accessible CUA prototype, collecting 1,258 commands across 12 applications with screenshots, UI trees, model responses, and action traces. We evaluate GPT-5 during deployment and re-execute the same commands with four additional models. GPT-5 achieved the highest success rate at 52.5%. Trace analysis reveals grounding, planning, constraint-tracking, and termination failures, while interviews reveal beyond-automation needs.