The Pulse
OSWorld-Pro catches computer agents failing before the finish
A new arXiv benchmark evaluates computer-use agents through the sequence of actions leading to a result, exposing failures that final-state scoring can hide.
AI.info Team ·
Computer-use agents can receive the same failing score whether they break on the first click or make a mistake late in a long workflow. A new benchmark called OSWorld-Pro is designed to show that difference.
The paper, OSWorld-Pro: Process-based Evaluation for Computer Use Agents, was submitted to arXiv on September 21, 2026. Its author list includes Zhilin Wang, Shaokun Zhang, Yifan Zhang and nine other authors. The work examines how computer-use agents perform across a sequence of actions instead of judging only the final file or application state.
“Evaluation of Computer-Use Agents (CUAs) is often limited to the final deliverables they create (at the end of hundreds of steps) and assessed with functional verifiers, as seen in OSWorld.”
Zhilin Wang, Shaokun Zhang, Yifan Zhang and nine co-authors, authors of the paper
Why the final result can hide failure
OSWorld-Pro is presented as a procedural evaluation for computer-use agents. It contains more than 300 tasks and more than 2,800 subgoals, supported by more than 67,000 human annotations. The subgoals are sequentially dependent, allowing the benchmark to track the progress an agent makes through a task rather than recording only whether it eventually reaches the intended end state.
That structure can distinguish between different kinds of failure. An agent that makes a keyboard-input error may need a different remedy from one that cannot provide a precise click on a graphical interface. A final-state score can conceal that distinction when both systems fail to produce the requested result.
The benchmark uses large-language-model judges aligned with human assessments to evaluate whether agents fulfill individual subgoals. This gives researchers a way to examine where an agent loses progress during a workflow and which actions appear connected to the failure.
A harder test for computer-use systems
The paper says OSWorld-Pro is challenging even for state-of-the-art large language models. Its abstract reports that Claude Opus 5 achieves 75.7% on OSWorld-Pro, compared with 83.4% on OSWorld. The comparison suggests that agents can perform worse when evaluation follows the process through which they complete a task, rather than focusing only on the final deliverable.
OSWorld-Pro is described as a companion to OSWorld, an evaluation setting centered on the end state produced by an agent. By adding subgoal-level assessment, the new benchmark is intended to provide more information about how a system behaves before it reaches—or fails to reach—the final result.
Process-focused failure modes
The authors identify subgoal-irrelevant actions and click-based mistakes as critical failure modes. These categories point to problems that may not be visible in an outcome-only evaluation. An agent can continue issuing actions that appear reasonable in isolation while no longer contributing to the current subgoal.
Click-based mistakes can also change the visible state of a graphical interface, leaving the agent to operate from an unintended screen or selection. When a benchmark records the sequence of subgoals and actions, researchers can study that progression rather than treating the entire attempt as a single success or failure.
OSWorld-Pro does not replace final-outcome testing. It adds a process-focused view of computer-use performance, giving developers a way to examine not only whether an agent completed a task but also how its progress changed along the way.