OSWorld-Pro: Process-based Evaluation for Computer Use Agents

AI summary

Overview

• Research area: Evaluation of Computer-Use Agents (CUAs) — agents that operate desktop graphical interfaces — with a focus on process-based rather than outcome-based benchmarking. The paper is labeled under Natural Language Processing (cs.CL) on arXiv.
• Technical level: Advanced.
• Scope: The paper introduces OSWorld-Pro, a benchmark of 305 long-horizon computer-use tasks broken into 2,814 sequentially dependent subgoals with 67,264 human-annotated step-level labels, plus an evaluation of LLM-Judges and of closed-source and open-weight CUAs on it.

What This Paper Is About

Existing CUA benchmarks such as OSWorld score agents only on the final deliverables they produce, so a trajectory that fails at the first subgoal and one that fails at the ninth both receive a zero. OSWorld-Pro addresses this by decomposing each task into progressive subgoals and annotating every step of an agent trajectory for which subgoal it targets, whether that subgoal is feasible, and whether the step makes progress or completes it. The goal is to reveal how and why agents fail, not just whether they finished.

Key Contributions

1. A process-based CUA benchmark. OSWorld-Pro contains 305 distinct tasks with 2,814 progressive subgoals and 67,264 human-annotated step-level labels, collected over more than 5,000 person-hours, with tasks averaging 9.2 sequentially dependent subgoals each.
2. Broader environment and application coverage. The benchmark spans 31 applications (vs 13 in OSWorld) and 19 Linux distributions plus 13 graphical interfaces, versus OSWorld's Ubuntu Jammy with GNOME only, organized into Diversity (117 tasks), Coordination (109 tasks) and Robustness (79 tasks) categories.
3. Human-aligned LLM-Judges for procedural evaluation. The paper evaluates several GPT-5.6 judge configurations, showing that GPT-5.6-Sol at max reasoning effort reaches 94.1% subgoal-level and 93.0% task-level agreement, approaching human performance of 95.6% and 96.0%.
4. Diagnostic failure-mode and efficiency analysis. The paper reports action-level progression likelihoods by action type (click, drag+move, keyboard, scroll, execution) and subgoal-level efficiency statistics such as successful effort, failure persistence and infeasible persistence.

Main Findings

• OSWorld-Pro is harder than OSWorld. Claude Opus 5 reaches only 75.7% on OSWorld-Pro versus 83.4% on OSWorld. The best overall result in the model comparison table is Claude Opus 4.8 Max at 77.7%.
• Open-weight models lag substantially. The top open-weight model, Qwen 3.8 Flash Next Xhigh (125B), reaches 55.1%, whereas open-weight models score above 80% on OSWorld. Minimax M3 Xhigh (428B) scores 28.9% on OSWorld-Pro but 75.2% on OSWorld.
• Category difficulty differs by model type. Among open-weight models, scores are highest on Diversity, then Coordination, and lowest on Robustness, indicating weakest generalization to non-default Linux distributions and graphical interfaces.
• Larger judges help, but with diminishing returns. Step-level judge agreement rises from 75.7 (GPT-5.6-Luna) to 80.9 (GPT-5.6-Terra) and then only to 82.0 (GPT-5.6-Sol) at max reasoning, while total cost first increases about 10x and then 1.6x.
• Judges are weakest on feasibility and progress. GPT-5.6-Sol Max scores 97.0 on identifying targeted subgoals (human: 98.4) and 94.3 on completion (human: 97.6), but only 61.9 on feasibility (human: 91.6) and 74.7 on progress (human: 94.1).
• Higher reasoning effort can reduce feasibility accuracy. The proportion of feasibility=No predictions falls from 9.7% for GPT-5.6-Sol with no reasoning to 5.1% at max reasoning, against a human ground-truth rate of 15.3%, suggesting models over-estimate feasibility as they reason more.
• Human annotations are reliable. Agreement between annotators and reviewers (Cohen's kappa) is 0.987 for subgoal targeted, 0.847 for feasibility, 0.869 for progression and 0.940 for completion, all above 0.8.
• Pointer actions separate weak from strong models. Progression likelihood on click operations is 39.0-41.9% for Minimax M3 and Kimi K3 versus 72.4-92.5% for stronger models. Minimax M3 frequently fails to click correct coordinates, such as missing a window's close button.
• Claude models lead on keyboard and scroll. Claude models score at least 89.2% on keyboard and at least 84.1% on scroll actions; no other model reaches 80% on either category. Claude Opus 4.8 also improves over Opus 4.7 on click (86.3 to 92.5%) and drag+move (41.0 to 74.3%).
• Strong models can be highly inefficient. Claude Opus 5 Max completed all subgoals but was stuck on subgoals without progress, including once for 59 steps. GPT-5.6-Sol and Kimi K3 average 4.9 and 5.1 steps per successful subgoal, versus 9.9 for Sonnet 5 and 21.2 for Minimax M3.
• Recognizing infeasibility varies enormously. Claude Opus 5 takes 9.5 steps per infeasible subgoal and GPT-5.6-Terra 12.1, while Sonnet 5 persists for 39.8.
• Parameter scaling is not a clear driver. Sonnet 5 outperforms Opus 5, and GPT-5.6 Luna outperforms Terra but not Sol. Within the Qwen family, Qwen 3.8 27B scores 32.1% versus 10.2% for Qwen 3.6 27B and 55.1% for Qwen 3.8 Flash Next (125B), and Qwen 3.5 122B scores 16.7%.
• Cost varies widely at similar performance. Per-task token cost is $0.51 for GPT-5.6 Luna Max versus $9.56 for GPT-5.6-Sol Max. Kimi K3 Max uses 125,870.7 output tokens in only 42.2 steps, which the authors attribute to multiple actions per step supported by the pyautogui library.

Methodology in Plain English

The authors selected tasks from the ProCUA-SFT generation approach, deliberately including rarely benchmarked applications (Diversity), tasks needing coordination across at least four applications (Coordination), and tasks in non-default Linux distributions and graphical interfaces (Robustness). They used DeepSeek-V4-Pro to decompose each overall goal into atomic subgoals plus the application each subgoal requires, then had human annotators validate tasks and delete any whose subgoals were not sequentially dependent.

Annotation proceeded in four stages: task validation, assignment to technical or non-technical annotator pools, step-level labeling, and independent plus interactive review. For each step of a pre-generated model trajectory, annotators marked which subgoal the step targets (almost always one, with under 2% of steps targeting two or more), whether that subgoal is feasible in that environment, and if feasible, whether the step makes progress and/or completes it. Infeasible subgoals were retained, following OSWorld. Reviewers first annotated each task independently, with the Super Annotate platform highlighting differences before they gave feedback. Annotators were barred from using LLMs, with compliance checks based on repetition patterns and annotation durations. Twenty-five annotators and reviewers from five countries participated; annotators spent between 5 and 20 hours per task.

To test whether LLM-Judges can replace expensive human evaluation, the authors compared judge labels against human labels using Macro-F1, considering each field only when its prerequisite field was correct (feasibility only if the targeted subgoals were right, progression only if feasibility was right, completion only if progression was right). Their final judge design evaluates an entire trajectory in a single API request, which requires payloads of up to 500 MBs and hundreds of screenshots; only OpenAI GPT-5.6 models handled this without payload, image-count or context-window errors. Model evaluation follows OSWorld's GitHub setup and only includes vision-language models with publicly available harnesses there, scoring the percentage of tasks on which all human-deemed-feasible subgoals are completed, using GPT-5.6-Sol Max as the judge.

Why This Matters

Process-based evaluation changes what can be measured about agents: partial credit for progress, identification of specific action types that fail, and quantification of wasted steps. Because CUAs are deployed to operate real desktop software, these diagnostics map directly to where engineering effort should go.

• Debugging agent design: Distinguishing keyboard-input failures from click-precision failures points to different fixes, such as coordinate grounding versus keyboard action handling.
• Cost and latency budgeting: Step counts, output tokens and per-task token cost let developers trade off model size, reasoning effort and price; the same benchmark can be judged at lower cost with a smaller judge model.
• Reinforcement learning and harness optimization: The paper notes subgoal progress signals could serve as process rewards or guide harness improvements.
• Generalization testing across environments: The Robustness category probes non-default Linux distributions and graphical interfaces, which matters for deployments outside Ubuntu/GNOME.

Industry relevance: The benchmark targets frontier model developers and agent platform teams; the paper reports results for 14 closed-source and open-weight model configurations, the top open-weight score of 55.1% leaves clear headroom for improvement, and the paper argues the benchmark is neither saturated nor overly difficult for open-weight models to hill-climb against.

Future Directions

• Using OSWorld-Pro subgoal signals as process rewards in reinforcement learning, or for harness optimization, which the conclusion explicitly leaves to future work.
• Closing the judge gap on feasibility and progression, the two fields where GPT-5.6-Sol Max scores 61.9 and 74.7 against human 91.6 and 94.1.
• Understanding and mitigating subgoal-irrelevant behavior and inefficiency in otherwise high-scoring models, such as Claude Opus 5 Max spending up to 59 steps stuck without progress.
• Improving open-weight generalization to different Linux distributions and graphical interfaces, where the Robustness category produced the lowest scores and one model (Qwen 3.6 27B) scored 0.0.

Target Audience

Agent and benchmark researchers, evaluation scientists at model laboratories, and engineers building computer-use agents who need failure diagnostics rather than a single success rate. The paper also suits readers interested in LLM-as-judge methodology, human annotation design, and the reliability limits of procedural evaluation. Prior familiarity with OSWorld and desktop GUI automation helps but the paper defines its terms and includes worked examples.

Authors’ abstract

Evaluation of Computer-Use Agents (CUAs) is often limited to the final deliverables they create (at the end of hundreds of steps) and assessed with functional verifiers, as seen in OSWorld. However, such evaluation of end-state performance lacks transparency into how and why agents fail in various tasks, obfuscating critical insight for subsequent improvement. For instance, agents that err during keyboard inputs would require a different mitigation strategy from those that fail to precisely provide click-based inputs on the graphical UI. We introduce OSWorld-Pro: a set of over 300 tasks containing over 2800 subgoals to enable the procedural evaluation of CUAs grounded in over 67,000 human annotations. We use robust human-aligned LLM-Judges to evaluate the fulfillment of OSWorld-Pro subgoals and thereby reveal the progress that models make throughout a series of sequentially dependent subgoals. Our findings reveal that OSWorld-Pro is challenging even for state-of-the-art LLMs, with top performers like Claude Opus 5 achieving only 75.7% vs. 83.4% on OSWorld. Furthermore, we identify critical process-focused failure modes of various models (e.g. subgoal-irrelevant actions and click-based mistakes) to provide insights to improve performance and efficiency of CUAs.