ProBench: Benchmarking GUI Agents with Accurate Process Information

AI summary

Overview

Research area: Evaluation benchmarks for Graphical User Interface (GUI) agents — AI systems that operate real mobile apps on a user's behalf.

Technical level: Advanced. The paper assumes familiarity with multimodal LLMs, Android accessibility (a11y) trees, and agent evaluation pipelines.

Scope: ProBench is a mobile GUI agent benchmark of over 200 tasks across 34 bilingual (Chinese and English) online applications, distinguished by evaluating the process an agent follows, not only the final screen state.

What This Paper Is About

Existing GUI agent benchmarks usually decide success by looking only at the final screenshot. But GUI tasks are chains of steps, and critical information — such as whether the agent sorted by price or applied a filter — is often not visible on the last screen, so a correct-looking result can be reached by a wrong trajectory. ProBench addresses this by adding a second task type (Process-related Task) plus an automated "Process Provider" that captures intermediate operations, so agents can be graded on both the destination and the route.

Key Contributions

1. ProBench, a mobile benchmark of over 200 challenging GUI tasks spanning 34 mainstream Chinese and English online applications (14 English, 20 Chinese) across media, news, social, shopping, and other categories — built from seed tasks written by hand, expanded with Qwen3, and then human-filtered and edited.
2. A dual task typology. The benchmark retains State-related Tasks (judged only from the final screenshot) and adds Process-related Tasks, which require both the final state and critical intermediate steps such as applying a price filter or specifying a delivery address.
3. An automated evaluation pipeline with a Process Provider, containing two optional components — the Structure Description Converter (parses the a11y tree after each click and describes the clicked node via text, content_desc, resource_id, and child nodes) and the MLLM-based Summarizer (compares before/after screenshots around a click to produce a textual summary of the action).
4. A systematic evaluation and error analysis of proprietary, general open-source, and GUI-specific agents, exposing shared weaknesses in grounding, use of historical actions, and task planning.

Main Findings

• Even the best agent stays under 50%. Gemini 2.5 Pro ranks first with an average accuracy of 40.1% (English average 37.3%, Chinese average 41.5%). The conclusion states that the best-performing models successfully complete less than 50% of the tasks.
• Process-related Tasks are consistently harder than State-related Tasks. For Gemini 2.5 Pro, overall State-related accuracy is 45.6% versus 27.9% on Process-related; the same gap appears for every model that scored above zero.
• Only one model clears 60% on English State-related Tasks. Qwen2.5-VL-72B reaches 63.5% on English State-related Tasks and an English average of 53.3%, the highest in the table, but its Chinese average is 28.2%. Its overall average is 36.9%.
• Two proprietary models and one GUI-specific model score zero. GPT-4o, Claude 4 Sonnet, and UI-R1-E-3B each record 0.0 on every column. The paper attributes GPT-4o's and Claude 4 Sonnet's failures primarily to limited grounding capability, while UI-R1-E-3B is linked to the absence of COMPLETE action examples in its training phase, so it never emits a completion signal.
• Scale helps general open-source models. Qwen2.5-VL improves steadily from 7B (overall average 5.1%) to 32B (16.6%) to 72B (36.9%), with Qwen2.5-VL-72B outperforming every other model on English tasks.
• GUI-specific models generalize poorly. UI-TARS-1.5-7B beats its base model Qwen2.5-VL-7B on English tasks (11.4% versus 6.7% overall State-related) but still trails large general models. GUI-R1-3B has an overall average of 8.3% and scores 0.0 on both Process-related columns.
• Category-level weakness. Agents generally do better on production tools and system applications and struggle with social-networking and lifestyle applications, which refresh content frequently and present fragmented, icon-heavy, deeply nested layouts with sparse signals and many visual distractors. Per-category numeric accuracies appear only in Figure 5 and are not reported in the text.
• Evaluation pipeline is reliable against human labels. With Gemini 2.5 Pro used for both the Evaluator and the Summarizer, correctness versus human judgment is 96.0% for the Evaluator on State-related Tasks, 89.7% for the Evaluator plus Structure Description Converter on Process-related Tasks, and 94.1% for the Evaluator plus MLLM-based Summarizer on Process-related Tasks.
• Agents loop and fail to reflect. An early-stop mechanism marks a task failed in advance if the model outputs the same operation five consecutive times. Among failed tasks, uncompleted ratios were 90.0% (Gemini 2.5 Pro), 93.7% (Qwen2.5-VL-7B), 92.3% (32B), 71.5% (72B), 59.2% (InternVL3-8B), 98.0% (UI-TARS-1.5-7B), and 25.6% (GUI-R1-3B); among those uncompleted tasks, early-stop ratios were 49.6%, 63.7%, 67.1%, 50.0%, 77.9%, 43.8%, and 58.8% respectively. GPT-4o, Claude 4 Sonnet, and UI-R1-E-3B were excluded from this table because of their poor accuracy.
• Oversimplified planning. Agents often reduce a task to typing the whole instruction into a search box, skipping necessary subtasks such as locating the right mini-program and configuring conditions step by step.

Methodology in Plain English

ProBench connects to a real Android device through adbutils, an open-source Python Android Debug Bridge library. The system takes a live screenshot, sends it to the agent along with the task instruction and history of past actions, receives a predicted operation, converts it into a device command, and repeats until the agent signals completion or hits the limit of 15 interaction steps. The inherited action space from AITW covers CLICK, SWIPE, TYPE, ENTER, BACK, and COMPLETE; HOME is removed because all tasks stay inside one app, and a new WAIT action was added for slow network loading.

For grading, State-related Tasks are judged from the last screenshot alone. Process-related Tasks additionally need process information, which is supplied either by the Structure Description Converter — which walks the accessibility tree after each click, finds the smallest clickable node containing the click coordinates, and reads its text and content_desc, falling back to resource_id and child-node details when those are empty — or by the MLLM-based Summarizer, which stitches the before and after screenshots horizontally, marks the click coordinate, and asks the model to describe what the action did, outputting "Invalid click" when nothing changed. An evaluator model then combines the textualized action sequence with the final screenshot. Final outcomes are Uncompleted, Failure, or Success.

English apps were run on the Android emulator and Chinese apps on a physical Android phone equipped with AdbKeyboard for Chinese-character input. In-app histories were manually cleared before each run to guarantee a consistent starting state. Evaluation used the Structure Description Converter for process information and Gemini 2.5 Pro as the judger, with coordinates scaled to the original image size for models outputting 0–1000 ranges.

Why This Matters

The paper argues that judging GUI agents only by the final screen rewards lucky outcomes and hides whether an agent actually reasoned through a task, which makes reported progress misleading for anyone trying to build or choose a mobile agent.

Real-world applications:

• Personal assistants that must filter, sort, and configure settings correctly, not just land on a plausible screen — for example buying the cheapest item, which requires an actual low-to-high price sort.
• Shopping and booking flows where criteria like delivery address, room count, or date range only appear in the final state if intermediate filters were applied.
• Social and lifestyle apps such as messaging, delivery, and package tracking, identified as the weakest area for today's agents.
• Category and model selection for teams deciding whether to deploy a large generalist model or a smaller GUI-specific one.

Industry relevance: ProBench is positioned as an automatically verifiable, Chinese-and-English benchmark for third-party online apps, which the comparison table shows earlier benchmarks do not fully cover — AndroidArena, AndroidWorld, B-MoCA, and AndroidLab do not combine online third-party apps, Chinese apps, manual-verification-free evaluation, and automatic process capture, while SPA-BENCH and A3 address process information only partially. The work was supported by the National Natural Science Foundation of China (Grant No. 62372408) and the Ant Group Research Fund.

Future Directions

• Move beyond binary grading to capture degrees of task progress; the authors explicitly name this as a limitation of the current evaluation.
• Improve grounding capability in models without GUI-specific pretraining, since inaccurate element location blocked whole apps for some agents.
• Build agents that attend to their own history, so they recognize when an action has already taken effect instead of repeating it and triggering early stops.
• Develop hierarchical task decomposition so agents can break abstract instructions into subtasks and pursue them step by step rather than dumping everything into a search box.

Target Audience

Researchers and engineers working on multimodal LLM agents, mobile automation, and GUI grounding; benchmark designers who need an automatically verifiable protocol that includes intermediate process information; and practitioners evaluating whether a generalist or GUI-specific model is ready for real mobile tasks in Chinese or English apps.

Authors’ abstract

With the deep integration of artificial intelligence and interactive technology, Graphical User Interface (GUI) Agent, as the carrier connecting goal-oriented natural language and real-world devices, has received widespread attention from the community. Contemporary benchmarks aim to evaluate the comprehensive capabilities of GUI agents in GUI operation tasks, generally determining task completion solely by inspecting the final screen state. However, GUI operation tasks consist of multiple chained steps while not all critical information is presented in the final few pages. Although a few research has begun to incorporate intermediate steps into evaluation, accurately and automatically capturing this process information still remains an open challenge. To address this weakness, we introduce ProBench, a comprehensive mobile benchmark with over 200 challenging GUI tasks covering widely-used scenarios. Remaining the traditional State-related Task evaluation, we extend our dataset to include Process-related Task and design a specialized evaluation method. A newly introduced Process Provider automatically supplies accurate process information, enabling presice assessment of agent's performance. Our evaluation of advanced GUI agents reveals significant limitations for real-world GUI scenarios. These shortcomings are prevalent across diverse models, including both large-scale generalist models and smaller, GUI-specific models. A detailed error analysis further exposes several universal problems, outlining concrete directions for future improvements.