VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics

AI summary

Overview

Research area: Human-Computer Interaction / mobile GUI agents (vision-language-model-driven agents that operate smartphone interfaces).

Technical level: Intermediate. The paper is a benchmark-and-evaluation study; it assumes familiarity with vision-language models, agent loops, and success-rate metrics, but contains little mathematical machinery beyond counting metrics.

Scope: This paper introduces VenusBench-Mobile, an online benchmark of 149 user-intent-driven tasks plus 80 stability variations across 27 Android apps, annotated with a five-dimension capability taxonomy (PUDAM) for diagnosing why mobile GUI agents fail.

What This Paper Is About

Existing online benchmarks for mobile GUI agents are built "bottom-up" from app functionality, producing narrow, homogeneous task distributions that do not match how people actually use phones, and they report only coarse success rates that cannot explain failures. The authors build VenusBench-Mobile to fix both problems: a "top-down" task design derived from real user intents, and a capability-oriented annotation scheme that attributes failures to specific underlying abilities. The goal is to measure how close current agents are to reliable real-world deployment, and to show where they break.

Key Contributions

1. A user-intent-driven online benchmark. VenusBench-Mobile organizes evaluation around 10 major user-intent categories (Function Assistance, Conflict, Vague Instructions, Multi-round Interaction, GUI State Awareness, Visual GUI Manipulation, Hard GUI Browsing, Noise Resistance, Browsecomp-like, and Stability Evaluation), yielding a primary pool of 149 tasks across 27 open-source Android apps — the 20 apps from AndroidWorld's suite plus 7 newly added apps.

2. A capability-oriented diagnostic annotation scheme. Every task is annotated along five dimensions — Perception (P), Understanding (U), Decision (D), Action (A), and Memory (M) — each split into four progressive proficiency levels (Level 1 basic functional automation through Level 4 sophisticated embodied intelligence), so failures can be attributed to specific deficiencies rather than reported as one aggregate number.

3. Systematic environment variations for stability testing. A dedicated Stability Evaluation subset takes 20 representative tasks and generates four variants each (Chinese translation, semantically equivalent English "Variation," Dark theme, and Tablet/Pad at 2560×1600 in landscape), producing 80 additional instances and 100 total instances in that subset.

4. An extended evaluation infrastructure. Built on AndroidWorld's Android emulator platform, the framework adds task-specific initialization and verification functions (programmatic OS state inspection or MLLM-based judgment), dynamic noise injection for four noise types, and multi-round dialogue support for follow-up instructions and user clarifications.

Main Findings

• Large performance drop versus prior benchmarks. Compared with AndroidWorld, state-of-the-art agents' average success rates decrease by around 50 points. On the 149-task primary pool, the strongest agent (Gemini-3-Pro + UI-Venus-72B) reaches a total success rate of only 36.9%, GPT-5.1 + UI-Venus-72B reaches 26.9%, and most open-source models fall below 15%. The reported average across all evaluated agents is 15.2%.

• Blind spots in earlier benchmarks. GUI State Awareness and Visual GUI Manipulation appear in no prior benchmark in the paper's comparison table, and they are the hardest categories for current agents, with average success rates of only 2.5% (GSA) and 4.2% (GUIM).

• High discriminative power. On the Noise Resistance category, Gemini-3-Pro reaches 75.0% success while Qwen3-VL-8B drops to 6.3%, showing the benchmark separates robust general-purpose agents from weaker ones.

• Hierarchical capabilities are validated. Capability-level accuracy falls consistently from Basic (L1-2) to Advanced (L3-4) tiers across essentially all dimensions; average Perception, for example, drops from 17.5% to 10.3%.

• Decision and Perception collapse under complexity. Most open-source models drop to single digits (under 5%) in L3-4 Decision and Perception, which the authors attribute to missing autonomous reflection and dynamic temporal perception.

• Memory is the absolute bottleneck. Performance in Memory shows the steepest drop at advanced levels. Smaller open-source models such as UI-Venus-7B and GUI-Owl-7B fall to near-zero success rates at L3-4, which the authors argue cannot be fixed by simply enlarging the context window.

• Near-zero stability. Under the Stability Pass Rate (requiring success on all five variants of a task), most models score 0% and the best model reaches only 15%. Tablet/Pad mode causes the largest drop, with many open-source agents near zero, indicating dependence on vertical phone layouts; language variation exposes uneven multilingual robustness.

• Agent-framework overhead is severe. The multi-agent framework MA3 (GUI-Owl-7B) consumes 1,640.0K total tokens at 438.7 tokens per step, versus 373.7K / 146.5 for the standalone GUI-Owl-7B base model. GPT-5.1 is the most efficient, at 167.5K total and 54.6 per step; UI-Venus-72B uses 850.0K total, nearly double UI-Venus-7B's 447.4K.

• Per-category success rates are highly uneven. Averages across agents are 27.3% (FA), 4.0% (CF), 23.5% (VA), 2.5% (MR), 2.5% (GSA), 4.2% (GUIM), 13.2% (HGB), 32.1% (NR), and 15.0% (BC).

Methodology in Plain English

The authors start from user needs rather than from apps. They first enumerate the kinds of things people actually ask a phone assistant to do — including asking how an app works, being told their instruction contradicts the screen, giving underspecified requests, interrupting mid-task, reacting to changing on-screen state, drawing or marking on images, browsing across many pages and formats, and coping with notifications or crashes — and turn those into task categories.

They then build the tasks on top of AndroidWorld, the widely used dynamic Android emulator platform, adding 7 apps so the suite covers more GUI diversity. Each task gets an initialization function that sets a consistent starting state (importing files, configuring settings) and a verification function that checks completion either by inspecting OS state programmatically or, for tasks needing visual or semantic judgment, by an MLLM judge.

Separately, each task is labeled along the five PUDAM dimensions at a proficiency level, which lets the authors compute accuracy per dimension and per difficulty tier rather than a single pass rate. For stability, they take 20 representative tasks and produce four systematic variants each. For cost, they record output token counts rather than wall-clock time, arguing tokens are hardware-independent and map directly to API billing.

Finally, they run a purely vision-based evaluation — the environment supplies only screenshots and agents emit GUI actions step by step — over open-source GUI-specialized models (UI-Venus-72B, UI-Venus-7B, GUI-Owl-7B, MAI-UI-8B), general-purpose VLMs (Qwen3-VL-30B-A3B, Qwen3-VL-8B), one multi-agent framework (Mobile-Agent-v3 with GUI-Owl-7B), and two closed-source models (Gemini-3-Pro, GPT-5.1) used in a planner–executor setup with UI-Venus-72B as the grounding executor.

Why This Matters

Impact on research. The paper argues that a benchmark's usefulness depends on two questions — what to evaluate and how to evaluate — and that prior online benchmarks answer both poorly for general-purpose assistants. By adding categories absent from earlier suites and by replacing single success rates with capability-level diagnosis, it gives researchers a way to tell why an agent failed and where to invest next, at a point when aggregate scores on older benchmarks are saturating.

Real-world applications.

• Everyday phone assistance, such as explaining how to accomplish something in an unfamiliar app or discovering whether an app supports a feature.
• Cross-app information gathering and comparison, including browsing videos, PDFs, images, non-searchable app GUIs, and HTML pages to synthesize an answer.
• Handling interruptions during automation — incoming calls, app crashes, operation failures, and unrelated popups — without losing the original task.
• Fine-grained visual tasks on device, such as drawing a specific shape or marking content in an image with a colored pen.

Industry relevance. The authors are from the Venus Team at Ant Group, and the paper explicitly links evaluation to deployment concerns: commercial systems such as Doubao AI Smartphone and AutoGLM indicate practical demand for end-to-end task automation. Token cost is framed as an economic and edge-deployment constraint — the 438.7 per-step tokens of an agentic framework versus 146.5 for its base model is presented as a barrier for real-time use on devices with limited battery and bandwidth. The near-zero Stability Pass Rate is framed as a direct warning that current agents are not ready for reliable deployment.

Future Directions

1. Online learning as a standalone dimension. The authors describe their current coverage of online learning — HTML tasks with randomized button functionalities — as rudimentary, and call for richer randomization covering page transition logic and navigation depth, plus evaluation of both intra-task adaptation and inter-task self-evolution.

2. Lifelong assistants. Current benchmarks, including this one, evaluate isolated task episodes. The paper proposes extending evaluation over long periods to assess long-term memory, user-preference accumulation, and personalization across app ecosystems.

3. Agent-simulated users. Because manual task design limits benchmark scale, the authors suggest using LLM-based agents to simulate realistic user behavior and automatically generate diverse instructions, multi-round dialogues, and environmental noise, with automated validation preserving quality.

4. Dedicated state-tracking mechanisms. The finding that Memory is the dominant bottleneck and that larger context windows alone are insufficient raises the open question of what architectural mechanism — beyond context length — could let agents aggregate cross-screen information and maintain context over long sequences.

Target Audience

Researchers and engineers building or evaluating mobile GUI agents and vision-language-model agents will get the most from this paper, particularly those who need fine-grained failure diagnosis rather than a single score. Benchmark designers will find the user-intent taxonomy and the PUDAM annotation scheme directly reusable. Product and applied-ML teams assessing whether an agent is ready for real deployment will benefit from the stability and token-cost results. Readers without an agent background can follow the findings, but the paper is best suited to those already familiar with online GUI benchmarks such as AndroidWorld.

Authors’ abstract

Existing online benchmarks for mobile GUI agents remain largely app-centric and task-homogeneous, failing to reflect the diversity and instability of real-world mobile usage. To this end, we introduce VenusBench-Mobile, a challenging online benchmark for evaluating general-purpose mobile GUI agents under realistic, user-centric conditions. VenusBench-Mobile builds two core evaluation pillars: defining what to evaluate via user-intent-driven task design that reflects real mobile usage, and how to evaluate through a capability-oriented annotation scheme for fine-grained agent behavior analysis. Extensive evaluation of state-of-the-art mobile GUI agents reveals large performance gaps relative to prior benchmarks, indicating that VenusBench-Mobile poses substantially more challenging and realistic tasks and that current agents remain far from reliable real-world deployment. Diagnostic analysis further shows that failures are dominated by deficiencies in perception and memory, which are largely obscured by coarse-grained evaluations. Moreover, even the strongest agents exhibit near-zero success under environment variations, highlighting their brittleness in realistic settings. Based on these insights, we believe VenusBench-Mobile provides an important stepping stone toward robust real-world deployment of mobile GUI agents. Code and data are available at https://github.com/inclusionAI/UI-Venus/tree/VenusBench-Mobile.