D-GARA: A Dynamic Benchmarking Framework for GUI Agent Robustness in Real-World Anomalies

AI summary

Overview

Research area: Artificial intelligence — GUI agents, multimodal large language models, and agent evaluation/benchmarking, with a focus on Android mobile interfaces.

Technical level: Intermediate. The paper assumes familiarity with GUI agents and vision-language models, but its central idea (testing agents against interruptions inside a live Android environment) is easy to grasp.

Scope: The paper introduces D-GARA, a dynamic benchmarking framework that injects realistic interruptions into live Android task execution in order to measure how well GUI agents hold up when the interface does not behave as expected.

What This Paper Is About

Existing benchmarks for GUI agents are mostly static and idealized: an agent is shown a sequence of screenshots and asked to predict the next action, so unpredictable real-world events such as permission dialogs, battery warnings, or app crashes never appear. Because real interfaces are dynamic and full of such interruptions, strong benchmark scores may not translate into practical reliability. D-GARA addresses this by running agents inside a live Android simulator, injecting context-aware anomalies mid-task, and judging success from the actual final UI state rather than the agent's self-reported "done" signal.

Key Contributions

1. Explicit treatment of interruptions during live GUI execution. The paper states it is the first to explicitly consider interruptions during real-world execution of GUI agents, and introduces D-GARA, a benchmarking framework that supports both dynamic evaluation and real-world data collection.

2. A real-world Android benchmark with embedded anomalies. Using the framework, the authors collect and annotate D-GARA-152, described as one of the first benchmarks built on commonly used applications with deliberately triggered real-world interruptions.

3. A state-centered success validator and a robustness metric. Unlike static evaluation, which checks whether a fixed action sequence was followed, D-GARA judges completion from the stabilized UI state against declarative goal rules, and adds Robust Success Rate (RSR) to isolate interruption robustness from raw task-solving ability.

4. Extensive robustness experiments on state-of-the-art agents and general MLLMs. The authors evaluate UI-TARS-1.5-72B, AgentCPM-GUI-8B, GPT-4o, Gemini-2.5, and Qwen2.5-VL-7B, and report that all of them degrade substantially under interruptions.

Main Findings

• A motivating gap at the outset: evaluating UI-TARS-72B and AgentCPM-GUI-8B showed a decline in task success rate under real-world anomalies, with performance degradation reaching up to 33%.

• Every evaluated model loses accuracy under interruptions. In Table 1, Gemini2.5-flash drops from 80.26% SR (no interruption) to 68.42% (with interruption), RSR 73.77%; GPT-4o from 69.08% to 60.53%, RSR 66.67%; Qwen2.5-VL-7B from 69.08% to 46.05%, RSR 53.33%; UI-TARS-1.5-72B from 50.66% to 39.47%, RSR 48.05%; and AgentCPM-GUI-8B from 59.87% to 26.97%, RSR 39.56%. The average drop exceeds 17.5%.

• Larger general models are more robust than GUI-specialized ones. GPT-4o and Gemini-2.5 withstood interruptions better than the GUI-trained models; the paper suggests this may be attributed to stronger planning capabilities. AgentCPM-GUI and UI-TARS-72B showed the weakest robustness by RSR, which the authors read as evidence that GUI training mainly adapts visual perception of the interface while planning stays dependent on the base model.

• Difficulty varies sharply by interruption category. Across the five categories (System Resource, System Network, App Malfunction, Permission Control, UX Disruption), Gemini and GPT-4o strongly outperformed other methods on Permission Control and System Network, reaching 100% on Permission Control. UI-TARS performed notably better on app-level interruptions, and AgentCPM showed a strong peak there despite ranking lowest overall.

• Skipping an interruption is not the same as handling it. When agents could choose a simple exit such as "Close," Qwen2.5-VL-7B reached 96.15% RSR and AgentCPM-GUI-8B 82.35%. When only the complex path was available (for example "Install Now" instead of "Close"), those numbers fell to 41.27% and 9.30% respectively.

• Agents know what to do but struggle with where to do it. With screenshots only, Gemini2.5-flash fell to 45.33% SR (no interruption) versus 80.26% with screenshots plus XML, an almost 35% drop. AgentCPM-GUI dropped less (56.58% versus 59.87%), suggesting it has partially learned coordinate prediction.

• Perception drift can persist after recovery. In a documented GPT-4o case, the model correctly identified an app crash and relaunched the app, but on returning to the search page it clicked search directly instead of re-entering the keyword, failing the task. The paper attributes this to the model's decision-making being overly influenced by its action history in the prompt, which no longer matched the current visual state.

• Coverage of the benchmark. D-GARA-152 contains 152 tasks over the applications JD.com (27 tasks), Weibo (27), Bilibili (25), Amap (24), Ctrip (20), Amazon (18), Facebook (8), and Google Maps (3). The text describes these as 8 applications, while the comparison table in the appendix lists 7 for D-GARA-152.

• Interruption mix is weighted toward everyday device issues. System Network makes up 42.8% and System Resource 28.3% of interruptions, together over 70%. UX Disruption accounts for 10.5%, and App Malfunction and Permission Control are each 9.2%.

• Positioning against prior benchmarks. The comparison table lists Mind2Web (137 apps/web, 2,350 tasks, desktop web), GUI Odyssey (201, 7,735, Android apps), OSWorld (9, 369, desktop apps and web), Windows AgentArena (11, 154, desktop apps and web), WorldGUI (10, 315, desktop apps), Android World (20, 116, Android apps), SPA-Bench (58, 340, Android apps), and GUI-Robust (392, 5,318, desktop apps and web). The paper notes GUI-Robust remains largely static in practice, with fewer than 4% of cases involving actual anomalies, and states D-GARA-152 stands out by supporting both anomaly injection and a dynamic environment, with English and Chinese support.

Methodology in Plain English

The authors built a loop around a live Android device. At each step the framework grabs a screenshot and the UI layout as an XML hierarchy and decides whether to inject an interruption. If it does, a new screenshot and XML file are captured and handed to the agent, which outputs an action such as a tap or text entry, sent to the device through ADB. After a short pause for the interface to settle, the framework checks the new state.

Interruptions are chosen by a rule-based evaluator that scans the XML text for keywords; a rule fires only when enough of its keywords are present. The example in the paper uses the keywords Drive, Nearby, Metro, and Mine with a threshold of 0.75, and a rule can specify follow-up outcomes. An Amap location permission dialog illustrates a two-stage pipeline: the agent deals with the foreground dialog first, then ADB executes the consequence — accepting redirects to the settings interface to enable the permission, while denying terminates the app. All trigger logic lives in external configuration files rather than hard-coded code, so researchers can add or edit anomaly scenarios without touching the framework.

Success is judged by a state-centered validator. Rather than trusting a model's "done" signal, which small models may never emit and large models may fire prematurely, the validator inspects the XML after each action against a declarative goal condition. One listed rule checks whether the resource-id tv.app:id/like_button has content-desc "Liked" for a video-liking task. Because completion depends only on the final interface state, an agent may take detours or backtrack as long as it reaches the goal. Automatic validations are additionally reviewed by human evaluators.

Interruptions are injected through a template approach: dialog layouts are designed in Android Studio, compiled into a standalone APK ("ADB Smart Test"), and filled at runtime with values from configuration files. The benchmark itself was built in four steps — anomaly design and implementation, task definition, manual trajectory collection with the DataCollector tool, and creation of injection and validation rules. Evaluation reports Success Rate (SR) under baseline and interrupted conditions, plus Robust Success Rate (RSR), which counts the share of baseline-solvable tasks that also succeed under interruption.

Why This Matters

Impact on research: Static, idealized benchmarks can overstate how capable GUI agents really are. D-GARA provides a way to measure robustness directly and reports that current state-of-the-art systems degrade substantially, which the authors frame as an argument for robustness-aware learning and evaluation. Its modular, extensible design lets other researchers add tasks, anomaly types, and interaction scenarios.

Real-world applications:

• Mobile assistants that must survive permission prompts, low-battery and thermal warnings, network drops, and app crashes while completing a user's task.
• E-commerce and travel apps, where rating prompts, update dialogs, and interruptions can derail an automated checkout or booking flow.
• Navigation and location-based apps, where an agent's choice to accept or deny a location permission leads to entirely different execution paths or app termination.
• Social media and content apps, where post-crash recovery and remembering what was already done determines whether a task finishes.

Industry relevance: The paper is directly relevant to teams deploying on-device or local GUI agents, particularly given its finding that GUI-specialized models were the least robust to interruption and that handling complex interruption paths (rather than dismissing pop-ups) remains a major weakness. Its observation that agents depend heavily on XML coordinates rather than vision alone points to a concrete engineering target for perception and grounding.

Future Directions

• The authors state that future work will explore involving a large foundation model in the success-verification step, extending beyond the current rule-based validator plus human review.
• Training and evaluation methods that produce robustness-aware agents, rather than models optimized only on idealized trajectory data, are identified as an open need.
• The perception-drift problem after app crashes suggests a need for agents that retain useful memory while discarding misleading action history when it no longer matches the current visual state.
• Strengthening visual perception and coordinate prediction, so that agents can act without XML-provided coordinates, is raised as a promising direction given the large screenshot-only performance drops.
• The paper also invites the community to extend D-GARA with new tasks, interruption types, and evaluation strategies.

Target Audience

Researchers and engineers working on GUI agents, multimodal large language models, and agent benchmarking. It is also useful for practitioners evaluating or deploying mobile automation in production, and for anyone designing evaluation suites who needs to understand why static benchmarks miss failure modes that only appear during live interaction.

Authors’ abstract

Developing intelligent agents capable of operating a wide range of Graphical User Interfaces (GUIs) with human-level proficiency is a key milestone on the path toward Artificial General Intelligence. While most existing datasets and benchmarks for training and evaluating GUI agents are static and idealized, failing to reflect the complexity and unpredictability of real-world environments, particularly the presence of anomalies. To bridge this research gap, we propose D-GARA, a dynamic benchmarking framework, to evaluate Android GUI agent robustness in real-world anomalies. D-GARA introduces a diverse set of real-world anomalies that GUI agents commonly face in practice, including interruptions such as permission dialogs, battery warnings, and update prompts. Based on D-GARA framework, we construct and annotate a benchmark featuring commonly used Android applications with embedded anomalies to support broader community research. Comprehensive experiments and results demonstrate substantial performance degradation in state-of-the-art GUI agents when exposed to anomaly-rich environments, highlighting the need for robustness-aware learning. D-GARA is modular and extensible, supporting the seamless integration of new tasks, anomaly types, and interaction scenarios to meet specific evaluation goals.