HATS: Hardness-Aware Trajectory Synthesis for GUI Agents

AI summary

Overview

Research area: Computer Vision / multimodal GUI agents, specifically automated training-data (trajectory) synthesis for vision-language-model agents that operate mobile and web interfaces.

Technical level: Advanced (assumes familiarity with Monte Carlo Tree Search, vision-language model fine-tuning, and GUI benchmark environments).

Scope: A single training-data synthesis framework — HATS — that targets semantically ambiguous GUI actions through closed-loop hardness-driven exploration and alignment-guided refinement, evaluated on AndroidWorld and WebArena.

What This Paper Is About

GUI agents built on large vision-language models need large amounts of high-quality trajectory data (task instruction, step description, action-state pairs) to learn. Current automatic synthesis pipelines, especially the exploration-driven OS-Genesis approach, over-collect trivial "semantic-intuitive" actions and barely capture "semantic-ambiguous" actions — interactions whose meaning depends on context, on prerequisite steps, or on subtle visual cues — and their one-shot instruction generation produces misaligned, noisy supervision. HATS attacks both problems at once by defining a measurable "hardness" (semantic ambiguity of an action) and using it to steer data collection while iteratively validating and repairing the instruction-execution alignment of every synthesized trajectory before it enters the training corpus.

Key Contributions

1. HATS framework. A unified closed-loop trajectory synthesis framework that explicitly represents and handles semantic-ambiguous actions, coupling exploration and refinement through a Hardness-Driven Monte Carlo Tree Search (HD-MCTS) procedure rather than treating them as separate offline stages.

2. Hardness-driven exploration. A module that prioritizes rare, high-value, semantically challenging actions while reducing redundancy from intuitive ones, replacing the random walks and shallow breadth-first search used by prior exploration-driven pipelines.

3. Alignment-guided refinement. A module that replaces one-shot instruction generation with multi-round validation — sub-trajectory selection, instruction synthesis, execution replay, similarity verification, and contextual correction — ensuring instruction-execution alignment.

4. Hardness as a reward signal. A formal definition of an alignment-derived hardness reward, r(A,B) = (R(A,B) + ε)^(−α), that converts misalignment (normally noise) into a backpropagated signal guiding future exploration toward ambiguous regions.

Main Findings

• AndroidWorld overall success: HATS reaches 22.60% with InternVL2-4B versus OS-Genesis at 11.30%, a 100% increase; with InternVL2-8B, 24.35% versus 14.81%; with Qwen2-VL-7B, 24.35% versus 11.50%. The GPT-4o reference baseline is reported at 45.22%.

• WebArena overall success: HATS reaches 20.60% with InternVL2-4B versus OS-Genesis at 6.53%, a 215% increase; with InternVL2-8B, 24.87% versus 8.16%; with Qwen2-VL-7B, 23.28% versus 12.81%. The GPT-4o reference baseline is reported at 22.82%.

• Largest gains in weak categories: On AndroidWorld Social & Communication, OS-Genesis reaches 11.11% (InternVL2-4B) and 0.00% (Qwen2-VL-7B) while HATS attains 38.46% and 30.77%. The paper also reports a P&W comparison of 1.85% versus 9.88% and a Gitlab domain comparison of 7.94% versus 22.73%.

• Default harness parameters are best: In the hardness reward sensitivity study, ε = 0.01 and α = 1 gives 50.8% success. Other settings degrade: (0.01, 0.50) gives 22.7% (−55.3%), (0.01, 2.00) gives 39.2% (−22.8%), (0.10, 0.50) gives 13.8% (−72.8%), (0.10, 1.00) gives 40.0% (−21.3%), (0.10, 2.00) gives 37.9% (−25.3%).

• Recall beats precision as a hardness metric: Recall-based alignment scoring yields 50.8% success versus 36.0% for the analogous precision-based metric; the authors argue precision does not penalize missing steps.

• Exploration coverage shifts toward harder categories: Relative to the baseline, HATS raises the sampling weight of Productivity & Work from 31.2% to 46.6% and lowers System & Utility from 30.8% to 18.9%. A third-party GPT-4o auditor labels a consistently larger fraction of HATS trajectories as context-dependent, sequentially dependent, or visually ambiguous compared to OS-Genesis.

• Multi-round refinement helps monotonically at the endpoint: Average action-level reconstruction recall R_avg rises from 0.26 (one-shot) to 0.40 (round 3), with success rate rising from 16.00% to 23.07%. Intermediate rounds: round 1 gives R_avg 0.19 / 15.56%, round 2 gives 0.29 / 21.43%.

• Added Self-Instruct data can hurt: During baseline reproduction, the authors observed that adding Self-Instruct data sometimes degraded Task-Driven performance due to instruction-execution misalignment and noisy supervision.

Methodology in Plain English

The team starts from the observation that most automatically collected GUI interaction traces are boring — over 70% collapse into trivial actions like "open menu" or "tap back" — while the interactions that actually trip agents up are the ambiguous ones. They reframe ambiguity as a measurable quantity they call hardness.

Their pipeline runs a modified Monte Carlo Tree Search over the GUI. Each node is a screen state and each edge is an action (Tap, Type, Scroll, Back, Long-Press, Swipe). Four stages run per iteration: Selection traverses the tree using an Upper Confidence Bound rule with a constant C; Expansion executes an unvisited action and grows a new node; Simulation rolls out a bounded-depth trajectory, reverse-synthesizes a natural-language instruction for a coherent sub-trajectory, replays that instruction, and compares the replayed execution against the reference using action-level reconstruction recall R(A,B); Backpropagation converts that recall into a hardness reward r = (R + ε)^(−α) and pushes it up the tree.

Because classical MCTS normally backpropagates an environment-supplied task reward — which GUI trajectory synthesis does not provide — the team substitutes this alignment-derived hardness. Low recall (hard, ambiguous behavior) yields high reward, so the search keeps returning to difficult regions. Instructions are refined in a replay-repair loop until recall meets a threshold R_min = 0.7 or a refinement budget F_max is exhausted. Only trajectories that pass verification are admitted. The resulting verified trajectories, annotated with hardness, are used for downstream supervised fine-tuning in the two-stage planning-level and action-level paradigm of prior work.

Training used three VLM backbones (InternVL2-4B/8B, Qwen2-VL-7B) with GPT-4o for instruction synthesis and reward modeling, all under the same SFT setting with screenshots and a11ytree inputs, and HATS used the same 1K-trajectory synthesis budget as OS-Genesis. AndroidWorld was run in emulators with Pixel 6 hardware profiles and the Tiramisu system image (API Level 33); WebArena was run via AgentLab on Gitlab (180 tasks), Maps (109 tasks), and Reddit (106 tasks). AndroidWorld's 116 tasks were grouped into DLS, S&C, S&U, and P&W.

Why This Matters

Impact on research. The paper reframes data quality for GUI agents as a problem of coverage of hard cases rather than raw volume, and shows that semantic ambiguity can be quantified into a usable reward signal. That is a reusable idea: any domain where automated supervision is noisy could in principle turn its own misalignment into an exploration gradient.

Real-world applications:

• Mobile assistants that must decide what a generic "plus" or "menu" icon does in a specific app context, rather than what it did in the last app.
• Web automation for enterprise tools (repository management on Gitlab-style platforms, booking flows, CRM configuration) where a step only works after prerequisites are completed.
• Accessibility tooling and robotic process automation, where an agent must follow an instruction across screens it has never seen.
• Customer-support or IT-workflow bots that fail today because they replay an instruction inconsistently across visually similar pages.

Industry relevance. Annotation of GUI trajectories is expensive and does not scale, so automated synthesis quality directly determines training cost and agent reliability. A framework that produces smaller but harder, cleaner corpora — verified before admission — matters to anyone shipping GUI agents on a fixed data budget, including the mobile and browser automation vendors whose environments (Android, WebArena-style suites) this paper evaluates on.

Future Directions

• Scaling beyond the 1K-trajectory budget. The paper deliberately matches OS-Genesis at 1K trajectories with additional cost bounded by the HD-MCTS iteration budget and F_max; whether the gap widens, narrows, or saturates at much larger budgets is not reported.
• Closing the remaining gap to GPT-4o. HATS-trained 4B, 8B, and 7B backbones still trail the GPT-4o reference numbers (22.60 vs. 45.22 on AndroidWorld; 20.60 vs. 22.82 on WebArena), so the relationship between synthesized data quality and model capacity remains open.
• Broadening evaluation. Only three of WebArena's domains (Gitlab, Maps, Reddit) were evaluated to reduce computational cost, and component analysis ran on a representative AndroidWorld subset; extending to the full suite and to desktop environments would test generality.
• Generalizing the hardness signal. Hardness is defined entirely through action-level reconstruction recall against a reference sequence; whether other ambiguity notions, other thresholds than R_min = 0.7, or other refinement budgets than F_max behave similarly is not reported.

Target Audience

Researchers and engineers working on GUI automation, multimodal agent training, and synthetic data generation — particularly those who need to build instruction-aligned trajectory corpora without manual annotation. It is also relevant to practitioners in reinforcement-learning-style data curation and to teams evaluating whether difficulty-aware sampling can replace brute-force data scaling. Readers without background in MCTS or VLM fine-tuning will need to consult the appendices and cited baselines (OS-Genesis, AndroidWorld, WebArena) first.

Authors’ abstract

Graphical user interface (GUI) agents powered by large vision-language models (VLMs) have shown remarkable potential in automating digital tasks, highlighting the need for high-quality trajectory data to support effective agent training. Yet existing trajectory synthesis pipelines often yield agents that fail to generalize beyond simple interactions. We identify this limitation as stemming from the neglect of semantically ambiguous actions, whose meanings are context-dependent, sequentially dependent, or visually ambiguous. Such actions are crucial for real-world robustness but are under-represented and poorly processed in current datasets, leading to semantic misalignment between task instructions and execution. To address these issues, we propose HATS, a Hardness-Aware Trajectory Synthesis framework designed to mitigate the impact of semantic ambiguity. We define hardness as the degree of semantic ambiguity associated with an action and develop two complementary modules: (1) hardness-driven exploration, which guides data collection toward ambiguous yet informative interactions, and (2) alignment-guided refinement, which iteratively validates and repairs instruction-execution alignment. The two modules operate in a closed loop: exploration supplies refinement with challenging trajectories, while refinement feedback updates the hardness signal to guide future exploration. Extensive experiments show that agents trained with HATS consistently outperform state-of-the-art baselines across benchmark GUI environments.