EchoTrail-GUI: Building Actionable Memory for GUI Agents via Critic-Guided Self-Exploration

AI summary

Overview

Research area: Artificial intelligence, specifically autonomous GUI (graphical user interface) agents built on Large Vision-Language Models, with a focus on retrieval-augmented memory and automated experience collection.

Technical level: Intermediate. The paper assumes familiarity with vision-language models, in-context learning, retrieval-augmented generation concepts, and Android GUI automation benchmarks, but its architecture is described without heavy mathematical formalism.

Scope: The paper proposes a three-stage, training-free framework that lets GUI agents autonomously collect, filter, and reuse their own successful task trajectories as in-context "memory," evaluated on the AndroidWorld and AndroidLab benchmarks.

What This Paper Is About

Most GUI agents today are stateless: they attempt every task from scratch and cannot accumulate knowledge from tasks they have already solved successfully, a limitation the authors call "digital amnesia." EchoTrail-GUI addresses this by having an agent explore Android environments on its own, have a critic model score the resulting trajectories, and archive only the high-quality ones into a memory database. When a new instruction arrives, the most relevant stored trajectories are retrieved and inserted into the agent's prompt as worked examples, turning past experience into actionable guidance without any model retraining.

Key Contributions

1. A three-stage memory framework. EchoTrail-GUI combines Critic-Guided Self-Exploration (autonomous trajectory generation and filtering), Dynamic Memory Injection (hybrid dense-sparse retrieval and formatting), and Memory-Augmented Inference (injecting retrieved trajectories into the agent's prompt), forming a fully automated pipeline with no human supervision.

2. The EchoTrail-4K dataset. A curated dataset of over 4,000 high-quality trajectories collected in Android environments, built from 4,143 exploration episodes, supporting both retrieval-based inference and downstream training.

3. A dual-database learning system. A volatile Processing Database holds both successful and failed in-progress trajectories to provide real-time guidance during exploration, while a permanent Memory Database stores only trajectories that pass a critic quality filter.

4. Empirical validation on two benchmarks. Experiments on AndroidWorld and AndroidLab showing that memory augmentation improves success rate, sub-goal completion, step efficiency, and operational robustness across two different backbone models (GPT-4o and Qwen2.5-VL-72B-Instruct).

Main Findings

• AndroidWorld success rate: EchoTrail-GUI with GPT-4o reaches 51.7% SR in the training-free setting, above the GPT-4o baseline at 34.5% and above GUI-explorer at 47.4%. Applied to Qwen2.5-VL-72B-Instruct, it reaches 46.6% SR, matching UI-TARS-72B-SFT at 46.6% and RAG-GUI-72B-RSF at 45.7%, both of which require fine-tuning or task-specific training.

• AndroidLab multi-metric gains: With Qwen2.5-VL, EchoTrail-GUI raises SR from 23.9% to 37.5%, Sub-SR from 26.1% to 41.1% (a reported +15.0 gain), RRR from 68.7% to 89.4%, and ROR from 81.4% to 92.1%. With GPT-4o, SR rises from 31.2% to 48.1%, Sub-SR from 35.0% to 50.7%, RRR from 87.3% to 97.9%, and ROR from 85.4% to 88.5%.

• Low-quality memory is worse than none: In the AndroidWorld ablation, removing Critic-based Filtering drops average SR to 31.0%, below the no-memory backbone baseline of 34.1%. The authors conclude that unfiltered, low-quality trajectories actively mislead the agent.

• Every component contributes: The full system achieves 46.6% average SR on AndroidWorld (65.6% easy, 30.6% medium, 15.8% hard), compared with 40.5% when Hybrid Retrieval is removed and 42.7% when Real-time Guidance is removed.

• Two memories is the sweet spot: Injecting any small number of relevant trajectories (K > 0) improves over the memory-free baseline (K = 0), but performance peaks at K = 2 and degrades beyond that due to longer prompts, conflicting advice, and contextual dilution.

• Critic reliability: Comparing critic decisions with human expert labels on 100 stratified trajectories yields Cohen's kappa of approximately 0.72, described as substantial agreement against a standard threshold of kappa at or above 0.61.

• Open-weight critics work: Qwen3-VL-235B-A22B matches or exceeds GPT-4o as a critic (F1 = 0.86, kappa = 0.68, accuracy 0.84), showing proprietary APIs are not required.

• Compact abstraction: Storing trajectories as text (interface description, agent intent, action) rather than screenshots keeps an average 5-step trajectory at roughly 1k tokens versus more than 10k tokens for an equivalent screenshot history, an approximately 90% reduction.

• Exploration quality improves over time: Across four sequential exploration stages, the share of high-quality trajectories rises for all tracked apps, with roughly 20 percentage-point gains on complex apps such as OsmAnd and VLC.

• Generated tasks align with real ones: A UMAP visualization of EchoTrail-4K task intents against AndroidLab ground-truth instructions shows dense overlap (realism) while the generated space extends beyond the test set (diversity).

Methodology in Plain English

The authors treat GUI automation as a partially observable problem in which the agent sees a screenshot and must pick the next action. Their fix is to give the agent a memory it builds itself.

First, an exploration agent powered by Gemini 2.5 Flash interacts with Android apps and produces trajectories up to 30 steps long. The agent starts in a "curiosity-driven" mode that encourages it to poke at unfamiliar UI elements, then switches after a few steps to a "target-focused" mode where it commits to a concrete sub-goal. Throughout, it consults a temporary Processing Database holding both successful and failed partial runs, so it can avoid repeating recent mistakes in real time.

Second, when a trajectory finishes, a reward model (Gemini 2.5 Flash Lite) scores it on a 1-to-5 scale. Only trajectories scoring at least 4 are kept in the permanent Memory Database. Kept trajectories are not stored as raw pixels but converted into a compact text form: a description of each screen, the agent's stated intent, and the action taken.

Third, at inference time on a new task, the system retrieves the two most relevant stored trajectories using a weighted blend of dense embedding similarity (via FAISS) and BM25 keyword matching. These are reformatted as step-by-step human-readable guides and injected into the prompt of an off-the-shelf VLM such as GPT-4o or Qwen2.5-VL-72B-Instruct, alongside the instruction, action history, current screenshot, and a summarized screen description. Summaries and embeddings come from Qwen3-30B-Instruct-2507 and Qwen3-Embedding-4B respectively. No part of the pipeline fine-tunes the inference model.

Why This Matters

Impact on research. The paper argues that structured, non-parametric memory is a lightweight alternative to fine-tuning for GUI agents, and that the abstracted intent-and-action format is transferable across models. It also provides evidence on a design question with practical weight: filtering matters more than retrieval volume, since bad memories hurt more than no memories.

Real-world applications:

• Mobile assistants that get better at a user's specific apps over time by remembering how past tasks were completed.
• Enterprise automation of repetitive multi-step workflows, such as order processing or form entry, without training a bespoke model per workflow.
• Customer-support and QA tooling where an agent replays a known-good sequence of clicks rather than rediscovering it.
• Accessibility tools that convert complex app navigation into guided, previously validated action sequences.

Industry relevance. The work comes from Taobao & Tmall Group of Alibaba alongside East China Normal University and the Shanghai Innovation Institute, and its results are framed around commercial Android deployment rather than lab-only settings. The finding that open-weight models can serve as both critic and inference backbone is directly relevant to teams weighing proprietary API costs against self-hosted options.

Future Directions

• Extending the framework beyond Android to desktop, web, and other interactive environments, which the authors suggest is plausible but do not test.
• Determining whether the EchoTrail-4K trajectories can be used for downstream training, not just retrieval, since the paper describes the dataset as supporting both but reports only retrieval-based inference results.
• Investigating why more than two retrieved memories degrade performance, and whether better selection or compression could allow larger memory budgets to help.
• Reducing the cost of memory construction, given that the exploration pipeline required approximately 4.3 minutes per valid trajectory on a single device over 4,143 episodes.

Target Audience

Researchers and engineers working on GUI automation, computer-use agents, or multimodal LLM agents who want a training-free way to improve task success through experience reuse. It is also useful for practitioners evaluating retrieval-augmented approaches for interactive systems, and for readers interested in automated data generation and quality filtering with LLM-based critics. Some familiarity with vision-language models and retrieval systems is assumed.

Authors’ abstract

Contemporary GUI agents, while increasingly capable due to advances in Large Vision-Language Models (VLMs), often operate with a critical limitation: they treat each task in isolation, lacking a mechanism to systematically learn from past successes. This digital ''amnesia'' results in sub-optimal performance, repeated errors, and poor generalization to novel challenges. To bridge this gap, we introduce EchoTrail-GUI, a novel framework designed to mimic human-like experiential learning by equipping agents with a dynamic, accessible memory. Our framework operates in three distinct stages. First, during Experience Exploration, an agent autonomously interacts with GUI environments to build a curated database of successful task trajectories, validated by a reward model. Crucially, the entire knowledge base construction is thus fully automated, requiring no human supervision. Second, in the Memory Injection stage, upon receiving a new task, our system efficiently retrieves the most relevant past trajectories to serve as actionable ''memories''. Finally, during GUI Task Inference, these memories are injected as in-context guidance to inform the agent's reasoning and decision-making process. We demonstrate the efficacy of our approach on benchmarks including Android World and AndroidLab. The results show that EchoTrail-GUI significantly improves the task success rate and operational efficiency of baseline agents, validating the power of structured memory in creating more robust and intelligent GUI automation.