Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agents

AI summary

Overview

Research area: Machine learning / large language model (LLM) GUI agents, specifically reusable "agent skills" and self-evolving agents that improve at inference time without training.

Technical level: Advanced.

Scope: The paper introduces EvoSkill-GUI, a training-free framework in which each GUI skill is a structured, editable multi-file package that is revised from execution feedback through a reflect-revise-reuse loop, and reports gains across the MobileWorld, AndroidWorld, and OSWorld benchmarks.

Note on source content: the provided paper text is truncated inside Appendix A.3, so the remaining appendix material (including Appendix A.4, which the main text references for the two negative OSWorld domain-level entries) is not available and is therefore not summarized here.

What This Paper Is About

GUI agents that operate real screens fail in predictable ways: pop-ups appear, pages load slowly, and widgets move, so plans fixed before execution stop being valid, and the same mistakes get repeated because nothing is written down. This paper's goal is to let a deployed agent turn each failure into an edit of the skill it was following, so procedural knowledge accumulates across tasks instead of being rebuilt from scratch or baked into model weights.

Key Contributions

1. EvoSkill-GUI, a training-free self-evolution framework. A single backbone model constructs, retrieves, executes, and revises skill packages entirely at inference time, using information-isolated reflection and tool-restricted edits (no training and no external reflection model).
2. A GUI-oriented structured multi-file skill package. Each skill is defined as S = (D, A, P, B, C, F), separating retrieval metadata, accessibility-tree utilities, executable planning knowledge, backup localization strategies, failure-recovery rules, and failure cases, so a revision can target the component responsible for a failure.
3. A metadata-indexed retrieval and reuse mechanism. Skills are indexed by structured metadata fields (id, intent, app, platform, keywords, args, history, status) rather than by long procedural text, scored by a weighted lexical/tightness formula plus app and keyword bonuses minus a divergence penalty, and reused when the score exceeds a threshold θ_r (default 0.6).
4. Consistent empirical gains without training. Evaluation across MobileWorld, AndroidWorld, and OSWorld spanning mobile and desktop platforms, with maximum gains of +16.2%, +6.0%, and +10.5% respectively, plus ablations isolating structure, instant revision, information isolation, and metadata retrieval.

Main Findings

• MobileWorld (GUI-only setting) improves for every evaluated base model. Claude-Sonnet-4.6 rises from 57.1% to 67.6%, Qwen3.6-Plus from 53.3% to 69.5%, Qwen3.6-35B-A3B from 32.4% to 44.8%, and MAI-UI-8B from 29.5% to 37.1%. The later round-by-round analysis reports MAI-UI-8B rising from 28.5% to 37.1% over rounds, while the earlier section lists 29.5% as the starting point.

• AndroidWorld improves under both evaluation phases. On the 116+116 task set, Phase 1 (seed 30, 37.9% reuse rate) goes from 68.1% to 70.7% (+2.6), and Phase 2 (seed 42, 100.0% reuse rate) from 55.2% to 61.2% (+6.0).

• Desktop transfer holds on OSWorld. GUI-Owl-1.5-8B's overall success rate rises from 46.7% to 54.8% (+8.1), with large reported gains on Thunderbird (+20.0), VLC (+42.7), and VS Code (+5.2); Qwen3-VL-8B-Instruct rises from 23.8% to 34.3% (+10.5). Two negative domain-level entries are noted and deferred to Appendix A.4.

• Structured multi-file packages beat single-file skills. On MobileWorld the structured package reaches 69.52% versus 66.67% for a single-file skill, a +2.85 gain.

• Instant in-rollout revision prevents cascading failures. Removing it lowers MobileWorld success rate from 69.52% to 62.86%, a drop of 6.66 points.

• Information isolation improves reflection quality. Removing it lowers MobileWorld success rate from 69.52% to 60.95%, a drop of 8.57 points.

• Metadata retrieval outperforms full-text matching. Across twelve MobileWorld reuse cases, metadata scoring averages 0.8753 versus 0.4050 for full-text retrieval; at θ_r = 0.6, metadata retrieval recovers the correct skill in all 12 cases while full-text retrieval succeeds in only one.

• Every skill-package component contributes. Ablating planning documents drops success from 69.52% to 65.71% (−3.81, the largest drop), and ablating failure reflection drops it to 66.67% (−2.85).

• Accessibility-tree utilities help grounding across backbones. Adding accessibility-tree inputs improves Qwen3.6-Plus by +3.80, Qwen3.6-35B-A3B by +1.90, and MAI-UI-8B by +2.85.

• The retrieval threshold needs to be moderate. On MobileWorld, θ_r = 0.4 yields 55.2%, θ_r = 0.6 yields 69.5%, and θ_r = 0.8 yields 66.7%.

• Evolution gains accumulate and largely saturate after three rounds. Claude-Sonnet-4.6 moves from 58.1% in Round 1 to 67.6% in Round 3; Qwen3.6-35B-A3B from 32.4% to 44.8%; MAI-UI-8B from 28.5% to 37.1%. Extending Qwen3.6-Plus to five rounds gives 56.2% to 61.9% (+5.7) and 68.6% (+6.7) over the first three rounds, stays at 68.6% in Round 4, and reaches 69.5% in Round 5 (+0.9), so the first three rounds yield a 12.4-point gain while two extra rounds add only 0.9 point.

• Evolution beats repeated sampling at comparable budget. Three-round EvoSkill-GUI reaches 69.5% using 94.75M tokens, versus 62.8% for a Qwen3.6-Plus pass@3 baseline using 103.00M tokens, a 6.7-point accuracy improvement at slightly lower token use.

• Skill libraries grow and remain reusable. On the sequential 116+116 AndroidWorld set, the library grows from 9 skills at task 10 to 69 skills at task 110 in Phase 1, then reaches 98 skills and a 56.1% reuse ratio in Phase 2. Overall, EvoSkill-GUI recovers 27 of 89 initially failed executions, including 19 of 65 failures on tasks using reused skills; by source, reused skills recover 19/65 (29.2%) and new skills recover 8/24 (33.3%).

• Evolved skills transfer to held-out tasks. Splitting MobileWorld into 87 library-building and 30 held-out tasks, EvoSkill-GUI solves 22/30 (73.3%) versus 12/30 (40.0%) for a pass@3 baseline; among the 16 tasks that trigger retrieval, 15 retrieve an appropriate skill (93.8%) and one produces a mismatch (6.3%).

• Failure types differ in repairability. Plan-level errors are recovered 16/23 times (69.6%) via plan.md, grounding errors 6/12 times (50.0%) via backup.md, and missing contingencies only 1/4 times (25.0%) via recover.md.

Methodology in Plain English

The researchers model a GUI task as a partially observable process: the agent sees a screenshot and an accessibility tree at each step, acts, and receives a terminal reward of 0 or 1 when the task finishes. A skill is a persistent procedural knowledge object that conditions the agent's policy. Instead of one long prompt, a skill is stored as a folder-like package of separate files, each holding one kind of knowledge: metadata for retrieval, executable plans, backup localization strategies, recovery rules, accessibility utilities, and recorded failure cases.

The agent can only touch these files through a restricted set of tools (read, write, append, list, search, and create_failure), which keeps edits inside the package schema, while a read-only accessibility-tree tool augments observation without modifying the skill. This separation is deliberate: interface understanding and skill revision stay distinct at the tool level.

Each iteration runs a rollout, during which the executor may make instant edits when reality contradicts the current plan, producing an intermediate package. After the rollout, the same backbone model switches roles to act as a critic on the trajectory, but under strict information isolation: the critic may only see the instruction, the observations, and the actions, and is explicitly barred from the skill package, the executor's chain of thought, and ground-truth labels, while the executor has access to a superset. This is meant to prevent the critic from relying on privileged information a deployed agent would not have, and to stop it inheriting the executor's mistaken plan. The critic returns failure-step localization, cause analysis, and revision suggestions; the executor then edits specific files, and failed rollouts are appended as failure cases.

For reuse, each skill is indexed by short structured metadata rather than its full procedural body, so surface-token overlap between unrelated tasks matters less. A new instruction is scored against candidates using a weighted combination of lexical coverage and set tightness, plus bonuses for matched app and keyword fields and a penalty for query-specific concepts the candidate lacks, clipped to [0, 1]. If the top score exceeds θ_r = 0.6, the agent reuses that package; otherwise it builds a new one, which enters the library only after successful execution or revision. Evaluation uses up to 50 interaction steps per task, a 3-second per-step wait, and at most three execution iterations (the initial rollout plus at most two revision rounds), with small open-weight models served via vLLM on one RTX PRO 6000 GPU 96GB and larger models accessed through provider APIs.

Why This Matters

Impact on research. The paper reframes agent skills as living artifacts revised at deployment time rather than static assets built before it, and contrasts this with systems that construct skills offline, evolve them only during training, or update memory and weights rather than the deployed skill itself. It also offers a concrete, reproducible design for information-isolated same-backbone critique, which is a reusable protocol for anyone studying self-improving agents.

Real-world applications:

• Mobile assistants that automate long-horizon, cross-app

Authors’ abstract

GUI agents execute long-horizon tasks on dynamic graphical user interfaces, where pop-ups, delayed loads, and relocated widgets routinely invalidate plans fixed before execution. Recent agent-skill frameworks encapsulate reusable procedural knowledge to mitigate this, yet existing skill designs are largely developed without targeting GUI execution dynamics and treat skills as static artifacts produced before deployment rather than living procedural knowledge that improves through it. We argue that what GUI agents need is not better static skills, but skills that can be revised from execution feedback at deployment time, without additional training. We propose \textbf{EvoSkill-GUI}, a training-free framework in which each skill is a structured multi-file package containing retrieval metadata, executable plans, backup localization, failure-recovery rules, accessibility utilities, and failure cases. EvoSkill-GUI operates through a \textbf{\emph{reflect-revise-reuse}} loop: the executor performs instant in-rollout revisions, an isolated critic diagnoses failed trajectories under strict information isolation, and the executor edits specific skill files through a restricted tool interface. Across MobileWorld, AndroidWorld, and OSWorld, three mainstream GUI benchmarks spanning mobile and desktop platforms, EvoSkill-GUI consistently improves multiple base models without any training, with maximum gains of $+16.2\%$, $+6.0\%$, and $+10.5\%$ respectively, and evolved skill libraries continue to benefit related tasks rather than being rebuilt from scratch. Our code is available at https://github.com/ZJU-REAL/EvoSkill-GUI.