Training High-Level Schedulers with Execution-Feedback Reinforcement Learning for Long-Horizon GUI Automation

AI summary

Overview

• Research area: GUI automation agents built on vision-language models (VLMs), multi-agent system design, and reinforcement learning (GRPO-style) for long-horizon task planning.
• Technical level: Advanced. The paper assumes familiarity with reinforcement learning post-training, VLM-based GUI agents, and multi-agent architectures, although the core idea can be understood without the math.
• Scope: The paper introduces CES (Coordinator-Executor-State Tracker), a multi-agent framework whose high-level components are trained with a staged execution-feedback reinforcement learning algorithm, and evaluates it on three long-horizon GUI benchmarks. (Note: the provided content is truncated partway through Appendix A, so appendices referenced for reward-function details and additional case studies are not included here.)

What This Paper Is About

GUI agents powered by VLMs still struggle with long-horizon tasks, where a user gives an ambiguous high-level instruction and the agent must follow a long, complex trajectory. The authors argue this happens for two reasons: a single policy model is overloaded by having to do both high-level planning and low-level execution at once (responsibility coupling and capability conflict), and agents have no reliable sense of where they are in a task because screenshots alone are a poor representation of progress. Their goal is to fix this by decoupling the roles and training only the high-level schedulers, using feedback from a frozen executor as the reward signal.

Key Contributions

1. The CES multi-agent framework, composed of a Coordinator (strategic planning and task decomposition), an Executor (low-level action, kept frozen and swappable), and a State Tracker (context compression and state summarization). The high-level components are described as general-purpose and plug-and-play with various Executors.
2. A dedicated State Tracker whose core job is dynamic context compression and state summarization in natural language, addressing the state-unawareness problem and maintaining logical coherence over long-horizon tasks.
3. A staged execution-feedback reinforcement learning strategy that decouples high-level capability from low-level execution: a pre-trained Executor is frozen, and the verifiable reward signals it produces are used to train only the Coordinator and the State Tracker, in two separate stages.
4. Extensive experiments showing that the trained high-level modules improve long-horizon scheduling and state management for multiple Executor models and surpass existing baselines, with code released at https://github.com/hehehahi4/CES.

Main Findings

• Screenshots do not encode task progress. In a preliminary temporal reasoning experiment, three strong GUI agents were asked to determine the temporal order of two screenshots from the same task trajectory. Accuracy was high for adjacent steps but dropped dramatically as the step interval increased. The authors attribute this to repeated screens such as the Home screen and to out-of-distribution interfaces that the model has not seen in training.
• Main benchmark gains (Table 1). Using GUI-R1-7B as the Executor, CES reached AITZ Type 64.44 / GR 64.58 / SR 43.05; AMEX 77.57 / 61.64 / 48.48; and GUI-Odyssey 79.24 / 63.82 / 53.69. The same executor baseline (GUI-R1-7B) scored 52.73 / 54.92 / 30.59, 67.26 / 57.12 / 43.69, and 65.49 / 43.64 / 38.79 respectively. The authors report an average improvement of 10.38% in Type accuracy across all benchmarks.
• Prompting a very strong general model is not a substitute. Using GPT-5 via prompting as Coordinator and State Tracker yielded only an average increase of about 4% in Type accuracy, and some metrics degraded (for example, AMEX SR of 35.80, below the 43.69 of the untrained GUI-R1-7B baseline on the same benchmark).
• CES beats both the direct baseline and prompted cooperation (Table 2). For UI-R1-3B on AMEX, CES reached 70.39 Type / 66.28 GR / 43.38 SR versus the baseline's 60.23 / 41.78 / 35.81, while the prompted variant CES-P fell to 42.52 / 52.15 / 29.12. For GUI-Owl-7B, CES reached AMEX 75.72 / 61.19 / 47.24 and GUI-Odyssey 74.87 / 61.39 / 46.65, versus baselines of 61.56 / 48.38 / 40.48 and 60.60 / 45.96 / 35.82. For GUI-Owl-32B, CES reached AMEX 78.55 / 63.11 / 52.05 and GUI-Odyssey 79.58 / 65.42 / 56.75.
• Evidence for the capability-conflict hypothesis. The prompted multi-role setup (CES-P) degraded the small UI-R1-3B model, dropping SR by 18.05% on GUI-Odyssey, while the large GUI-Owl-32B improved by 13.18% on AMEX under the same setup. The authors read this as smaller models being unable to acquire planning and execution capabilities together, while the 32B model already had latent high-level skills that prompting could surface.
• Specialized training outperforms prompting for those roles. On GUI-Odyssey, training raised GUI-Owl-32B's SR from 39.60% to 56.75% and GUI-Owl-7B's SR from 37.53% to 46.65%. The authors state that comparable performance to GUI-Owl-32B can be obtained with a 7B Coordinator and 4B State Tracker.
• Ablations show both roles and the RL stage matter (Table 3). Removing the Coordinator dropped GUI-Odyssey SR from 53.69 to 39.15 (a 12.77% drop). Removing the State Tracker dropped it to 42.52. Using only the SFT warm-up stage gave 36.54 SR on AMEX and 42.89 on GUI-Odyssey, well below the full model's 48.48 and 53.69.
• Failure analysis locates where the gains come from. CES reduced State Loss errors from 14% to 2% and Planning Error from 12% to 4%, while Perception Error and Generalization Failure, attributed to the frozen Executor, remained largely unchanged, shifting the bottleneck to the Executor's perceptual limits.
• Plug-and-play generality. For every Executor tested, the full CES framework improved on the direct baseline, supporting the claim that the trained high-level modules generalize across Executor models.

Methodology in Plain English

The authors treat the GUI task as a Markov Decision Process and then break the single decision-making policy into three collaborating roles, inspired by how an operating system splits work: the Coordinator is the CPU (planning), the Executor is the I/O device (action), and the State Tracker is the dynamic memory (state management).

In the loop, the Coordinator reads the user's high-level instruction, the current screenshot, and the previous natural-language state summary, and emits a clear atomic instruction. The Executor receives only that atomic instruction and the current screen, and produces an action in pixel space. The State Tracker, a language model that never looks at the screen directly, reads the Executor's output along with the user intent and the previous state, and writes an updated high-semantic state summary that feeds the next planning step.

Training proceeds in three parts. First, a warm-up supervised fine-tuning stage teaches each agent its role and strict output format, using data pairs built from existing datasets with automated scripts. Second, rule-based reinforcement learning with GRPO optimizes the policies using an Execution-Feedback Reward: because judging an intermediate planning sentence or state summary has no objective standard, the authors instead pass these outputs through the frozen Executor and score the resulting action with verifiable rules — a format reward plus an executor reward for correct action type and correct action parameters (R = α1·R_format + α2·R_executor, with the executor reward split into γ1·R_type + γ2·R_param). Third, the optimization is staged: in Stage 1 the Coordinator is trained while state inputs come from ground-truth annotated states in the preprocessed dataset; in Stage 2 the Coordinator is frozen and the State Tracker is trained so that its summaries steer the fixed Coordinator and Executor toward the best final reward.

Implementation: Qwen2.5-VL-7B serves as the Coordinator base and Qwen3-4B as the State Tracker base; GUI-R1-7B is used as the frozen Executor during reward computation. Warm-up SFT used the LLaMA Factory framework for 1 epoch at a learning rate

Authors’ abstract

The rapid development of large vision-language model (VLM) has greatly promoted the research of GUI agent. However, GUI agents still face significant challenges in handling long-horizon tasks. First, single-agent models struggle to balance high-level capabilities and low-level execution capability, facing prevalent issues of responsibility coupling and capability conflicts. Second, agents lack awareness of the task state, leading to progress loss in long-horizon tasks. To address these challenges, we propose a staged execution-feedback reinforcement learning algorithm. Unlike training a unified policy model, we focus on training high-level scheduling models. Specifically, we propose and train two agents: a Coordinator, responsible for the strategic planning and task decomposition; and a State Tracker, responsible for context compression and information management to maintain the task's state and coherence. Based on this, we built the Coordinator-Executor-State Tracker (CES) multi-agent framework, which can be integrated with any low-level Executor model, assisting the Executor in solving long-horizon tasks through task scheduling and state management. Experiments on long-horizon task benchmarks demonstrate that CES significantly enhances the system's planning and state management capabilities. Furthermore, analysis confirms that our trained high-level scheduling module is a generalizable, plug-and-play module that significantly enhances the long-horizon capabilities of various Executors. Code can be available at https://github.com/hehehahi4/CES.