Co-EPG: A Framework for Co-Evolution of Planning and Grounding in Autonomous GUI Agents

AI summary

Overview

Research area: Autonomous GUI (Graphical User Interface) agents that combine multimodal vision-language models with reinforcement learning to operate software by clicking, typing, and selecting on screens.

Technical level: Advanced. The paper assumes familiarity with vision-language models (VLMs), reinforcement learning (specifically Group Relative Policy Optimization, GRPO), reward shaping, and supervised fine-tuning pipelines.

Scope: The paper proposes Co-EPG, a self-iterative training framework in which a planning model and a grounding model improve each other in a closed loop without external data, and evaluates it on web, mobile, and desktop GUI benchmarks.

What This Paper Is About

Most GUI agents need two abilities: planning (deciding which action to take and with what value) and grounding (locating the exact screen element to act on). Existing modular systems typically train these two components separately and lean heavily on large synthetic datasets. The paper's goal is a training framework where planning and grounding co-evolve: the grounding model supplies rewards that steer the planner, and the improved planner generates better data that strengthens the grounder.

Key Contributions

1. Co-EPG framework. A self-iterative training framework for Co-Evolution of Planning and Grounding that establishes a positive feedback loop: the grounding model guides the planning model's strategy exploration through rewards (via GRPO), while the optimized planning model generates high-quality data that further enhances the grounding model.

2. C-DREM (Confidence-based Dynamic Reward Ensemble Mechanism). A mechanism that aggregates rewards from multiple grounding models, weighting each by a static prior plus a dynamic confidence score (the length-normalized sum of log-likelihoods of predicted coordinate tokens), producing a more robust composite reward signal for GRPO training.

3. Data self-enhancement mechanism. At iteration 0, open-source VLM pools (a Planner and a Verifier) build an initial dataset of successfully verified plans; in later iterations, the trained planning and grounding models join the pools, with only the latest two versions of each retained to keep pool size constant.

4. Benchmark results without external data. Co-EPG reports state-of-the-art performance on Multimodal-Mind2Web (58.4% average Step SR) and AndroidControl (83.1% average Step Acc) after three iterations, using only the benchmark data.

Main Findings

• Multimodal-Mind2Web performance: Co-EPG-Web-7B reaches 58.4% average Step SR across the three subtasks (cross-task, cross-website, cross-domain), exceeding Explorer-7B (54.3%) and AGUVIS-7B (57.2%). Co-EPG-Web-3B reaches 51.4% average Step SR.

• Smaller-model gains: Co-EPG-Web-3B outperforms Explorer-4B by nearly 1.6% on average Step SR, and Co-EPG-Mob-3B reaches 81.8% average Acc versus InfiGUI-R1-3B at 81.6%.

• AndroidControl performance: Co-EPG-Mob-7B achieves 83.1% average Step Acc (high-level 74.2, low-level 92.0), a 1.4% advantage over UI-TARS-7B (81.7%). Co-EPG-Mob-3B reaches 81.8%.

• Decoupled architecture beats end-to-end: An end-to-end model scores 50.1 average SR, while Co-EPG-Web-7B-M1 scores 53.5, a 3.4% improvement attributed to the P-G dual-model decoupling.

• Iterative improvement: Performance rises steadily across iterations. Co-EPG-Web-7B moves from 53.5 (M1) to 55.0 (M2) to 58.4 (M3); Co-EPG-Web-3B from 46.6 to 48.8 to 51.4; Co-EPG-Mob-7B from 79.0 to 80.9 to 83.1; Co-EPG-Mob-3B from 75.8 to 79.0 to 81.8.

• Both data iteration and GRPO matter: Removing GRPO and relying on SFT alone still improves results (e.g., Co-EPG-Web-7B iteration 3 w/o GRPO: 56.5; Co-EPG-Mob-7B iteration 3 w/o GRPO: 82.0) but underperforms the full framework, indicating GRPO collaborative training acts as an accelerator.

• C-DREM ablation: Single reward model (w/o C-DREM) gives 56.50 average SR; average weighting (w/o Confidence & Prior Weights) gives 57.01 (+0.51%); prior-only weighting (w/o Confidence Weights) gives 57.67 (+0.66%); full C-DREM gives 58.41 (+1.91%).

• Prior weight ratio: Best performance occurs at a 1:1:2 ratio of Qwen2.5-VL-72B-Instruct : Qwen2.5-VL-32B-Instruct : trained model φ_k (58.4 average SR), compared with 57.3 at 1:1:1, 57.9 at 1:1:1.5, and 58.0 at 1:1:3.

• Data efficiency: Co-EPG-Web-7B surpasses AGUVIS-7B on average Step SR using only 2.42% of the labeled step data (6862 vs. 283500).

• Data quality evolution: Across iterations, plan purity (proportion of plans successfully executed by grounding models) improves by 8.84%, and the diversity metric (average number of generated plans per task) increases by nearly 4.

• Convergence stability: Compared with a single grounding model, C-DREM mitigates reward noise and speeds up the planning model's convergence on Multimodal-Mind2Web.

• Additional environments: The authors report experiments on the OmniACT desktop/web dataset (9,802 annotated instances, MacOS 22 / Linux 8 / Windows 8 native applications plus 27 web applications, 7:1:2 split, Action Score metric). The numerical results in that table are not included in the provided content.

Methodology in Plain English

The system is split into two models. A planning model sees the screen observation, the task description, and the history of past actions, and outputs a textual plan, an action type, and an action value. A grounding model then takes that textual plan plus the screenshot and predicts the exact coordinates of the target element. The two outputs combine into the action that is executed.

Training proceeds in a loop:

1. Iteration 0 data: Open-source VLM pools act as a Planner (proposes plans) and a Verifier (keeps only plans that succeed), producing an initial dataset.

2. Iterative training: Models are fine-tuned on the current dataset. The planning model is then further refined with GRPO, a reinforcement learning method that compares a group of sampled rollouts against each other to compute advantages.

3. Reward construction: The plan reward is the grounding accuracy when using that plan, ensembled across several grounding models with confidence-based weights (C-DREM). Action type is rewarded by exact match, and action value is rewarded when its F1 score against ground truth exceeds 0.5. If either the type or value reward is zero, the total reward is zero.

4. Data enhancement: The trained planner and grounder join the data-production pools, generating richer training data for the next iteration.

GPT-style models are not used; the backbone is Qwen2.5-VL in 3B and 7B variants, trained with MS-SWIFT, DeepSpeed ZeRO-3, and FlashAttention-2. The paper runs three iterations, uses averaged results over three independent runs, and reports the training environment as PyTorch 2.6.0 on a Linux server with 984GB RAM, an Intel Xeon Platinum 8369B CPU @ 2.90GHz, and Nvidia A100 80GB GPUs.

Why This Matters

Impact on research: The work argues for a shift from isolated optimization of modular GUI agent components toward integrated, self-driven co-evolution. It claims to be the first co-evolution framework for decoupled planning-and-grounding architectures, and it shows that a small amount of benchmark data (2.42% of the labeled steps used by a prior state-of-the-art method) can support competitive results when data value is mined iteratively rather than expanded synthetically.

Real-world applications:

• Web automation, such as filling forms, clicking through sites, and completing multi-step transactions from natural-language instructions.
• Mobile app automation on Android, covering both high-level (e.g., "book a hotel") and low-level (e.g., "tap this button") instructions.
• Desktop application control across MacOS, Linux, and Windows environments, as explored in the OmniACT experiments.
• Digital accessibility tools that let users describe an intent and have an agent operate the interface on their behalf.

Industry relevance: The framework targets a practical cost problem. Synthetic trajectory generation is expensive (the paper cites Explorer generating over 94K successful trajectories and Winclick building a 60k-sample dataset, among others), while Co-EPG aims to extract more value from existing data. The authors are affiliated with Alibaba Cloud Computing and the University of Tokyo, and the framework is intended for production deployment considerations, reflected in the higher static prior weight assigned to the trained grounding model.

Future Directions

• Combining with data synthesis techniques: The conclusion states that applying Co-EPG alongside other data synthesis methods could unlock greater data potential.

• Extending beyond the reported benchmarks: The OmniACT desktop experiments are introduced to test generalization, but the results are incomplete in the provided content; broader desktop and cross-platform validation remains open.

• Scaling the iteration and model budget: All reported results come from three iterations and 3B/7B backbones; whether the co-evolution loop continues to pay off at larger scales or more iterations is not established.

• Reward design beyond the current signals: The plan reward depends on grounding accuracy falling inside the target bounding box, and the action value reward uses a 0.5 F1 threshold; whether richer or finer-grained rewards change the co-evolution dynamics is an open question.

Target Audience

Researchers and engineers working on GUI agents, multimodal vision-language model training, and reinforcement learning for agentic systems. The paper is most useful to readers already comfortable with VLM fine-tuning and policy-optimization methods such as GRPO, and to practitioners building modular planning-and-grounding pipelines who want to reduce dependence on large synthetic datasets.

Authors’ abstract

Graphical User Interface (GUI) task automation constitutes a critical frontier in artificial intelligence research. While effective GUI agents synergistically integrate planning and grounding capabilities, current methodologies exhibit two fundamental limitations: (1) insufficient exploitation of cross-model synergies, and (2) over-reliance on synthetic data generation without sufficient utilization. To address these challenges, we propose Co-EPG, a self-iterative training framework for Co-Evolution of Planning and Grounding. Co-EPG establishes an iterative positive feedback loop: through this loop, the planning model explores superior strategies under grounding-based reward guidance via Group Relative Policy Optimization (GRPO), generating diverse data to optimize the grounding model. Concurrently, the optimized Grounding model provides more effective rewards for subsequent GRPO training of the planning model, fostering continuous improvement. Co-EPG thus enables iterative enhancement of agent capabilities through self-play optimization and training data distillation. On the Multimodal-Mind2Web and AndroidControl benchmarks, our framework outperforms existing state-of-the-art methods after just three iterations without requiring external data. The agent consistently improves with each iteration, demonstrating robust self-enhancement capabilities. This work establishes a novel training paradigm for GUI agents, shifting from isolated optimization to an integrated, self-driven co-evolution approach.