GUI-Rise: Structured Reasoning and History Summarization for GUI Navigation

AI summary

Overview

Research area: Multimodal large language model (MLLM) agents for Graphical User Interface (GUI) navigation, combining structured chain-of-thought reasoning, history summarization, and reinforcement learning.

Technical level: Advanced.

Scope: The paper proposes a reasoning-enhanced agent framework and a two-stage training recipe (supervised cold-start plus GRPO reinforcement learning) that improves multi-step GUI navigation, particularly out-of-domain generalization.

Authors and affiliation note: Tao Liu, Chongyu Wang, Rongjie Li, Yingchen Yu, Xuming He, and Bai Song; work from ShanghaiTech University, ByteDance, and the Shanghai Engineering Research Center of Intelligent Vision and Imaging. Code is listed at https://leon022.github.io/GUI-Rise.

What This Paper Is About

GUI agents built on MLLMs still struggle to stay coherent across long multi-step interactions and to make good use of their own history. Existing systems either record only past actions (losing visual state) or feed full screenshots back into a heavily truncated context window (expensive and lossy). GUI-Rise addresses both problems by having the agent compress its entire interaction trace into a short textual history summary, reason explicitly about task progress and next decisions, and be trained with reinforcement learning whose rewards directly link summary quality to future action success.

Key Contributions

1. A structured reasoning pipeline for GUI tasks. The agent formally decomposes each step into three subtasks — structured reasoning (progress estimation plus decision reasoning), action prediction, and history summarization — with history represented as a compact textual memory rather than raw screenshots or action lists.

2. A composite reward for reinforcement learning. Training combines a format reward (enforcing XML-style tagged output), an action reward (action type, format, and coordinate accuracy), and a novel history summary reward that scores a summary by whether it enables correct future actions.

3. A two-stage training paradigm. A cold-start supervised fine-tuning stage on pseudo-labeled trajectories establishes reasoning and summarization skills, followed by GRPO-based reinforcement learning in a simulated GUI environment.

4. State-of-the-art results under identical training data, with emphasis on out-of-domain generalization, demonstrated across Mind2Web, AITW, GUIAct, MiniWob, AndroidWorld, and OSWorld.

Main Findings

• Offline web generalization (Mind2Web, standard setting): With the Qwen2-VL-2B backbone, GUI-Rise reaches the highest step success rates across all splits, including 39.7 Step SR in cross-domain. With Qwen2.5-VL-3B, Step SR improves by 1.2 and 3.5 points in the cross-website and cross-domain settings respectively. On the cross-task split with Qwen2.5-VL-3B, the reported Step SR is 46.2 versus 48.3 for the unmodified Qwen2.5-VL-3B baseline.

• Zero-shot web generalization: Trained on GUIAct and evaluated on Mind2Web, GUI-Rise achieves a 38.7% performance improvement over the previous state-of-the-art ShowUI in the cross-domain split (33.4 and 35.1 Element Accuracy for the 2B and 3B backbones respectively).

• Zero-shot mobile generalization (AITW): GUI-Rise shows a 50.7% relative gain in the overall metric over ShowUI, with a +15.5 point improvement on the more complex WebShop category.

• In-domain mobile performance (AITW): With Qwen2-VL-2B, GUI-Rise reaches 71.1% overall versus 70.0% for ShowUI-2B, including a 1.6-point improvement in the WebShop shopping scenario (68.2% vs. 66.6%). With Qwen2.5-VL-3B, GUI-Rise reaches 73.7% overall versus 72.5%.

• Online environments: GUI-Rise-2B scores 30.6 on MiniWob zero-shot, 72.8 on MiniWob fine-tuned, 10.4 on AndroidWorld, and 8.7 on OSWorld (chrome split), outperforming prior results such as ShowUI-2B (27.1 / 71.5 / 7.0) and UI-Tars-2B (6.5 on OSWorld).

• Ablation on AITW: Two-stage training alone gives limited gains (66.0 overall vs. 67.2 baseline). Training structured CoT via SFT alone causes a large drop (42.6), while applying RL to structured reasoning recovers and exceeds the baseline (69.8). Adding history summaries gives minor gains (70.7), and adding the history summary reward yields the best result (71.1).

• Source of the largest gains: The paper attributes the WebShop improvement to structured reasoning, and the AndroidWorld/OSWorld results to the history summarization design, which must compress and integrate context in dynamic, long-horizon online environments.

Methodology in Plain English

The agent runs in a loop. At each step it sees the current screenshot, the user's instruction, and a short text summary of everything that happened before. It then produces four tagged outputs in a fixed order: a progress estimate, a decision reasoning block, an action, and an updated memory summary. The action is parsed from text into an executable form (action type, value, and screen coordinates).

Training happens in two stages. In the cold-start stage, a stronger model (GPT-4o-mini) generates pseudo-labels for the reasoning trace and the history summary using a retrospective labeling strategy: because the correct action for each step is already known from the dataset, the labeler can write reasoning and summaries that are consistent with the right answer. The agent is then fine-tuned with a standard token-level cross-entropy loss on the serialized sequence of reasoning, ground-truth action, and summary.

In the reinforcement learning stage, GRPO optimizes the policy without a value network by comparing samples within a group and normalizing rewards into advantages. Three rewards are summed with weights for action and history terms: a format reward that checks the XML-style tag sequence, an action reward that checks structure, action-type match, and whether predicted coordinates fall inside the target element's bounding box, and a history summary reward. The history reward gives zero if the current action was wrong; otherwise the model performs k extra rollouts using the generated summary as input, without gradient backpropagation, and scores how well the predicted next action matches ground truth. This makes the summary itself a trained object rather than an artifact of input formatting.

Why This Matters

Research impact. The paper reframes agent memory as a learned, reward-shaped output rather than a fixed window of past screenshots or actions, and it extends GRPO from single-step GUI tasks (as in UI-R1) to multi-step navigation. It also shows that structured reasoning helps mainly when trained with RL rather than supervised fine-tuning alone — a finding with implications beyond GUI agents.

Real-world applications:

• Web automation across unfamiliar sites, such as form filling and multi-step booking flows (the Mind2Web setting, with over 2,000 tasks across 137 websites and 31 domains).
• Mobile phone assistants operating real Android interfaces (the AITW setting, 30K instructions and 715K trajectories).
• Shopping and transactional agents, where the WebShop results show the largest relative gains.
• Desktop or browser-based long-horizon automation, as tested on OSWorld and AndroidWorld.

Industry relevance. The work targets deployable open-source MLLM backbones rather than frozen proprietary APIs, and it is co-authored with ByteDance researchers, indicating direct relevance to production assistant and automation products. The emphasis on cross-domain generalization addresses a known failure mode of fine-tuned agents.

Future Directions

• Online learning. The stated limitation is that GUI-Rise is trained entirely offline even though it is evaluated in online environments, preventing real-time adaptation. The authors suggest enabling the model to reflect on its successes and failures directly from live interaction.
• Better supervision for history summaries. The ablation shows history summaries add little without the summary reward, suggesting that how summaries are supervised and scored remains an open design space.
• Scaling and backbone transfer. The framework is demonstrated on Qwen2-VL-2B and Qwen2.5-VL-3B and is described as compatible with other similar MLLMs, leaving the effect of larger or different backbones unstated.
• Handling dynamic, long-horizon environments. Performance on AndroidWorld (10.4) and OSWorld (8.7), while above prior art, is far below the offline benchmarks, leaving substantial room for improvement in real-time, multi-step settings.

Target Audience

Researchers and engineers working on multimodal LLM agents, GUI and web automation, and reinforcement learning for language models — particularly those interested in agent memory design, reward shaping, and out-of-domain generalization. It is also relevant to practitioners building production automation on open-source vision-language backbones, though the dense training details and RL formulation make it best suited to readers with prior exposure to MLLM fine-tuning and policy optimization.

Authors’ abstract

While Multimodal Large Language Models (MLLMs) have advanced GUI navigation agents, current approaches face limitations in cross-domain generalization and effective history utilization. We present a reasoning-enhanced framework that systematically integrates structured reasoning, action prediction, and history summarization. The structured reasoning component generates coherent Chain-of-Thought analyses combining progress estimation and decision reasoning, which inform both immediate action predictions and compact history summaries for future steps. Based on this framework, we train a GUI agent, \textbf{GUI-Rise}, through supervised fine-tuning on pseudo-labeled trajectories and reinforcement learning with Group Relative Policy Optimization (GRPO). This framework employs specialized rewards, including a history-aware objective, directly linking summary quality to subsequent action performance. Comprehensive evaluations on standard benchmarks demonstrate state-of-the-art results under identical training data conditions, with particularly strong performance in out-of-domain scenarios. These findings validate our framework's ability to maintain robust reasoning and generalization across diverse GUI navigation tasks. Code is available at https://leon022.github.io/GUI-Rise.