Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live IOS Apps

AI summary

Overview

Research area: AI agents for graphical user-interface (UI) navigation — specifically, training small vision-language models to operate real mobile apps via reinforcement learning.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning (PPO, advantage estimation, value critics), vision-language models, and mobile UI automation.

Scope: The paper describes a system that trains an 8B-parameter planning agent with online reinforcement learning on live iOS devices, using an LLM-as-a-Judge for dense step-level and terminal rewards, and evaluates it on UINavBench, iOSWorld, and AndroidWorld.

What This Paper Is About

Frontier vision-language models can navigate app interfaces well but cost too much to run for everyday tasks (the paper cites costs of "several dollars per task"), while small models trained with supervised fine-tuning (SFT) memorize fixed trajectories and fail to recover when something goes wrong. The authors build an online reinforcement learning framework that trains a small planner directly on live iOS applications, scoring each step with an LLM judge to give the model richer feedback than a single end-of-task success/failure signal. The goal is an agent that is cheap to run, learns generalizable navigation strategies, and transfers to unfamiliar apps — even on Android, an operating system it never trained on.

Key Contributions

1. A decoupled planning–grounding architecture trained at scale on live iOS. The agent splits into a Planning Agent (trained with RL) and a frozen UI Grounding model (trained with SFT), with an Ask_User action primitive for underspecified tasks. The authors state this is the first at-scale training of UI navigation agents entirely on live iOS devices and real iOS applications.

2. Asymmetric reward discounting to fix reward hacking in step-level RL. The dense reward is discounted at γ = 0.5 while the terminal reward is carried undiscounted to every step, preventing the agent from artificially prolonging trajectories to farm per-step rewards.

3. GiGPPO, a neighbor-conditioned value estimation method. Each state is paired with the state whose 64-bit perceptual hash is closest in Hamming distance among sibling trajectories for the same task; a small MLP predicts the value from both states' hidden embeddings, the neighbor's return, and the Hamming distance, stabilizing the PPO critic.

4. Empirical generalization results. 8B RL agents substantially beat SFT and terminal-only RL baselines, achieve nearly a sevenfold improvement over SFT on out-of-distribution AndroidWorld (32.8% vs. 4.7%), and on UINavBench outperform Qwen3.8-27B and Gemini-3.5-Flash while tying Gemini-3.1-Pro.

Main Findings

• Large gains over SFT on real iOS apps. On UINavBench (116 tasks, 36 real iOS apps), the SFT baseline scores 16.7%, while PPO at Step 336 reaches 32.4% — tying Gemini-3.1-Pro (32.4%) and beating Gemini-3.5-Flash (28.4%) and Qwen3.8-27B (19.6%). Six RL checkpoints outperform Qwen3.8-27B, which is described as over three times larger than the 8B agents.

• Strong out-of-distribution transfer to Android. On AndroidWorld (116 tasks, 20 real applications), the SFT baseline scores 4.7% and PPO Step 336 scores 32.8%, described as nearly seven times higher. Every RL checkpoint beats the SFT baseline on this benchmark. Frontier models still lead here: Gemini-3.1-Pro 60.8%, Gemini-3.5-Flash 57.8%, Qwen3.8-27B 49.1%, UI-TARS-72B 46.6%.

• Dense rewards accelerate convergence. Standard PPO at Step 168 (one epoch) matches or exceeds terminal-only PPO at Step 336 (two epochs) across benchmarks — for example 25.5% vs. 21.6% on UINavBench and 20.7% vs. 19.8% on AndroidWorld.

• GiGPPO speeds learning on the in-distribution benchmark. On UINavBench, GiGPPO at Step 75 (21.6%) and Step 168 (30.4%) is comparable to standard PPO at Step 168 (25.5%) and Step 336 (32.4%) respectively.

• iOSWorld remains the hardest benchmark. The SFT baseline scores 4.5% there; most RL checkpoints show only modest gains, and PPO Step 336 drops to 2.3%. The exception is GiGPPO Step 336, which more than doubles the baseline at 9.8%. The authors attribute this to iOSWorld's synthetic apps, personalization workflows, and multi-app tasks spanning more than two apps — all out-of-distribution relative to training. Gemini-3.1-Pro reaches 39.5% and Qwen3.8-27B 19.5% on this benchmark; UI-TARS was not evaluated on iOSWorld.

• Undiscounted returns cause reward hacking. In a training run documented in the appendix, γ = 1 led the agent to prolong trajectories to collect dense rewards rather than finish tasks, motivating the γ = 0.5 dense discount.

• Exact-state matching fails in this setting. With G = 4 or G = 8 rollouts per task, subgroups defined by exact state equality are "essentially singletons," which is why the authors route cross-trajectory signal through a learned critic rather than through averaging as in GiGPO.

Methodology in Plain English

Splitting the job in two. Rather than one model that both decides what to do and finds where to tap, the system uses two. The Planning Agent looks at the screenshot, the task instruction, its history of past steps, and its remaining plan, then produces chain-of-thought reasoning plus the next step and an updated remaining plan. That step goes to a Grounding Model — an internal 3B model fine-tuned on human-annotated data that maps instructions to action primitives and x,y coordinates, reported at over 90% accuracy on its held-out test set. Because that model only handles directional swipes, swipe instructions are routed to Gemini-3.5-Flash. The reasoning: grounding is a fairly deterministic mapping problem that SFT solves well, while planning is open-ended and benefits from RL.

Warm start. The planner begins as Qwen3-VL-8B-Instruct, supervised-fine-tuned on human-annotated iOS trajectories from two in-house datasets plus the public GUIOdyssey Android dataset (included specifically for its multi-app trajectories). Since none of these datasets record reasoning, Gemini was used to generate chain-of-thought and remaining-plan targets. Because GUIOdyssey's multi-app tasks are Android-only, the authors synthetically generated multi-app iOS tasks in two stages — pairing logically related apps, then generating tasks per pair — with every generated task manually verified by an author.

Running on real devices. The agent operates cloud-hosted iOS virtual machines, each configured once with apps installed and signed in, then snapshotted. Episodes end by rolling back to the snapshot rather than tearing down the session, cutting restart latency to roughly 10 seconds. The system runs 512 trajectories simultaneously, each with its own device and agent-environment loop.

Scoring with an LLM judge. Gemini-3.5-Flash acts as a per-step judge, returning success, fail, or overcomplete (acting after the task is already done), and also penalizes unproductive loops and progress that comes from hallucinated reasoning. Gemini-3.1-Pro acts as the terminal judge on the final three screenshots plus full action history, returning success, fail, or blocked_complete (used when an unavoidable barrier such as a paywall or CAPTCHA prevents completion). A separate dense judge checks whether Ask_User requests are genuinely warranted; Gemini-3.5-Flash also plays the simulated user during training and evaluation.

Reward arithmetic. Dense rewards are +1.0 for judged success, 0.0 for fail or an ungroundable plan, −1.0 for an empty plan or truncated generation, and −2.0 for acting after completion. Terminal rewards are +2.0 for completing the task, +0.5 for blocked_complete, −0.75 for exhausting the step budget, −1.0 for a false completion claim, and −2.0 for unparseable or empty output. Since the dense return is bounded by 1/(1 − 0.5) = 2.0, a perfectly successful trajectory earns dense credit worth exactly one terminal success, so neither channel dominates.

Optimization. PPO with GAE at λ = 1 (a Monte Carlo estimator) rather than GRPO, because PPO's critic can consume dense rewards step by step. The critic is a linear head on the hidden embedding of the last token before the action tokens. Critic training is warm-started for 15 steps with the policy frozen, then trained jointly with a learning rate of 1×10⁻⁴ — two orders of magnitude above the policy's 1×10⁻⁶. Trajectories lost to infrastructure failures (app never renders, failed install, dropped connection, exceeded generation deadline) are masked out entirely rather than penalized.

Why This Matters

Research impact. The paper argues that no prior GUI-agent system learns a value function for step-level credit assignment, since group-relative methods like GRPO dominate the field; dense signals elsewhere are collapsed into trajectory-level returns or subgroup averages. Training a critic that consumes the dense channel directly is what makes the asymmetric discounting possible, and the finding that undiscounted dense rewards cause reward hacking is a concrete, reusable lesson for anyone doing step-level RL.

Real-world applications:

• Automating routine multi-step tasks on a user's own phone (shopping, scheduling, reordering) at a cost low enough for daily use rather than "several dollars per task."
• Automated app testing and A/B testing, which the paper names as an application area for UI navigation agents.
• Cross-platform agents: an iOS-trained agent transferring to Android suggests one training pipeline can serve multiple mobile operating systems.
• Assistive automation that can ask clarifying questions via Ask_User when a task is underspecified, instead of guessing.

Industry relevance. The infrastructure contribution matters as much as the algorithm: orchestrating hundreds of concurrent virtual devices with snapshot-based resets is what makes online RL on real mobile apps practical. The result that an 8B model can match Gemini-3.1-Pro on a real-iOS benchmark is directly relevant to anyone weighing on-device or low-cost deployment against frontier-model API calls.

Future Directions

• Improving performance on synthetic and personalization-heavy benchmarks. iOSWorld remains weak (best result 9.8% for GiGPPO Step 336 vs. 39.5% for Gemini-3.1-Pro). The authors specifically flag the naming-collision failure mode they observed — the agent trying to install a real App Store app sharing a synthetic app's name, or searching Safari for it — and say they will investigate improvements in future work.

• Closing the gap with frontier models on Android. RL substantially improved AndroidWorld performance but frontier models still led; the authors suggest this partly reflects the abundance of public Android UI datasets versus scarce iOS ones, raising the question of whether more iOS data would close the gap.

• Scaling the training environment further. The system runs 512 simultaneous trajectories, versus roughly 10,000 concurrent environments reported for Qwen-UI-Agent's multistage pipeline and 64 for the concurrent ClawGUI-RL work — leaving open how much further scale changes results.

• Broadening the decoupling. The architecture currently trains RL only on the planner and freezes the grounder; whether end-to-end RL, or RL on the grounding model, adds value beyond the current split is untested here.

Target Audience

Researchers and engineers working on GUI/computer-use agents, mobile automation, or agentic RL; practitioners who need to train small models to replace expensive frontier-model inference; and infrastructure engineers interested in large-scale device orchestration for online RL. Readers without RL background will find the reward-design and infrastructure sections accessible, but the value-estimation and advantage-estimation material requires prior familiarity with policy-gradient methods.

Authors’ abstract

While frontier vision-language models demonstrate strong capabilities in user interface (UI) navigation, their high inference costs prohibit everyday use. Conversely, smaller models trained via supervised fine-tuning (SFT) or offline reinforcement learning often memorize static trajectories, leading to brittle execution and an inability to recover from errors. To address these limitations, we propose a novel online reinforcement learning framework that trains UI navigation agents directly on live iOS applications. Our framework decouples the agent into two specialized components: a Planning agent and a UI Grounding model. We rely on an LLM-as-a-Judge to provide comprehensive reward signals, combining dense, step-level feedback to evaluate intermediate progress with a terminal reward for overall task completion. To effectively leverage this dual reward structure, we optimize the agent using Proximal Policy Optimization (PPO) with step-level advantage estimation. Furthermore, we introduce GiGPPO, a novel neighbor-conditioned value estimation method that improves critic stability by leveraging reward signals from visually similar states across concurrent trajectories. To support this online training paradigm at scale, we introduce an efficient infrastructure that orchestrates hundreds of concurrent virtual iOS devices. Evaluations on rigorous benchmarks, including UINavBench, iOSWorld, and AndroidWorld, demonstrate that our online RL trained agents substantially outperform both SFT and terminal-only RL baselines while accelerating training convergence. Notably, our 8B RL agents achieve nearly a sevenfold improvement over the SFT baseline in out-of-distribution generalization to Android environments, while outperforming Gemini-3.5-Flash and tying with Gemini-3.1-Pro on real iOS applications.