The Pulse

Apple Trains UI Agents on Live iOS Apps

Apple researchers report training an 8-billion-parameter agent through live iOS apps, with a near-sevenfold gain over its supervised baseline on Android tasks. The paper’s results distinguish performance on real iOS apps from a separate ben

AI.info Team · Written with AI · fact-checked · 2026-11-02

Apple researchers say they trained an 8-billion-parameter agent by letting it operate live iOS apps, then scoring its actions as it worked through tasks. The paper, posted to arXiv on October 29, 2026, reports an online reinforcement-learning approach designed to help agents recover when an app’s screen or workflow diverges from a memorized sequence. The results include a near-sevenfold gain over the team’s supervised fine-tuning baseline on Android tasks.

Two models divide the work

The paper’s authors—Peitong Duan, Khashayar Gatmiry, Jeffrey Nichols, Samy Bengio and Emmanuel Abbe—write:

“To address these limitations, we propose a novel online reinforcement learning framework that trains UI navigation agents directly on live iOS applications.”

Their system separates planning—deciding what to do next—from grounding, which maps an instruction such as tapping a control to a screen location and action.

The planning agent receives the task, a screenshot, and a history of previous steps. A separate grounding model turns its immediate instruction into an action, such as a tap, swipe or text entry. The authors use reinforcement learning for planning, where the agent must adapt to changing screens and recover from mistakes, while training the grounding component with supervised examples.

The paper describes training on cloud-hosted iOS virtual-machine devices. The system can run 512 independent agent-and-device trajectories at once; each device can be reset to a saved app state between tasks. Before reinforcement learning, the planning model receives supervised fine-tuning using human-annotated trajectories from first- and third-party iOS apps, alongside multi-app examples from the public Android dataset GUIOdyssey.

Judges score progress along the way

A large language model acting as a judge supplies two types of reward: feedback on each step and a separate score for completing the overall task. That gives the training process a way to recognize useful intermediate actions even when an agent does not finish, while preserving task completion as the final objective.

The researchers also introduce GiGPPO, a method that uses visually similar screens from other concurrent task attempts to help estimate the value of a state. Their motivation is that two runs may reach similar screens without reproducing an identical sequence of actions. The paper describes an asymmetric reward setup that discounts step-level signals but leaves the terminal score undiscounted, aiming to keep local feedback from overwhelming the actual goal.

The authors’ internal 3-billion-parameter grounding model scores above 90% on its held-out test set, according to the paper. It handles directional swipes but not swipes tied to precise screen locations, so the researchers route those instructions to Gemini-3.5-Flash in their training setup. The reported configuration therefore uses a separate model for location-based swipes alongside the planning agent.

Android gains stand out; iOS results depend on the benchmark

On AndroidWorld, the strongest listed PPO checkpoint reaches 32.8% task success, against 4.7% for the supervised baseline. The evaluation uses 116 tasks across 20 Android apps, even though the reinforcement-learning phase trains on iOS apps. On UINavBench, the best PPO result is 32.4%, matching the listed result for Gemini-3.1-Pro.

The iOS results come from two different kinds of benchmark. UINavBench uses real iOS apps, while iOSWorld uses synthetic apps. On UINavBench, the paper reports that PPO at step 336 ties Gemini-3.1-Pro at 32.4%, and three RL checkpoints score at least as high as Gemini-3.5-Flash’s 28.4%. Those results align with the abstract’s comparison on real iOS apps.

On iOSWorld, the strongest listed RL result is 9.8% for GiGPPO, compared with 12.9% for Gemini-3.5-Flash and 39.5% for Gemini-3.1-Pro. The paper says iOSWorld’s synthetic apps differ visually from the real apps used in training, and describes the results there as modest gains over the supervised baseline, with the strongest checkpoint reaching more than double that baseline’s 4.5%.

Across its three benchmarks, the study reports improvements over the supervised baseline, including a substantial transfer result on Android. Performance varies by benchmark, training checkpoint and grounding configuration. Apple’s research page lists the work as a paper and links to its arXiv publication.