OpenMobile-2: Building Versatile Mobile Agents with Scalable Environments and App-Native Tools

AI summary

Overview

Research area: Mobile agents built on vision-language models (cs.SE), spanning environment/benchmark construction, interaction-data synthesis, supervised fine-tuning, and reinforcement learning for GUI-based smartphone automation.

Technical level: Intermediate — the paper is readable with general familiarity with LLM agents and supervised fine-tuning, though the appendix assumes comfort with RL objectives (GRPO/DAPO), LoRA, and benchmark evaluation protocols.

Scope: OpenMobile-2 is a fully open stack of simulated and emulator-based mobile environments, ~12K demonstration trajectories, 2K+ verifiable RL tasks, and trained 9B/27B agent models that can act through either GUI actions or app-native tool calls.

What This Paper Is About

Progress in mobile agents depends on large amounts of interaction experience, but high-quality trajectories and controllable environments are largely proprietary, leaving open models behind. Existing open environments are limited: emulator-based setups rely on a small set of mostly open-source apps that resist clean reset, while simulated environments are efficient to reset and parallelize but contain few, shallow applications. The paper's goal is to close that gap by scaling application coverage in a fully controllable simulator (MobileGym++, 35 new commercial-style apps, over 110 apps total in the playground), releasing open training data and RL tasks built on it, and testing an experimental setting where apps expose their own native tools alongside their GUIs.

Key Contributions

1. Environment scaling with replicated commercial applications. MobileGym++ adds 35 high-fidelity re-implementations of commercial apps with persistent business state and cross-app workflows, bringing the simulator to 63 installable clients and the overall playground (including newly configured Android emulator apps) to over 110 apps — with full reset and programmatic verification preserved.

2. Open training data and recipes at scale. The release includes nearly 12K (11.6K) demonstration trajectories for supervised fine-tuning — of which 1.7K interleave GUI actions with app-native tool calls — plus the largest open collection of verifiable RL data for mobile agents, over 2K executable tasks with automatic rewards.

3. A unified framework for GUI and app-native tool use. Over 300 carefully scoped app-native tools across 50+ apps are exposed alongside the GUI, and MobileGym++ Bench (215 long-horizon tasks) supports both GUI-only and hybrid GUI–tool evaluation on the same tasks, initial states, and checkers.

4. Trained open agents with measurable transfer. OpenMobile-2 models (9B and 27B) reach 79.9 on AndroidWorld and 50.4 on MobileWorld, and nearly double SPA-Bench real-device performance from 31.9 to 59.6 for the 27B model.

Main Findings

• Fine-tuning on OpenMobile-Data-v2 lifts both model scales substantially. The 9B model gains 19.7 points on AndroidWorld over its base (54.3 to 74.0), and the 27B model gains 12.8 points (67.1 to 79.9), with consistent improvements on MobileWorld and MobileGym++ Bench as well.

• RL helps most where training and evaluation environments align. Starting from the hybrid SFT checkpoint, RL raises MobileGym success rate from 35.9 to 38.3 (9B) and from 48.4 to 51.2 (27B), and improves 27B SPA-Bench from 57.4 to 59.6; gains on other benchmarks remain limited.

• Hybrid training improves the hybrid channel without hurting the GUI channel. On MobileGym++ Bench, hybrid SFT improves hybrid-setting success over GUI-only SFT at both scales (9B: 41.9 to 44.7; 27B: 57.7 to 65.6), while GUI-only success stays comparable.

• Tool-use ability generalizes to unseen tools. On MobileWorld MCP, whose tools are not seen in training, hybrid SFT scores 27.5 and 45.0 versus 12.5 and 35.0 for GUI-only SFT, suggesting learned strategies rather than memorized APIs.

• Tool access alone is not enough. Without hybrid training, tool calls appear in only 20–102 of the 215 tasks and yield small gains (Qwen models improve by only 2.8–4.2 points, and GUI-Owl-1.5 results are inconsistent or worse). After hybrid SFT, calls appear in 113 and 128 tasks covering 110 and 115 distinct tools, gains rise to 5.6 and 10.3 points, and trajectories shorten by 4.8 and 5.8 steps.

• Broader app coverage improves generalization at a fixed data budget. Holding training to 2K emulator trajectories while varying coverage from 15 to 53 apps raises held-out AndroidWorld Pass@1 from 60.7 to 67.5, even as the in-distribution data share falls from 100% to 39.1% and in-distribution Pass@1 stays roughly flat (73.7 to 71.4).

• A new apps are measurably richer than prior ones. The median app exposes 37 routes, 166 interactive actions, and 86 page transitions with a median interaction depth of 3; the new clients reach 1.48× the interactive actions and 1.91× the page transitions of the 12 prior MobileGym business apps.

• The training data is diverse and not trivially overlapping with benchmarks. Domain balance yields a Gini coefficient of 0.43 overall (0.52 emulator, 0.25 simulated), mean pairwise task-goal cosine similarity is 0.43, and 30% of trajectories span multiple applications. Only 1.11%, 0.39%, 0.46%, and 2.25% of SFT instructions exceed 0.7 cosine similarity with AndroidWorld, MobileWorld, MobileGym, and MobileGym++ Bench respectively, with none exceeding 0.9; 6.1% of the 2,475 RL task goals exceed 0.7 against MobileGym++ Bench, none above 0.9.

Methodology in Plain English

The team built a controllable simulated phone environment in the browser, extending the earlier MobileGym framework. For each new app, they started from the source app's data schema and interaction graph — specifying pages, actions, and business objects before rendering any interface — then had human annotators compare each replica against the live app until state transitions and interface details matched. Each app was checked twice before tasks were written: a coverage check mapping declared interactions to implementations, and a replay of core flows confirming each transition produces the declared state change. Observations are raw screen pixels with no accessibility tree, and the whole device state can be serialized, so episodes reset cleanly and run in parallel.

To generate data, they built a state–action transition graph per application (explored directly on the Android emulator, or derived from the backend specification in MobileGym++). A model, Gemini-3.1-Pro-Preview, generated realistic user instructions from global and local views of that graph, and then acted as a teacher agent to execute them. Only rollouts that completed the task and passed the environment's verifier were kept — a vision-language judge on the emulator, a programmatic verifier in MobileGym++ — followed by step-level filtering to remove action loops and no-op actions.

For app-native tools, each tool exposes a structured lookup or write over an app's own business objects, designed to be semantically equivalent to a corresponding GUI interaction, so both channels modify the same state and are judged by the same terminal condition. Agents were trained in two 5-epoch supervised runs from each base model (GUI-only data versus the full mixture including hybrid trajectories), then a subset of 512 RL tasks was used with a group-relative objective and a reward combining a state verifier, a tool-use bonus, a length factor, and penalties for malformed turns and false success claims. Evaluation spanned five benchmarks covering emulator apps, physical devices, and simulated apps.

Why This Matters

The paper argues that application coverage is the key bottleneck for scaling interaction experience in mobile agents, and it releases environments, data, benchmark, and models openly rather than keeping them proprietary. It also provides one of the first controlled comparisons between GUI-only interaction and hybrid GUI-plus-tool interaction on identical mobile tasks, showing that tool access is only valuable once models are trained to use it.

Real-world applications:

• Travel and shopping workflows, such as booking on apps like Trip and purchasing on apps like Taobao, where tasks span multiple apps and persistent user state.
• Everyday personal automation across the seven covered domains — travel, shopping, finance, content, social, office, and local services — including system clients like Clock and Notes.
• Cross-app information handoff, using clipboard transfer, app-to-app sharing, payment redirection, and media selection to move a value from one app into another.
• Real-device assistance, since SPA-Bench results (31.9 to 59.6) indicate the trained behavior transfers off the simulator onto physical Android phones.

Industry relevance: the release lowers the cost of building and evaluating mobile agents for teams without proprietary device farms or human-annotation budgets, and the app-native tool framing mirrors the broader shift toward structured interfaces (CLIs, MCP) complementing GUI automation in production computer-use systems.

Future Directions

• Closing the RL transfer gap. RL gains concentrate in training-aligned environments, so how to make reward-driven improvement carry over to unseen benchmarks and real devices remains open.

• Extending simulated commercial coverage further. The analysis shows coverage improves generalization under a fixed data budget, which invites scaling beyond the current 35 replicated apps and 63 installable clients.

• Teaching when to call a tool. Tools alone yielded limited or inconsistent gains before hybrid training; better policies for choosing between a GUI action and an app-native call, and for tool sets larger than the current 300+ tools, are natural next steps.

• Broadening the hybrid interface. The setting currently exposes selected functionalities per app; how far app-native tools can approach full API coverage while remaining semantically equivalent to the GUI is unresolved.

Target Audience

Researchers and engineers working on GUI agents, mobile automation, and computer-use models; teams building agent training environments or benchmark suites; and practitioners who need open, resettable mobile environments and verifiable RL task data rather than proprietary interaction logs. It is most useful to readers who already understand supervised fine-tuning and RL for LLM agents and want a concrete recipe for scaling environment and data coverage.

Authors’ abstract

We introduce OpenMobile-2, a near-frontier mobile agent with fully open training environments and recipes. We make three key advances: (1) Diverse environments with simulated commercial apps: We build MobileGym++, featuring 35 realistic, functionally rich commercial-style apps with cross-app workflows, while preserving full controllability for reset and verification. Together with newly configured apps in Android emulators, this yields a diverse playground spanning over 110 apps. (2) Open training data at scale: Building on this foundation, we curate nearly 12K mobile interaction trajectories for supervised fine-tuning and the largest open collection of verifiable RL training data for mobile agents, comprising over 2K executable tasks with automatic rewards. (3) Hybrid GUI and app-native tool use: We explore an experimental mobile-use setting where apps expose selected functionalities as app-native tools alongside their GUIs, allowing agents to interleave GUI actions and tool calls within a task. We implement this setting in MobileGym++ with over 300 carefully scoped tools across 50+ apps. Additionally, we introduce MobileGym++ Bench for complex, long-horizon commercial mobile scenarios, supporting both GUI-only and hybrid GUI-tool evaluation on a shared task suite. OpenMobile-2 performs competitively across established benchmarks, including AndroidWorld (79.9) and MobileWorld (50.4), while showing promising transfer to real-device mobile use, nearly doubling SPA-Bench performance from 31.9 to 59.6. All resources are available at https://os-copilot.github.io/OpenMobile2-Home/.