MAI-Gym: An Open All-in-One Platform for GUI Agent Evaluation and RL Training across Mobile and Computer Environments

AI summary

Overview

Research area: AI agents that operate graphical user interfaces (GUI agents) for mobile, desktop, and web; specifically, the infrastructure and reinforcement-learning (RL) methods needed to evaluate and train them at scale.

Technical level: Intermediate. The paper is readable without deep RL mathematics, but familiarity with policy-gradient methods (GRPO), multimodal model inference, and container/VM orchestration helps.

One-sentence scope: MAI-Gym is an open, all-in-one platform that unifies GUI agent evaluation, data collection, and online RL training across mobile, computer, and web environments, and adds automated synthesis of resettable applications and execution-verified tasks; it is used to train MAI-Gym-9B, a 9B mobile-and-computer agent.

What This Paper Is About

RL with verifiable rewards has driven fast progress in math, coding, and tool use, but GUI agents have lagged because GUI environments are expensive: every attempt needs its own phone, desktop, or browser in an exact starting state, resets constantly, and makes many screenshot-based model calls before producing one reward. At the same time, realistic training tasks are scarce — real workflows span multiple apps and files, and existing synthesis methods rely on isolated input files or shared sample data.

The paper's goal is a single open codebase where evaluation, data collection, and training run the same agent through the same rollout path, scaled to thousands of parallel environment instances, and paired with a pipeline that automatically builds new apps, coherent multi-app "worlds," and execution-validated task verifiers.

Key Contributions

1. A unified, scalable platform for evaluation and online RL. MAI-Gym shares one agent harness and one rollout path across evaluation, data collection, and training, supports ten benchmarks (three mobile, four computer, three web), and has been run with 20,000 parallel environment instances.

2. Automated application and task synthesis. The pipeline builds resettable "mocked apps" (seven real-world clones plus 42 novel apps), constructs coherent cross-app worlds (persona- and scenario-driven files and app records sharing people, events, and facts), generates task requirements before verifier code, validates verifiers by execution on initial/completed/reset states, and supports failure-driven task generation from diagnoses of failed rollouts.

3. A strong open GUI agent under 10B parameters. MAI-Gym-9B reaches 78.4% on MobileWorld and 76.2% on OSWorld-Verified, described as state of the art among open GUI agents under 10B parameters, outperforming UI-Venus-2-9B (75.2 / 70.8), SCALECUA-Qwen3.5-9B (12.8 / 68.7), and the larger GUI-Owl-1.5-32B (43.9 / 56.5).

4. Open resources for reproducible research. The release includes the framework (evaluation harnesses and online-RL training code), the synthesis skills, 49 mocked apps with accompanying tasks, 2,000 tasks with execution-tested verifiers (1,000 each for MobileWorld and OSWorld), and the model checkpoint.

Main Findings

• State-of-the-art results under 10B parameters: MAI-Gym-9B scores 78.4% on MobileWorld and 76.2% on OSWorld-Verified, benchmarks whose tasks never appear in training.
• Sim-to-real transfer from a cloned app: RL in a clone of the train-ticketing app (12306) raises success on 52 tasks in the real app, evaluated on a physical phone, from 28.8% to 48.1%.
• Transfer from entirely novel apps: training on invented apps improves held-out task success from 25.8% to 45.2%, which the authors present as evidence of generalizable GUI capability learned through interaction.
• Generalization beyond training environments: the paper reports that RL consistently improves over the SFT warm-up checkpoint across all five benchmarks outside its training environments (per-benchmark values are in the experimental section, which is truncated in the supplied content).
• Joint mobile-plus-computer training: a single RL run draws tasks from both phone and desktop domains through one rollout path, instead of running per-domain RL and merging experts by parameter averaging.
• Infrastructure performance: median reset time of 165 seconds on Android, 37 seconds on desktop virtual machines, and 6 seconds in the browser; 1,000 fresh instances come up in 328.4 seconds for mobile and 88.9 seconds for computer.
• Failure isolation as a design requirement: environment failures are recorded separately from zero rewards so infrastructure faults do not enter the training signal; evaluation reruns attempts that ended in environment failure rather than agent failure.
• Not reported in the supplied content: the detailed per-benchmark result tables, ablations, and the comparison of GRPO versus asynchronous optimizers such as SAO referenced in Section 4.3.6 fall beyond the truncated text; only the headline numbers and baseline comparisons above appear.

Methodology in Plain English

One harness, one path. The same model can score differently under different harnesses because copies drift apart in screenshot preprocessing, history truncation, prompt templates, or action parsing. MAI-Gym therefore keeps a single harness per environment-and-agent configuration. A scheduler queues task attempts; workers get an instance, run the harness, query the inference engine at each step, and write the trajectory plus verifier reward to a buffer. Evaluation scores those trajectories, data collection stores them, and RL trains on them — so a fix to the agent code reaches all three.

Environment manager and adapters. Five operations (request instances, observe, reset, step, evaluate) hide the details of provisioning, session management, task registration, reward computation, and trajectory persistence. Adapters translate these into each benchmark's native APIs and keep its own initialization and verification; deployment backends run instances on Kubernetes or Docker or attach to physical devices and browser sessions.

Making resets and scale work. On mobile, the standard Android emulator needs nested virtualization, which most clusters lack, so each phone runs as a ReDroid container on the host kernel and resets by restoring a golden /data snapshot. On desktop, the team added stricter readiness checks, mid-attempt recovery, per-instance ports and artifact caches, and waits for apps to finish saving. On web, headless Chromium hosts both real sites and the mocked apps. Provisioning is asynchronous, warm pools keep initialized instances ready, and screenshots travel through object storage rather than control messages. Failures are confined to the session that caused them through leases, idempotent action identifiers, deadlines, and isolated verifier process groups.

Cheaper long rollouts. Instead of keeping the last k screenshots (which changes the prompt prefix every step and forces re-prefill), history is grouped into windows of K = 10 steps overlapping by O = 2. Within a window prompts only grow, so the inference engine reuses its KV cache; at boundaries old screenshots become placeholders while their text remains. The same windows are reused as training samples, with loss applied only to each window's new steps, so no output is trained twice.

Diagnose, then synthesize. A dashboard ties scores to per-task attempts and replays each attempt against screenshots, actions, and verifier results, separating agent failures from environment failures. Coding agents query these records to compare successful and failed attempts and report which capabilities are weak; the task generator targets those. Synthesis builds a coherent world from a persona and scenario, writes task requirements (including a hidden answer context) before any verifier exists, generates golden code and verifier code in two independent calls, and accepts a task only if the verifier scores 0 on the initial state, 1 after golden code, and 0 after reset — followed by model attempts reviewed for executability, difficulty, clarity, and reward-hacking risk.

Training. The trainer builds on slime, using Megatron-LM for training and SGLang for generation, and supports GRPO, DAPO, GSPO, REINFORCE++, SAO, and PPO over the same records. The main recipe is GRPO with a group baseline; task sampling estimates each task's success rate and gives full groups to unseen or sometimes-solved tasks, while never-solved and mastered tasks (above η = 0.8) get small budgets of 4 and 2 attempts so they are rechecked. Trainer-versus-rollout log-probability agreement is monitored with the k₃ estimator of KL divergence to catch corrupted generation or misaligned packed history. MAI-Gym-9B starts from Qwen3.5-9B, gets a supervised fine-tuning warm-up covering both mobile and desktop, then one joint RL stage on 5,000 synthesized tasks.

Why This Matters

Impact on research. The paper targets a measurement problem as much as a training problem: because harness differences can move a fixed model's benchmark score, released numbers are hard to compare. A shared harness plus per-attempt replay makes scores traceable, and the synthesis pipeline addresses the task-supply bottleneck that limits online RL for GUI agents. Releasing training code alongside benchmarks lets others reproduce results that previously required proprietary infrastructure.

Real-world applications:

• Transactional workflows. Clones of banking (China Construction Bank), ticketing (12306), ride hailing (DiDi), food delivery (Meituan), local services (Dianping), and social media (Douyin, Xiaohongshu) let agents practice money-moving and ordering flows thousands of times without side effects.
• Desktop knowledge work. Cross-app worlds with project briefs, budget spreadsheets, email threads, and calendar entries train agents to find and combine evidence across files and applications instead of following isolated navigation instructions.
• Adapting to unfamiliar software. Novel apps that no model has seen test whether an agent can learn a new interface through interaction, relevant to internal tools and newly shipped products.
• Web task automation. WebVoyager, OnlineMind2Web, and WebArena support on headless Chromium extend the same harness to browser-based workflows.

Industry relevance. The platform lowers the cost of standing up GUI-agent RL, offers an open 9B checkpoint that fits modest hardware, and was used to train Qwen-UI-Agent, indicating adoption beyond a single research artifact. The sim-to-real result in particular argues that mocked clones are a viable substitute for risky live systems.

Future Directions

• Closing the loop over multiple rounds. The paper explicitly states that repeated diagnosis–synthesis–retraining cycles are supported but not yet evaluated, leaving open whether returns compound.
• Broader environment and app coverage. Only seven real-world clones and 42 novel apps are released; whether the synthesis skills generalize to other app categories and platforms is untested here.
• Verifier reliability at scale. Verifiers combine code checks and VLM judges; the paper flags reward hacking as a risk and adds trajectory review, but the robustness of accepted rewards under sustained optimization is an open question.
• Wider sim-to-real evidence. Transfer is demonstrated for one clone (12306) on a physical phone; whether the effect holds across other clones, device types, and longer training is not established in the supplied content.

Target Audience

Researchers and engineers building GUI or computer-use agents; RL infrastructure teams who need scalable, resettable interactive environments; benchmark and evaluation maintainers concerned with harness-dependent scoring; and applied teams in banking, travel, delivery, and enterprise software who want to train agents on realistic workflows without touching production systems. Readers without RL background can still follow the systems and synthesis sections, but the training recipe assumes some familiarity with policy-gradient methods.

Authors’ abstract

Online reinforcement learning (RL) with verifiable rewards has driven rapid progress in mathematics, coding, and tool use, but scaling this success to GUI agents remains challenging due to the high cost of interactive environments and the scarcity of realistic training tasks. We introduce MAI-Gym, an open all-in-one platform that unifies GUI agent evaluation, data collection, and online RL training across mobile, computer, and web environments. Through a shared rollout pipeline, MAI-Gym ensures consistent agent behavior across evaluation and training, supports ten established benchmarks, and scales to 20,000 parallel environment instances. Beyond scalable infrastructure, MAI-Gym enables automated synthesis of diverse GUI applications and challenging tasks, including resettable mocked apps and coherent cross-app worlds with execution-validated task verifiers. Using MAI-Gym, we train MAI-Gym-9B, a unified 9B agent for mobile and computer use, achieving state-of-the-art performance among open GUI agents under 10B parameters: 78.4% on MobileWorld and 76.2% on OSWorld-Verified. Importantly, RL in a mocked train-ticketing app transfers to its real-world counterpart, improving task success on a phys- ical phone from 28.8% to 48.1%. We release the framework, synthesis skills, mocked apps, 2,000 verified tasks, and the model checkpoint to support repro- ducible research and further advances in scalable GUI agent training.