MintAct: A Unified Visual Agent for Digital Environments

AI summary

Overview

Research area: Computer vision and multimodal agents — specifically vision-language models that control digital devices (mobile, desktop, web) directly from screenshots and also invoke external tools.

Technical level: Advanced. The paper assumes familiarity with vision-language model training, supervised fine-tuning, reinforcement learning (GRPO-style group-relative objectives), asynchronous RL infrastructure, and GUI agent benchmarks.

One-sentence scope: The paper presents MintAct, a family of unified 2B, 4B, and 8B vision-language agents that combine UI grounding, multi-step navigation across mobile/desktop/web, and visual tool use in a single set of weights, trained with a multi-stage SFT-plus-RL recipe on scalable heterogeneous environment infrastructure.

What This Paper Is About

Today's digital-use agents are fragmented: UI grounding, mobile navigation, desktop control, web navigation, and visual tool use are each built, trained, and benchmarked as separate specialist models. This is costly to serve and impractical at the compact model sizes needed for on-device use. MintAct asks whether a single model can unify all of these capabilities across three scales without giving up per-domain performance, and answers that this requires solving not just a modeling problem but also data, environment, and training-distribution problems.

Key Contributions

1. A unified model family at three scales. MintAct is trained at 2B, 4B, and 8B parameters and unifies three capabilities: UI grounding, multi-step navigation across mobile, desktop, and web, and visual tool use — all with one set of weights.

2. Evidence that unification does not degrade per-domain quality. The authors report that a single MintAct model matches or exceeds size-matched specialist baselines (each trained on an individual domain) across grounding, mobile navigation, desktop navigation, web navigation, and visual tool use simultaneously.

3. An asynchronous RL framework for unified visual agents. The framework explicitly controls the cross-domain training distribution and remains stable under noisy environment feedback and off-policy drift, addressing heterogeneous environment interaction and long multimodal trajectories.

4. Scalable environment infrastructure. The same infrastructure serves both online RL rollouts and offline SFT trajectory collection at high throughput, sustaining hundreds of concurrent environment instances across domains.

Main Findings

• State-of-the-art at comparable sizes: MintAct-8B reaches 48.9 on OSWorld-Verified, 39.1 on Online-Mind2Web, and 67.0 on AndroidWorld. The abstract describes this as state-of-the-art performance "across a wide range of benchmarks at comparable model sizes."

• Specialist-matching, not specialist-sacrificing: Across all three scales (2B, 4B, 8B), MintAct models match or exceed per-domain specialists of similar size across grounding, mobile/desktop/web navigation, and visual tool use — and are described as competitive with the best public models at comparable sizes.

• Every training stage contributes: Ablations are reported to show that each stage of the multi-stage recipe adds value: high-resolution single-step SFT, low-resolution multi-step SFT, per-domain RL specialists distilled back via RFT, and final joint agentic RL.

• Three design choices enable unification: (i) a shared observation and grounding space — raw screenshots only, with pixel coordinates normalized to 999×999 across domains and no DOM elements, accessibility trees, or platform APIs; (ii) prompt-conditioned action sets — each domain's actions are exposed through a domain-specific system prompt rather than collapsing all action sets into one merged space; (iii) balanced cross-domain mixing, maintained through both SFT and RL.

• Environment scale achieved: The pipeline concurrently hosts more than 200 desktop environment instances and more than 100 mobile emulator instances during RL training. Desktop instances are deployed on a Linux cluster with 10 CPU cores and 40 GB of memory each.

• Data scale reported: Multi-step SFT uses roughly 42.7k mobile, 23.5k desktop, 51.7k web, and 59.8k visual-tool-use trajectories, mixed at a uniform 25%:25%:25%:25% ratio. The RFT stage adds roughly 37.8k (mobile), 21k (desktop), 7k (web), and 24k (visual tool use) filtered trajectories. RL curation yields roughly 3k OSWorld tasks, roughly 3k AndroidWorld tasks, a 10k-task web RL suite, and roughly 800 visual-tool-use scenarios.

• Tool-use environment is large: MM-ToolSandBox contains more than 500 native tools across 16 application domains, managed dynamically via a search_tool for on-demand retrieval and a coding_tool for code-based operations.

• Thinking-trace relabeling matters for mobile: The teacher model's thinking traces lacked reasoning detail, so the authors added an offline labeling pass with a frontier VLM that, given the rollout up to the current step plus the ground-truth action, produces the thinking trace for that step. This relabeling is described as vital for learning a capable agent.

Methodology in Plain English

MintAct is trained in stages, each targeting a specific gap:

1. High-resolution single-step SFT. The model first learns perception and grounding from public static datasets (GroundCUA, MolmoPoint, GUI-Odyssey, AndroidControl). Input is capped at 2,116,800 pixels per image, trained for one epoch at a learning rate of 1×10⁻⁵. Grounding outputs go into <grounding_think>…</grounding_think> followed by <answer>…</answer>; single-step navigation uses a Thought: … Action: … format.

2. Low-resolution multi-step SFT. To handle long screenshot histories efficiently, resolution drops to 921,600 pixels per screenshot. Training mixes trajectories from mobile, desktop, web, and visual tool use at an equal 25% split. The format is append-only (each turn added to a running sequence) so that earlier KV cache can be reused during later RL. Trajectories are capped at 30 turns for mobile, desktop, and web, and 100 turns for tool use.

3. Per-domain RL specialists, distilled back. One RL specialist is trained per domain, then distilled into a single model via rejection-sampling RFT: specialists roll out on their domain's SFT tasks, a strong VLM judge keeps only successful trajectories, and the SFT model is fine-tuned on the result for one epoch at 3×10⁻⁶ with the same balanced 25% mixing.

4. Joint agentic asynchronous RL. The final stage has the model interact directly with the multiple environments. Rollout generation and policy optimization are decoupled: a Rollouter drives many concurrent rollouts through an SGLang router; an Environment Manager holds separate isolated pools for mobile, web, desktop, synthetic, and visual-tool backends and hands out exclusive environments per rollout; a FIFO message queue buffers trajectory groups with bounded staleness; a Trainer consumes groups, reconstructs trajectories from token indices and multimodal inputs, and recomputes proximal-policy log probabilities before updating with Megatron; a Parameter Synchronizer periodically pushes updated weights back to the Rollouter. Historical images are offloaded to disk during rollout, with only file references kept in the queue, and loaded on demand for training.

Environment construction. Desktop builds on OSWorld inside Docker containers, with HTTP-based middleware; SFT tasks are expanded iteratively using EvoCUA-32B to roll out current tasks and another strong VLM to propose new tasks grounded in what those rollouts reveal, with a VLM-as-judge filtering failed rollouts. Mobile builds on the AndroidWorld emulator. Web uses Weblica-Cache (recorded Playwright sessions replayed deterministically offline, with volatile tokens stripped) and Weblica-Synth (an autonomous coding agent generating self-contained HTML/CSS/JS websites); web SFT rollouts are generated by Qwen3-VL-32B-Instruct over domains sampled from InstaV3, with only fully verified successful trajectories retained. Visual tool use builds on MM-ToolSandBox with a multi-stage scenario generation pipeline using 100K images from COYO and Aria UI.

Difficulty-aware RL curation. Across domains, the authors sample multiple trajectories per task and keep tasks with a mix of successes and failures, favoring those near a 50% success rate (centered around 0.5 for visual tool use) to maximize learning signal for the group-relative objective.

Why This Matters

Impact on research. The paper argues that unification of visual agent capabilities is not just a modeling problem but a data, environment, and training-distribution problem — and demonstrates a concrete recipe (prompt-conditioned action sets, shared grounding space, enforced balanced mixing) that avoids the domain interference that naive action-set merging or data mixing causes. It also extends asynchronous RL from single-domain or single-modality settings to a unified multi-domain visual-agent setting, which the authors note prior systems largely did not target.

Real-world applications:

• On-device digital assistants that operate phones and desktops from pixels, at compact 2B–4B scales suited to on-device use.
• Web automation agents that complete multi-step browsing tasks and stateful interactions in reproducible offline environments.
• Accessibility and productivity tooling where a single model can both manipulate a GUI and call APIs or run code when UI actions are insufficient.
• Visual assistants that reason across multiple images over multi-turn conversations, using tools and code for information retrieval, application operations, and visual inspection.

Industry relevance. Maintaining one specialist per domain is costly to serve and scale, especially at small sizes; a single unified model family simplifies deployment and infrastructure. The environment and RL infrastructure — hundreds of concurrent instances, HTTP-decoupled environments with fault tolerance (a failed instance can be terminated and restarted without interrupting training) — speaks directly to production-scale training and evaluation pipelines. The work comes from Apple.

Future Directions

• Scaling and latency characterization. The paper reports results at 2B, 4B, and 8B scales but does not report throughput, latency, or compute-cost comparisons between the unified model and per-domain specialists; whether the advantages hold at larger or much smaller scales is an open question.
• Extending the environment and domain coverage. The current domains are mobile, desktop, web, and visual tool use, built on AndroidWorld, OSWorld, Weblica, and MM-ToolSandBox. Whether the same recipe transfers to additional platforms or interaction settings (e.g., cross-device tasks) is not addressed in the paper.
• Robustness of the asynchronous RL stability claims. The framework is described as stable under noisy environment feedback and off-policy drift, but the paper content does not report the sensitivity of this stability to queue capacity, rollout staleness bounds, or the choice of the 25%:25%:25%:25% mixing ratio.
• Longer horizons and richer reasoning supervision. The trajectory caps (30 turns for mobile, desktop, and web; 100 turns for tool use) and the mobile thinking-trace relabeling point to horizon length and trace quality as active constraints; whether longer-horizon tasks require new training mechanisms is left open in the available content.

Target Audience

Researchers and engineers working on GUI agents, vision-language models, and multimodal agent training — particularly those interested in unifying multi-domain capabilities, on-device compact models, or asynchronous reinforcement learning at scale. It will also be useful to practitioners building digital-assistant products who need to weigh a single unified model against per-domain specialist deployments. The paper is advanced reading: readers should already be comfortable with RL post-training terminology (GRPO-style grouping, off-policy staleness, micro-batching) and with GUI agent environments and benchmarks.

Authors’ abstract

We present MintAct, a family of vision-language models that unifies UI grounding, multi-step navigation across mobile, desktop, and web, and visual tool use, trained at 2B, 4B, and 8B scales. Through careful design of our environments, data, and training recipes, MintAct models match the performance of per-domain specialists across all of these capabilities. To enable this, we develop a scalable environment and reinforcement learning (RL) infrastructure. On the environment side, we host hundreds of concurrent instances across heterogeneous per-domain backends, serving both trajectory data collection and online RL. To enable efficient and scalable RL training, an asynchronous framework keeps explicit control over the cross-domain training distribution and remains stable under noisy environment feedback and off-policy drift. Experimental results show that MintAct achieves state-of-the-art performance (48.9 on OSWorld-Verified) across a wide range of benchmarks at comparable model sizes.