Mage-VL-Cowork: A Full Training Recipe to Learn Hybrid Computer Use From Evolved Harnesses

AI summary

Overview

Research area: Computer-use agents (CUAs) — vision-language models that operate desktop and web applications — with a focus on post-training recipes, synthetic environment generation, and reinforcement learning infrastructure.

Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, RL for LLM agents (GRPO, importance sampling), GUI grounding benchmarks, and distributed rollout infrastructure.

Scope (one sentence): The paper presents a complete end-to-end training recipe — data synthesis pipeline, five-stage curriculum, and asynchronous RL system — for a 4B parameter agent that acts through both a graphical interface and a terminal, reaching 66.0% on OSWorld-Verified.

What This Paper Is About

Most computer-use agents are restricted to clicking and typing through screenshots, which is expensive (a screenshot can cost thousands of tokens) and forces the model to re-derive through the GUI what a single command could accomplish. This paper argues that a capable agent should instead choose freely between GUI actions and terminal/code execution within one policy, and that the hard part is teaching that channel-routing behavior. The authors build a 4B hybrid agent, Mage-VL-Cowork, by first evolving the harness (tools, prompts, scaffold, terminal environment) of a stronger teacher model so its demonstrations show good hybrid behavior, then distilling those demonstrations into a small model through a five-stage curriculum.

Key Contributions

1. An evolved teacher harness instead of default distillation. Rather than rolling out a teacher under its stock configuration, the authors iterate on the harness components — scaffold, tool interface, system prompt, and terminal execution environment — selecting variants that improve the success–cost trade-off. Evolution includes removing tools (e.g., dropping xdotool in favor of native screenshot-grounded GUI actions) as well as adding domain libraries such as python-pptx.

2. Mage-CUA-Forge, a workflow-first synthesis pipeline. Instead of an open-ended agent loop that constructs each task from scratch, an authoring agent builds one reusable generation workflow per atomic skill, combining LLM nodes with application-specific "preparers" that materialize input artifacts (e.g., a Calc workbook or a Thunderbird profile). This yields 293,806 tasks across 6,915 atomic skills and 318,837 retained teacher trajectories, while cutting recurring per-task expenditure by 82.6%.

3. A single policy with an internal hybrid execution policy. Mage-VL-Cowork observes full-desktop screenshots, emits coordinate mouse/keyboard actions, and in the same episode may run code and terminal commands. Unlike orchestrator-level systems that hand off between a separate GUI agent and a coding agent, the channel choice is internal to the policy and must therefore be taught by the training data.

4. A progressive five-stage curriculum from a general checkpoint. Starting from an intermediate 4B Mage-VL checkpoint rather than a model already specialized for GUI or coding, the pipeline runs GUI-oriented midtraining, GUI agent SFT, coding & terminal SFT, hybrid CUA SFT, and online hybrid RL, supported by a decoupled Rollout-as-a-Service infrastructure.

Main Findings

• State-of-the-art at 4B scale: Mage-VL-Cowork-4B reaches 66.0% on OSWorld-Verified with a 50-step budget, above Muse Glimmer-30B (65.9%), Qwen3.6-27B (63.9%), EvoCUA-32B (56.7%), and OpenCUA-72B (44.9%). Human performance is marked at 72.4%.

• Large gain over the same-size baseline: It exceeds Fara1.5-4B (13.9%) by 52.1 percentage points overall.

• Category strengths: 87.5% on OS and 62.7% on multi-application Workflow tasks — the highest OS score in the table, and 14.6 percentage points above ScaleCUA-Qwen3.5-9B (48.1%) on Workflow.

• Curriculum drives steady improvement: OSWorld-Verified rises from 45.56% after GUI SFT to 55.84% after Hybrid SFT (+10.28 pp) and 66.0% after RL (+10.16 pp), a total climb of 20.44 percentage points.

• Cheaper and shorter execution: Mean model calls per task fall from 25.36 (GUI SFT) to 21.57 (Hybrid SFT) to 18.77 (RL); GPT-5.4-mini averages 18.24. RL cuts model calls by 13.0% and reduces output and think tokens by roughly 25.4%, bringing mean output tokens to 5,499 per task (from 7,374 at Hybrid SFT and 6,401 at GUI SFT).

• Grounding improves during midtraining: OSWorld-G accuracy rises from 30.6% to 56.9% (surpassing Qwen3.5-4B at 54.5%), and ScreenSpot-v2 from 78.8% to 87.2% (exceeding Qwen3-VL-4B at 85.3%).

• Synthesis cost collapses: Per-task expenditure drops from $0.906 to $0.158 (82.6% reduction) and tokens per task from 1,276K to 85K (93.3% reduction), both using gpt-5.6-terra.

• Harness evolution matters for the student: The GUI-and-coding harness variant yields a 12.5-percentage-point gain when distilled into the 4B student, compared with its AX-augmented counterpart.

• Browser performance: On the offline WebVoyager-Static benchmark the SFT model reaches 58.6% (377/643 tasks), second to the specialized Fara1.5-4B (68.0%) but above Qwen3-VL-8B-Thinking (53.7%), Qwen3.5-9B (53.3%), OpenWebRL-4B (47.1%), MolmoWeb-8B (46.8%), and Fara-7B (40.4%).

• Live web benchmarks inflate scores: Comparing published live WebVoyager results to the offline re-evaluation shows drops of 12.8 to 33.1 percentage points across the five models with self-reported live results, attributed to a judge that inspects only the final screenshot and instructs the evaluator to believe unsupported claims.

• Dataset composition: 61.76% single-application and 38.24% multi-application tasks; 877,608 task–file references across 278 suffix categories; GUI and terminal actions account for 68.19% and 31.81% of teacher tool calls respectively.

Methodology in Plain English

The authors start from an already multimodally aligned 4B checkpoint and treat training as a staged apprenticeship.

First, they fix the teacher rather than the model: they rewrite the surrounding tooling — the execution loop, the available tools, the system prompt, and which command-line utilities and Python libraries are installed in the sandbox — and keep whichever version scores best while using fewer tokens. This matters because a strong model with a clumsy toolset produces demonstrations with bad habits, and those habits are exactly what a small student copies.

Second, they generate tasks efficiently. Rather than paying an agent to invent each task independently, they build one reusable workflow per atomic skill. Those workflows share "preparers" that create the starting files and application state (a spreadsheet, an email profile), plus automated gates that check the instruction, setup script, and grading rubric agree with each other. Tasks from the same workflow are cheap variations. For web tasks, they use self-hosted deterministic site replicas so state resets cleanly and grading can check the backend database rather than trusting a screenshot. Student evaluation results feed back into expanding and refining the skill taxonomy over three rounds.

Third, they train in five stages: build visual grounding and structured tool-calling (midtraining), learn GUI-only execution, learn coding and terminal execution, then learn to combine both channels, and finally refine with online RL on 1,988 verifiable tasks using GRPO. Long trajectories are folded into turn-level examples that keep only the three most recent screenshots and four complete responses, with earlier steps compressed into short imperative summaries that act as working memory.

Finally, the RL infrastructure separates concerns: CPU workers drive desktop environments, GPU services handle inference and training, and a relay carries control, inference, and trajectory traffic so that episodes running over 40 minutes do not block optimization. The reference deployment uses 1,120 GUI slots across 70 CPU hosts and 32 B200 GPUs (24 for training, 8 for inference).

Why This Matters

The work shows that for computer-use agents, how you collect demonstrations can matter as much as model size — a 4B model trained on well-engineered teacher trajectories beats open-source agents 7.5–18× larger. It also argues that hybrid GUI-plus-terminal operation should be a property of a single policy rather than an orchestration layer bolted on top, and that live web benchmarks substantially overstate progress.

Real-world applications:

• On-device desktop assistants that edit spreadsheets and slide decks via library calls but inspect rendered results visually.
• Enterprise workflow automation across office suites, email clients, and internal web tools, where deterministic self-hosted replicas make training and evaluation reproducible.
• Accessible computing, where a small local agent can operate applications on modest hardware.
• Reliable agent evaluation, via offline replicas that grade against backend state instead of a final screenshot.

Industry relevance: the 82.6% reduction in per-task generation cost and the reusable workflow/preparer abstraction speak directly to the economics of building agent training data at scale, and the demonstrated gap between live and offline web scores is a practical caution for anyone reporting agent benchmark numbers.

Future Directions

• Extending the curriculum beyond the current scope — the paper evaluates the final RL checkpoint on 361 OSWorld-Verified tasks excluding Google Drive, so broader application and benchmark coverage remains open.
• Whether harness evolution can continue during training rather than being frozen after selection, given that the harness is held essentially fixed across later generation rounds.
• Closing the remaining gap to human performance (66.0% versus 72.4%) and understanding which failure categories dominate at the frontier.
• Generalizing the workflow-first synthesis abstraction to applications where a "preparer" is hard to define, and to domains beyond desktop and browser productivity software.

Target Audience

Researchers and engineers building computer-use or GUI agents, particularly those working on post-training data pipelines, synthetic environment generation, or RL infrastructure for long-horizon agentic tasks. It is also useful for practitioners who need to evaluate desktop and web agents reliably, and for readers interested in how small models can be brought to competitive performance through supervision design rather than scale. The density of systems and data-engineering detail makes it most accessible to readers with prior exposure to agent training.

Authors’ abstract

Effective computer-use agents (CUAs) should seamlessly combine visual GUI interaction with code and terminal execution within a single policy. We present Mage-VL-Cowork, a 4B hybrid CUA trained from a general multimodally aligned checkpoint through an end-to-end, progressive five-stage curriculum across midtraining, SFT, and RL. To power this curriculum at scale, Mage-CUA-Forge replaces per-task agent construction with reusable, skill-conditioned generation workflows and application-specific preparers, synthesizing 293,806 desktop and web tasks across 6,915 skills while reducing recurring per-task expenditure by 82.6%. Rather than directly distilling a default teacher, we evolve the teacher harness and instantiate specialized CUA harnesses to yield high-quality trajectories alongside stage-specific supervision. Mage-VL-Cowork-4B finally climbs to 66.0% on OSWorld-Verified through different training stages with more efficient reasoning cost, achieving state-of-the-art performance at its parameter size and pushing forward the frontier of on-device intelligence.