OS-Helix: Unifying GUI Interaction and Code Execution in Native Computer-Using Agents

AI summary

Overview

Research area: Machine learning for computer-using agents (CUAs) — specifically, training a single vision-language policy that can both click through a graphical user interface and execute Python/Bash commands on a real desktop (Ubuntu, plus Windows transfer).

Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, online reinforcement learning for LLM/VLM policies (GRPO/GSPO-family objectives), importance ratios and clipping, and desktop agent benchmarks such as OSWorld.

One-sentence scope: OS-Helix is an end-to-end trained multimodal agent that unifies GUI actions and code execution in one action space, built on a synthetic desktop-environment/task pipeline plus a modified RL objective ("Step-balanced GSPO") that stops long code actions from dominating training.

What This Paper Is About

Desktop work is neither purely visual nor purely programmatic: sometimes you must look at a screen and click, and sometimes you should just run a shell command on the file. The paper's starting point is that you cannot get this coordination for free by handing a GUI agent a code tool at inference time — in the authors' preliminary experiment, Qwen3-VL-32B-Instruct scores over ~34% on OSWorld-Verified with GUI actions alone but drops to roughly ~32% when code execution is naively exposed. The goal is therefore to internalize the routing decision between GUI and code into the model through training on high-quality hybrid data, in reproducible executable desktop environments.

Key Contributions

1. Problem formulation plus a negative control. The authors frame native GUI/code hybrid computer use as an end-to-end policy-learning problem, and show by controlled experiment that merely exposing a GUI policy to code execution degrades it rather than producing reliable routing.
2. Data infrastructure for executable desktop states. They introduce application–artifact composition — pairing versioned applications with authentic file artifacts through an application–artifact compatibility graph to instantiate reproducible Ubuntu states — plus a decoupled instruction/evaluator synthesis pipeline in which an Exploration Agent proposes grounded instructions and an independent Evaluator Agent builds mostly rule-based verifiers, with an "everything-is-a-file" abstraction so the same task can be verified regardless of whether it was solved via GUI, code, or both.
3. Step-balanced GSPO. A modification to the RL objective that aggregates the sample-level importance ratio over action steps rather than raw assistant tokens, so a long code action counts the same as a short click, preventing code-heavy drift in heterogeneous-action optimization.
4. Released artifacts. The authors state they will release the model family, the desktop data-synthesis and RL environments, task/trajectory collection pipelines, approximately 9K generated instructions, 83K unfiltered SFT samples, and about 31K successful filtered SFT samples in the unified GUI/code action space.

Main Findings

• Hybrid training beats tool access alone. On OSWorld-Verified, OS-Helix-27B reaches 62.75% success and OS-Helix-35B-A3B reaches 57.27%, versus matched Qwen3.5 hybrid baselines of 46.50% and 42.84% — absolute gains of 16.25 and 14.43 points, reported as relative improvements of 35.0% and 33.7%.
• RL adds more on top. OS-Helix-27B-RL reaches 66.44% success (a further reported 42.9% relative improvement over the hybrid baseline comparison framing), with incremental gains concentrated in Professional (+23.10 points), OS (+7.61), and Daily (+6.69), while Office and Work. do not improve further over already strong SFT results.
• Largest SFT gains are in Office and Workflow. Office improves by +33.3 (27B) and +29.1 (35B-A3B) points; Workflow by +16.0 and +14.1 points. These are the categories that mix visually grounded interaction with precise transformation of structured artifacts, and multi-application handoff.
• Standard GSPO drifts toward code and hurts performance. Under identical SFT initialization, data, sampling, reward, and budget, standard GSPO scores 60.47% (−2.28 vs. SFT) and raises the code-action fraction from 27.9% to 40.0%. Step-balanced GSPO scores 66.44% (+3.69) while holding code actions at 26.1%.
• SFT is the main driver; RL alone is weak. Hybrid backbone 46.50 → RL only 48.20 → SFT only 62.75 → SFT then RL 66.44.
• Code access helps, but learned coordination helps more. The same OS-Helix-27B checkpoint scores 52.13% with code disabled and 62.75% with code enabled (a 10.62-point contribution). At a fixed 10K-sample SFT budget, GUI-only data gives 57.56% (11.7% code actions) versus 60.61% for hybrid GUI/code data (37.1% code actions), a 3.05-point gap.
• Broadening search, not just sharpening one path. RL improves Pass@1 from 62.75% to 66.44% (+3.69) and Pass@3 from 70.88% to 76.44% (+5.56, with a 95% paired-bootstrap interval of [2.24, 8.97]).
• Data robustness. With the same 800 synthesized tasks, Kimi K2.5 teacher trajectories yield 54.03% (+7.53 over base) versus Claude Sonnet 4.6 at 58.77% (+12.27). Removing all samples from the nine OSWorld application domains shrinks the desktop SFT set from ~31K to ~17K and moves success only from 62.75% to 62.31%.
• Cross-platform transfer. On WindowsAgentArena, OS-Helix-27B scores 57.95% and OS-Helix-27B-RL 63.87%, versus the reproduced Qwen3.5-27B baseline at 56.41%, Agent S3 w/ GPT-5 at 54.10%, and OS-Symphony w/ GPT-5 at 63.45%.
• Competitive but not dominant in open comparisons. OS-Helix-27B-RL is competitive with GPT-5-based agentic frameworks, while newer, much larger general models — Kimi K2.6 at 73.07% and Qwen3.7-Plus at 73.33% — remain stronger overall.
• Efficiency. On the full OSWorld set, successful episodes, and outcome-matched subsets, OS-Helix-27B uses fewer interaction steps while covering more successes under small step budgets.

Methodology in Plain English

The pipeline has four stages.

Building worlds. The authors treat a desktop state as the pairing of a fixed-version application with a real file artifact, organized in a compatibility graph keyed on file types (a .pptx connects to LibreOffice Impress and WPS Presentation, for example). These pairings are instantiated on reproducible Ubuntu virtual machines, so states are both diverse and replayable.

Turning worlds into verifiable tasks. A synthesis episode samples an application, selects compatible artifacts, and hands an Exploration Agent the application metadata, file paths, launch configuration, tutorial, and initial screenshot. The agent does a short, non-destructive poke-around (open, click, scroll, terminate) and proposes a realistic user instruction with success conditions, without prescribing a GUI or code solution. A global application memory records which functions have already been exercised per application, pushing exploration toward uncovered functionality. The instruction is then frozen and passed to a separate Evaluator Agent, which compiles verification logic — mostly rule-based checks over VM files and command-line state, with VLM checks for conditions that are only visually observable. Evaluators must compile and must assign zero reward at the initial state; near-duplicates are removed.

Cold start. Teacher trajectories from Claude Sonnet 4.6, with Computer Use and Bash Tool enabled, are normalized into the OS-Helix action schema and rejection-sampled by evaluator score, with malformed calls and schema mismatches filtered out. Training uses one epoch on 31K successful filtered desktop samples plus 40K general-purpose multimodal samples.

Online RL. Rollouts run in live sandboxes — a cross-node system managing over one hundred Ubuntu environments behind a unified port. The evaluator score is the sole outcome reward; any turn with an unparsable tool call is isolated and given reward 0 even if the rest of the trajectory would have scored. Three efficiency measures make this affordable: asynchronous RL where the rollout policy lags the latest policy by at most one version, sampling biased toward tasks near the current success frontier, and bounded sliding-window visual-context segmentation applied consistently at training and inference. RL is a 1,000-step run from the dense OS-Helix-27B checkpoint.

The objective change. Standard GSPO averages token-level log-importance weights over all assistant tokens, so a long code action can dominate both the sample ratio and the clipping decision. Step-balanced GSPO first averages log-importance weights within each action step, then averages across steps with equal weights for GUI and code, so each step contributes equally regardless of token count. The authors note this substitution does not in general equalize full parameter-gradient norms or variances across action lengths.

Evaluation uses OSWorld-Verified (369 tasks, minus 8 Google Drive tasks, leaving 361), with a 50-step cap, up to 8 historical screenshots at 1504x832, and full textual history.

Why This Matters

Impact on research. The paper reframes hybrid GUI/code computer use as a learned capability rather than an orchestration trick. Its two sharper methodological points are negative and transferable: adding a tool to a GUI policy can actively hurt (the ~34% → ~32% drop), and a standard sequence-level RL objective silently reweights behavior when action granularities are heterogeneous. The step-balancing idea and the decoupled instruction/evaluator synthesis are both reusable outside desktop agents.

Real-world applications.

• Office and document automation — the Office category (Calc, Impress, Writer) is where SFT gains are largest, matching the common case where a file must be transformed precisely but checked visually.
• Multi-application workflows — the Workflow category, where information must be handed between applications and action modalities, improves by +16.0 and +14.1 points.
• IT operations and system administration — OS-level tasks benefit from code execution directly (OS +7.61 points under RL).
• Cross-platform enterprise deployment — the WindowsAgentArena results indicate the learned routing is not tied to Ubuntu.

Industry relevance. The efficiency finding matters commercially: fewer interaction steps and more successes under small step budgets translate to lower inference cost per completed task, which is the binding constraint for deploying desktop agents at scale. The released 9K instructions, 83K unfiltered and ~31K filtered SFT samples, and RL environments also lower the entry cost for teams that cannot build desktop sandboxes themselves.

Future Directions

• Why does RL help in some domains and not others? RL gains concentrate in Professional, OS, and Daily, with Office and Workflow showing no further improvement over SFT. Understanding that saturation — whether it is reward sparsity, task difficulty, or verifier coverage — is an open question the paper leaves implicit.
• Closing the gap to large general models. OS-Helix-27B-RL at 66.44% trails Kimi K2.6 (73.07%) and Qwen3.7-Plus (73.33%), so scaling the backbone and the hybrid data jointly is the obvious next axis.
• Generalizing the step-balancing principle. The authors are explicit that step-balanced aggregation does not equalize gradient norms or variances across action lengths. Whether a more principled per-modality weighting (beyond the equal GUI/code weights used here) yields further stability is untested.
• Safety and containment outside research VMs. The ethics statement flags that agents combining screen observation, file modification, and shell execution may encounter personally identifiable information on real devices; the paper's own experiments are confined to isolated VMs, leaving deployment-time safeguards as future work.

Target Audience

Researchers and engineers working on GUI agents, multimodal LLM post-training, or agentic reinforcement learning will get the most from this paper — particularly those who have tried prompt-level tool composition and found it degrades their agent. It is also useful for practitioners building desktop automation products who want to know where hybrid execution actually pays off (Office, OS-level, multi-app workflows) and what the training data and reward design require. Readers without background in policy-gradient methods can still follow the data-infrastructure and results sections, but the Step-balanced GSPO derivation assumes prior familiarity.

Authors’ abstract

Real desktop automation is neither purely visual nor purely programmatic. Agents must understand user-facing interfaces while efficiently manipulating files, scripts, and system state. However, native GUI/code coordination cannot be obtained by prompt-level tool composition. In our preliminary experiments, naively adding a code tool degrades a GUI agent, revealing that GUI/code routing must be internalized through training on high-quality hybrid data. We present OS-Helix, an end-to-end trained multimodal computer-using agent that unifies GUI actions and code execution in one action space. To construct the required supervision, we introduce application-artifact composition, pairing desktop applications with authentic file artifacts to instantiate executable Ubuntu states, and a decoupled instruction/evaluator generation pipeline that produces environment-grounded tasks whose final states can be verified regardless of the solution modality. We cold-start the policy with SFT on filtered hybrid data and further improve it through online RL in live desktop environments. For RL, we propose Step-balanced GSPO, which aggregates importance weight by action step rather than raw assistant tokens to prevent long code actions from dominating heterogeneous-action optimization. On OSWorld-Verified, OS-Helix-27B and OS-Helix-35B-A3B achieve relative improvements of 35.0% and 33.7% over their matched Qwen3.5 hybrid baselines, while OS-Helix-27B-RL further improves by 42.9% to reach 66.44% success. Ablations provide evidence consistent with contributions from learned GUI/code coordination, scalable verifiable supervision, and action step balanced optimization, while demonstrating cross-platform transfer on WindowsAgentArena. Our project homepage is at https://os-copilot.github.io/OS-Helix/.