SoftwareGym: Scaling Computer-Use Agent Training with Diverse Synthesized Software Environments

AI summary

Overview

Research area: Computer-use agents (CUAs), GUI software environment synthesis, and training-data generation for multimodal language models.

Technical level: Advanced. The paper assumes familiarity with agent architectures (multi-agent pipelines, GUI agents), desktop application frameworks (Avalonia/XAML), tree-edit distance and Hamming distance metrics, and supervised fine-tuning (SFT) of vision-language models.

Scope: The paper presents SoftwareGym, an automated multi-agent pipeline that synthesizes diverse, executable desktop GUI applications and the tasks to train and evaluate computer-use agents on them.

Published as arXiv:2610.12611v1 [cs.MA] on 08 Oct 2026 under a CC BY 4.0 license. Authors are affiliated with Shanghai Jiaotong University and Zhejiang University. Code is released at github.com/MrKiing-Jin/SoftwareGym and the model at huggingface.co/GothamLegend/SoftwareGym.

What This Paper Is About

Training computer-use agents requires large amounts of interaction data collected from real, executable software, but building such environments at scale is expensive and slow. Existing options are limited: native desktop applications are costly to integrate and risk the agent overfitting to specific applications, while browser-hosted environments are easier to deploy but restrict how deep and complex a workflow can be. SoftwareGym addresses this by automatically generating complete desktop applications, verifying that they actually work through GUI testing, and using them to produce training trajectories for CUA models.

Key Contributions

1. An automated environment-synthesis pipeline. SoftwareGym produces functionally reliable and structurally diverse software at scale, with substantially lower development time and monetary cost than comparable manually developed applications. It uses a design–implementation–evaluation loop with multi-agent roles (FunctionDesigner, StyleDesigner, SubStyleDesigner, IconDesigner, SubIconDesigner, Coder, DevCoder, EvalTaskGenerateAgent, GUIAgent).

2. A released application library and benchmark. The paper releases 182 runnable applications across twelve categories and constructs SoftwareGym-Bench, a dedicated 120-task test set spanning those same twelve categories.

3. Evidence that synthesized environments transfer. A model trained only on SoftwareGym trajectories improves OSWorld Pass@3 by 15.58 points, from 41.39% to 56.97%, without using any OSWorld applications, tasks, or trajectories for training.

4. Extensive evaluations and ablations. The paper isolates three components: structured style design, GUI-guided implementation repair, and the source of training trajectories.

Main Findings

• Synthesis scale and reliability: The pipeline generates 182 runnable applications across twelve categories with a mean GUI task pass rate of 91.5%.
• Iterative repair works: The mean GUI task pass rate rises from 54.9% at version v1 to 81.1% at v2 and 91.5% at v3, with consistent gains across categories.
• Out-of-domain transfer on OSWorld: Training only on SoftwareGym trajectories lifts Pass@3 from 41.39% (Qwen3.5-9B Base) to 56.97% (Qwen3.5-9B SFT), improving every evaluated domain. OSWorld evaluation uses 244 tasks from eight application domains.
• In-domain performance on SoftwareGym-Bench: The same model raises Pass@3 by 14.2 points, from 25.8% to 40.0%, and exceeds Qwen3.7-Plus (38.3%) and several substantially larger specialized CUAs.
• Comparison to frontier baselines: On OSWorld, Seed 2.1 Pro scores 79.51%, Gemini 3.1 Pro 79.10%, GPT-5.4 73.77%, and GPT-5.5 71.31%, so the trained 9B model remains below the API foundation agents but competitive with larger open-weight agents. On SoftwareGym-Bench, GPT-5.4 leads with 74.2%, followed by GPT-5.5 at 73.3% and Gemini 3.1 Pro at 70.8%.
• Style design prevents interface collapse: Full Design improves the macro-average structural diversity score from 4.9 to 83.6 compared with a simplified baseline, with higher diversity in all categories.
• GUI-guided repair beats source-only repair: Starting from the same initial pass rate of 50.4%, GUI-guided repair reaches 83.1% while source-only repair ends at 48.8%.
• Synthesized and native data complement each other: Training on SoftwareGym alone gives 56.97% on OSWorld and 40.0% on SoftwareGym-Bench; training on OSWorld alone gives 61.48% and 33.3%; combining both gives 58.61% and 47.5%.
• Functional coverage: Across 121 applications in eight categories aligned with OSWorld, GUI evaluations cover all 49 major functional modules and 78.8% of concrete functions on average.
• Cost profile: Sampling 20 applications, each requires on average 237.3 minutes, 14.8 million tokens, 1,718 model calls, and an estimated $70.76.

Methodology in Plain English

The pipeline has three stages, and applications are only admitted to the library after passing all three.

1. Software design. Starting from minimal human input (a software type and supporting resources), a FunctionDesigner agent expands the idea into a full functional specification by mimicking a mature product: it defines target users, core objects, and operations such as creating, editing, organizing, searching, recovering, and saving; then adds professional features like multi-condition retrieval, batch operations, history management, and input validation; then links these into end-to-end workflows. Every function is recorded as a contract specifying user entry, user action, expected state update, visible GUI response, and acceptance checkpoint. Separately, a StyleDesigner generates multiple distinct Avalonia UI blueprints for the same functionality, each following a layout archetype. To stop variants from converging, every pair must differ along at least two of four layout dimensions (navigation, spatial organization, object browsing, action placement), measured as a Hamming distance. Generated XAML files are compiled to confirm they are valid, and blueprints are compared using normalized tree-edit distance over layout trees; if the minimum pairwise distance falls below a 0.60 gate, the least distinct blueprint is regenerated. Icons are generated as style-specific SVG assets with a platform-side validator checking completeness, validity, and safety.

2. Implementation and evaluation. A Coder agent turns each Verified Design Package into a complete runnable Avalonia application, implementing every function as a full GUI workflow and recording acceptance checks in a functional manual. A DevCoder agent audits the code against that manual, checking command bindings, application logic, persistence, startup behavior, and file interfaces. Then EvalTaskGenerateAgent writes GUI test cases (control reachability, navigation, object creation and editing, persistence, retrieval, settings, dialogs, file workflows), each starting from an explicit state and defining visible checkpoints. A GUIAgent executes these cases with mouse and keyboard actions and records traces, checkpoint evidence, failure location, and repair suggestions. Functional pass rate is the number of passed tasks divided by total generated cases. A visual-fidelity gate checks that repairs do not change the intended interface identity, scoring color palette, primary color, corner radii, font, and resource-based binding; the threshold is F_vis ≥ 0.80. The evaluation case set is fixed after the first implementation and reused across repair rounds, with up to three versions (v1–v3) and early stopping when all cases pass.

3. Task synthesis. Only capabilities verified through GUI interaction can be turned into tasks. The pipeline builds an evidence-grounded function tree where a capability counts as "supported" only if it cites a passed case, checkpoint, or interaction step. Tasks are then assigned to easy, normal, or hard difficulty based on dependencies among verified capabilities, and each level has two task types: file-based (starts with an input file) and blank-state (starts from the default GUI state). Finally, each planned workflow becomes a natural user request plus an internal specification recording initial state, outcome criteria, process criteria, and expected final state. Deterministic validation rejects unsupported capabilities, inconsistent initial states, incompatible assets, and unreachable final states.

Experimental setup. Applications were synthesized across twelve categories using Codex CLI agents with GPT-5.5 for design, implementation, and evaluation-case generation. Seed 2.1 Pro evaluated each version through mouse-and-keyboard interaction for up to 50 steps per case in isolated virtual machines running Ubuntu 22.04.3 LTS (x86_64). Multimodal SFT was performed on Qwen3.5-9B using trajectories collected only from SoftwareGym applications. Evaluation used the native OSWorld context structure and tool protocol, temperature 1, an interaction budget of 100 steps, and three independent rollouts per task, reported as Pass@3.

Why This Matters

Impact on research. The paper reframes environment construction as a generative problem rather than an integration problem. Instead of adapting fixed real applications, it expands the software distribution itself, which the authors argue reduces application-specific overfitting. The result that SoftwareGym-only training improves OSWorld while OSWorld-only training is better in-domain suggests synthesized and native data play complementary roles rather than substituting for one another.

Real-world applications:

• Automating desktop workflows, such as batch image conversion with specific format, resize width, and quality settings, or file attachment in a chat application.
• Data-entry and document editing, such as replacing a specific cell value in a spreadsheet while leaving a find-and-replace panel open.
• Software configuration management, such as removing a leftover editor extension while keeping built-in extensions enabled.
• Community and content administration, such as creating a public community with an exact summary string.

Industry relevance. The cost figures matter commercially: 237.3 minutes, 14.8 million tokens, 1,718 model calls, and about $70.76 per application, versus manual development of comparable software. Companies building GUI agents, robotic process automation, or synthetic training data pipelines could adopt this approach to generate supervision in domains where no real applications exist. The public release of 182 applications, a 120-task benchmark, code, and the trained model lowers the barrier to reproducing and extending the work.

Future Directions

• Closing the gap to frontier agents. The trained model reaches 56.97% OSWorld Pass@3 while API foundation agents reach up to 79.51%, so scaling model size, trajectory volume, or data quality remains open.
• Broadening beyond desktop applications. The authors position SoftwareGym as a complement to native software and a contrast to browser-hosted environments; extending the pipeline to other platforms and to persistent multi-application workflows is a natural next step.
• Coverage completeness. GUI evidence covers all 49 major functional modules but only 78.8% of concrete native functions across the 121 aligned applications, leaving room to determine whether the remaining functions are unverifiable, unimplemented, or simply untested.
• Training-data mixing strategy. Combining SoftwareGym and OSWorld trajectories improves SoftwareGym-Bench to 47.5% but gives 58.61% on OSWorld, lower than SoftwareGym-only training; the optimal mixing ratio and the reason for this trade-off are not resolved.
• Note on missing detail: the supplied paper content is truncated partway through Appendix C, so the complete ablation results and the full training configuration in Appendix C.3 are not available in the provided text.

Target Audience

Researchers and engineers working on computer-use agents, GUI automation, and agent training data. Also relevant to practitioners in synthetic data generation, multimodal model fine-tuning, and software engineering automation who need scalable, verifiable executable environments. Readers seeking a beginner-level introduction will find the metrics and multi-agent pipeline architecture require prior background in agent systems and GUI frameworks.

Authors’ abstract

Computer-use agents (CUAs) are rapidly improving, yet synthesizing diverse and reliable software environments at scale remains challenging. Native application environments require substantial construction and integration effort, which limits application coverage and increases the risk of application-specific overfitting. Browser-hosted environments provide limited interaction depth and workflow complexity. These limitations motivate SoftwareGym, an automated pipeline for synthesizing GUI software environments with diverse interfaces, executable functionality, and runtime-verified behavior. It generates structurally diverse designs, realizes them as runnable applications, and uses GUI evaluation with iterative repair to ensure functional reliability. Using SoftwareGym, we synthesize 182 applications across twelve categories and achieve a mean GUI task pass rate of 91.5%. We construct SoftwareGym-Bench, a 120-task evaluation suite spanning the same twelve categories, and use trajectories collected from these environments for CUA training. The resulting model improves Pass@3 from 41.39% to 56.97% on the strictly out-of-domain OSWorld benchmark and achieves 40.0% on SoftwareGym-Bench. These results demonstrate the potential of synthesized software environments as scalable and transferable supervision for CUA training.