Research
OSWorld-Science: A Benchmark of Computer Use Agents for Learning and Using Scientific Software
Provides 146 artifact-evaluated tasks across scientific software for computer-use agents, letting researchers compare VLMs on real research workflows instead of toy GUIs.

- arXiv
- 2609.39903
- Published
- 2026-09-30
- Authors
- Dingyuan Dai, Heli Qi, Lei Liu, Yinxi Li, Baiding Chen, Zijun Dou, Qingcheng Zeng, Qi Kang, Oliver Sun, Eric Wang, Bo Zhou, Haixin Wang, Yufan Du, Shi Bo, Ruihan Lin, Mengqi Yuan, Dunjie Lu, Steven Dillmann, Yiming Shi, Tina Su, Amy Xin, Minghao Liu, Xi Wang, Xu Huang, Ge Zhang, Pengyu Nie, Zhen Yang, Jie Tang, Juanzi Li, Weihao Xuan, Tianyu Liu
AI summary
OSWorld-Science: A Benchmark of Computer Use Agents for Learning and Using Scientific SoftwareOverview
Research area: Artificial Intelligence — evaluation of visual-language-model (VLM) based computer-using agents (CUAs) on real scientific software workflows.
Technical level: Intermediate. The paper assumes familiarity with agent benchmarks, GUI/CLI action spaces, and VLM evaluation, but its methods and findings are described in applied terms.
Scope: The paper introduces OSWorld-Science, a benchmark, environment, and evaluation harness for measuring how well VLM-driven agents can operate professional scientific software and produce verifiable scientific artifacts.
What This Paper Is About
Computer-using agents can already perceive screenshots, reason over instructions, and issue mouse and keyboard actions, but ordinary desktop benchmarks say little about whether they can navigate specialized scientific software, manipulate scientific objects such as molecular structures or segmentation masks, and leave behind results that a domain expert would accept. The authors build a benchmark of 146 tasks across scientific domains and software packages, pair each task with an execution-based evaluator that inspects the resulting application state and artifacts, and add a harness that lets many different VLMs be compared under identical interaction rules. The goal is to determine how much of scientific computer use depends on the backbone model versus the harness, and where current agents still fail.
Key Contributions
-
An importance-driven scientific task set. 146 tasks spanning scientific domains and 14 recorded software configurations, collected through expert proposals and iterative human–AI co-design, with selection guided by scientific value and difficulty rather than convenience.
-
Evidence-based, artifact-level evaluators. Task-specific evaluators inspect application states and generated artifacts — molecular structures, segmentation masks, plots, statistical measures, numerical results — and award partial credit, normalizing heterogeneous outputs into a 0–1 pass rate.
-
An efficient, portable agent harness. A harness integrating model adapters, a loop controller, trajectory logging, retry and rollback logic, and a user-interaction platform, used to run 12 VLMs under a single root agent (the multimodal PromptAgent "mmagent" of the OSWorld base release, ported unchanged in its message construction and prompting).
-
Diagnostic analysis of trajectories and harness design. Fine-grained error and trajectory labeling plus controlled ablations of reasoning effort, history window length, and prompt language, used to separate model capability from harness configuration.
Main Findings
-
Frontier models still fail on scientific software. The strongest agent reaches a mean task-specific score of 73.7% (Claude Fable 5.1, at $6.5 per task), with the introduction reporting roughly 70% average and roughly 40% on the most difficult set.
-
Performance and cost are only loosely coupled. Claude Fable 5.1 scores 73.7% at $6.5 per task; Claude Opus 5 scores 57.9% at $6.7; GPT-6 Astra scores 57.4% at $7.7; GPT-5.6 sol scores 50.1% at $2.79; GPT-5.6 terra 31.8% at $2.0; GPT-5.6 luna 22.6% at $0.23. Kimi K3 reaches 30.2% at $11.7 per task, approximately 51 times the cost of GPT-5.6 luna, and Claude Sonnet 5 reaches 26.3% at $3.6, approximately 16 times that cost.
-
Domain difficulty is highly uneven. Statistics mean scores range from approximately 66% to 97%, with GPT-5.6 luna reaching 88% at about $0.06 per task. Chemistry is far harder: Claude Fable 5.1 and Opus 5 lead at approximately 44% and 39%, GPT-5.6 sol reaches 20%, and eight models score approximately 6% or less. EEG ranges from 89% (Fable 5.1) to 54% (Astra) to 33% (Opus 5), with all remaining models at 15% or less. Astra leads medical imaging and geoscience at approximately 69% each, and Fable 5.1 reaches approximately 96% in biology.
-
Model rankings invert across domains. Astra scores about 6% in chemistry yet 100% on the two linguistics tasks.
-
Most failures produce no artifact at all. Of the released runs, 402 (26.3%) receive full credit. Of the remaining 1,100 (excluding 20 harness-ended and 8 unresolved runs), 667 (60.6%) end without the graded artifact, 400 contain wrong content, and 33 produce an invalid artifact; 613 of the 667 exhaust their budget. Claude Fable 5.1, Claude Opus 5, GPT-6 Astra, and GPT-5.6 sol deliver nothing in 1–25% of their runs, while Claude Sonnet 5, Gemini 3.1 Pro, Qwen3.7-Plus, Qwen-CUA, and MiniMax M3 deliver nothing in 60–78%.
-
Placeholder artifacts are concentrated in chemistry. The GPT models produce 78 of the 84 placeholder-only artifacts, 77 of them in chemistry, typically writing an empty skeleton and running out of budget before filling it.
-
Budget exhaustion is rarely a literal loop. Only 15% of budget-exhausted runs without a deliverable repeat at least 30% of their replies, and 84% of those come from the two Qwen models.
-
Narration and thinking are model signatures, not success predictors. Narration and thinking are stable across software for each model (mean rank correlation 0.74 and 0.69, against 0.14 for GUI share). The GPT models narrate in under 5% of replies, Qwen-CUA in 91%; Gemini 3.1 Pro spends 87% of its output tokens on thinking, MiniMax M3 11%. Within a task, full-credit runs devote more steps to CLI (median +11 points, in 83% of 86 tasks). With model and task fixed effects, no interaction measure predicts full credit after Bonferroni correction.
-
Reasoning effort is an expensive lever. On the 23 QuPath tasks, Opus 5 rises from 44.1% at medium effort to 59.7% at max, and from 34.8% binary at low/medium/xhigh to 43.5% at high and 52.2% at max. Sonnet 5 is non-monotone and never exceeds 22.0%, reached at high rather than at max (18.3%). Max effort costs roughly five to ten times the output tokens of low effort, gaining 17.0 points for Opus 5 but only 3.3 points for Sonnet 5.
-
Longer history helps, then stops helping. Raising the history window shortens the mean trajectory from 52 steps at w=1 to 22 at w=10 and cuts output tokens from 25k to 11k while the score rises from 35.2% to 57.0%; VOID runs disappear from w=10 on, and unsolved runs shrink from 62 steps at w=1 to 23 at w=10. At w=15, output tokens are the same as w=10 but the score is lower (50.2%).
-
Prompt language changes style more than reasoning. Chinese and Japanese prompts leave Opus 5's scripting strategy and trajectory length intact, whereas Spanish, French, and Thai prompts shift it toward GUI manipulation and lengthen trajectories from 27 to 36–44 steps. Opus 5 writes prose in the prompt language for Chinese (89%) and Japanese (65%), partly for Spanish (43%), and in English for French and Thai; Sonnet 5 replies in English regardless of prompt language. Both backbones score highest with Japanese prompts (53.1% and 18.0%) and below English with Chinese (29.7% and 10.9%). No ablation cell differs significantly from its default after Bonferroni correction over the 22 non-default cells (smallest uncorrected p = 0.04, Opus 5 with French prompts).
-
Self-consistency is not validation. In a paired case study, an Opus 5 run on a Lenacapavir SAR task separated a shared scaffold from 17 variable R¹ substituents, cross-checked assay columns and footnotes, parsed every SMILES with RDKit, and scored 1.0 after 70 steps. On an Archangiumide structure-reconstruction task in 68 steps, the same model validated its hypothesis by round-tripping its own SMILES, producing an internally consistent but stereochemically wrong result, for a binary score of 0.0.
Methodology in Plain English
The authors treat each task as a goal-conditioned partially observable Markov decision process: the environment has a full state (operating system, application internals, loaded data, latent scientific quantities, files), but the agent sees only partial observations such as the instruction, a screenshot, or accessibility-tree information. The agent emits pyautogui code, special tokens (<WAIT>, <DONE>, <FAIL>), or nothing parseable; the harness executes actions inside a guest virtual machine, waits, and captures the next screenshot. Most tasks allow 100 steps, with a 16k-token reply limit, a maximum of 10 executed actions per reply, retries up to four times with back-off and a doubled token limit on truncation, and rollback of failed predictions so history stays consistent. Scoring uses only the files and application state collected from the VM after the episode, graded by task-specific evaluators in the 0–1 range, with VOID runs scored zero.
Tasks were built two ways: domain experts with PhD-level knowledge designed tasks from a template, and an AI-assisted "co-worker" framework iterated on anchor tasks to propose and refine new ones. Environments use Ubuntu base images with per-domain images to avoid software conflicts, covering 17 software packages including open-source tools such as QuPath and closed-source tools such as SAS, for which licenses were obtained in advance with a commitment not to use trajectory data for training. To keep model comparisons fair, every model is driven by the same root agent.
Why This Matters
Impact on research. The benchmark reframes scientific agent evaluation around what domain experts consider important and difficult, and around artifacts a grader can verify, rather than around generic desktop chores. Its results suggest that many operational failure modes — missing artifacts, placeholder outputs, wasted budget — cannot be fixed simply by raising reasoning effort or enlarging the context window, which points future work toward agent design rather than raw model scaling.
Real-world applications:
- Assessing scientific copilots before deployment. Labs can use the task set and artifact-level evaluators to check whether an agent can actually complete a workflow in software such as QuPath, ChemDraw, ASKCOS, OpenFOAM, or QGIS.
- Model and harness procurement. The performance-versus-cost data (for example, 73.7% at $6.5 per task versus 22.6% at $0.23 per task) supports budget-aware selection among 12 VLMs.
- Screening tools and protocols. Workflows involving pathology image analysis, radiology inspection, molecular drawing and retrosynthesis, statistical computing, and physical simulation are directly represented, as are cross-application file and PDF workflows.
- Agent interface design. Results on CLI versus GUI step shares and history-window length give concrete guidance for building scientific agent harnesses.
Industry relevance. The findings matter to vendors of scientific software and agent infrastructure, since harness design shifts results as much as backbone choice, and to organizations that must decide whether a given model's cost is justified by measurable completion of verifiable scientific work.
Future Directions
-
Build agents that maintain explicit uncertainty. The case study implies agents should track uncertainty at each stereocenter and compare geometry and derived CIP labels against the source or an independent reference before canonicalization, rather than validating against their own reconstruction.
-
Improve problem understanding and software learning. The conclusion calls for agents that better understand scientific problems and learn to operate unfamiliar software, as opposed to relying on higher reasoning effort or longer context.
-
Extend coverage and balance. The paper notes uneven domain distribution (chemistry 43 tasks at 29.5% versus linguistics 2 at 1.4%) and invites public contributions, suggesting expansion of tasks and software configurations.
-
Investigate harness and interaction variables further. Since none of the ablated settings differed significantly from the default after correction, and since GUI share depends mostly on the software (29% of steps in OpenFOAM versus 90% in Praat), the paper leaves open which harness levers genuinely move scientific performance.
Target Audience
Researchers and engineers working on computer-using agents, multimodal foundation models, and scientific AI who need a rigorous evaluation of software operation; benchmark designers interested in artifact-based grading of heterogeneous outputs; domain scientists who want to know how far current agents are from handling their tools; and practitioners selecting models and harness configurations for scientific-agent deployment.
Authors’ abstract
Scientific software presents a demanding test for computer-using agents based on visual language models (VLMs): completing a research workflow requires interpreting specialized interfaces, manipulating scientific objects, and producing verifiable results. We thus introduce OSWorld-Science, a benchmark and evaluation environment that combines scientifically meaningful tasks, artifact-based evaluation, and an efficient agent harness for studying computer use in the scientific…
The full abstract on arXiv