OmegaUse-SOP: SOP Engineering for Professional Computer Use from Human Demonstrations

AI summary

Overview

Research area: Human-Computer Interaction, specifically LLM-based graphical user interface (GUI) agents and how they can be taught professional, domain-specific workflows.

Technical level: Intermediate. The paper assumes familiarity with vision-language models, GUI grounding, and agent execution loops, but it explains the system architecture in accessible terms and centers on a practical engineering methodology rather than new model architecture.

Scope: The paper introduces a human-in-the-loop system that converts expert screen demonstrations into reusable, parameterized "SOP skills," and validates it on five photovoltaic simulation tasks in PVsyst 7.2.

What This Paper Is About

General-purpose computer-use agents perform well on open-ended benchmarks but break down on professional software, where success depends on implicit domain knowledge, software-specific conventions, configurable parameters, and verification habits that benchmarks do not capture. The authors propose treating professional procedural knowledge as an engineering artifact — analogous to prompt engineering — that can be captured from human demonstrations, refined by experts, and executed reliably by GUI agents. Their goal is a system that turns a recorded expert workflow into an editable, reusable skill rather than a brittle sequence of replayable mouse coordinates.

Key Contributions

1. The SOP Engineering methodology. A formalization of professional computer use as an iterative refine loop over demonstrations, execution rules, domain knowledge, and task parameters — positioned explicitly as the procedural analogue of prompt engineering.

2. The OmegaUse-SOP system with four modules (Observe, Reason, Configure, Execute). Observe records multimodal GUI traces including pre-action screenshots and cropped visual targets; Reason uses a VLM to abstract low-level events into semantic step instructions; Configure exposes domain guidance and task variables for human editing; Execute runs the skill with progressive step disclosure and post-action verification.

3. A progressive-disclosure execution design. Rather than passing all step information to the model at once — infeasible for SOPs with hundreds of steps — Execute retrieves only the current step's trace, semantic instruction, domain rules, and parameters, and verifies each step against the expected post-action screen while tolerating harmless visual differences.

4. A real-world industrial case study with an ablation. Five PVsyst 7.2 tasks derived from an actual power-sector client workflow, evaluated across three VLMs, plus an ablation isolating the contribution of the Reason module. Code is open-sourced and a demo video released.

Main Findings

• Baseline agents fail on professional tasks. Without SOP guidance, Qwen3-VL-235B-A22B-Instruct completed 1/5 tasks, GPT-5.5 completed 3/5, and Opus-4.7 completed 2/5. Failures clustered on tasks requiring procedural domain knowledge, most notably detailed losses setting, which spans thermal behavior, ohmic losses, and ageing across multiple sub-panels.

• SOP guidance produces uniform, large gains. With OmegaUse-SOP, all three models reached 5/5 task completion, including the open-weight Qwen3-VL. The improvement held across both open and proprietary models, suggesting the gains come from the SOP representation rather than a particular model's capability.

• The Reason module is load-bearing. Removing semantic abstraction and letting the agent work directly from the raw low-level trace dropped Qwen3-VL from 5/5 to 2/5. Failures without Reason appeared on meteorological data importation, grid-connected system setting, and detailed losses setting — cases where knowing where the human clicked is insufficient without knowing what interface element was targeted and why.

• Expert-authored rules fix specific, recurring model errors. Two documented examples: a spinner-control rule (down-arrow decreases, up-arrow increases the number of modules in series) correcting reversed adjustment clicks, and a parameter-priority rule ensuring user-specified values such as a 35° plane tilt override values carried in operation instructions or historical inputs.

• Semantic grounding outperforms coordinate replay. The system's value comes from describing the operated object, its visual context, and its role in the procedure — enabling the agent to locate the same target under different screen states rather than reproducing pixel positions.

Methodology in Plain English

The researchers start from the observation that an expert doing professional work on a computer is doing something a raw click recording cannot express. So they record the expert carefully, then have a vision-language model explain what happened, then let a human add knowledge, then run it back.

In the Observe stage, the system watches the expert's mouse and keyboard while capturing screenshots. Crucially, it saves the screenshot taken immediately before each event, preserving the "see-act" ordering of human behavior. Clicks are linked to detected UI elements via OmniParser and PaddleOCRv5, which crop the bounding box of the clicked element as its visual target. Continuous typing is merged into a single text-input event, while special keys and shortcuts are recorded separately.

In the Reason stage, a VLM converts these raw events into plain-language step instructions. For clicks, it does not decide what action to take — the action and coordinate are already known. It identifies the semantic target using the pre-action screenshot, a highlighted crop of the clicked region, and the post-action screenshot as evidence of effect. For typing and hotkeys, it uses the preceding screen to explain where and why the input is used — turning "entered the recorded text" into "typed the project name into the project-name field."

In the Configure stage, a human adds two things that screenshots cannot supply: domain SOP guidance (professional rules and software constraints, often written after observing model failures) and task-specific parameters (which recorded values are variables rather than fixed literals). This is what allows the skill to generalize to new task instances instead of replaying one demonstration.

In the Execute stage, the agent works through the live interface step by step. Inspired by how coding agents invoke skills, it retrieves only the current step's information to avoid context overload, generates a computer-use action (click, double click, text input, shortcut, scroll, wait, or terminate), and executes it via low-level mouse and keyboard control. After each step it compares the resulting screen to the expected post-action screen and asks the model whether to continue, tolerating cursor position, timestamps, and dynamic content. If verification detects deviation, a human-in-the-loop mechanism lets the user continue, retry, or stop.

Evaluation was a case study, not a large-scale benchmark: five tasks, three models, three trials each, with success judged manually by domain experts.

Why This Matters

Research impact. The paper reframes the professional-GUI-agent problem away from "train a stronger perception and action model" toward "engineer and reuse procedural knowledge." The strong ablation result on the Reason module gives a concrete, testable claim: semantic abstraction over demonstrations is a distinct and necessary capability, not a cosmetic layer over coordinate replay. It also suggests that the gap between general computer-use benchmarks and professional software is a data-representation gap as much as a model-capability gap.

Real-world applications.

• Energy and power system design. The demonstrated case: configuring PVsyst simulations for site-specific meteorological data, plane orientation, grid connection, and loss parameters to produce annual production estimates.
• Enterprise software onboarding. Capturing how experienced operators use ERP, CAD, EHR, or financial systems and turning it into agent-executable skills, reducing dependence on tribal knowledge.
• Regulated and auditable workflows. Where step-wise verification and explicit domain rules make an automated procedure inspectable and debuggable by compliance reviewers, rather than an opaque policy.
• Long-tail professional tools. Niche, GUI-only software with no API — the exact category where GUI agents have the most leverage and general models have the least training data.

Industry relevance. The work is a collaboration between Baidu and a power-sector client (Ningxia Electric Power Engineering), and the paper is explicit that the SOP Engineering loop is meant to be run by domain experts, not ML engineers. That positions the method as a deployment pattern for enterprises that already have expert operators and brittle, expensive software workflows — a natural product shape for agent platforms. The open-source release and separate open-source human-in-the-loop implementation lower the barrier to reproduction.

Future Directions

• Scaling beyond a single-domain case study. Five tasks in one software package with three trials each is a proof of concept. Whether SOP Engineering holds across dozens of tasks, multiple software packages, and workflow branching remains untested.

• Automating or reducing the human configuration burden. The Configure module currently depends on experts writing rules and identifying variables, often after watching models fail. Making that loop faster — automatic failure clustering, suggested guidance, parameter inference from repeated demonstrations — would determine whether the approach scales economically.

• Generalizing and composing SOP skills. Open questions include whether a skill recorded in one software version survives UI updates, whether partial SOPs can be composed into larger workflows, and whether skills transfer across structurally similar applications.

• Richer evaluation and verification. Task success was judged manually by domain experts. Automated, fine-grained verification of intermediate states, plus an error taxonomy for professional SOP failures, would make the reliability claims more precise and comparable across systems.

Target Audience

Researchers and practitioners working on GUI agents, computer-use agents, and human-in-the-loop automation will find the four-module architecture and the progressive-disclosure execution design directly applicable. Enterprise AI teams evaluating agent deployment on internal, API-less professional software are the primary industrial audience, since the paper reads as a deployment methodology rather than a model contribution. Domain experts in engineering software — particularly energy and simulation tooling — will find the PVsyst case study concrete, though the paper's contribution to them is the capture-and-reuse pattern rather than the specific task details. Readers looking for new model architecture or large-scale benchmark results will find less here; the value is in the methodology, the ablation, and the practical framing of professional procedural knowledge as an engineering artifact.

Authors’ abstract

Large language models (LLMs) are increasingly evolving from conversational assistants into agents capable of operating external digital environments. Graphical user interface (GUI) agents play an important role in this transition, as many real-world workflows remain accessible only through user-facing software interfaces. However, despite recent progress on general computer-use benchmarks, domain-specific professional standard operating procedures (SOPs) remain challenging for GUI agents because they often involve implicit domain knowledge, software-specific conventions, and task-level verification requirements. We introduce OmegaUse-SOP, a human-in-the-loop SOP Engineering system for transforming human demonstrations of professional computer use into reusable SOP skills for GUI agents. Analogous to prompt engineering, SOP Engineering iteratively refines demonstrations, execution rules, and domain knowledge to convert professional SOPs into reusable GUI-agent skills. OmegaUse-SOP consists of four modules: Observe, Reason, Configure, and Execute. Together, these modules record expert operations as multimodal GUI traces, abstract low-level events into semantic step-level instructions, incorporate domain rules and task-specific parameters, and execute the resulting skills in live GUI environments through step-wise grounding, action generation, and verification. To demonstrate its effectiveness, we collaborate with a power-sector client and test OmegaUse-SOP on photovoltaic simulation workflows in PVsyst 7.2. The results suggest that OmegaUse-SOP can improve GUI-agent reliability on professional SOP tasks, highlighting a practical path toward deploying GUI agents in domain-specific professional software environments.