Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill Induction

AI summary

Overview

Research area: Computer-use GUI agents, demonstration-based learning, and reusable procedural skill induction.

Technical level: Advanced. The core idea is intuitive, but the paper combines a multi-platform recording system (macOS capture plus Linux AT-SPI accessibility collection), a curated multimodal dataset, and an agentic pipeline with tool interfaces and validation hooks.

Scope: The paper introduces WorkRecorder (an infrastructure for capturing aligned screen video, input events, and accessibility-tree evidence), WorkDemo-350 (350 curated human desktop GUI demonstrations spanning 64 MyPCBench Bounded Action tasks and a 286-task OSWorld-Verified subset), and an agentic induction method that turns those demonstrations into reusable natural-language skills, evaluated across four execution models and two benchmarks.

What This Paper Is About

Computer-use agents still fail on unfamiliar software and on workflows that depend on user- or organization-specific procedures that a request never spells out. Humans and agents share the same GUI interaction space, so human demonstrations are a natural source of that procedural knowledge, but reusing them requires figuring out what each recorded action meant in context and separating durable workflow rules from instance-specific details. Work2Skill builds the capture infrastructure, the curated dataset, and the induction method needed to turn human GUI experience into reusable, recording-free skills, and then measures whether those skills actually help agents execute.

Key Contributions

1. WorkRecorder, a human GUI experience collection infrastructure that aligns screen recordings, mouse and keyboard events, and available accessibility-tree (AX) information on a common video-relative timeline, so a recorded click can be inspected alongside its screen image, target control metadata, and surrounding video.

2. WorkDemo-350, described as the first synchronized multimodal dataset of human desktop GUI demonstrations for reusable agent skill induction and downstream evaluation, containing 350 demonstrations (64 MyPCBench Bounded Action tasks and a 286-task OSWorld-Verified subset), screened for completeness and task success.

3. An agentic skill-induction method that interleaves on-demand evidence inspection (files, images, video intervals) with skill writing and revision in a single conversation, producing SKILL.md files that capture workflow rules, step dependencies, and completion checks without the source recording.

4. A controlled downstream evaluation using official task evaluators and a shared text-guidance interface, comparing Work2Skill against No Skill and text-adapted ShowUI-Aloha and Demo2Tutorial guidance across four execution models, plus reuse tests under changed task requirements, cross-model transfer tests, evidence ablations, and a pipeline-prompt comparison.

Main Findings

• Highest scores and success rates among compared methods: Across all four executors (GPT-5.5, Opus-4.8, Qwen3.5-35B-A3B, Qwen3.5-9B) and both benchmark sets, Work2Skill records the best Score and SR in Table 2. Mean Score gains over the corresponding No Skill baseline are 4.67–10.54 points on MyPCBench Bounded Action and 3.11–7.50 points on the 286-task OSWorld-Verified subset, with success-rate gains of 6.25–25.00 and 1.40–7.69 percentage points respectively.

• Advantage over demonstration-based baselines: Against the higher-scoring of ShowUI-Aloha and Demo2Tutorial for each executor, Work2Skill improves Score by 3.95–7.79 points on MyPCBench and 1.65–6.96 points on the evaluated OSWorld-Verified subset. For GPT-5.5 on MyPCBench, SR reaches 48.44% versus 37.50% for both baselines.

• Model-dependent efficiency: Work2Skill uses fewer execution steps than No Skill for Opus-4.8 and both Qwen models, but more for GPT-5.5, and its skill-induction token usage is not consistently lower than that of the demonstration-based baselines.

• Reuse under changed requirements: On the 64 MyPCBench Jitter tasks (one paired variant per original task, changing entities, parameters, or constraints within the same application), skills reused without re-induction improve Score over No Skill by 3.93–8.64 points and SR by 6.25–9.38 percentage points, with a 4.28–6.25 point Score advantage over the stronger demonstration-based baseline and 2.49–5.44 fewer execution steps.

• Cross-model transfer: Crossing GPT-5.5 and Qwen3.5-9B as induction and execution models, skills induced by GPT-5.5 outperform those induced by Qwen3.5-9B under both executors; the advantage reaches 9.46 score points and 9.38 percentage points in success rate with GPT-5.5 execution, and all four pairings improve on their executor's No Skill baseline. Execution with Qwen3.5-9B remains substantially lower regardless of which model induced the skill.

• Evidence matters, unevenly by modality: With GPT-5.5 at high reasoning effort on 64 MyPCBench tasks, Full Evidence scores 85.27 and fully solves 31 tasks. Removing indexed images, UI annotations, or KeyLogs reduces Score by 5.02–10.63 points and SR by 14.06–21.88 percentage points, with the largest Score drop without indexed images and the largest SR drop without KeyLogs. The UI-Grounded Action Text Only condition uses the fewest steps but trails Full Evidence on Score and SR.

• Concise prompt beats a prescribed pipeline: Holding evidence fixed, the concise Work2Skill system prompt improves Score by 3.97 points and SR by 6.25 percentage points over a Pipeline Prompt that dictates induction stages, while cutting average execution steps from 48.98 to 47.38, at 12.7% higher induction token usage.

• Dependent subgoals are what separate the methods: In a Cotogna reservation case with ten people, three fallback restaurants, and two favorites, ShowUI-Aloha and Demo2Tutorial guidance both score 55/100, leaving fallback inspection, favoriting, and persistence checks incomplete, while the Work2Skill-indused skill satisfies all eight rubric items for 100/100 by making intermediate checks and ordering explicit.

Methodology in Plain English

The team built a recorder that watches a human do a task in the benchmark's official environment and keeps several synchronized streams: screen video, keyframe images, mouse and keyboard events, and whatever structured UI information the operating system exposes through its accessibility layer. On Linux this uses AT-SPI to look up names, roles, and bounds for the control at the pointer location; macOS recording runs locally, and remote Linux workflows are captured through the Mac while a collector inside the target desktop supplies UI evidence. Each demonstration is packaged as an "evidence episode" with a manifest, ordered action trace, and linked images and video, so any individual step can be looked up without replaying the whole recording.

They then recorded 350 demonstrations across two benchmarks, excluded OSWorld tasks where a valid human demonstration could not be completed, and reviewed each recording for task success and completeness, discarding ones that missed key operations or ended in the wrong state. The result is WorkDemo-350, roughly 8.48 hours of recorded experience.

Induction is handled by a tool-using agent (NexAU). Rather than loading a long recording into context or following a fixed parse-then-abstract pipeline, the agent reads the task definition, then alternates between inspecting selected files, images, and video intervals and drafting the skill, returning to the evidence when a draft exposes an unverified transition. A bundled authoring policy pushes it to generalize instance-specific values into task-dependent descriptions, describe UI targets semantically, and preserve ordering, dependencies, stop conditions, and visible completion checks. A finalize_extraction call validates output structure and required tool use, though it does not verify skill correctness or task success.

For evaluation, each generated skill is appended as a delimited text block to the executor's base system prompt, and all methods share the same wrapper and execution settings, so only the guidance changes. Skills are never executed during induction, and downstream results are not fed back as revision signal.

Why This Matters

Impact on research: The paper isolates a variable that prior GUI-demonstration work conflated, since studies used different demonstrations, task sets, and execution protocols. By fixing the source demonstrations, the task evaluators, and the execution settings, and by releasing the recordings alongside the method, it makes induction-method comparisons possible on shared evidence. The evidence ablation and prompt comparison also give concrete, quantified guidance about which modalities and prompt styles actually help downstream execution.

Real-world applications:

• Enterprise software onboarding: Encoding internal procedures such as reimbursement submission, where the request omits which portal, which receipts, and what to check before submitting, into reusable skills an agent can follow.
• Record-and-replay style desktop automation: Turning a single expert walkthrough into adaptable guidance rather than a brittle fixed macro, so variants in dates, entities, or parameters still work.
• Supporting heterogeneous agent deployments: Because skills are plain text injected through a shared wrapper, one induced skill can be reused across different agent harnesses and execution models, which the cross-model transfer experiment tests directly.
• Procedure capture in accessibility-rich environments: The AX-based target association gives explicit control names and roles, which is useful wherever applications expose structured UI metadata and visual grounding alone is ambiguous.

Industry relevance: The approach parallels user-facing features such as OpenAI's Record & Replay and Claude's Record a skill, which the paper cites. The finding that a concise, agent-directed prompt outperforms a prescribed pipeline, and that induction cost varies by model, is directly actionable for teams building demonstration-to-automation products. The dataset construction and screening pipeline is also a reusable template for anyone assembling demonstration corpora.

Future Directions

• Closing the executor capability gap: Stronger induction did not close the performance gap between executors on these tasks, so whether better skills, better executors, or better skill-executor pairing is the binding constraint remains open.

• Verifying skill correctness, not just structure: The finalize_extraction checks cover task-file access, skill structure, write-then-read of each skill, and metadata inspection before video sampling, but explicitly do not verify correctness or task success. How to validate induced skills without leaking evaluation signal is unresolved.

• Using execution feedback for revision: The method deliberately freezes skills and withholds downstream results during induction. Whether controlled execution feedback could improve skills without overfitting to benchmark instances is an obvious next question, and the MyPCBench Jitter results provide a reuse testbed for it.

• Scaling and broadening the evidence base: The dataset covers 350 demonstrations over desktop workflows with 64 and 286 tasks respectively; whether the approach holds for larger, noisier, non-expert recordings, applications with weaker accessibility support, and non-desktop environments such as mobile (where prior work like LearnGUI operates) is not established here.

Target Audience

Researchers and engineers working on computer-use and GUI agents, demonstration-based learning, and agent memory or skill libraries. It is also relevant to practitioners building record-and-replay automation products who need to know which capture modalities and prompt designs actually improve downstream task success, and to dataset builders interested in how to align and screen multimodal human demonstration data. Readers without background in GUI agent evaluation will find the high-level argument accessible, but the experimental and infrastructure sections assume familiarity with accessibility trees, benchmark harnesses, and agent tool-use pipelines.

Authors’ abstract

Computer-use agents can perform increasingly complex GUI tasks, yet unfamiliar software and workflows requiring human domain knowledge remain challenging. Human demonstrations convey procedural knowledge through the same GUI interaction space, but reuse requires interpreting actions in context and separating workflow rules from task-specific details. We thus introduce Work2Skill, a framework that captures human GUI experience for reusable agent skill induction. WorkRecorder provides the framework's infrastructure for collecting this experience, aligning screen recordings, mouse and keyboard events, and available accessibility-tree information. Using WorkRecorder, we construct WorkDemo-350, the first synchronized multimodal dataset of human desktop GUI demonstrations for reusable agent skill induction and downstream evaluation. Its 350 demonstrations are screened for completeness and task success. We then propose an agentic skill-induction method that inspects evidence on demand to produce reusable skills capturing workflow rules, step dependencies, and completion checks. When the same four execution models are equipped with skills induced by each method under shared execution settings, Work2Skill achieves the highest scores and success rates. Mean score improvements over the corresponding No Skill baseline are 4.67--10.54 points on MyPCBench Bounded Action and 3.11--7.50 points on a 286-task subset of OSWorld-Verified. Our code and data are available at https://github.com/china-qijizhifeng/Work2Skill.