JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition

AI summary

Overview

Research Area: Artificial Intelligence Technical Level: Beginner-Friendly Reading Time: ~3 minutes Published: September 9, 2026

What This Paper Is About

Real-world GUI usage frequently involves workflows that span multiple devices and platforms, requiring the transfer of intermediate results, maintenance of shared state, and coordination across heterogeneous environments. However, existing GUI benchmarks overwhelmingly evaluate agents on single-device, statically defined tasks, thus leaving such cross-device capabilities largely unexamined, resulting in an overly optimistic assessment of agents' readiness for real-world usage.

Key Contributions

1. We introduce JarvisGUI, a dynamic benchmark that evaluates GUI agents on cross-device workflows requiring coordinated interaction across heterogeneous platforms, including Android, Windows, and Ubuntu
2. However, existing GUI benchmarks overwhelmingly evaluate agents on single-device, statically defined tasks, thus leaving such cross-device capabilities largely unexamined, resulting in an overly optimistic assessment of agents' readiness for real-world usage

Main Findings

• Finding 1: Real-world GUI usage frequently involves workflows that span multiple devices and platforms, requiring the transfer of intermediate results, maintenance of shared state, and coordination across heterogeneous environments
• Finding 2: However, existing GUI benchmarks overwhelmingly evaluate agents on single-device, statically defined tasks, thus leaving such cross-device capabilities largely unexamined, resulting in an overly optimistic assessment of agents' readiness for real-world usage

Methodology in Plain English

Specifically, JarvisGUI formulates GUI tasks as input-output transformations under a lightweight type system, which allows us to automatically compose multi-step, cross-device workflows and dynamically evaluate agent performance within a unified framework

Why This Matters

Impact on Research: This work opens new directions for researchers in Artificial Intelligence.

Real-World Applications:

• Improving AI system performance and reliability
• Developing more efficient and scalable solutions
• Addressing practical challenges in Artificial Intelligence
• Enabling new capabilities in AI-powered applications

Industry Relevance: Companies working with AI can leverage these findings to enhance their products and reduce development friction.

Future Directions

• Extending the approach to related domains and use cases
• Improving scalability and efficiency of the proposed methods
• Investigating practical deployment considerations
• Exploring combinations with complementary techniques

Target Audience

• Researchers in Artificial Intelligence and related fields
• Engineers building AI systems and applications
• Students studying advanced topics in artificial intelligence
• Industry practitioners seeking to apply cutting-edge research

This summary was generated from the paper abstract. For full technical details, refer to the original paper.

Authors’ abstract

Real-world GUI usage frequently involves workflows that span multiple devices and platforms, requiring the transfer of intermediate results, maintenance of shared state, and coordination across heterogeneous environments. However, existing GUI benchmarks overwhelmingly evaluate agents on single-device, statically defined tasks, thus leaving such cross-device capabilities largely unexamined, resulting in an overly optimistic assessment of agents' readiness for real-world usage. We introduce JarvisGUI, a dynamic benchmark that evaluates GUI agents on cross-device workflows requiring coordinated interaction across heterogeneous platforms, including Android, Windows, and Ubuntu. Specifically, JarvisGUI formulates GUI tasks as input-output transformations under a lightweight type system, which allows us to automatically compose multi-step, cross-device workflows and dynamically evaluate agent performance within a unified framework. By evaluating agents in virtual environments spanning multiple operating systems, JarvisGUI reveals that state-of-the-art open-source GUI agents struggle with the state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management required for real-world workflows, exposing a critical capability gap invisible to existing benchmarks.