DevicesWorld: Benchmarking Cross-Device Agents in Heterogeneous Environments

AI summary

Overview

Research area: LLM-based autonomous agents that operate digital environments, specifically benchmarking cross-device collaborative operation across mobile, desktop, and IoT systems.

Technical level: Intermediate. The paper is readable without deep systems knowledge, but familiarity with agent benchmarks, GUI agents, and executable evaluation environments helps.

Scope (1 sentence): The paper introduces DevicesWorld, a 6,140-task executable benchmark for agents that must acquire, transform, and deliver information across multiple heterogeneous devices, and reports that five frontier LLM-agent baselines top out at 12.5% task success.

What This Paper Is About

Existing agent benchmarks mostly test a single environment — a phone, a desktop, or a smart home — but real user goals routinely span devices: a code may arrive on a phone, a file may need processing on a desktop, and the result may need to land on a third device. The paper's core problem is that there is no executable benchmark that measures whether an agent can preserve information, device roles, and task progress across heterogeneous devices and satisfy completion conditions distributed over several of them. DevicesWorld is the authors' answer: a unified cross-device interaction and evaluation environment paired with a large, automatically verifiable task suite.

Key Contributions

1. A large-scale cross-device task set. 6,140 unified executable task specifications spanning three classes of device environments (Android, Linux, SmartHome), with diverse device combinations, application interfaces, file types, and workflow patterns. Each task defines a natural-language user goal, participating devices and initial states, executable actions, rule-based verifiers, and a cleanup procedure, and all tasks pass multi-stage design, review, instantiation, verification, and runtime testing.

2. A unified executable cross-device environment. The environment handles initialization, observation, target-device selection, action routing, state updates, result verification, environment cleanup, and trajectory recording across multiple heterogeneous devices within a single evaluation workflow. Android is implemented on top of AndroidWorld, Linux on top of OSWorld, and SmartHome is a home-IoT simulation built for DevicesWorld covering 11 device categories.

3. A systematic evaluation of five representative LLM-agent baselines with trajectory-level analysis. The experiments expose characteristic failures in required-information acquisition, target-side operation, error recovery, device-role maintenance, final-result generation, and task-completion checking.

4. Evidence that successful local operations do not translate into complete cross-device task execution. Roughly 28.7% of failed runs satisfy at least one scoring condition, showing that many failures occur after partial progress.

Main Findings

• All five baselines perform poorly. Success rates range from 9.5% to 12.5%. UFO³ with GPT-5.5 achieves the highest success rate at 12.5%; GPT-5.5 and Gemini-3.1-Pro-Preview both reach 12.0%; Claude Opus 4.8 reaches 10.5%; Qwen3.7-Plus is lowest at 9.5%. The differences among baselines are relatively small.

• Partial credit is common even when tasks fail. Approximately 28.7% of failed runs satisfy at least one scoring condition. Agents may have read information, created a Linux file, or completed a SmartHome control operation, yet still fail because another output is missing, the result is written to the wrong device, its content or structure is incorrect, or the final state remains incomplete.

• Three-device tasks are never solved. For tasks jointly involving Android, Linux, and SmartHome, none of the five baselines successfully completes a task. The authors caution this is not purely a coordination gap, since some failures also involve mobile-GUI localization, target-side operations, structured-output generation, and single-device result persistence.

• Execution statistics per baseline (Success %, Mean Score, Steps, Duration in minutes, Budget Exhaustion %, Tokens per task): GPT-5.5: 12.0 / 0.262 / 22.1 / 7.9 / 22.0 / 413k. Qwen3.7-Plus: 9.5 / 0.201 / 33.5 / 4.5 / 56.0 / 606k. Gemini-3.1-Pro-Preview: 12.0 / 0.252 / 26.0 / 6.6 / 33.5 / 406k. Claude Opus 4.8: 10.5 / 0.213 / 26.0 / 12.0 / 35.0 / 561k. UFO³: 12.5 / 0.212 / 25.4 / 11.5 / 38.5 / 278k.

• More effort does not mean better results. Qwen3.7-Plus uses the largest number of average steps and tokens and has the highest budget-exhaustion rate, yet records both the lowest success rate and lowest mean score. UFO³ uses the fewest average tokens and the highest success rate, but its mean score does not improve correspondingly and its budget exhaustion remains substantial (38.5%).

• Failed runs end in two distinct ways. Overall, 41.7% of failed runs exhaust the interaction budget, while 58.3% are terminated by the agent declaring the task complete even though the final state fails verification. Qwen3.7-Plus is more likely to keep operating without converging (61.9% of its failed runs are budget exhaustion), while GPT-5.5 more often declares completion prematurely (75.0% of its failed runs). Gemini-3.1-Pro-Preview, Claude Opus 4.8, and UFO³ also more often declare completion and fail verification, though budget exhaustion still accounts for 38.1%, 39.1%, and 44.0% of their failed runs respectively.

• Budget-exhausted failures split into three categories. Across all five baselines, failure to acquire required source information accounts for 44.6%, failure to complete the target-side operation 34.9%, and failure to recover after an explicit execution error 20.5% of budget-exhausted runs. Claude Opus 4.8 and UFO³ concentrate more heavily in source-information acquisition; Qwen3.7-Plus has the highest proportion of target-side operation failures; Gemini-3.1-Pro-Preview has a comparatively high proportion of explicit-error recovery failures.

• Incorrect completion declarations have five defined failure types. The paper defines omission of a required subgoal (D1), confusion about device roles or target location (D2), incorrect final output or device state (D3), completion without confirming result persistence (D4), and incomplete satisfaction of final conditions across multiple devices (D5). The per-type distributions within this mode are not reported in the available content.

• Agents get stuck on the source side. In the B1 category, models often select the correct information source and may even enter the correct application, but repeatedly fail to open the page containing the needed content (the excerpt ends at this point).

Methodology in Plain English

The authors first formalize cross-device operation as a partially observable sequential decision problem: the agent sees observations from a set of devices, picks both a target device and an action, and the environment updates the joint device state. Success requires all final conditions to hold, and a partial score is aggregated from enabled scoring conditions.

To build tasks, they run a multi-stage pipeline. A task space is sampled over device counts and combinations, applications or interaction surfaces, input and output file types, scenario, and feasibility conditions. An LLM then designs a concrete task following the chain "visible sources → transformation or decision → required outputs → expected outcomes," with every verified outcome derivable from information visible to the agent. An independent review-and-repair stage checks naturalness, whether the specified devices genuinely participate, whether information sources are sufficient, whether required operations are supported, and whether the instruction matches the evaluation criteria; unfixable designs are discarded. Deterministic programs then generate format-valid resource files (text, CSV, JSON, webpages, Office documents, PDFs, images, audio, video, archives, code projects) and instantiate setup, evaluation, and cleanup, where setup must not pre-complete the target outcome and cleanup must remove task-induced states.

Candidates then enter an iterative loop of deterministic validation (schema, device–interface compatibility, file paths, formats, verifier coverage, whether setup accidentally satisfies the objective), runtime testing (does initialization work, is the intended outcome recognized by the verifier, do missing or incorrect outcomes fail verification, does cleanup remove principal task states), and targeted repair. Candidates that repeatedly fail are excluded. LLMs handle scenario and semantics design; deterministic programs handle files, specification assembly, and structural checks; release review handles naturalness, requirement–verifier alignment, and real-environment executability.

For evaluation, a coverage-driven stratified selection picks a fixed evaluation set covering major device combinations, representative applications and interaction surfaces, task types, and difficulty levels. Each task allows at most 50 interaction steps. The four direct LLM baselines receive the task instruction, currently visible observations and screenshots, the previous action and its feedback, and textual history from the most recent ten steps; decoding temperature is 0. SmartHome state is not automatically included and must be queried with explicit actions. UFO³ uses GPT-5.5 as its underlying model, decomposes tasks into device-level subtasks, and uses an adapter to convert outputs into DevicesWorld's target-device and action format. Reported metrics are Task Success Rate, Mean Score, Average Steps, Average Duration, Budget Exhaustion Rate, and Average Tokens per Task.

Why This Matters

Impact on research. The paper reframes cross-device collaboration as an executable, reproducible, and diagnostically useful evaluation problem rather than a planning exercise. It shows why concatenating single-device benchmarks is insufficient — observation formats, action interfaces, and state representations are incompatible, and outcomes are distributed across devices so that local success does not imply task success. The failure taxonomy gives the field a shared vocabulary for diagnosing agent breakdowns.

Real-world applications:

• Personal assistant agents that read a verification code from one phone, retrieve a contact, and submit a browser form on a desktop.
• Data pipeline automation that combines a phone calendar, a desktop spreadsheet, and household device state to create a scheduled task.
• Policy-driven home automation, where a Linux policy file determines whether a SmartHome control action should execute.
• Multi-endpoint document workflows, such as saving a note on a designated phone while writing a structured file on a desktop.

Industry relevance. The work is co-authored with Honor Device Co., Ltd, reflecting direct commercial interest in agents that operate across a user's phone, computer, and connected home devices. The results — a best success rate of 12.5% and zero successes on three-device tasks — give device makers and agent developers a concrete capability baseline and a set of failure modes to target.

Future Directions

• Close the source-acquisition gap. Since 44.6% of budget-exhausted failures involve failing to acquire required source information, improving navigation to the right page or application inside a device is the single largest lever.
• Improve self-assessment before termination. 58.3% of failed runs end with the agent declaring completion incorrectly, and D4 specifically covers completion without confirming result persistence. Agents need mechanisms to check that all distributed conditions are jointly satisfied.
• Reduce budget exhaustion without inflating steps. Qwen3.7-Plus shows that more steps and tokens do not yield better outcomes, so the question is how to make existing observations more useful and revise plans after becoming blocked.
• Advance three-device coordination. No baseline solves a task that jointly involves Android, Linux, and SmartHome, and the authors note this combines coordination difficulty with single-device weaknesses, so progress likely requires both better cross-device state tracking and stronger per-device execution.

Target Audience

Researchers and engineers working on LLM agents, GUI agents, and agent evaluation will benefit most, along with product teams building assistant features that span phones, computers, and connected home devices. The paper is also useful to benchmark designers, because its multi-stage construction and quality-control pipeline — covering task contracts, deterministic file generation, verifier validation, runtime testing, and cleanup — is a reusable template for building executable, automatically verifiable task suites.

Authors’ abstract

LLM-based agents have rapidly improved at operating individual digital environments such as mobile applications, desktop systems, and smart homes. However, real-world user goals often span multiple devices: information may come from a phone, be processed on a desktop, and the result may need to appear on another device. Most existing benchmarks center on a single dominant execution environment, making it difficult to evaluate whether agents can acquire and integrate information across heterogeneous devices and complete end-to-end tasks with cross-device dependencies. We introduce DevicesWorld, a large-scale executable benchmark for cross-device collaborative operation. DevicesWorld contains 6,140 tasks and integrates three classes of device environments -- mobile, desktop, and IoT -- into a unified cross-device interaction and evaluation framework. Each task defines a natural-language user goal, participating devices and initial states, executable actions, rule-based verifiers, and a cleanup procedure. A multi-stage construction and quality-control pipeline keeps tasks close to realistic user needs while allowing final outcomes to be automatically verified from device states and generated files. We evaluate five frontier LLM-agent systems on a fixed evaluation set. All methods achieve low success rates, with the best reaching only 12.5%. Among failed runs, about 28.7% satisfy at least one scoring condition yet still fail the full task. Trajectories show that agents become stuck acquiring information or manipulating interfaces, confuse source and output devices, or terminate before all conditions are jointly satisfied. DevicesWorld turns cross-device collaborative operation into an executable, reproducible, and diagnostically useful evaluation problem for research on reliable cross-device agents.