Empowering Edge Agents: A Systematic Analysis of Edge-Cloud Collaboration in LLM Agent Execution

AI summary

Overview

Research area: Systems and evaluation for LLM-based agents — specifically, how to split long-horizon agent execution between a small on-device ("edge") model and a stronger cloud-hosted model.

Technical level: Intermediate. Readers should be comfortable with the idea of LLM agents that take actions and receive observations in a loop, and with basic cost/accuracy trade-offs across model sizes. No specialized systems background is required.

Scope (1 sentence): The paper introduces EC²-Bench, a 128-task executable agent benchmark with a unified evaluation of task performance, resource usage and privacy exposure, and uses it to empirically compare four edge–cloud collaboration strategies across four edge-model capability levels.

What This Paper Is About

Deploying LLM agents on edge devices is limited by device resources, so practical setups use small models (the paper describes edge models as typically ≤10B parameters), which struggle with demanding reasoning, planning and tool use. A natural fix is to let a lightweight edge model handle most of the work and call a much stronger cloud model when needed — but in long-horizon agent execution, cloud participation is interleaved across many steps rather than decided once per query, so decisions accumulate costs and leak information over the whole trajectory. The paper's goal is to measure, systematically, whether such collaboration actually helps, how it changes as edge models get stronger, and which collaboration design is best at which capability level.

Key Contributions

1. EC²-Bench, a benchmark for edge–cloud collaboration in long-horizon agent execution, built from 128 executable digital-workspace tasks drawn from six existing executable agent benchmarks (Claw-Eval, WildClawBench, QwenClawBench, LiveClawBench, PinchBench, ClawBench) and adapted to a common runtime and evaluation interface.
2. A three-dimensional evaluation framework applied to the full collaborative execution trace: task performance (completion score and Pass³), resource usage (cloud tokens and USD cost, plus edge FLOPs), and privacy exposure (fraction of annotated sensitive units visible to the cloud), with fine-grained sensitivity annotations for 100 of the 128 tasks.
3. A formal formulation of edge–cloud collaboration as a per-step model-selection rule over the partial trajectory, plus concrete instantiations of four representative strategies: Query Routing, Per-step Routing, Edge-Controlled Cloud Delegation, and Edge-Controlled Cloud Advisory.
4. An empirical study organized around three research questions (is collaboration effective; how does rising edge capability change it; how should collaboration design evolve), including analyses of cloud-invocation decision-making and instruction following as edge-side abilities.

Main Findings

• Collaboration beats both extremes on balance, not on raw accuracy. Cloud-only execution with GLM-5.3 reaches 82.70 completion with 87.01% privacy exposure and 56.42 USD cloud cost; edge-only with Qwen3.5-2B reaches 12.75 completion with 0.00 exposure and no cloud cost. Collaborative configurations sit between these: with Qwen3.5-2B as the edge model, Per-step Routing raises completion from 12.75 to 39.97 while using only 22.7% of the cloud-only cost.
• Gains exceed a simple mixture of the two endpoints. Using linear interpolation between each edge model's edge-only and cloud-only endpoints as a reference, several collaborative configurations lie above that baseline in both cost–performance and privacy–performance space.
• Stronger edge models close the performance gap to cloud-only. The best collaborative configuration rises from 39.97% completion with Qwen3.5-2B to 76.84% with Qwen3.7-Plus, approaching cloud-only's 82.70%.
• Strategy choice stops mattering for accuracy — but not for privacy. The completion gap between the best and worst collaboration strategy narrows from 25.22% with Qwen3.5-2B to 3.67% with Qwen3.7-Plus. With Qwen3.7-Plus, strategies differ by only 3.67% in completion but by 66.23% in privacy exposure.
• The preferred collaboration mode flips with edge capability. Weaker edge models favor externally controlled routing and direct cloud execution (with Qwen3.5-2B, Per-step Routing scores 39.97 vs. 15.86 for Edge-Controlled Delegation), whereas stronger edge models favor edge-controlled invocation and cloud advisory.
• Good invocation decisions are an edge-side skill. Compared against a random-invocation baseline matched to the same cloud-call rate, edge-controlled delegation wins by only 1.2% for Qwen3.5-2B but by 3.0–4.4% for the more capable edge models.
• Instruction following tracks advisory benefit. Applying targeted instruction-following training to Qwen3.5-2B and evaluating under a fixed cloud-advisory setting raises task completion from 24.8% before training to 28.8% after 600 steps.
• The privacy judge is reliable. The LLM-based exposure detector achieves an F1 of 96.9% against adjudicated human labels on 200 sensitivity unit–context pairs.

Methodology in Plain English

The authors first build a task suite. They take agent tasks from six existing benchmarks that already run end-to-end in a workspace with tools and verifiable outcomes, port them onto one shared runtime so all tasks execute and are scored the same way, and then filter for difficulty: tasks that both the edge-only and cloud-only setups always solve are dropped (no room to differentiate), as are tasks neither endpoint can meaningfully complete. Underrepresented scenarios are added back in, yielding 128 tasks. For privacy measurement, they annotate sensitive information already present in workspace artifacts and use an LLM-based procedure to insert realistic synthetic sensitive content into 100 tasks, down to atomic sensitivity units.

Execution is formalized as a loop: at each step a collaboration strategy picks which model(s) participate given the task and the trajectory so far; the chosen model(s) produce an action, the environment returns an observation, and the pair is appended to the trajectory. Crucially, the benchmark also records what context was shown to the cloud at each step, which is what makes trajectory-level privacy measurable.

The four strategies span who decides and what the cloud does: Query Routing decides once per task with an external router; Per-step Routing re-decides at every step; Edge-Controlled Cloud Delegation lets the edge agent call a delegation tool to hand a step to the cloud; Edge-Controlled Cloud Advisory lets the edge agent request plans, suggestions or reasoning from the cloud while still generating the action itself.

Experiments use Qwen3.5-2B and Qwen3.5-9B as current edge models, with Qwen3.5-27B and Qwen3.7-Plus as proxies for future stronger edge models; GLM-5.3 is the primary cloud model, with DeepSeek-V4.1-Flash used to check robustness. Edge-only and cloud-only runs are included as reference points. Every task–configuration pair runs three times, with Pass³ defined as the fraction of tasks completed in all three runs (a run counts as successful above a 0.7 completion score). Scoring mixes rule-based checks and environment-state auditing with rubric-based LLM-as-a-judge using GPT-5.4-mini; cloud cost is computed from OpenRouter pricing over raw input, cache-read input and output tokens, and edge computation is estimated in FLOPs following Kaplan et al. (2020).

Why This Matters

Impact on research. Prior edge–cloud benchmarks (RouterBench, LLMRouterBench, LLMSwitchBench, M4A1) evaluate routing over individual queries or dialogue turns and score task performance plus cost. EC²-Bench reframes the object of study as the whole agent trajectory, adds privacy exposure as a first-class metric, and covers four collaboration modes rather than routing alone. It also reframes edge-model development: the paper argues future edge models should be optimized for both standalone capability and the ability to collaborate with cloud models.

Real-world applications:

• Personal-assistant and workplace-automation agents running on phones or laptops, where sensitive documents should stay local unless cloud help is genuinely needed.
• Coding and data-analysis agents on developer machines, where workspace contents are proprietary but hard subtasks benefit from frontier models.
• Enterprise deployments with compliance constraints, where the 66.23% spread in privacy exposure across strategies at fixed accuracy is a directly actionable design lever.
• Model and runtime vendors choosing invocation policies — external routing vs. agent-controlled delegation vs. advisory — based on the capability of the edge model they ship.

Industry relevance. The paper gives concrete cost and privacy numbers per strategy and per edge-model tier, which is the kind of evidence needed to pick a deployment architecture rather than guess. The finding that accuracy differences collapse to 3.67% while privacy differences stay at 66.23% once edge models are strong suggests that in a strong-edge regime, the collaboration decision becomes primarily a privacy and governance decision.

Future Directions

• Train edge models explicitly for collaboration. The instruction-following experiment (24.8% to 28.8% completion under advisory after 600 steps) is a small probe; the paper motivates broader work on building cloud-invocation decision-making and guidance-following into edge models rather than treating them as fixed components.
• Learn when to invoke the cloud. The random-invocation comparison provides only a matched-rate baseline, not ground-truth labels of when cloud help was necessary; better supervision or learned invocation policies remain open.
• Extend beyond the studied configurations. Results are reported for Qwen-family edge models with GLM-5.3 as the primary cloud model and one alternative cloud model (DeepSeek-V4.1-Flash) for robustness; other model families, hardware, and latency-constrained settings are not covered here.
• Operationalize privacy-aware collaboration. Since privacy exposure stays strongly strategy-dependent even when performance converges, designing strategies that are explicitly privacy-targeted — rather than incidentally private — is a natural next step.

Target Audience

Researchers and engineers working on on-device or hybrid LLM systems, agent runtime and harness developers, and evaluation/benchmark designers who care about trajectory-level rather than query-level behavior. It is also useful for practitioners making deployment decisions about when an agent should call a cloud model, and for privacy/compliance teams who need evidence about how much workspace information actually leaves the device under different designs. Readers new to agents will find the formal setup and the four-strategy taxonomy approachable, but will need the related-work context to appreciate the benchmark comparison.

Authors’ abstract

Edge-cloud collaboration combines the lower inference overhead and greater data locality of edge-side execution with the stronger capabilities of cloud-hosted models. However, as LLM agents increasingly operate in long-horizon scenarios, cloud participation becomes interleaved throughout execution rather than a one-shot decision for each query. Such interleaved participation introduces trajectory-level effects that remain insufficiently understood. To address this gap, we introduce EC2-Bench, a benchmark for evaluating edge-cloud collaboration in long-horizon agent execution. It provides executable digital-workspace agent tasks and a unified evaluation framework covering task performance, resource usage, and privacy exposure. Using this benchmark, we empirically study diverse forms of edge-cloud collaboration. Our study reveals three findings. First, collaboration achieves a favorable balance among task performance, resource usage, and privacy exposure compared with edge-only and cloud-only execution. Second, as edge models become stronger, collaborative execution approaches cloud-only performance, and performance differences across strategies shrink, while privacy exposure remains strongly strategy-dependent. Third, stronger edge capability shifts preferred collaboration from externally controlled cloud participation and direct cloud execution toward more edge-controlled collaboration. Further analyses identify cloud-invocation decision-making and instruction following as important edge-side abilities for effective collaboration. These findings provide a systematic understanding of edge-cloud collaboration in long-horizon agent execution and insights for designing future edge agents that effectively collaborate with cloud models. Code and resources are available at https://github.com/OpenBMB/EC2-Bench.