EdgeWise: Bringing Long-Horizon Agents to the Edge via Selective Cloud Delegation

AI summary

Overview

Research area: Efficient agent systems / edge–cloud model collaboration for long-horizon LLM agents (Natural Language Processing, cs.CL).

Technical level: Intermediate. The paper is readable with a basic grasp of LLM agents, model routing, and reinforcement learning, but the RL section and the evaluation tables assume some familiarity with training pipelines.

Scope: The paper proposes EdgeWise, a framework in which a small language model (SLM) on a local device solves long-horizon coding-agent tasks and selectively hands only critical steps to a stronger cloud LLM, and it evaluates the resulting accuracy-versus-cost trade-offs on Terminal-Bench 2.1 and SWE-ReBench.

What This Paper Is About

Long-horizon agent tasks such as software engineering are normally run end-to-end with a frontier model in the cloud, which is expensive even though many intermediate steps are simple. The authors ask whether a small local model can do most of the work and call the cloud model only at a few key moments. Their goal is a framework that keeps task performance comparable to full cloud execution while cutting inference cost, and that runs on real edge hardware.

Key Contributions

1. A step-level edge–cloud collaboration framework (EdgeWise). A locally deployed SLM acts as the primary solver interacting with the environment, and delegates individual steps to a stronger cloud LLM when more capability is needed.
2. A systematic study of design factors. The paper examines when the cloud LLM is needed, how delegation decisions should be made, what context should be shared between local and cloud models, and how frequently the cloud LLM should be invoked. The central finding is that the cloud model's benefit is concentrated in a small subset of critical steps.
3. Two delegation mechanisms. One is a training-free external-judge strategy that uses GPT-5.4-mini as the judge. The other is self-delegation, where a step-level RL method with fine-grained credit assignment teaches the SLM to both solve the task and decide when to delegate, with no extra decision model at inference time.
4. Extensive evaluation plus a real deployment. Experiments on long-horizon coding benchmarks show an average 46% inference-cost reduction across models and benchmarks with competitive performance, and the self-delegation model is also deployed on an NVIDIA DGX Spark.

Main Findings

• Random delegation already shows placement matters: Using Qwen3.5-9B locally and GPT-5.5 (xhigh) in the cloud with up to 80 turns per trajectory, the authors randomly delegated a fixed fraction of steps. At a 40% cloud ratio, some random allocations matched full cloud execution while reducing inference cost by at least 25%, but performance varied substantially across allocations.
• Write/Edit steps are the most valuable to delegate: Across 2.4k coding-agent trajectories split into six action types (Write/Edit, Read/Search, Diagnose/Reproduce, Test/Validate, Discover/Navigate, Wait/Poll), Write/Edit cloud coverage had the strongest association with success, with a Pearson correlation of 0.200 and a 95% confidence interval of [0.099, 0.297]. Read/Search showed a weaker positive association, and the other action types had confidence intervals overlapping zero.
• Successful trajectories delegate differently: Successful runs allocated 9.4 percentage points more cloud coverage to Write/Edit on Terminal-Bench 2.1 and 5.6 more on SWE-ReBench, while differences for other action types remained small.
• Self-delegation roughly preserves accuracy at much lower cost (Terminal-Bench 2.1, GPT-5.5): Vanilla LLM reached 74.16 Pass@1 at $90.31 and 82.02 Pass@3 at $279.73; self-delegation reached 71.91 Pass@1 at $56.96 and 82.02 Pass@3 at $181.88 (deltas of −2.25%, −36.93%, +0.00%, −34.98%).
• Same pattern with Opus 4.8 (Terminal-Bench 2.1): Vanilla was 67.42 Pass@1 at $64.78 and 75.28 Pass@3 at $187.72; self-delegation was 64.04 Pass@1 at $42.98 and 75.28 Pass@3 at $126.53 (deltas of −3.38%, −33.65%, +0.00%, −32.60%).
• Self-delegation actually improves accuracy on SWE-ReBench (GPT-5.5): Vanilla was 56.36 Pass@1 at $200.50 and 63.64 Pass@3 at $598.08; self-delegation was 58.18 Pass@1 at $97.80 and 66.36 Pass@3 at $283.02 (deltas of +1.82%, −51.22%, +2.72%, −52.68%).
• SWE-ReBench with Opus 4.8: Vanilla was 60.00 Pass@1 at $199.06 and 69.09 Pass@3 at $584.70; self-delegation was 57.27 Pass@1 at $74.28 and 66.36 Pass@3 at $221.39 (deltas of −2.73%, −62.68%, −2.73%, −62.14%).
• Headline cost savings: The abstract reports reductions of up to 62% on SWE-Rebench and 34% on Terminal-Bench 2.1, and the introduction reports an average 46% reduction across models and benchmarks. Self-delegation reduces Pass@3 cost by 53%/63% on SWE-ReBench and 35%/33% on Terminal-Bench 2.1 with GPT-5.5/Opus 4.8.
• Better than routing baselines: On Terminal-Bench 2.1 with GPT-5.5, self-delegation (71.91 Pass@1, $56.96) beat Random Delegation (66.29, $74.34), RSD (62.92, $45.61), STEER (66.29, $61.81) and MTRouter (68.54, $73.55).
• Full context to the LLM works best: Among five context configurations passed from SLM to LLM, Full History gave the best performance at the highest cost; No SLM Reasoning was second-best at lower cost. Passing only the previous step or recent k=3 steps lost information needed for delegated steps.
• Full history is also the best context for the judge: Among the three judge configurations tested, Full History performed best; adding the SLM's current analysis reduced cloud usage and cost but lowered task performance.
• A threshold gives budget control: Extending the judge to emit a continuous delegation score with threshold τ, validation on a split of SWE-Smith showed thresholds around 0.2–0.4 gave the better performance–cost trade-offs, a trend that held on Terminal-Bench 2.1 and SWE-ReBench across LLMs.
• Real edge deployment holds up: On an NVIDIA DGX Spark, the self-delegation model achieved Pass@3 comparable to the A100-based evaluation while reducing inference cost by approximately 50% versus the vanilla LLM.
• RL is essential, not just SFT: The SFT-only model reached 57.30% Pass@1 at $53.22, versus 71.91% at $56.96 for the RL self-delegation model.
• Step-level credit assignment changes where delegation goes: With token-level advantage, 42.82% of delegated steps were Write/Edit and 32.11% Read/Search, with 5.80 delegations per task and 71.91% accuracy at $56.96. Without it, only 29.20% were Write/Edit and 40.73% Read/Search, with 8.97 delegations per task and 65.17% accuracy at $55.05.
• Stronger SLM capability alone is not enough: Vanilla Qwen3.5-9B scored 19.10% Pass@1 without cloud access; a version trained with standard GRPO then SFT for delegation scored 32.58%; self-delegation reached 71.91%.
• Generalizes to another SLM: With TMax-9B as the local model and GPT-5.5 xhigh in the cloud, the method matched GPT-5.5's Pass@1 while reducing inference cost by 30.6%.

Methodology in Plain English

The authors start with a diagnostic experiment rather than a system. They let a small local model run the agent loop and flip a weighted coin at every step to decide whether the cloud model takes that step; comparing many random allocations at the same cloud budget shows how much the placement of cloud calls matters.

They then look at what distinguishes valuable steps by labeling each step in 2.4k trajectories with one of six action types, measuring how often each type was sent to the cloud, and correlating that with task success after controlling for benchmark, task, and delegation budget. Write/Edit and complex Read/Search emerge as the key step types, and these become the criteria for delegation.

Two delegation mechanisms follow. The training-free version adds a separate judge model (GPT-5.4-mini) that reads the trajectory and scores whether the next step is a key step; a threshold on that score controls how often the cloud is used. The trained version removes the judge entirely: the SLM is first fine-tuned with supervised data generated by GPT-5.5, where delegation labels mainly teach the output format of delegating, and then trained with reinforcement learning. The reward gives credit for a correct final answer plus a bonus for correct answers that used fewer cloud calls. To teach which steps deserve the limited cloud budget, a lightweight critic scores each individual delegation decision against the key-step criteria, and that step-level signal is added to the usual trajectory-level advantage, with a scaling factor so it is not diluted when delegation tokens are rare. Tokens produced by the cloud model are excluded from the policy loss.

For training they use SWE-Smith with Qwen3.5-9B as the SLM and Kimi-K3 as the LLM, 2K examples for SFT, then RL on 40 NVIDIA A100 80GB GPUs for 175 steps with batch size 8 and 16 rollouts per example, yielding a checkpoint called EdgeWise-9B. Evaluation uses Terminal-Bench 2.1 and SWE-ReBench-03 with the Terminus2 harness, capped at 80 turns, and cost is computed from cloud API token pricing with local SLM compute excluded and judge API cost included for judge-based delegation.

Why This Matters

Impact on research: The paper reframes cost-efficient agents as a step-allocation problem rather than a query-routing problem. It argues that prior routing work is mostly evaluated on single-query reasoning or multi-hop QA rather than long-horizon tasks, and that routers trained over a fixed model pool become brittle as that pool changes. Self-delegation removes the extra decision model at inference time, which is a meaningful shift for how such systems are built.

Real-world applications:

• On-device coding assistants that run mostly locally on a workstation or laptop and only call a cloud model for edits and complex searches, which is directly what the DGX Spark deployment demonstrates.
• Cost-controlled software-engineering agents in CI or issue-triage pipelines, where the threshold mechanism lets operators set a cloud budget in advance.
• Privacy- or latency-sensitive deployments where most of the trajectory stays on local hardware, with only selected steps leaving the device.
• Autonomous long-running agents with many turns per task, where per-step cloud pricing compounds and a 30–60% cost reduction changes what is economically viable.

Industry relevance: The work comes from an industrial research lab (the project page is hosted under NVIDIA research) and is evaluated on production-relevant harnesses (Terminus2, OpenHands) and hardware (NVIDIA DGX Spark, A100), which makes the cost numbers directly interpretable for teams deciding whether to ship local-plus-cloud agent products.

Future Directions

• Beyond coding tasks: The limitations section states the framework is only evaluated on long-horizon coding and not on deep research or computer-use agents (CUA), leaving validation in those settings to future work.
• Generalization across more SLMs, judges, and harnesses: The paper begins this in its appendix (for example TMax-9B as the SLM and alternative judge models), but a broader sweep would test how tightly the delegation policy is bound to a particular local model.
• Learning the delegation budget instead of thresholding it: The threshold τ is currently a knob tuned on validation data (with 0.2–0.4 working best); making the budget itself part of the training objective is a natural extension of the existing cost-regularization term.
• Understanding why collaboration sometimes beats the pure LLM: The authors hypothesize that SLM–LLM collaboration mitigates overthinking and increases exploration diversity across trajectories, since self-delegation matched the LLM on Pass@3 while trailing on Pass@1 in some settings, but this is offered as a hypothesis rather than established.

Target Audience

Researchers and engineers working on agent systems, model routing and cascades, or on-device inference. It is most useful to readers who care about the cost of running long multi-turn agents and who want both a practical recipe (judge-based delegation with a threshold) and a training recipe (step-level RL for self-delegation). Readers primarily interested in pure model-capability research will find the empirical cost analysis more relevant than the training details.

Authors’ abstract

Complex and long-horizon agent tasks often rely on frontier models deployed in the cloud to achieve strong performance. However, using these models for every step of a long-horizon trajectory incurs substantial inference cost, even though not all steps may require the same level of model capability. In this paper, we find that allocating the LLM to only a small subset of key steps can achieve performance comparable to full LLM execution while significantly reducing inference cost. Motivated by this observation, we propose EdgeWise, an edge-cloud collaboration framework that deploys an SLM on the local device as the primary solver and selectively delegates critical steps to a cloud LLM when stronger model capacity is needed. To enable autonomous delegation without additional models at inference time, we introduce a step-level RL method that jointly optimizes task solving and delegation, with fine-grained credit assignment for individual delegation decisions. We also introduce an external judge-based delegation strategy and use it to systematically investigate key design choices in edge-cloud collaboration, including when to delegate, how delegation decisions should be made, what context should be shared, and how frequently the cloud LLM should be invoked. Experiments show that EdgeWise matches or outperforms pure-LLM execution while reducing inference cost up to 62% on SWE-Rebench and 34% on Terminal-Bench 2.1. We further deploy EdgeWise on a local workstation, demonstrating comparable cost savings in a practical edge-cloud setting.