Not Every Call Needs a Frontier Model: Per-Call-Site Evaluation of Small Language Models in a Deployed Agentic Home-Automation System

AI summary

Overview

Research area: Applied evaluation of small language models (SLMs) inside a deployed agentic system, sitting at the intersection of agent architecture, model routing, LLM cost engineering and smart-home automation.

Technical level: Advanced. The paper combines a benchmark release, paired statistical testing, cost accounting and a live deployment study.

Scope (one sentence): The authors measure nine models from 0.8B parameters up to a frontier hosted model on the five distinct LLM call sites of a running open-source home-automation framework, and show that per-call-site model assignment — not whole-system model choice — is where capability, safety and cost are decided.

What This Paper Is About

An agentic system makes several structurally different LLM calls per user turn, and those calls differ enormously in difficulty — classifying a short command is not the same task as writing asynchronous Python against an unfamiliar API. In practice, one model sized for the hardest call serves every call, so cheap, frequent calls are paid for at frontier prices. This paper asks, for each call site separately, which calls can be moved to a small local model without the deployed system getting worse.

Key Contributions

1. A 280-case benchmark over five call sites of a real agentic system. The cases are grounded in two live Home Assistant installations with different inventories, and include cross-site cases that are answerable at one site and impossible at the other. Sub-groups (flip, nearmiss, refusal, control, same) are encoded in case ids.
2. A 9-model measurement with paired testing. The results show per-site orderings that differ from each other, are non-monotonic in model size, and leave only one call site (codegen) separating local models from hosted ones statistically.
3. An actuation-safety metric that separates deployable models from models with the same aggregate score. The paper reports the joint rate of acting when asked and refusing when the device does not exist, because aggregate accuracy hides both failure directions.
4. Routing policies tested end to end on the live deployment, where the offline prediction held and three failures appeared that no per-call benchmark could see. All 2520 offline records, the harness and the benchmark are released at https://github.com/waldiez/slm-callsite-eval.

Main Findings

• No model wins every call site, and size does not settle the ordering. Three different local models take the five sites. Qwen3.5 4B scores below Qwen3.5 2B on actuation (50.9% vs. 62.1%), and Gemma4 26B falls below Gemma4 12B on code generation (80.0% vs. 84.0%). The hosted frontier model is not uniformly best either: Table 1 gives Gemma4 26B 88.0% at planning against 84.0% for Claude Sonnet 5, a gap driven by the infeasible sub-group, where the hosted model plans a pipeline for a request the site cannot support 20% of the time. (The paper's prose reverses these two planning figures in parentheses relative to its own Table 1.)

• Four of five call sites are statistically indistinguishable between the best local model and either hosted model. Paired McNemar exact tests compare Gemma4 26B against Claude Haiku 4.5 and Claude Sonnet 5 on the same cases. Intent, HA action, actuator and planner show no significant difference; the actuator comparison against Sonnet 5 misses significance at p = 0.062, but all five discordant cases fall in the same direction, which the authors read as insufficient power over a fixed set of 116 cases rather than a true null.

• Only code generation separates local from hosted. Gemma4 26B is beaten at codegen by Sonnet 5 (p = 0.002) and by Haiku 4.5 (p = 0.039), with every discordant case favouring the hosted model. Because the small hosted model reproduces the gap, the authors read the deficit as a property of the call site rather than of frontier scale.

• The codegen gap is narrow, not broad. Gemma4 26B already matches the hosted models on five of twelve API shapes (blocking work, topic contracts, one-shot agents, publication and state). It loses ground on the callback and lifecycle shapes: window 50, poll 67, cleanup 67 and subscribe 71, against 100 for both hosted models. One shape resists both small models: llm scores 33 for Gemma4 26B and 33 for Haiku 4.5, where Sonnet 5 scores 100.

• Small hosted is not uniformly better than local. Claude Haiku 4.5 scores 80.6% at intent, below five of the seven local models, and 89.6% overall, below Gemma4 26B's 90.7%. The two reach near-identical totals with opposite profiles: the local model is stronger at routing, the hosted one stronger at code generation.

• Aggregate accuracy hides a two-sided actuation failure. Small models resolve the accuracy/refusal trade-off in degenerate ways. Gemma4 E2B actuates on 87.2% of requests for devices the site does not own, while another model refuses every request it receives. Qwen3.5 0.8B emits the same two-token empty array across all 116 actuator cases, so its perfect 0.0% over-actuation rate and 0.0% execution accuracy are the same fact, and its 39 correct refusals carry no evidence of recognition. Within one family, Qwen3.5 2B over-actuates on 66.7% of impossible requests, while Qwen3.5 4B returns the empty action list on 72.7% of actionable ones.

• Constrained decoding would not close the actuation gap. Of the 70 service calls models emitted on impossible requests, 53 (76%) already named an entity that exists in the site's registry: the model chose a real but wrong device, so a grammar over valid entity ids would not have blocked them.

• Routing beats model choice on cost. All-local per-site routing costs nothing and gives up 3.6 points against all-hosted Sonnet 5 (91.8% vs. 95.4%). Hosting only the two generative sites gives up 1.4 points and cuts spend by 82%. Routing the generative sites to Haiku 4.5 instead of Sonnet 5 costs 92.5% against 93.9% for $0.56 against $2.15. The reason is that 81% of hosted spend goes to the actuator site, whose prompt carries the site's whole entity registry at 28,122 input tokens on average — and that is the site a mid-sized local model already serves at 95.7%.

• The offline prediction held on live hardware. Hosting only planner and codegen matched hosting everything on pipeline cases (15/20 and 15/20) for 28% of the spend ($0.70 against $2.47); excluding the 3 system-limited cases, the two configurations were indistinguishable at 39/43. The direct cases act as a control and behave like one (24/26 and 24/26), while moving actuation from the frontier model to a local one cost exactly one case in twenty-six (25/26 against 24/26) — four points, matching the four points Table 1 predicted (99.1% against 94.8%).

• Scale was not monotone in the live run either. The all-hosted configuration was the only one to fail "in two minutes turn off the livingroom lights", classifying it as conversation and actuating nothing, and it answered "when the livingroom door opens turn on the lights there" by switching on every light in the house, where the local configurations replied that the site has no door sensor. Both are cases the cheaper system got right.

• Three failures were visible only in deployment. Two pipeline cases failed because agent.window() returns a fresh empty buffer on every call, so an agent that counts or averages reads zero forever; a third degraded silently to a statistical summary because a spawned agent could not reach an LLM client. Several requests never reached the codegen site at all, because the framework matches them to a pre-built catalogue recipe, which bounds how much the codegen site matters in practice.

Methodology in Plain English

The authors took an already-deployed open-source home-automation framework, Wactorz v0.5.3 (a multi-agent runtime with agents supervised over MQTT), and identified the five LLM calls it makes per user turn. Three of them are classification-style: intent (a four-way label under a ten-token budget), HA action (a Home Assistant request classified into one of the framework's action labels), and actuator (mapping language into a JSON array of service calls against the site's live entity registry). Two are generative: planner (emitting a pipeline specification with concrete entity ids, topics and schedules) and codegen (writing the Python body of each agent against the framework's agent API). Only the last two form a chain.

They built a 280-case benchmark (36 intent, 28 HA action, 116 actuator, 50 planner, 50 codegen) from two real installations — a household and an office — with different inventories, using the framework's unmodified production prompts for the first three sites and harness-condensed variants that preserve the output contract for the last two. The actuator cases deliberately include devices the other installation owns, so a wrong answer is a substitution rather than a hallucination, and it would drive real hardware.

Nine models were run: Qwen3.5 at 0.8B/2B/4B, Gemma4 at E2B/E4B/12B/26B, and Claude Sonnet 5 and Haiku 4.5 as hosted references. Local models ran on one NVIDIA DGX Spark (GB10 Blackwell, 128 GB unified memory) via llama.cpp behind a LiteLLM proxy, all at 4-bit GGUF and 128k context so no prompt was truncated. Every call ran at temperature 0, one pass per case. Scoring is exact: exact label match for intent and HA action, exact action-set match for actuation (an unrequested extra actuation counts as a failure), a valid plan resolving the given entity ids and topics for planner, and code that compiles and defines the required async functions for codegen. Statistical separation uses paired McNemar exact tests on the same cases.

Finally, they drove the running deployment through its chat endpoint under three routing configurations and judged every case as a user would: the requested effect happened, nothing else was actuated, and the reply claimed nothing untrue. Judgement was by a person rather than a model or a side-effect check, because neither would catch an agent that registers, runs and does the wrong thing.

Why This Matters

The paper reframes model procurement for agentic systems as a per-call-site routing decision. It shows that the expensive call is not the difficult call, that the standard measure — aggregate accuracy — can hide both a model that actuates on devices a site does not own and a model that refuses everything, and that a per-call-site assignment is a static configuration value that cannot mis-route between sites. It also connects a position-paper argument (most agentic calls are repetitive and narrow) to measured numbers on a deployed system, and provides an end-to-end check that the offline benchmark actually predicts what happens to real hardware.

Real-world applications:

• Smart-home and building automation assistants that run locally on a home hub instead of paying per request to a hosted API, with the actuator call — the one that dominates spend — served by a mid-sized local model.
• Safety-critical actuation, such as locks, doors, covers and thermostats, where the metric that matters is the joint rate of acting when asked and refusing when the device does not exist, not a single accuracy number.
• Edge and on-premises agent deployments in which privacy, latency or connectivity rule out hosted calls for routine traffic while leaving a small hosted budget for code generation.
• Cost governance for agentic platforms that issue mixed call types per turn, where the paper's measured 82% and 95% savings at 1.4-point cost illustrate how configuration alone changes spend.

Industry relevance: The results give teams a concrete migration procedure — instrument the running system, find the calls a smaller model can serve, migrate them — and a warning that a model chosen on aggregate benchmarks may be unusable for hardware control. The framework-specific finding, that local models lose ground specifically on callback and lifecycle idioms (subscribe, window, poll, cleanup) rather than on ordinary Python, points at a targeted failure mode for anyone generating code against a proprietary SDK or internal API.

Future Directions

• Multiple sampling passes and variance reporting. The paper runs one pass per case at temperature 0 and reports no variance at the generative sites, even though one end-to-end case passed in two configurations and failed in the third.
• More installations and registry sizes. The evaluation covers two installations with author-annotated ground truth, and the entity registry — the dominant part of the actuator prompt — was never varied in size, so how small models degrade as a home grows is open.
• Blinded, multi-rater end-to-end evaluation. The live run covers one installation and its verdicts come from a single unblinded rater; the criteria are observable and the annotations are released for re-judging.
• Comparison against learned per-query routers, and decomposition of compound requests. Static per-site assignments must be revisited

Authors’ abstract

An agentic system issues several structurally different kinds of LLM calls. It routes intent, classifies actions, grounds language in a device registry, plans multi-agent pipelines and writes the Python code those pipelines run. The difficulty of these call sites varies by an order of magnitude, yet in practice a single model, chosen for the hardest site, serves all of them. In this work, we evaluate 9 models from 0.8B to a frontier hosted model across the five call sites of a deployed open-source home-automation framework (Wactorz), using its unmodified production prompts and two real Home Assistant installations (280 cases, 2520 scored calls). We find that capability is not ordered the same way at every site, and that larger models are not uniformly better: one 4B model is worse than its 2B sibling at grounded actuation. Paired testing shows the best local model to be statistically indistinguishable from both hosted models at four of five sites. Only code generation separates them, against a small hosted model (p = 0.039) as well as a frontier one (p = 0.002). Aggregate accuracy also hides a safety failure specific to actuation, where small models resolve the accuracy/refusal trade-off in degenerate ways: one model (Gemma4 E2B) actuates on 87.2% of requests for devices the site does not own, while another refuses every request it receives. Routing each site to its best local model reaches 91.8% against 95.4% at no per-call cost. In a live deployment judged by a user, hosting only the two generative sites matches hosting everything (39/43 against 39/43) for 28% of the spend, and the actuation gap the benchmark predicted appears as exactly one case in twenty-six. Benchmark, harness and all records are released at https://github.com/waldiez/slm-callsite-eval.