Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation

AI summary

Overview

• Research area: Evaluation methodology for local LLM coding agents — specifically the serving layer (Ollama, llama.cpp, vLLM, SGLang) that sits between an agent harness and model weights when tool calls are requested.
• Technical level: Intermediate. The mechanisms are explained plainly, but comfort with HTTP status codes, function-calling schemas, and benchmark terminology helps.
• Scope: A measurement study showing that a standard local tool-use setup can measure the serving stack rather than the model, with a control arm, a four-stack cross-check, and a reproducibility checklist.

What This Paper Is About

Before an agent can act, it must emit a tool call the harness can parse and execute — a protocol step separate from choosing the right tool. The authors set out to measure how often small local models succeed at this step, and found that the measurement itself was broken. On Ollama, the default tools= request is gated per model by a static template flag, and the harness records a refused or empty request as an ordinary assistant turn, so a capable model can be reported at 0% fidelity without ever running.

Key Contributions

1. Isolation of a silent serving-layer confound in native function calling. The tools= request is accepted or rejected per model by a static flag in the model template, before the model is consulted; Phi-3 and Gemma-3 are rejected with HTTP 400 "does not support tools", yet a naive per-turn analysis reports them at 0%.

2. A control arm that separates the serving channel from the prompt. Adding a text tool list and call format while retaining the native tools= channel recovers most of the measured fidelity for accepted models, while a uniform text-tools protocol lowers fidelity for Llama-3.2, which has real native tool_calls support — so the best interface is model-dependent.

3. A four-stack cross-check. Identical tools= requests are handled differently by Ollama, llama.cpp, vLLM, and SGLang, showing the outcome is a serving-stack property rather than a model property.

4. A reporting checklist plus a negative result. The paper argues for reporting per-seed rates with intervals, logging transport failures as a distinct outcome, and refusing to claim scale, family, or reasoning effects at this measurement precision.

Main Findings

• Rejection masquerades as model failure. Under native function calling, Phi-3 and Gemma-3 are reported at 0%, but 100% of their turns are rejected requests — the model never ran, so the rate is properly undefined ("rej" in the figures and Table 1).

• Non-response is a second, separate mechanism. Gemma-3-4B emits one valid call per seed and then fails to respond; a naive denominator counts this against it. Fidelity is therefore defined over turns the model actually produced, with non-responses excluded and reported separately.

• The prompt, not the channel, drives most of the accepted-model gap. Per-seed fidelity for Qwen 0.5B–14B rises from 0/38/23/59/60% (native) to 35/58/82/89/80% (native+hint), close to the uniform text-tools rate. The low native numbers reflect a default call path without explicit format guidance, not model inability.

• Llama-3.2 is the informative exception. native+hint reaches 82%, but the uniform text-tools protocol drops it to 44%, because Llama-3.2 has real native tool_calls support that the text protocol discards. No single serving configuration maximizes measured fidelity for every model.

• The rejection is stack policy, not the model. The same GGUF weights Ollama rejects run on llama.cpp and return a text response under an identical tools= request. vLLM refuses the same request by default (HTTP 400, "'auto' tool choice requires --enable-auto-tool-choice and --tool-call-parser"), independent of the model and before model dispatch; enabling those flags with the hermes parser yields HTTP 200 but the call still did not surface as a native tool_call for the two small models probed. SGLang accepts by default (HTTP 200) but returns the call as text unless launched with --tool-call-parser.

• The fidelity number is not robust to pooling. Qwen-0.5B under text-tools is 85% pooled but 34% per-seed: seven of eight episodes fail in one or two turns while a single 41-turn episode of repeated valid calls dominates the pool. Pooled and per-seed estimates differ by up to about 55 points. With denominators as small as 8–16 turns over 8 seeds, bootstrap intervals are wide: Qwen-0.5B text-tools 34% [9, 59], Phi-3 38% [12, 75], Gemma-3-270m 38% [12, 75] — too wide to rank these models.

• Constrained decoding removes parse failures but moves the failure. With decoding constrained to a JSON schema whose name field enumerates the available tools, all nine local models emit a valid in-schema call on 8 of 8 single-step trials. But forced to emit a call every turn, weaker models never terminate, producing 600–900-turn loops; Qwen-0.5B reached 869 assistant turns in one seed before being cut off.

• Replication on a second task and on HumanEval. On a dependency-chain task (4 seeds), the native gating replicates exactly, per-seed fidelity rises roughly with Qwen scale (44% to 100%), Gemma-3-4B again emits a valid call then fails to respond (4 non-responses), and Llama-3.2 falls to 12% per-seed versus 60% on aggregation. On HumanEval (n=6), the gating replicates and text-tools valid-call rates are 4/6, 5/6, 6/6 for Qwen 0.5B/1.5B/3B, with Gemma-3-270m at 0/6 and Llama-3.2 at 6/6.

• Explicit negative result. The intervals overlap and estimates depend on the pooling choice, so the authors claim no scale law, no family or tool-training effect, and no dissociation from reasoning. A small no-tools reasoning probe was inconclusive and is omitted.

Methodology in Plain English

The authors instrument an off-the-shelf ReAct coding-agent harness (an extension of LOCA-bench) on a fixed aggregation task requiring several tool calls, and label every assistant turn: valid in-schema call, hallucinated call (parseable but tool not found), unparseable text, no-call prose, and non-response (the harness's retry-exhaustion error). Protocol fidelity is the valid-in-schema-call rate over turns the model actually produced.

They run 8 seeds per model — "seed" here means a task instance, not a decoder RNG seed, and decoding is sampled at T=1.0 and not held fixed — and report both a turn-pooled rate and a per-seed mean, with a seed-level bootstrap 95% CI over 10,000 resamples.

They compare three serving conditions: native (harness default, sends tools=, gated per model on Ollama), native+hint (still sends tools=, plus a single fixed plain-text tool list, an explicit JSON call format, and an allowed-name list in the prompt), and text-tools (tools= dropped, same text guidance, calls parsed from text, uniform across all models). They add a constrained-decoding probe and then probe the same request across four serving stacks. Models are Qwen2.5-Coder 0.5B/1.5B/3B/7B/14B, Llama-3.2-3B, Phi-3-mini, Gemma-3-4B, and Gemma-3-270m via Ollama at default quantization (Q4_K_M), plus deepseek-v4-flash as a cloud anchor.

Why This Matters

Research impact. Existing function-calling and agent benchmarks — BFCL, AgentBench, tau-bench, ToolSandbox, SWE-agent — assume the serving layer is transparent. BFCL already reports empty-turn and API-error categories, but it evaluates through per-model handlers on a held-fixed stack, so per-model request gating inside a local serving layer is not isolated. This paper treats the serving interface as an experimental variable instead of infrastructure, and argues that a refused or empty request is a first-class evaluation outcome rather than a framework-compatibility bug.

Real-world applications:

• Local coding agents for cost and privacy. Practitioners running small models on laptops to avoid hosted APIs are exactly the users exposed to the per-model tools= gate in Ollama.
• Benchmark and leaderboard design. Anyone reporting tool-call accuracy should report the serving configuration — stack, version, tool-call parser — alongside model identity, as they already report decoding parameters and hardware.
• Debugging a 0% tool-call rate. The paper's diagnostic order (was the request refused or emptied, is a parser configured, only then blame the model) is directly usable as an operational triage.
• Agent framework engineering. The confound is a correctable defect in trajectory representation: the transport evidence arrives but the structured failure type is not persisted, so only the message stream remains.

Industry relevance. The authors argue the trend is toward more layers between harness and model, not fewer, and that tool-calling standards like the Model Context Protocol plausibly add another translation step — offered explicitly as conjecture, not a measured finding. They also note hosted APIs are not exempt in principle, though that gate is opaque and was not probed.

Future Directions

• Cross-stack magnitude transfer. Whether the per-seed fidelity magnitudes transfer across serving stacks is untested; the vLLM and SGLang probes cover only Qwen-0.5B and Phi-3, so their results are claims about those model–stack pairs rather than either stack in general.
• Agentic constrained decoding. The constrained-decoding baseline is single-turn because the agentic version did not terminate on weak models; whether multi-turn behavior under constraint is recoverable remains open.
• Hint sensitivity. The text hint includes an allowed-name list, so "the prompt drives it" is specific to this guidance; a weaker hint might recover less.
• Release-level policy drift. The mechanism (a per-model template flag consulted before dispatch) is structural, but which Ollama tags are gated is a release-level policy that can change; the authors scope their native-mode claims to Ollama 0.30.8 and make no claim that the specific per-model outcomes hold for later releases.
• Reasoning and scale questions. The authors collected a reasoning probe but omit it as inconclusive, and explicitly decline to claim any scale law or family effect at this precision.

Target Audience

Researchers and engineers who evaluate or deploy local coding agents: benchmark maintainers and leaderboard authors, agent-harness developers, and practitioners running small models through Ollama, llama.cpp, vLLM, or SGLang who need to know when a reported tool-call number reflects the model and when it reflects the stack. It is also useful to anyone designing evaluation protocols for function calling, and to readers interested in a candid example of a negative result and self-reported measurement fragility.

Authors’ abstract

A coding agent must emit a valid tool call--a parseable invocation of a tool in the provided schema--before the harness can execute its chosen action. We study how local serving stacks affect this protocol step and show that measured outcomes can depend on the serving layer rather than model behavior alone. In Ollama, the default tools= request is gated per model by a static template flag: some models are accepted and return calls as text, some return native tool_calls, while Phi-3 and Gemma-3 are rejected before inference. In our harness, rejection and retry exhaustion are not preserved as structured failure metadata, so downstream analysis can misclassify them as model non-calls and naively report 0% fidelity. Adding a text tool list while retaining the native channel recovers much of the measured fidelity for accepted models, whereas a uniform text protocol reduces fidelity for Llama-3.2, which has native tool-call support. Cross-stack probes on Ollama, llama.cpp, vLLM, and SGLang show different handling of the same request. Constrained decoding removes parse failures but can induce non-termination, and turn-pooled versus per-instance estimates differ by up to about 55 points. We conclude with a checklist for treating serving behavior as part of the evaluation protocol.