Beyond Benchmarks: Can Small Open-Weight Models Get Real Coding Work Done?

AI summary

Overview

Research area: Software engineering for AI coding agents — specifically, model serving and inference-systems engineering for small open-weight language models used inside coding-agent workflows (cs.SE).

Technical level: Intermediate. The paper is an experience report rather than a formal methods paper; it requires some familiarity with model serving runtimes (vLLM, SGLang, llama.cpp, MLX), tool calling, and KV/prefix caching, but the argument is carried by operational narrative rather than mathematics.

Scope (one sentence): A retrospective, single-author account of four months (July–October 2026) of engineering work to make small open-weight models complete verified coding work behind a frontier-model parent agent, supported by 4,165 logged gateway turns, 623 debug jobs, and two bounded delegation studies with hidden behavioral checks.

What This Paper Is About

Getting an open-weight model to produce text is not the same as getting it to finish coding work. The author documents, layer by layer, everything that had to function between a loaded checkpoint and a code change a developer could trust: distributed inference across laptops, reproducible serving recipes, tool-call parsing, gateway translation, session and cache continuity, artifact contracts, and independent behavioral verification.

The goal is not to rank models or runtimes, but to give an evidence-qualified account of where progress actually stops — between a model being served, producing coherent text, having its tool calls parsed and executed, and finally producing code that passes task-specific hidden checks.

Key Contributions

1. A four-level success framework distinguishing served, coherent text, parsed and executed tool calls, and verified task completion, with documented cases where models satisfied earlier levels while failing later ones.
2. Counted operational evidence from real sessions: 4,165 coding-agent gateway turns over three weeks in October 2026, 623 unique debug jobs (predominantly Gemma 4 26B-A4B), including 85 narrowly defined cold-prefill records and 280 high-reuse records with performance metadata.
3. A bounded 45-trial coding study across five TypeScript task categories (quote, prorate, customer, split, parts) and three execution conditions, scored by hidden behavioral tests separate from the agents' own success claims.
4. A second 15-task suite under a fixed artifact contract, comparing 45 direct and 45 delegated sessions with a Nemotron Cascade 2 worker, plus a separate 45-session Qwen3-Coder follow-up, with inputs, settings, and model revisions hash-recorded before scoring.

Main Findings

• Cold versus warm prefix cost is large (observational). Saved cold-prefix requests had a median time to first token of about 69.4 s, versus 2.8 s for high-reuse requests. Among the 85 cold jobs, 54 fell between 65 and 82 seconds; 59 cold and 167 warm jobs were attributed to llama.cpp, with the remaining 26 cold and 113 warm jobs attributed to MLX.

• Warm decode throughput: in warm jobs that recorded decode throughput, the median was 38.5 tokens/s across 206 records; some selected warm requests had TTFT near one second.

• Per-recipe gateway-turn timings differ substantially (warm = at least 8,000 reused prefix tokens; cold = none): Gemma 4 26B-A4B 1,128 warm at 3.0 s versus 259 cold at 67.0 s; Nemotron Cascade 2 338 warm at 10.8 s versus 90 cold at 86.6 s; Qwen3-Coder 30B-A3B IQ3 282 warm at 2.0 s versus 29 cold at 135.6 s. The author stresses these are not controlled model comparisons.

• Tool-protocol events interrupt work before correctness can be judged. Of 492 Gemma prompts containing tools, 294 ended in parsed tool_calls, 144 ended stop with no parsed tool call, and 54 had no saved parsed response (not an exhaustive partition). Of 2,816 timed gateway turns, the disjoint error classes totalled 154 turns: empty_content 74, stream_failed 44, repetition_cut 21, job_failed 15, with two rarer classes covering another three turns; a separate fallback flag appeared on 40 turns, 19 of which also had empty_content.

• Direct frontier coding beat delegation in both studies with a direct arm. In the 45-trial study, direct coding passed 15/15 (95% Wilson interval [0.80, 1.00]); optimized delegation to a quantized Qwen3-Coder 30B-A3B worker passed 12/15 ([0.55, 0.93]); batch-labeled adaptive delegation passed 14/15 ([0.70, 0.99]). All 26 verified delegated trials retained worker-produced implementations, though the harness cannot rule out intermediate parent edits.

• The batch-labeled condition was not a clean batching comparison. Only 5/15 batch-labeled trials actually used a batch call (three packages each); the remaining 10/15 used a single delegation. Among the 14 verified batch-labeled trials, five used batch calls and nine used single delegation.

• The second suite showed a much wider gap under a fixed artifact contract. The completed replacement cohort scored 45/45 for direct coding and 12/45 for the Nemotron Cascade 2 worker, with 11 worker substitutions. Sixteen sessions stopped by a subscription limit before any model call were replaced; counting them as failures gives 37/45 and 8/45. A Qwen3-Coder worker passed 0/45.

• Interface contracts can reject correct code. A post-hoc diagnostic found that 19 of 36 artifacts rejected solely for renamed parameters (removed leading underscores) passed the hidden checks unchanged when that rule was bypassed. They are explicitly not counted as successes, since they never went through parent review and application.

• Cost direction was consistent. In both studies that included a direct arm, direct coding had a lower API list-price-equivalent frontier cost per verified success than delegation. This measure is derived from published API prices and recorded token usage, not the author's actual subscription bill, and excludes local worker cost.

• Auxiliary requests can evict the main conversation's cache. Of 65 requests classified as auxiliary, 44 recorded no cached-prefix reuse. Auxiliary requests had a median TTFT of about 12.2 seconds (13 observations) versus about 2.7 seconds for main requests (267 observations).

• Compaction is not completion. Across 12 OpenCode/Qwen3.5-122B-A10B tests with deliberately lowered compaction thresholds, sub-agent compaction triggered in six (main-session compaction never triggered), and none of those six sub-agent tasks completed; one recorded a PASS label that contradicted the incomplete-task checks.

• Distributed inference over Wi-Fi was feasible but not productive for agents. One saved 32B load transferred about 8.7 GB and took roughly five minutes over Wi-Fi; chat was usable, but coding-agent sessions remained too slow, and no matched distributed-versus-single-node benchmark was preserved.

• Gateway-level validation had to be exercised end to end. A Python API harness (not the Claude Code application) sending Anthropic Messages requests to Qwen3.5-122B-A10B on SGLang produced a preserved synthetic run sequence of 2/12, a separate 0/1, 10/12, 11/12, 12/12, and 12/12; separate existing-codebase checks returned 1/3 and, in a later run with a different task mix, 1/1.

• Individual integration failures produced rejected artifacts rather than scored failures. A bounded invoice attempt with a Nemotron Cascade 2 worker failed export-preservation validation and returned malformed before parent inspection. A twelve-adapter Nemotron request stopped after approximately 120 seconds with an unavailable/http result and left the stub untouched.

Methodology in Plain English

This is a retrospective empirical systems experience report, not a controlled benchmark campaign. The author reconstructs the work from what he saved along the way — configurations, code changes, integration-test results, inference logs, debugging conversations, and study results — and re-checked the reported counts against those saved results rather than re-running experiments. Four kinds of evidence are kept separate throughout: operational measurements (timings, event counts), task-verified outcomes (hidden behavioral checks), implementation changes (what changed, not necessarily what improved), and engineering observations (problems encountered without an independent measurement).

The engineering proceeded through stages, each forced by a practical constraint: llama.cpp RPC across two laptops (abandoned when restart and weight-transfer costs stalled iteration); single-node serving with reproducible recipes across vLLM, SGLang, llama.cpp, and MLX; a deliberately thinned gateway that translates between client protocols and model templates; session-affinity and cache-discipline work; and finally bounded coding tasks with sealed hidden checks. The target hardware for the local studies was a 32 GB Apple M1 Pro with substantial swap use, running one 4-bit 30B-class worker at a time, chosen to represent a working developer laptop rather than a dedicated benchmark machine.

The coding studies held Claude Code with Claude Opus 5.5 fixed as the parent harness. In the 45-trial study, the worker received the artifact contract, task text, and supplied file context, but the parent's separate constraints field was not forwarded, and a gateway path dropped a duplicate system message and truncated messages above 1,800 characters (of 41 reconstructed worker prompts, two lost approximately 220 final characters, both in failed trials). A trial counts as verified only when task-specific hidden behavioral checks pass; a verified substitution additionally requires a proposed worker artifact, parent inspection, confirmed application, a passing sealed check, and no subsequent parent source rewrite. The studies are explicitly not pooled and do not rank models or runtimes.

Why This Matters

Impact on research. The paper argues that benchmark scores and "the model is running" are weak success criteria for agentic software work. By separating serving, text coherence, tool execution, artifact acceptance, and behavioral verification into distinct levels, it gives evaluation researchers a vocabulary for the failure modes that sit between a model and a scored outcome — including the finding that an interface contract can reject functionally correct code before review begins.

Real-world applications:

• Local-first developer tooling: guidance on choosing quantized worker sizes that leave headroom for context, KV cache, and ordinary applications on a developer laptop, rather than maximizing checkpoint size.
• Agent gateway design: concrete lessons on append-only message sequences, frozen system prefixes, consistent streamed tool-argument appending, session affinity, and propagating upstream failures as errors instead of empty completed streams.
• Auxiliary-traffic planning: the observation that title generation and permission operations can evict the main conversation's cached state, with median TTFT around 12.2 s versus 2.7 s in the recorded groups.
• Delegation policy design: evidence that a quantized worker behind a frontier parent can complete bounded tasks, but that frontier cost per verified success was lower for direct coding in both studies with a direct arm, and that contract rules need testing against functionally correct artifacts.

Industry relevance. Teams deploying open-weight models behind coding agents face exactly the portability problem described: a recipe is a tuple of checkpoint revision × quantization × runtime build × parser/template × context configuration × agent client, and a recipe that works for one combination cannot safely be copied to another. The paper's counts of tool-parse failures, stream failures, and cache-state losses quantify the integration surface that nominally compatible endpoints still expose.

Future Directions

• Test harness-independence. The studies fix Claude Code with Claude Opus 5.5 as the parent; the author notes OpenCode and other harnesses might behave differently and were not tested as interchangeable parents, leaving open how much of the result is system-specific.
• Isolate the contract effect. The post-hoc diagnostic (19 of 36 rejected artifacts passing unchanged) shows a contract rule rejecting correct code, but the author states it does not establish what any study would have scored without that rule — a controlled follow-up would.
• Revisit multi-machine inference. The distributed-laptop direction was parked because of iteration cost, not disproven; a wired Thunderbolt link was never measured against Wi-Fi, and expert-level mixture-of-experts splitting was never made to work.
• Re-run studies at greater scale. The 45-trial study repeats only five task categories with three runs per cell and one worker configuration, and the Wilson intervals overlap; the author explicitly does not claim general success-rate differences from these data.

Target Audience

Engineers building or operating coding-agent infrastructure — inference-serving engineers, agent-framework and gateway developers, and platform teams evaluating open-weight models for developer tooling — will get the most from this paper, particularly from its failure taxonomies and cache/session-continuity guidance. It is also useful to evaluation researchers interested in why benchmark scores and task-verified outcomes diverge, and to technical leads deciding between direct frontier use and delegation to a local worker. Readers seeking a statistically controlled model comparison should look elsewhere; the author is explicit that these are case studies on selected TypeScript fixtures, that the underlying records are private, and that the studies are not pooled and do not rank models or runtimes.

Note: the supplied paper content is truncated partway through Section 9.2, so the cost assumptions (Section 10), the limitations discussion (Section 12), and the data availability statement are referenced but not included in the text available for this summary.

Authors’ abstract

Making an open-weight language model generate text is not the same as making it finish coding work. This retrospective empirical systems experience report follows my engineering work from July to October 2026 across coding clients, model-specific serving recipes, vLLM, SGLang, llama.cpp, and MLX. Models behind a nominally compatible endpoint still differed in tool-call formatting, turn control, context behavior, cache reuse, and reliability. A served model, coherent text, executed tool calls, and code that passes independent behavioral checks proved to be distinct levels of progress. The evidence includes 4,165 coding-agent gateway turns from my own sessions, 623 debug jobs, and three separate delegation studies with hidden behavioral checks and a fixed Claude Opus 5.5 parent. Saved cold-prefix requests had a median time to first token of about 69.4 s, versus 2.8 s for high-reuse requests (observational). In a 45-trial study, direct coding passed 15/15 and delegation to a quantized Qwen3-Coder 30B-A3B worker passed 12/15 and 14/15. On a second 15-task suite under a fixed artifact contract, the completed replacement cohort had 45/45 for direct coding and 12/45 for a Nemotron Cascade 2 worker (11 worker substitutions). Sixteen sessions stopped by a subscription limit before any model call were replaced; counting them as failures gives 37/45 and 8/45. A Qwen3-Coder worker passed 0/45. In both studies with a direct arm, direct coding had lower API list-price-equivalent frontier cost per verified success. A post-hoc diagnostic found that 19 of 36 artifacts rejected only for renamed parameters passed the hidden checks unchanged; they are not counted as successes. The contribution is an evidence-qualified account of serving, agent protocol, session continuity, artifact validation, and verification in practice. The studies are not pooled and do not rank models or runtimes.