The Pulse
Coding-Agent Harnesses Can Triple Costs Without Clear Score Gains
A study of Claude Code, mini-SWE-agent and OpenCode found that harness choice changed coding-agent costs by as much as threefold, while most score differences fell within run-to-run noise. The researchers tested five models on SWE-bench Ver
AI.info Team · Written with AI · fact-checked ·
Changing a coding agent’s harness can multiply its cost per task by as much as three without producing a clear improvement in pass rate, according to a study posted October 3. The researchers compared Claude Code, mini-SWE-agent and OpenCode while holding the underlying model and coding tasks constant. They found that the harness often changed the bill more consistently than it changed which tasks the agent solved.
Three harnesses, one benchmark
Yangze Liu and Zhongyi Han tested five models on SWE-bench Verified, a benchmark built around real GitHub issues, and reran configurations to measure how much scores shifted without a harness change. Their offline setup covered 492 tasks: a 45-task subset from the benchmark’s hardest difficulty bands and a 447-task pool used to measure smaller differences. Claude Opus 5 served as an additional comparison, but only inside Claude Code.
The three harnesses package different prompts, tools and context-management features around a model. Claude Code and OpenCode provide typed file tools and automatic context compaction; mini-SWE-agent gives the model a single Bash tool and no context management. The comparison therefore tests complete, production-style systems, rather than one isolated design choice.
“With the same model, the same tasks, and one price list, cost per task differs by up to 3× across harnesses.”
Yangze Liu and Zhongyi Han, “What Does a Harness Buy? Tokens, Mostly”
Most score changes looked like run noise
On the 45 hardest tasks, swapping harnesses changed the result on a median 13% of tasks—the same share that changed when the researchers reran the same harness. A harness’s wins also failed to repeat reliably on the same tasks in another run. On the larger pool, Claude Code and mini-SWE-agent scored within five percentage points of each other on the two models tested there.
The study did find a measurable loss: OpenCode trailed by as much as nine points on the 447-task pool. On Qwen3.8-27B, 33 OpenCode trials ended when the model hit an output limit, compared with one Claude Code trial on the same model. About half of that score gap disappeared when the researchers excluded OpenCode runs cut short by the limit, which they linked to the harness’s failure to prompt the model to continue.
The bill starts with the first prompt
The researchers traced much of the cost difference to material a harness resends with each model call: its system prompt, tool descriptions and message structure. They measured that fixed preamble at 16,581 tokens for Claude Code, 7,025 for OpenCode and 829 for mini-SWE-agent. Tool outputs contributed much less to the gap; the number of steps a model took also mattered, and could change which harness cost less for a given model.
Input-token cache prices affected the size of the bill but did not change the cost ordering in the researchers’ comparisons. Their accounting points to three practical cost drivers: the tokens sent on the first call, how quickly the input grows over later calls, and how many calls the agent makes. A lighter harness does not always mean a cheaper run if its model takes more steps.
Small score claims need larger tests
The task counts also limit what the benchmark can establish. At the rate tasks disagreed between runs, 45 tasks could detect a 13-point difference only about half the time and could not detect even a zero gap with 80% statistical power. The 447-task pool could resolve a difference of roughly five points, still larger than many advertised harness gains.
The authors caution that their results come mainly from one benchmark; their Terminal-Bench 2.1 check covered only one model. They also could not compare the three harnesses using Claude Opus 5, because that model ran only in Claude Code. Whether smaller score differences hold up on other coding tasks—or with closed models tested across all three harnesses—remains unresolved.