An Empirical Study of Harness Design for Coding Agents

AI summary

Overview

Research area: Autonomous software-engineering agents, specifically the design of the "coding harness" — the software layer that wraps a large language model and lets it plan, act, and manage context over long horizons.

Technical level: Intermediate to Advanced. Readers should be comfortable with LLM agents, tool calling, ReAct loops, context windows, and benchmark evaluation, but the paper is written to make each component's effect interpretable.

Scope: A 176-setting controlled ablation study that isolates how planning, action space, and context management each affect coding-agent success and cost across four models, two benchmarks, and four context-window budgets.

What This Paper Is About

Most published coding-agent systems (OpenHands, SWE-Agent, and similar) are evaluated as whole packages, so when one harness beats another on a benchmark it is impossible to tell whether the gain came from better planning prompts, a better tool set, smarter context handling, or the underlying model. This paper builds a deliberately lightweight harness in which the execution loop is frozen and only one component is changed at a time, then measures the conditional effect of each component across model capabilities and resource budgets. The goal is to answer whether harness components are universally useful or whether their value depends on the model, the task type, and the available context window.

Key Contributions

1. A modular, component-level harness. A from-scratch ReAct-based coding agent whose planning scaffold, action interface, and context-management policy can be toggled independently, with all other machinery (workspace guards, read-before-write checks, post-edit diagnostics, stuck detection) held fixed as a common substrate.

2. A five-tier context-management ladder. Three composable mechanisms — elision of stale tool outputs (M1), external storage with a recall_event tool (M2), and LLM summarization (M3) — are combined into Tiers T0 through T4, allowing matched comparisons of lossy versus lossless compression.

3. A 176-setting empirical study. Four models (Nemotron-3 at 30B, 120B, and 550B, plus Mistral-Medium-3.5-128B), two benchmarks (SWE-Bench Verified and Terminal-Bench 2.1), four context budgets (32k, 64k, 96k, 128k), five context tiers, and targeted planning and action-space ablations, all analyzed with paired significance testing.

4. Trajectory-level behavioral explanations. Every turn is annotated by an LLM judge into workflow phases (Localize, Reproduce, Fix, Verify, Understand, Write Code), turning aggregate success-rate numbers into mechanistic accounts of why each component helps or fails to help.

Main Findings

• Context management pays off most when the window is tight, and mostly by preventing overflow. The model-averaged success gap between managed tiers (T1–T4) and no management (T0) falls from 35.7 points at 32k to 15.9, 5.5, and 2.7 points at 64k, 96k, and 128k on SWE-Bench, and from 9.5 to 7.5, 4.8, and 2.8 on Terminal-Bench. This tracks the fall in T0 window-overflow failures (78.7% to 8.7% on SWE-Bench; 61.0% to 12.1% on Terminal-Bench), while every managed tier overflows on zero tasks.

• Staging elision before summarization (T4) gives the best accuracy–cost trade-off. T4 matches the success rates of T1–T3 while achieving the lowest cost in seven of eight model–benchmark panels and the lowest peak-context ratio at every budget. Early cheap elision absorbs many cases before expensive LLM summarization is needed, so T4 invokes M3 less often than T3 and M1 less often than T1/T2 at small windows.

• Lossless recall is essentially unused and buys no accuracy. Across 32 matched T1-versus-T2 comparisons, T2 wins 15, loses 14, and ties 3, for a mean difference of −0.36 percentage points. Among the 64 T2/T4 settings, 56.3% never call recall_event at all, the median invocation rate is zero, and mean calls per task fall from 0.540 at 32k to 0.069, 0.011, and 0.007 at larger budgets. Even the heaviest user (Nemotron-3 30B on Terminal-Bench at 32k under T2) scores 3.37 points below T1.

• Planning flips from an accuracy scaffold to a cost saver as models get stronger. For Nemotron-3 30B, enabling planning adds 11.6 points on SWE-Bench and 4.5 on Terminal-Bench, at higher cost. For Nemotron-3 550B and Mistral-Medium-3.5-128B, planning cuts cost by roughly 30% and 32% on SWE-Bench with only 2.0 and 0.4 point success drops. Nemotron-3 120B sits in between, with task-type-dependent outcomes.

• Predefined tools scaffold bash-weak models; bash-only favors bash-capable ones. The full tool set adds 15.0 points on SWE-Bench and 10.1 on Terminal-Bench for Nemotron-3 30B, largely because 66% of its bash-only Terminal-Bench runs terminate after emitting out-of-interface tool calls (average trajectory shortens from 71 to 15 turns). For Nemotron-3 550B, bash-only instead improves success by 3.6 and 5.6 points while cutting cost by 53% and 30%. Mistral-Medium-3.5-128B reveals the workload boundary: the full tool set is worth 23.2 points on SWE-Bench, but bash-only gains 6.7 points on the more shell-centric Terminal-Bench.

• The mechanisms differ qualitatively. Context management mainly extends trajectories — median lengths grow from 20–30 turns to roughly 50–180 turns at 32k — without changing phase ordering or proportions much. Planning changes where trajectories stop: with planning off, Nemotron-3 30B's median SWE-Bench run collapses from 40 to 5 turns, with 68.6% ending without any edit and 58.4% stalling in localization. For stronger models, planning removes redundant post-edit verification turns. The action space changes the granularity at which code is written: bash-only runs issue 24–32% fewer calls for Nemotron-3 550B, consistent with denser composite shell commands.

Methodology in Plain English

The authors build a minimal coding agent around a fixed ReAct loop: each turn the model reasons, emits an action, and receives an observation. Everything else is a knob. Planning is either on — a persistent task plan injected into every turn and updatable via update_plan — or off entirely, with instructions, reminders, and the tool all removed. The action space is either a typed tool set (read_file, write_file, edit_file, list_files, glob_files, grep_text, bash, web_fetch) or bare bash alone. Context management is one of five tiers built from the three mechanisms described above, operating under a soft threshold at 0.6 of the usable window and a hard threshold at 0.85, with the preamble and a recent window of at least two turns always kept verbatim.

Four models are served locally with SGLang in BF16 at temperature 0. Each task runs at most 300 steps with a 16,384-token output cap, tool results truncated to 24k characters, and up to eight concurrent read-only calls per turn. Stuck detection injects a reminder after five identical or five identically failing tool calls and terminates after eight consecutive identical failures. Costs are priced at OpenRouter rates. Within each comparison family (management vs. T0, planning on vs. off, full tools vs. bash-only), success rates are compared with two-sided exact McNemar tests on task-paired outcomes, with Benjamini–Hochberg control at a 0.05 false-discovery rate. An LLM judge labels each turn with its workflow phase to support the trajectory analysis.

Why This Matters

Impact on research. The paper argues that cross-harness leaderboards conflate planning, tooling, context policy, and model capability, and it demonstrates an alternative: hold the loop fixed, vary one component, and report conditional effects. This reframes harness comparison as an ablation methodology and yields a reusable diagnostic — are your gains coming from overflow prevention, trajectory extension, or changed action granularity? — that transfers to future models and scaffolds.

Real-world applications.

• Cost engineering for agent products. Staged elision before summarization cuts LLM summarization calls and peak context without hurting accuracy, directly reducing per-task inference spend in production coding agents.
• Model-aware harness configuration. Teams deploying a small model should keep planning on and use a rich predefined tool set; teams deploying a frontier model can often drop both, switch to bash-only, and cut cost substantially with no accuracy loss.
• Context-budget provisioning. The measured decay curve of context management's value tells infrastructure teams how much window they actually need to buy before overflow prevention stops mattering.
• Benchmark and evaluation design. The paired McNemar plus trajectory-labeling protocol offers a template for attributing benchmark gains to specific system components rather than to whole stacks.

Industry relevance. Harness design is currently empirical folklore inside agent vendors. This paper supplies quantitative crossover points — roughly where a model becomes bash-capable, roughly which window budget makes context management marginal — that translate directly into product configuration defaults and serving-cost models.

Future Directions

• Replicating the crossover thresholds on newer and larger models. The paper treats its four models as probes of capability, not as optimization targets; whether the planning and action-space crossovers sit at similar capability levels for future frontier models is open.
• Testing other harness components under the same methodology. Permission handling, post-edit diagnostics, and stuck detection were deliberately frozen here. Their conditional value under the same budget and capability axes is unmeasured.
• Designing better recall mechanisms — or abandoning them. Lossless recovery failed here because models rarely chose to invoke it. Whether retrieval can be made proactive, or whether retrieval should be dropped from harness designs entirely, is an unresolved design question.
• Extending the budget axis beyond the window. The study sweeps context tokens but not step budgets, wall-clock limits, or monetary caps. Whether the same conditional effects hold under a cost budget rather than a token budget is untested.

Target Audience

Researchers and engineers who build or evaluate LLM-based coding agents — agent framework developers, applied ML engineers choosing harness configurations for deployment, and benchmark designers who need a principled way to attribute performance changes to individual system components. The paper is also useful to infrastructure and serving teams who need to reason about context-window provisioning and per-task inference cost, and to academics studying long-horizon agent behavior who want a template for trajectory-level analysis rather than aggregate success metrics alone.

Authors’ abstract

Coding harnesses shape how autonomous coding agents translate model capabilities into long-horizon software-engineering performance, yet existing work typically evaluates harnesses as monolithic systems, leaving the effectiveness of individual components unclear. To enable component-level comparisons, we study this question with a lightweight coding harness whose execution loop is fixed while three components are varied: planning, action space, and context management. Across four models evaluated on SWE-Bench Verified and Terminal-Bench 2.1, we evaluate 176 matched settings spanning five context-management strategies, four context-window budgets, and targeted ablations of planning and action space. We find that: (1) Context management becomes increasingly valuable as the context-window budget tightens, with most of its benefit coming from preventing context-overflow failures. (2) Staging rule-based elision before LLM-based summarization provides the strongest overall efficiency among the context-management strategies, whereas making elided content recoverable adds machinery that models rarely use and yields no accuracy gain. (3) Planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger models, with little change in accuracy. (4) Predefined tools improve performance for models with weaker bash proficiency, whereas bash-capable models can operate effectively with a bash-only interface and achieve substantially lower cost, especially on command-line-centric tasks. Trajectory-level analysis explains these effects: context management extends execution trajectories without substantially altering agent behavior, planning changes where trajectories stop, and the action space changes the granularity at which code is written. These findings inform model- and budget-aware harness design and provide a modular framework for evaluating future harness components.