Agent Loops, Harnesses, and Runtimes

Every attempt got a hard ceiling of $4. When a run crossed it, the harness submitted whatever edits existed at that moment. SWE-agent, published in 2024, states the reason plainly: "Due to budget constraints, we set the per-instance budget to $4; if a run exceeded this budget, existing edits were submitted automatically."

The runs that worked never came close to that ceiling. Successful GPT-4 runs finished at a median of $1.21 and 12 steps. Unsuccessful ones averaged $2.52 and 21 steps. 93.0% of resolved instances were submitted before exhausting the budget, against 69.0% of instances overall. The authors traced one failure mode to agents inspecting every search hit exhaustively until the cost budget or the context window was gone.

Read the asymmetry rather than the averages. Agents succeed quickly and fail slowly. Twice the steps and twice the spend do not signal twice the effort. They signal a loop that has stopped making progress and has nothing inside it that can notice. The stop came from the harness. It did not come from a model deciding it was finished.

An agent loop repeatedly assembles context, asks the model for a decision, validates the proposal, executes an allowed action, records the result, and decides whether to continue.

That pattern is not folklore, and it is not new. ReAct measured it in 2022 and reported the gain in absolute points: "On two interactive decision making benchmarks (ALFWorld and WebShop), ReAct outperforms imitation and reinforcement learning methods by an absolute success rate of 34% and 10% respectively, while being prompted with only one or two in-context examples." Thirty-four and ten absolute percentage points, from one or two examples. The gain came from the shape of the loop around the model, not from more supervision inside it.

The harness owns this sequence and should remain inspectable outside the model. The model supplies judgment within the loop. The runtime supplies durability, policy enforcement, retries, budgets, tracing, and lifecycle control.

The first thing a healthy harness demonstrates is that a run can be resumed without reconstructing hidden model context. A second and independent check is that every loop consumes a visible budget and records a state transition. You read that off the log rather than infer it from the output.

The last two claims only prove themselves under repetition, and repetition has a published price. τ-bench, published in 2024, introduced pass^k: the chance that all k i.i.d. trials of a task succeed, averaged across tasks. Its abstract: "Our experiments show that even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail)."

Hold on to the per-domain figures behind that average. gpt-4o with function calling scores roughly 61% pass^1 on τ-retail and roughly 35% on τ-airline. Its retail score collapses to about 25% at pass^8. Running the same task eight times costs it about two thirds of its retail score. One green trace is therefore not evidence that a stopping rule or a policy check works. It is one draw from a distribution that pass^8 is designed to expose.

Assemble context: select instructions, state, observations, and available actions. Propose: the model returns a structured next action or final response. Validate: code checks schema, authority, policy, budget, and task state. Execute: a tool or service performs the permitted operation. Record and decide: the harness stores results and either continues, escalates, or stops.

The loop is worth tracing one stage at a time. Each stage fails differently, and execution and the record-and-decide step need separate owners and separate tests.

The last stage is the one people treat as free, and two production runtimes price it. An AWS Step Functions Standard workflow execution may run for up to 1 year. The quota on its history is explicit: "25,000 events in a single state machine execution history. If the execution history reaches this quota, the execution will fail." Temporal draws the same line in its own units. It terminates a Workflow Execution whose Event History exceeds 51,200 Events or 50 MB, and warns from 10,240 Events or 10 MB.

Two independent vendors, two different numbers, one shared conclusion: the resumable record is a bounded resource. A harness that never trims, checkpoints, or summarises its history does not fail gracefully at the ceiling. In Step Functions the execution fails outright. In Temporal it is terminated. Bounding the record is what makes resumption and a deterministic stop implementable rather than aspirational.

Design the smallest harness that still carries one real workflow from start to stop. Define the state object: goal, observations, pending actions, evidence, budgets, and status. Specify transition guards that validate every proposed action before execution. Record immutable events — model decisions, tool calls, results, approvals, and errors. Measure progress by choosing task-specific state changes that justify another loop. Implement terminal states: success, partial success, escalation, refusal, cancellation, and failure.

Two of those five steps are not engineering taste in the EU. The AI Act makes them duties for high-risk systems. Article 12(1): "High-risk AI systems shall technically allow for the automatic recording of events (logs) over the lifetime of the system." Over the lifetime of the system — not for the duration of a debugging session. Article 14(4)(e) requires that the people assigned human oversight be able to interrupt the system through a stop button or an equivalent procedure that brings it to a halt in a safe state.

Read those two together and the exercise acquires an acceptance test. "Record immutable events" means a log that survives the run. "Implement terminal states" means at least one terminal state reachable by a human at any moment, ending in a safe state rather than mid-transaction.

Then push the harness until runaway cost stops being an abstraction and shows up in your own run log. Set a budget near your successful median and watch what gets truncated. Finally, stop the process mid-task and restart it from the recorded state. What that restart produces is your evidence that resumption actually works.

The harness should remain useful when the underlying model changes. If the loop depends on hidden habits of one model, the architecture is not yet stable. The SWE-bench team built the test for that into its own leaderboard, where every language model runs through the same mini-SWE-agent bash environment precisely so that the loop stops being a variable.

A review of the runtime that never looks at looping cost has skipped the expensive part. That is where a small per-step uncertainty turns into a bill nobody budgeted for: 21 steps and $2.52 for the runs that failed, against 12 steps and $1.21 for the ones that worked. And where the record itself is the thing you are relying on, check its ceiling before a long-running agent discovers it for you — 25,000 events in Step Functions, 51,200 Events or 50 MB in Temporal.

The review should end with a working restart. Stop a run, bring it back from what the harness recorded, and confirm it carries on without the model's hidden state. If it cannot, the record was a transcript, not a state.