LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering

AI summary

Overview

Research area: Evaluation of AI models as runtime controllers for coding-agent loops (Loop Engineering), sitting at the intersection of coding-agent benchmarks, agent orchestration, and efficient benchmark evaluation.

Technical level: Advanced. The paper assumes familiarity with coding agents, ReAct-style execution loops, harness design, executable task evaluators, and rank-correlation validation.

Scope: The paper introduces and validates LoopArena, a benchmark that isolates and measures one model's ability to direct a separate, frozen coding agent through long-horizon repository-level tasks at three levels of execution cost (decision, task slice, full task). Authors are from DreamX Team, Alibaba Group; Beijing University of Posts and Telecommunications; UNSW Sydney; and Data61, CSIRO. It was published 2026-08-28 as arXiv:2608.28281v1 [cs.AI] under CC BY 4.0, with code and data at https://github.com/AMAP-ML/LoopArena.

What This Paper Is About

Coding agents are increasingly managed not by hand-written prompts but by "loops" that monitor progress, assign work, run checks, and decide what to do next. The problem is that a single end-to-end result cannot tell you whether success or failure came from the loop's guidance or from the coding agent's own ability. LoopArena addresses this by fixing the coding agent (the Worker) and evaluating only the model that guides it (the Controller), measuring whether that model can redirect, verify, recover, or correctly stop a long-running task.

Key Contributions

1. A benchmark for runtime loop control. LoopArena evaluates how well a model decides what a separate coding agent should do next and guides it through long-horizon repository-level tasks, making runtime loop control the direct object of evaluation rather than the whole agent system.

2. A controlled protocol for comparing Controller models. Across Controller-model comparisons, the benchmark holds the Worker and execution setup fixed, including tools, task environment, execution budget, evaluator, and control interface. The object of comparison is therefore the Controller model, not the complete coding-agent system.

3. Evaluation at decision, task-slice, and full-task levels. Type I shifts candidate execution to benchmark construction, producing low-cost control questions grounded in downstream outcomes; Type II and Type III evaluate runtime loop control on paired task slices and full tasks, letting the authors quantify cost reduction and compare Controller rankings across execution scopes.

4. Released benchmark data and evaluation code at https://github.com/AMAP-ML/LoopArena.

Main Findings

• Full-task control remains difficult. Across evaluated Controllers, Type III Strict Success Rate ranges from 16.05% to 24.69%, and the strongest observed value is 24.69% (GPT-5.5), leaving substantial room for improvement in long-horizon loop control.

• Persistent goal restatement helps only on a bounded slice. Fixed control (a deterministic policy inspired by Codex's /goal that restates the original objective at each nonterminal handoff) raises Type II success from 39.51% to 46.91%, but matches no control at 18.52% on Type III. The contrast suggests full-task control requires guidance that adapts as the run moves between implementation, verification, recovery, and stopping.

• Type II retains the full-task comparison at substantially lower cost. The paired Type II reduction in estimated inference cost averages 64.4% relative to Type III, and under the main Core criterion the observed Controller ordering is similar to Type III, with Spearman's rho = 0.9747.

• The Type II/Type III ordering agreement is near-complete. None of the nine Controller pairs strictly ordered in both settings reverses order, while one of the ten pairs contains a tie.

• Type I distinguishes Controller decision quality. Contract Accuracy ranges from 72.22% to 87.78% (Qwen3.7-Plus 72.22, GLM 5.2 74.44, Claude Opus 4.8 76.67, DeepSeek-V4-Flash-0731 77.78, GPT-5.5 87.78), with a 0% Invalid Rate.

• Simple shortcuts do not explain Type I performance. The strongest deterministic shortcut reaches 31.11%, and uniform choice, action alone, candidate length, and lexical overlap all score lower; none approaches the Controller scores.

• Controller results on the executable settings. Type II SSR / estimated cost per run: No control 39.51% / $1.04; Fixed control 46.91% / $1.08; GLM 5.2 37.04% / $1.63; DeepSeek-V4-Flash-0731 45.68% / $2.10; Qwen3.7-Plus 48.15% / $4.30; GPT-5.5 51.85% / $5.00; Claude Opus 4.8 48.15% / $5.87. Type III SSR / estimated cost per run: No control 18.52% / $2.01; Fixed control 18.52% / $5.58; GLM 5.2 16.05% / $4.86; DeepSeek-V4-Flash-0731 19.75% / $10.24; Claude Opus 4.8 20.99% / $16.82; Qwen3.7-Plus 23.46% / $6.89; GPT-5.5 24.69% / $18.84.

• Benchmark composition. Type I contains 90 questions (40 SCBench / 50 BeyondSWE). Type II and Type III each use the same 27 source tasks (11 SCBench / 16 BeyondSWE), with Type II requiring no-per-run inputs of 51.38–80.12 Worker turns and 3.04–4.19 control cycles, and Type III requiring 139.81–288.90 Worker turns and 8.60–13.46 control cycles.

Methodology in Plain English

The benchmark splits a coding run into two roles. The Worker is a fixed coding agent (Qwen3.7-Plus in these experiments) that is the only component allowed to edit the repository, run commands, and use coding tools inside its own ReAct loop. The model under evaluation is the Controller, which has no coding tools at all and can influence the task only by the instructions it sends.

After each Worker segment, the harness pauses the persistent Worker conversation and spins up a temporary Reporter agent (built from a copy of the Worker conversation, using the same model configuration) that may inspect the workspace read-only. The Reporter produces a four-part account: task context and constraints, work history and current state, verification and evidence, and open issues and uncertainty, with citations to the relevant Worker turns. The harness then deterministically formats this into an Evidence Packet, and the Controller returns a structured Loop Contract with an action in {advance, verify, stop}. Advancing or verifying contracts specify a bounded next assignment; a stop contract sends the workspace to the task evaluator.

Tasks come from two source benchmarks, SlopCodeBench (SCBench) for long-horizon iterative coding and BeyondSWE for software-engineering tasks beyond single-repository bug fixing. Success follows the source evaluators: SCBench requires every Core check in the frozen scored set to pass, and BeyondSWE requires a reward of one from the official Harbor evaluator. Each policy is run K = 3 times per task.

The three settings differ mainly in where execution cost is paid:

• Type I asks the Controller to pick one of four candidate Loop Contracts at a frozen control point, with no Worker execution at evaluation time. The correct option was established during construction by replaying all four candidates from the same restored state under two predeclared matched replay schedules (seeds S+1,000,000 and S+2,000,000), retaining an item only when both schedules identify the same unique winner; ties are broken by downstream Controller cycles, then downstream Worker turns.
• Type II evaluates one selected slice of each full task, starting from a prepared intermediate workspace.
• Type III evaluates the paired full task from its original state.

Two shared reference policies appear alongside Controllers but are excluded from the ranking: no control (the Worker runs autonomously), and fixed control (a deterministic policy that restates the same goal at each nonterminal handoff without reading the Packet or calling a model).

Why This Matters

Impact on research. Existing coding benchmarks score the complete coding-agent system, and harness-level benchmarks compare whole model-plus-harness configurations. LoopArena changes the measured object to the runtime control decision itself, giving researchers a way to attribute success or failure to guidance versus execution. The validation approach (paired task-slice and full-task runs, checked with Spearman's rho) also follows prior efficient-evaluation practice such as ConvCodeWorld and benchmark-compression work, extending that methodological principle to agentic loop control.

Real-world applications:

• Selecting which model should act as the orchestrator in a long-running coding-agent deployment, independently of which model writes the code.
• Auditing whether an autonomous agent pipeline should verify, recover, or stop, rather than trusting a plausible partial result as completion.
• Cost planning for agent pipelines, using the measured gap between task-slice and full-task evaluation cost.
• Regression testing of loop policies before deployment, since Type I control questions require no Worker execution.

Industry relevance. The paper quantifies worst-case underperformance at the top of the current field: even the best observed full-task Strict Success Rate is 24.69%, and fixed goal restatement does not beat unguided execution on full tasks (18.52% for both). For teams shipping always-on coding agents, that is a direct argument that adaptive loop control — not simply a persistent goal — is the missing capability, and it gives them a reproducible harness, fixed Worker, and shared budget rules for comparing candidates.

Future Directions

• Extend beyond repository-level coding. The paper states the current focus is repository-level coding tasks, so additional software domains would require domain-specific task construction and executable evaluators.
• Broaden the Worker side. Testing additional Worker families and multi-Worker settings would show whether Controller rankings and the 64.4% Type II cost reduction hold when the coding agent changes.
• Test other loop organizations. LoopArena currently assumes a Controller–Worker organization with structured handoffs; other loop designs and control interfaces are untested.
• Raise full-task performance. With Type III Strict Success Rate capped at 24.69% and fixed control matching no control at 18.52%, the open question is what form of state-adaptive guidance actually improves long-horizon control.

Target Audience

Researchers and engineers working on coding agents, agent harnesses, and model orchestration who need to evaluate a model's management ability separately from its coding ability. It is also relevant to benchmark designers interested in execution-validated construction, replay-based ground truth, and cost-reduced evaluation settings, and to industry teams deciding which model should lead an autonomous software pipeline.

Authors’ abstract

Loop Engineering is emerging as a practice for organizing development work around coding agents. Instead of writing each prompt by hand, practitioners design loops that monitor progress, assign work, run checks, and decide what the agent should do next. Even with a capable coding agent, a loop may trust a stale progress note, skip needed verification, spend its budget in the wrong direction, or stop before the task is safe to submit. Yet the final outcome of one end-to-end run cannot tell whether success or failure reflects the loop's guidance or the coding agent's ability to carry out the task. We introduce LoopArena, a benchmark for evaluating how well one model can guide a separate coding agent through a long-running task. The model under evaluation is the \textbf{Controller}: after each coding round, it receives a structured summary of the run and instructs a separate, fixed coding agent, the \textbf{Worker}, on what to do or verify next, or decides whether to stop. LoopArena evaluates this ability in three complementary settings that differ in execution scope and cost. Type I scores next-step Loop Contract selection through execution-validated questions without running the Worker at evaluation time. Type II executes repeated control over a selected slice of a full task, while Type III evaluates the paired full task from its original state. On full tasks, the best observed Strict Success Rate is \textbf{24.69\%}, leaving substantial room for improvement in long-horizon loop control. Across Controllers, the paired reduction in estimated inference cost averages \textbf{64.4\%}, and Type II produces a similar ordering under the main Core criterion (Spearman's \(ρ=\textbf{0.9747}\)). We release the benchmark data and evaluation code at https://github.com/AMAP-ML/LoopArena .