ControlScope: Workflow Revision and Reliability in LLM Agents

AI summary

Overview

Research area: Autonomous language-model agents that act by generating and executing code workflows, specifically the design question of how much of an in-progress program an agent should be allowed to revise.

Technical level: Intermediate. The central comparison is conceptually simple (three nested permission levels), but the paper assumes familiarity with agent architectures, tool-calling APIs, and benchmark families such as ALFWorld and AppWorld.

Scope: The paper introduces ControlScope, a controlled comparison of three nested revision permissions — Keep, Arg, and Full — evaluated across MCPMark-derived filesystem tasks, ALFWorld, and an AppWorld V1 official-test panel, with replay diagnostics that separate which repairs are available from which repairs an agent actually selects.

What This Paper Is About

LLM agents often act by writing a program that then executes tool calls on their behalf. The open question is what an agent should be permitted to change once that program is already running: nothing, only the data arguments of the very next tool call, or the entire remaining workflow. ControlScope holds the public execution state fixed — same goal, same observed tool results, same current program, same pending call — and varies only the set of permitted revisions, so that any difference in task success can be attributed to revision scope rather than to different starting conditions.

Key Contributions

1. A nested-scope formalism for agent revision. The paper defines three operation sets on a shared public state: Keep (continue), Arg (also patch the next primitive call's data arguments), and Full (also replace the remaining continuation). Because the sets are nested, Full can always fall back to a smaller-scope action, so any loss under Full reflects the deployed selection and execution policy rather than missing permissions.

2. A separation of "granted scope" from "realized scope." The paper explicitly distinguishes the operations an agent may take from the operation it actually chooses, and logs both. This is what allows it to show, for example, that a cheaper successful argument edit existed inside Full's own action set but went unused.

3. Replay-based diagnostics for repair availability and execution stability. Frozen prefix replays execute recorded revision candidates from the same state under two endpoints — a block/call-budget endpoint and a retained-edit-plus-ordinary-planning endpoint — establishing that useful repairs existed inside failing runs and that later reviews can interrupt them.

4. Evaluation across three task families with stated execution contracts. The comparison spans a 20-task filesystem panel selected from a 24-task MCPMark adaptation, ALFWorld cohorts of 87 and 134 tasks, and an AppWorld V1 official-test panel of 585 task instances drawn from 195 scenario templates, alongside an AppWorld V2 local-boundary diagnostic of 86 eligible boundaries from 28 templates.

Main Findings

• Workflow revision gives configuration-dependent gains on filesystem tasks. Across two source programs per task and three reasoning-reviewer draws on 20 tasks, Full completes 15–16 tasks versus 13 for the shared Keep baseline. The source program and the reviewer draw shape which tasks benefit: P1 gains contact inference and English-student selection; both P2 draws gain contact inference, file splitting, and grade scoring, while draw 1 also gains English-student selection and loses file arrangement, and draw 2 loses legal solution tracing.

• Fast-reviewer results are mixed and can favor the narrower scope. A matched fast panel completes 13, 11, and 13 tasks under Keep, Arg, and Full. The fast P2 required-tool configuration scores 13/13/10 for Keep/Arg/Full, while automatic-tool fast scores 13/11/13. The Arg/Full reviewers record 29/16 rejected proposals, which remain in the scored outcomes. The seven-category bootstrap 95% intervals for P2 Full − Keep are [-30.0, -4.2] and [-14.3, 16.7] percentage points for required-tool and automatic-tool fast, and [0, 38.9] and [-10.0, 27.8] for the two reasoning draws.

• ALFWorld outcomes differ sharply by cohort. Under a common fast protocol, Keep/Arg/Full score 85/86/87 on 87 tasks across 52 scenes, but 134/134/127 on 134 tasks across four scenes — so Full gains two tasks in one fixed cohort and loses seven in the other. Reasoning reviewers on the 87-task cohort also end at 85/86/87, at substantial review cost.

• Common recovery changes how much of the gain is visible. On the 87-task reasoning panel, initial-plan successes are 79/81/85 for Keep/Arg/Full; after ordinary recovery these become 85/86/87. Within 20 actions, completion corresponds to 68, 69, and 76 tasks. Full uses 1,124 actions versus 1,336 for Keep and invokes ordinary repair twice versus 17 times. Logged input tokens total 0.221, 15.926, and 11.881 million, with exact output totals of 42,481, 1,924,261, and 1,261,096 tokens. Full gains two terminal successes over Keep at 29.7 times its output tokens, and uses 34.5% less total output than Arg — a contrast concentrated 98.3% in two heating tasks where Arg records 28 invalid reviews and Full records none; Full logs more output on 60 of 87 tasks.

• AppWorld V1 shows small aggregate differences with a quota favoring Full. Across 1,755 completed runs, Full exceeds Keep by one task on each split, with paired task-goal-completion gains of +0.60 and +0.24 percentage points and scenario-cluster bootstrap intervals of [-2.38, 3.57] and [-2.40, 2.88]. Paired Full win/loss counts of 4/3 and 18/17 show changes in both directions, and each V1 Full replacement renews a 250-request block allowance, which favors Full.

• A single early intervention frequently changes nothing. All 216 paired local draws produce identical Arg and Full terminal outcomes. Of 432 decisions, 391 continue unchanged, including 13 malformed responses and 12 service-error fallbacks; the remainder contain 18 argument patches and 23 workflow replacements. All 41 actual revision branches pass saved-context, prefix, operation, and execution-budget checks. Twelve context-limit failures belong to one second-source paper-organization state, and excluding its draws preserves the observed equality.

• Workflow rewrites fix batching and parsing problems in student-record tasks. In one source, a retained batch-read replacement reaches the goal after 94 total calls, whereas the original loop exhausts the 100-call budget; both branches share the first 88 calls. A second source reproduces the mechanism at an earlier boundary, completing at 28 total calls from an identical 26-call prefix, with the resulting 19 selected students independently checked. A parser repair handles unquoted commas inside addresses and yields correct results for all 150 students.

• Fresh subsets confirm the repair but preserve source-program dependence. Across 25-, 75-, and 150-student subsets with two seeds, both reviewer modes score 5/6, 5/6, 6/6 for Keep/Arg/Full. Three new subsets with two independently generated source programs each reproduce 5/6, 5/6, 6/6. On seed 101's first source, Keep and Arg exhaust 100 calls while Full succeeds in 18 fast-mode calls and 15 reasoning-mode calls, following 13 identical API calls and receipts with zero rejected proposals or service errors. Relative to Arg, Full uses 23.4% fewer logged output tokens in fast mode and 81.1% fewer in reasoning mode; reasoning Arg/Full record 24/1 invalid-review fallbacks.

• Full can overlook a cheaper edit that sits inside its own action set. On seed 102's first source, reasoning Arg expands its path list from 2 to 75 and succeeds in 11 calls; Keep and the sampled Full also succeed but each use 25 calls. The public state at the sixth-call boundary matches between Arg and Full, which chooses to continue unchanged. Shared-patch replay succeeds under both labels in 11 calls with matching traces and files, so the sampled Full reviewer leaves the shortcut unused and adds 14 primitive calls while preserving success.

• Frozen replays show viable continuations interrupted by later revisions. In two failing runs of file-time classification, 49 of 71 actual replacements satisfy the goal in the first source and 46 of 62 in the second when retained through the block endpoint or original call budget; 48 of the first source's successful candidates also finish their replacement block. In a file-organization confirmation, Keep and Arg succeed on all three instances while Full succeeds on two; the failed run makes 35 replacements and reaches the 100-call budget, and retaining its first replacement with ordinary planning succeeds in 77 calls, whereas that first replacement alone ends after seven total calls with the goal incomplete.

• Tested retention rules do not produce panel-level gains. Across every replacement-producing run in the P2 automatic-tool fast panel and first P2 reasoning draw, sustained revision succeeds on 5/10 fast and 9/13 reasoning runs, while retaining only the first revision yields 4/10 and 7/13, with zero paired gains and three losses. On a source-trajectory-defined midpoint schedule, Arg succeeds on 12/20 under either schedule, Full on 13/20 with one review and 14/20 with continuous review; the continuous Full policy uses 294 reviews against 20 and logs 64.0k against 45.8k output tokens.

• Protection limits can trade a little success for substantial savings. Across 20 fresh source programs, Keep, Arg, continuous Full, and protected Full succeed on 13/12/14/13 tasks. Five-call protection activates on eight tasks with zero paired wins, one loss, and 19 equal outcomes against continuous Full, reducing reviews from 517 to 423 and logged output by 19.4%. In the sole flip, continuous Full selects a 300-path batch read and succeeds, while protected Full omits it and exhausts 100 calls.

• Runtime feedback changes selection without changing average success. At a matched boundary with Full authority once, a code-and-history view and a feedback view each succeed on 12/17 eligible pairs covering 11 tasks, with 11 joint successes. Feedback selects nine replacements versus three, with one paired gain and one loss, and accumulates 644 versus 466 tool calls.

Methodology in Plain English

The researchers start by fixing a task and one already-generated "source program," then pause execution at a specific decision boundary. From that same pause point — with the same goal, the same tool results so far, the same pending call, and the same exposed variables — they run branches that differ only in what the reviewer is allowed to do.

Three permissions are nested. Keep can only let the program continue. Arg can additionally rewrite the data arguments of the very next primitive tool call, which preserves the pending API and the surrounding program structure but can still change later behavior through existing branches. Full can additionally throw out the unfinished continuation and substitute new code, changing call selection, sequencing, loops, and dependencies. The same argument-patching implementation serves both Arg and Full, and Full can always choose a smaller-scope action instead.

Two experimental treatments are kept separate. A local intervention grants one revision opportunity at a fixed boundary and then reverts to Keep. A sustained revision policy offers the assigned scope repeatedly throughout execution. This distinction matters because a later review can interrupt an earlier repair before it completes. In every condition, ordinary planning still resumes when a code block or assistant reply finishes, so the comparison isolates control over an unfinished program inside an otherwise normally functioning agent loop.

For diagnostics, the team uses prefix replay: they rebuild all environment interactions before a selected boundary in an isolated workspace, check that API names, arguments, observations, and terminal files match the recorded source, and then execute selected candidates. Two endpoints are reported — goal completion by the replacement block or original budget, and full agent recovery after the retained edit plus ordinary planning. The reported statistics include positive, negative, and equal outcomes so that small denominators stay visible.

Model settings: source generation and ordinary planning use served deepseek-flash in fast mode, with reviewers in either fast or high-effort reasoning mode. Matched configurations share native tool schemas, an explicit tool-return instruction, automatic tool choice, and a 16,384-token per-request cap; fast sampling uses temperature 0.7. MCP conditions receive 100 primitive calls, 100 ordinary planning turns, and 1,800 execution seconds. ALFWorld uses 100 actions and up to five ordinary repairs.

Why This Matters

Impact on research. The paper reframes revision scope as an architectural permission set rather than a prompting trick, and it shows that availability and selection are different phenomena that need to be measured separately. Its replay methodology — branching from a shared public state and checking both a block endpoint and a full-recovery endpoint — gives the agent-evaluation community a template for attributing failures to missing permissions versus poor choices. The result that Arg and Full match on all 216 early local pairs, while AppWorld V1 shows only small net differences with a quota favoring Full, cautions against reading single benchmark deltas as evidence about scope.

Real-world applications:

• Tool-using assistants over files and databases, where the choice between patching one argument and rewriting a loop determines whether a batch read completes inside a call budget — the paper's student-record task is exactly this pattern, with one repair succeeding at 94 calls where the original loop exhausts 100.

• Household and embodied agents, where ALFWorld's opposite outcomes across two fixed cohorts (85/86/87 versus 134/134/127) indicate that scope effects depend on the environment rather than being universally beneficial.

• Enterprise application automation, where AppWorld's 585-instance official-test panel and the V2 local diagnostic of 86 boundaries show how permission sets interact with request quotas.

• Cost- and latency-sensitive deployments, where the five-call protection reduces reviews from 517 to 423 and logged output by 19.4% at the cost of one success in 20 tasks, and where a shared argument edit can save 14 primitive calls that a Full reviewer simply fails to take.

Industry relevance. Agent frameworks must decide what a reviewer is allowed to touch mid-execution. This paper quantifies the tradeoff: wider scope enables batch-read and parser repairs that narrower scope cannot reach, but it also introduces interruption risk, invalid-review fallbacks, and higher token totals — Full's two-success edge over Keep on ALFWorld came at 29.7 times Keep's output tokens.

Future Directions

• Review timing and valid-operation return rates need joint evaluation. The paper states that these belong in the same evaluation, and its midpoint schedule shows that continuous Full review uses 294 reviews against 20 for one-time review while adding one terminal success.

• Selection quality is a separate problem from permission breadth. The argument-only shortcut on seed 102 shows a successful 11-call edit available inside Full's action set that the sampled Full policy declined to take, adding 14 calls. Improving when an agent chooses a cheaper valid action is left open.

• Retention rules that preserve useful continuations without freezing progress. Both tested retention rules produce zero paired gains, and retaining only the first edit loses 3 of 23 replacement-producing runs, so a rule that protects a working repair while still permitting genuine later repairs remains unresolved.

• Reconciling the ALFWorld cohort disagreement. Full gains two tasks across 52 scenes and loses seven across four scenes, which the paper reports but does not resolve; identifying what distinguishes the four-scene cohort is a natural next question.

Target Audience

Researchers and engineers working on LLM agent architectures, tool-use frameworks, and code-generation agents will get the most from this paper, particularly those designing review, reflection, or self-repair loops and needing to decide how much control to grant mid-execution. It also suits benchmark designers interested in controlled same-state branching methodology, and practitioners in agent-platform engineering who must weigh task success against token cost and review overhead. Readers without prior exposure to agent benchmarks and tool-calling APIs will find the setup readable but the result tables dense.

Authors’ abstract

How much of a running workflow should a language model agent revise? ControlScope compares continuing generated code, editing the next tool call's data arguments, and replacing the unfinished workflow from the same public execution state. The nested permissions separate available repairs from the actions an agent selects. We evaluate one-time and repeated reviews across filesystem tasks, ALFWorld, and AppWorld. Across two source programs per task and three reasoning-reviewer draws on 20 filesystem tasks, FULL completes 15-16 tasks versus 13 for KEEP; across four fast draws it completes 10-13 versus 13. Fresh student-record confirmation reproduces a batch-read repair. ALFWorld fast panels yield KEEP/ARG/FULL scores of 85/86/87 on 87 tasks across 52 scenes and 134/134/127 on 134 tasks across four scenes; reasoning on the 87-task cohort also yields 85/86/87 with substantial review cost. An AppWorld V1 official-test panel of 585 task instances from 195 scenario templates shows small net differences. Frozen replays expose viable agent-written replacements interrupted by later revision in two failed file-organization runs. An offline source-trajectory midpoint comparison shows later reviews completing an insufficient repair. Five-call protection saves 19.4% of logged model output and loses one success across 20 fresh source runs. An argument-only shortcut shows that the broader sampled policy can overlook a cheaper successful edit available in both operation sets. These outcomes tie repair access to actual choices and subsequent execution.