ProgAgent: Learning Tool Orchestration through Programmatic Training

AI summary

Overview

• Research area: Agentic AI / large language model tool use, specifically tool orchestration in long-horizon environments, combined with reinforcement learning for agent training.
• Technical level: Advanced. The paper assumes familiarity with agent harnesses, POMDP framing, supervised fine-tuning, RL (GSPO), and the Model Context Protocol (MCP).
• Scope: The paper introduces ProgAgent, a full training stack (task synthesis pipeline, lightweight runtime, two-stage training recipe) for teaching LLMs to call tools through executable code rather than sequential direct calls, and evaluates it on four agentic benchmarks with Qwen3 8B and 14B models.

What This Paper Is About

Most LLM agents use Direct Tool Calling (DTC), where each tool result is appended to the model's context; as raw outputs pile up, the authors argue this causes "context rot" and "orchestration breakdown." Programmatic Tool Calling (PTC) instead has the model write code that invokes tools inside a sandbox, keeping intermediate results in runtime variables and printing only selected output into context. Prior work treated PTC as an inference-time trick, so the authors ask whether PTC is better understood as a training paradigm, and they build the missing pieces — verifiable long-horizon tasks, an isolated runtime, and a training recipe — to test that question.

Key Contributions

1. A task synthesis pipeline producing verifiable long-horizon tasks with controllable difficulty. Tasks are built from 23 curated domains, each packaged with a relational database, a file workspace, hidden single-purpose tools, a natural-language instruction, and a Python verifier returning a binary reward. Difficulty is controlled by scaling only the environment state (database rows at 1x/2x/4x/8x, workspace files at 1x/2x/3x/4x) while holding instruction, task logic, and tool schemas fixed. After auditing and difficulty filtering, 215 tasks remain, each instantiated at four scales (v0–v3), giving 860 task instances.

2. A lightweight PTC runtime. Tool-calling programs execute in a persistent, sandboxed IPython kernel (Bubblewrap isolation), exposed to the model as an MCP server. A tool proxy routes calls to environment tools and returns results as native Python objects; runtime variables persist across turns, only printed output returns to the agent, and the sandbox blocks access to the verifier and oracle solution.

3. A two-stage training recipe and a controlled comparison of training paradigms. Cold-start SFT on teacher trajectories followed by RL via GSPO, applied identically across three harnesses — DTC, PTC, and Hybrid (both interfaces available) — with the same task pool and hyperparameters, so the harness is the only variable.

4. Evidence that PTC training beats DTC training and transfers to DTC evaluation. Cross-harness experiments separate the effect of the interface from the effect of training under that interface.

Main Findings

• Large gains over comparable-size baselines. With only 215 tasks, ProgAgent-8B and ProgAgent-14B beat the evaluated baselines of comparable size across all four benchmarks. MCPMark averages: 18.3% (8B) and 29.7% (14B). Toolathlon-Verified: 9.3% (8B) and 17.1% (14B), versus at most 2.8% and 1.9% for the 8B and 14B baselines.

• The headline LOCA-bench result. ProgAgent-14B reaches a 30.7% pass rate at a 64k-token environment state, where the evaluated comparable-size baselines score zero. ProgAgent maintains pass rates between 17.3% and 30.7% at 64k, while comparable-size baselines score zero at both 32k and 64k.

• Near parity with a frontier model on MCP-Atlas. ProgAgent-14B reaches 45.6%, comparable to Claude Sonnet-4.5 at 43.8% under the DTC harness. Frontier models still lead overall: GPT-5.2 scores 52.4 on MCPMark, 63.7 on LOCA-bench, 41.7 on Toolathlon, and 67.6 on MCP-Atlas.

• PTC only pays off after training. For untrained backbones, switching to PTC gives limited benefit and often the lowest scores of the three harnesses. After SFT and RL, PTC gives the highest aggregate score on every benchmark at both scales — for example, ProgAgent-14B reaches 29.7% on MCPMark and 45.6% on MCP-Atlas versus 8.0% and 29.7% for the DTC recipe.

• Hybrid underperforms PTC. Despite offering a more flexible action space at both training and evaluation time, the Hybrid (PTC+DTC) recipe scores below pure PTC. The authors hypothesize that a unified tool-interaction interface lowers learning difficulty and yields a more generalizable representation.

• Training dynamics shift under PTC. At 14B, DTC reward rises only from 0.08 to 0.15, while PTC and Hybrid start near 0.50 and reach 0.70 and 0.72. Raw tool outputs account for around 30k context tokens per episode under DTC throughout training; under PTC, tool-output tokens fall by roughly 20% to below 20k per episode, and both programmatic paradigms use roughly half as many tool calls as DTC. PTC runtime errors per episode drop from 6.4 to 3.6 at 14B and from 7.5 to 5.0 at 8B.

• Environment scaling is a working difficulty knob. As the environment scales from v0 to v3, DeepSeek V4 Pro's pass@8 under DTC falls monotonically from 0.95 to 0.42 (database size 99 KB to 711 KB; mean workspace file count 12.7 to 40.3). Under PTC its pass@8 stays above 0.90 across all four scales in the main figure, and above 0.87 for Hybrid and PTC in the appendix figure. Training on all four scales matters: restricting training to v0 drops LOCA-bench averages from 24.3% to 16.0% at 8B and from 38.3% to 23.7% at 14B.

• Training gains transfer across interfaces. Under the same PTC harness, PTC-trained models beat DTC-trained models on both LOCA-bench and MCP-Atlas. The advantage also holds under DTC evaluation: on MCP-Atlas at 14B, the PTC-trained model beats its DTC-trained counterpart by 11.9 percentage points. Other baselines given access to PTC remain low, suggesting the interface alone is insufficient.

• Agents do use programming structures. During RL, 82.6% (8B) and 82.1% (14B) of generated code blocks used loops, 65.6% and 64.0% used conditional execution, and 8.2% and 9.5% used error handling.

• Two failure modes remain. On Playwright, 40.7% of ProgAgent-14B's PTC calls contained a single tool invocation with no control flow, error handling, or selective printing — a reversion to DTC-like behavior. Separately, because the harness relies on MCP's isError field, non-compliant servers can fail silently; disabling server-level error detection drops LOCA-bench pass rates from 24.3% to 19.3% at 8B and from 38.3% to 34.5% at 14B.

Methodology in Plain English

The authors first build a task factory. A language model proposes realistic user intents across 23 domains; a "constructor" agent (Claude Opus 4.6 via Claude Code) then creates a database, a file workspace, a set of narrow tools, a task instruction, and a Python verifier that checks both the final answer and the final environment state. To guarantee solvability, the constructor also writes an oracle solution and repairs the task in a loop until that oracle earns full reward. An independent "harness" agent then audits consistency, and a difficulty filter samples 8 DeepSeek V4 Pro trajectories per task per scale, keeping only tasks with pass rates strictly between 0 and 0.8 at every scale — this removed 35 of 250 candidate tasks.

Each surviving task is instantiated at four environment sizes by regenerating the data with a different scale setting while leaving the instruction, logic, tools, and verifier untouched.

For execution, the model writes Python programs that call tools inside a persistent IPython kernel that keeps variables alive between turns. Only what the program prints goes back into the model's context. Episodes are isolated so RL rollouts don't contaminate one another.

Training is two-stage: supervised fine-tuning on teacher trajectories for 3 epochs (batch size 64), then GSPO reinforcement learning for 150 steps with 16 rollouts per task over 32 tasks per step, using the binary verifier reward. The same pipeline is run three ways — DTC, PTC, and Hybrid — so the interface is the only difference. Models are Qwen3 8B and 14B, evaluated on MCPMark, MCP-Atlas, Toolathlon-Verified, and LOCA-bench at 8k–64k environment-state sizes, with each pass rate averaged over four independent runs.

Why This Matters

The paper reframes a deployment trick (write code instead of emitting tool-call JSON) as a training strategy, and provides evidence that the interface used during training shapes what the model learns, not just how efficiently it acts. It also supplies an openly described recipe for generating verifiable long-horizon tasks with a difficulty dial, which is a standing bottleneck in agentic RL — note that ProgAgent uses 215 tasks versus 1.53M trajectories for TOUCAN, 135k for Simulator, 11,572 for EnvScaler, 3,315 for AWM, 2,000+ for Agent-World, and 2,575 for EnvFactory.

Real-world applications:

• Enterprise MCP integrations — agents that must coordinate many SaaS tools where raw JSON responses would flood the context window.
• Data analysis and reporting agents — multi-hop aggregation across database tables combined with constraints buried in unstructured documents, exactly the task shape synthesized here.
• Long-running operational workflows — tasks requiring pagination, loops, conditional branching, and error recovery rather than single-shot calls.
• Cost and latency reduction in deployed agents — fewer tool calls and roughly 20% fewer tool-output tokens per episode under PTC translate into cheaper, longer-running agents.

Industry relevance: MCP has become the de facto standard for exposing tools to LLMs, and both the runtime (MCP server, IPython kernel) and the failure analysis (non-compliant servers omitting isError) speak directly to teams building agent platforms. The finding that PTC-trained models also do better under DTC evaluation lowers the risk of committing to a programmatic harness.

Future Directions

• Fixing DTC-like reversion. 40.7% of single-tool PTC calls on Playwright show agents abandoning programmatic orchestration when environments return unstructured, unpredictable content. Training objectives or environment designs that discourage this are open.
• Robust error semantics. Silent server failures cost up to 5.0 and 3.8 percentage points on LOCA-bench at 8B and 14B when server-level error detection is disabled; standardizing or inferring tool error signals is an unresolved infrastructure problem.
• Scaling the task pool and closing the frontier gap. ProgAgent trails GPT-5.2 substantially on every benchmark despite beating comparable-size baselines; whether more tasks, larger backbones, or richer domains close that gap is untested here.
• Understanding the Hybrid deficit and the transfer mechanism. The paper offers a hypothesis (unified interface, easier learning, more generalizable representation) rather than a mechanism for why PTC training improves DTC performance, and it explicitly motivates further study of how programmatic training shapes tool-use behavior across interfaces.

Target Audience

Researchers and graduate students working on LLM agents, agentic reinforcement learning, and synthetic environment/task generation; ML engineers building tool-using agent products on MCP who need to choose a harness and a training strategy; and practitioners evaluating whether to invest in programmatic tool interfaces. Readers without background in RL fine-tuning or agent harnesses will find the framing sections accessible but the experimental analysis dense.

Authors’ abstract

Large language models (LLMs) are increasingly expected to orchestrate diverse tools over long-horizon tasks. Existing agents typically use Direct Tool Calling (DTC), where the model invokes tools sequentially, and tool responses accumulate in the model context, which can lead to context rot and orchestration breakdown. In contrast, Programmatic Tool Calling (PTC) invokes tools through executable code and stores intermediate results in runtime variables rather than appending them to the context. Prior work has explored PTC mainly as an inference-time optimization, but its benefits for agentic training remain unclear, and suitable verifiable training tasks are scarce. In this work, we introduce ProgAgent, a full PTC training stack, comprising a pipeline for synthesizing verifiable long-horizon tasks, a lightweight PTC runtime, and a two-stage training recipe. Using only 215 tasks instantiated at four environment scales, ProgAgent yields higher pass rates than the evaluated baselines of comparable size across four benchmarks. On LOCA-bench, ProgAgent-14B achieves a 30.7% pass rate at a 64k-token environment state, where these baselines score zero. Cross-harness experiments show that PTC-trained models outperform DTC-trained models under both evaluation interfaces, suggesting that the benefits of PTC training extend beyond the interface.