Beyond ReAct: A Planner-Centric Framework for Complex Tool-Augmented LLM Reasoning

AI summary

Overview

Research area: Tool-augmented large language model (LLM) agents, specifically planning and orchestration of multi-tool workflows.

Technical level: Advanced. The paper assumes familiarity with LLM agents, supervised fine-tuning, reinforcement learning (GRPO), and graph-based planning (Directed Acyclic Graphs).

Scope in one sentence: The paper proposes a planner-centric "plan-then-execute" framework in which a dedicated, specially trained Planner model converts a complex user query into a global DAG execution plan, plus the ComplexTool-Plan benchmark and a two-stage SFT + GRPO training recipe for that Planner.

What This Paper Is About

Most tool-using LLM agents decide what to do one step at a time, in a reactive loop (the ReAct style). The paper argues that this incremental style gets stuck in "local optimization traps" on complex queries — it commits to a step before seeing the whole picture and typically produces only a linear sequence, missing opportunities to run independent tools in parallel. The goal is to replace step-by-step reacting with a single upfront, global plan: a dedicated Planner model reads the query and the available toolset and outputs a Directed Acyclic Graph (DAG) whose nodes are tools and whose edges are data dependencies, which an executor then carries out.

Key Contributions

1. A planner-centric plan-execute paradigm and a DAG-optimized Planner model. The framework decouples planning from execution and trains the Planner to improve both node accuracy (which tools to pick) and edge accuracy (how those tools depend on each other), producing global, parallelizable plans.
2. ComplexTool-Plan, a new large-scale, difficulty-graded benchmark. It is built from a library of 4,535 tool APIs sourced from ModelScope, spans Easy, Medium, and Hard levels, provides a curated SFT dataset of 3,000 instances and a filtered RL set of 787 instances (split 8:2 into train and test), and is used both as training ground and evaluation standard.
3. A two-stage training methodology: SFT followed by GRPO reinforcement learning. The paper pairs a cold-start supervised phase with an RL phase using a hierarchical reward that separates structural errors from strategic ones.
4. State-of-the-art end-to-end results on StableToolBench. When paired with GPT-4o as executor, the framework reports the best performance among the compared open-source methods for complex user queries, with the fewest inference steps.

Main Findings

• RL improves holistic plan accuracy over SFT alone. On the Easy evaluation set, Qwen3-8B's DAG Exact Match accuracy rises from 0.781 (SFT) to 0.803 (SFT+RL); the paper reports the trend is consistent across all model sizes.
• RL matters most on hard tasks. On the Hard set, Qwen3-8B's DAG Exact Match accuracy goes from 0.295 (SFT) to 0.319 (SFT+RL), which the paper describes as an 8.1% relative improvement. The authors attribute this to RL correcting subtle structural errors that appear in complex scenarios, supported by consistent Edge-level F1 gains.
• Small models can hit a capacity floor. The Qwen3-0.6B (SFT+RL) variant is excluded from the results due to training instability; the authors suggest sufficient model capacity is needed to prevent reward hacking, where a model learns a degenerate low-effort policy just to avoid penalties.
• Fine-tuned Planners beat strong proprietary baselines on plan quality. On the Easy set, GPT-4o reaches 0.635 DAG Exact Match, Claude-3.7 reaches 0.644, Ernie-X1 reaches 0.562, and DeepSeek-V3 reaches 0.511, while Qwen3-8B (SFT) reaches 0.781 and Qwen3-8B (SFT+RL) reaches 0.803. On the Hard set the same proprietary models score 0.098 (GPT-4o), 0.106 (Claude-3.7), 0.052 (Ernie-X1), and 0.082 (DeepSeek-V3), versus 0.295 (Qwen3-8B SFT) and 0.319 (Qwen3-8B SFT+RL).
• Performance scales monotonically with Planner size on the Hard set. DAG Exact Match accuracy of the SFT+RL models rises from 0.218 (Qwen3-1.7B) to 0.259 (Qwen3-4B) to 0.319 (Qwen3-8B).
• Larger Planners degrade more gracefully. Moving from the Easy to the Hard set, Qwen3-1.7B drops 71.2% (from 0.756 to 0.218) while Qwen3-8B drops 60.3% (from 0.803 to 0.319), indicating scaling improves resilience against complexity.
• Strong end-to-end results on StableToolBench. The Qwen3-8B (RL) model achieves an average SoPR of 59.8% and SoWR of 55.0%, compared with GPT-4 (ReAct) at 48.2% SoPR. Qwen3-4B (RL) reaches 56.3% SoPR / 51.8% SoWR and Qwen3-1.7B (RL) reaches 51.9% SoPR / 46.4% SoWR. The authors call the 8B variant the best among all open-source baselines in both metrics.
• The framework is architecturally efficient. The method requires an average of just 2.29 inference steps per task, which the paper states is fewer than all other compared methods, including the iterative parallel framework DTA-Llama (2.48 steps reported in the text).
• The authors acknowledge a trade-off with iterative methods. While DTA-Llama reports a higher SoPR, the paper notes its success is largely driven by a multi-turn framework allowing in-process error correction, whereas this non-iterative plan-then-execute approach has only one chance to produce a correct global plan upfront.

Methodology in Plain English

The problem setup. A Planner model is asked to map a query plus a toolset to a plan expressed as a DAG, where nodes are tools and edges are dependencies. "Good" means matching the ground-truth plan; the training objective is to maximize expected plan utility, realized through a reward function.

Building training data (ComplexTool-Plan, a three-stage automated pipeline). Because large-scale structured planning data did not exist, the authors generated it:

1. Workflow generation — DeepSeek-V3 is given a subset of tools from a predefined library and asked to write a structurally complex, logically consistent plan as a DAG (the "original" graph).
2. Query reverse-engineering — The same teacher model then writes a natural-language query that would justify each graph, turning hard plan generation into an easier text-to-text problem.
3. Intent analysis and re-planning — The teacher model re-solves the problem from the query alone, acting as an expert planner, so the final graph is a faithful solution derivable from the query. This filters out ambiguous queries.

Training the Planner in two stages.

• Stage 1 — Supervised fine-tuning (SFT). Starting from Qwen3 models at 0.6B, 1.7B, 4B, and 8B, the model is trained to reproduce ground-truth plans by minimizing negative log-likelihood loss. This acts as a cold start.
• Data curation for RL. The SFT model is used as a filter: tasks it always solves or always fails are removed, keeping only high-variance instances that carry a real learning signal.
• Stage 2 — RL with GRPO using a hierarchical reward. The reward checks plan quality in a fail-fast order, prioritizing structural correctness: invalid syntax or a cycle in the graph incurs a large penalty of -10.0 and terminates evaluation; lack of connectivity (isolated nodes) incurs -2.0. If the plan passes structural checks, it earns a positive reward equal to 5 times the edge-level F1 score against ground truth, plus a +5.0 perfect-match bonus if the plan exactly equals ground truth. The reward's domain is [-10.0, 10.0], meant to tell the model not just whether it is wrong but what kind of error it made.

Evaluation. Planning quality is measured on ComplexTool-Plan with node-level precision/recall/F1, edge-level precision/recall/F1, and DAG Exact Match, on a test set of 500 samples from the Easy split and 500 from the Hard split, macro-averaged. End-to-end effectiveness is measured on StableToolBench using Solvable Pass Rate (SoPR) and Solvable Win Rate (SoWR) against the benchmark's GPT-3.5 (ReAct) baseline, with all scores averaged over three independent runs. The Planner is paired with GPT-4o as the executor for these end-to-end runs. Baselines include proprietary planners (GPT-4o, Claude-3.7, DeepSeek-V3, Ernie-X1) and open-source frameworks on ToolLLaMA (ReAct, DFSDT, LLMCompiler).

Why This Matters

Impact on research. The paper reframes tool-use agent design: instead of improving the reasoning loop, it moves the intelligence into a separately trained planning model that emits a graph in one forward pass. It also shows that a specialized small model (Qwen3-8B) can outperform far larger general-purpose proprietary models purely on planning quality, supporting the view that a better planning strategy can matter more than raw model capability. The release of ComplexTool-Plan and the hierarchical reward formulation gives the field a concrete testbed and training signal for non-linear, parallelizable planning rather than sequential reasoning.

Potential real-world applications. The paper does not itself evaluate deployed applications; these follow from the capability it targets:

• Enterprise workflow automation where a request fans out to several APIs whose outputs must be combined in a specific order.
• Data and analytics pipelines where retrieval, computation, and formatting steps run partly in parallel.
• Customer-support or operations assistants that must chain multiple backend services to resolve one ticket.
• Research or information-gathering assistants that compose search, extraction, and synthesis tools with dependencies.

Industry relevance. The efficiency claim is the commercially interesting part: 2.29 average inference steps against methods requiring more high-level decision rounds. Fewer planning rounds per task translates into fewer model calls, which matters for latency and cost in production agent systems. The result that 1.7B, 4B, and 8B planners outperform GPT-4o, Claude-3.7, DeepSeek-V3, and Ernie-X1 on DAG planning suggests a company can train a small, specialized planner and pair it with whatever executor it already has, rather than relying on a frontier model at every step.

Future Directions

• Closing the gap with iterative frameworks. The paper states that DTA-Llama reports higher SoPR because it can correct errors mid-process, while this framework gets one shot at a global plan. Adding verification and repair without losing the single-pass efficiency is an open problem.
• Handling the small-model capacity floor. Qwen3-0.6B (SFT+RL) had to be excluded due to training instability attributed to reward hacking, so the minimum model size and the reward-design safeguards needed to avoid degenerate policies remain unresolved.
• Extending beyond the current benchmark and executor pairing. All end-to-end results pair the Planner with GPT-4o as executor on StableToolBench; how the approach behaves with other, smaller, or locally hosted executors is not reported.
• Generalizing the data pipeline. ComplexTool-Plan was generated with DeepSeek-V3 as the teacher model over tools sourced from ModelScope; whether the same three-stage pipeline transfers to other tool libraries and domains is untested here.

Target Audience

Researchers and engineers working on LLM agents and tool use, particularly those interested in planning architectures, agent orchestration, or reinforcement-learning post-training of tool-calling policies. It is also relevant to practitioners building multi-API automation systems who care about latency and inference-step counts, and to benchmark designers interested in how difficulty-graded, DAG-based planning data can be synthesized automatically. Readers without a background in reinforcement learning or graph-based planning will find the framework's motivation accessible but will need that background to follow the training methodology.

Authors’ abstract

Existing tool-augmented large language models (LLMs) encounter significant challenges when processing complex queries. Current frameworks such as ReAct are prone to local optimization traps due to their reliance on incremental decision-making processes. To address these limitations, we propose a novel Planner-centric Plan-Execute paradigm that fundamentally resolves local optimization bottlenecks through architectural innovation. Central to our approach is a novel Planner model that performs global Directed Acyclic Graph (DAG) planning for complex queries, enabling optimized execution beyond conventional tool coordination. We also introduce ComplexTool-Plan, a large-scale benchmark dataset featuring complex queries that demand sophisticated multi-tool composition and coordination capabilities. Additionally, we develop a two-stage training methodology that integrates Supervised Fine-Tuning (SFT) with Group Relative Policy Optimization (GRPO), systematically enhancing the Planner's tool selection accuracy and global planning awareness through structured DAG-based planning. When integrated with a capable executor, our framework achieves state-of-the-art performance on the StableToolBench benchmark for complex user queries, demonstrating superior end-to-end execution capabilities and robust handling of intricate multi-tool workflows.