Tool Zero: Training Tool-Augmented LLMs via Pure RL from Scratch

AI summary

Overview

Research area: Tool-augmented large language models, reinforcement learning from rule-based rewards (R1-style RL / GRPO), and function calling.

Technical level: Intermediate. Readers should have some familiarity with reinforcement learning for LLMs (GRPO, PPO, reward shaping) and with tool/function-calling benchmarks, but the paper's core ideas are explained in plain terms.

Scope in one sentence: The paper asks whether a base LLM can learn general-purpose tool use through pure rule-based reinforcement learning with no supervised fine-tuning at all, and answers with a reward design (GG-GRPO) plus a family of models called Tool-Zero.

What This Paper Is About

Current tool-augmented LLMs are mostly trained by supervised fine-tuning (SFT) on large synthesized datasets of tool-use trajectories. The paper argues that such models imitate surface-level patterns rather than internalizing reasoning, so they do well on data resembling their training distribution but generalize poorly to unfamiliar tools, formats, and multi-turn settings. The goal is to see whether pure reinforcement learning, applied directly to untouched base ("Zero") models, can elicit intrinsic reasoning and yield tool-agnostic generalization instead.

Key Contributions

1. A pure-RL recipe for tool learning. The authors extend rule-based R1-style RL (specifically GRPO) to the tool-use domain and scale it directly from "Zero" models, i.e., base models without any post-training, producing the Tool-Zero series (Tool-Zero-7B and Tool-Zero-32B).
2. A dynamic generalization-guided reward design (GG-GRPO). The total reward is split into a format reward and a tool reward, and the tool reward progressively shifts from a lenient, token-overlap "general" reward to a strict AST-based correctness reward via a sigmoid switching function, balancing exploration and exploitation.
3. Two tool-specific feedback signals added to the strict reward: a +0.3 reward for correct multi-tool collaboration, and a +0.3 penalty per invalid parameter value.
4. Empirical evidence of generalization. Cross-dataset and intra-dataset evaluations across BFCL-v3, API-Bank, SealTool, Tool-Alpaca, and Nexus Raven, plus ablations on the reward components, reward-switching hyperparameters, training data, and backbone scale.

Main Findings

• Pure RL beats SFT and SFT-then-RL. Under the same experimental settings, the Tool-Zero models achieve over 7% performance improvement compared to both SFT and RL-with-SFT models. Concretely, Tool-Zero-7B is reported as a 7.14% improvement over the SFT model ToolACE-8B and 7.18% over the RL-with-SFT model ToolRL-7B.
• BFCL-v3 results. Tool-Zero-7B scores 88.98 (Non-Live), 80.76 (Live), 25.93 (Multi-Turn), overall 65.22. Tool-Zero-32B scores 90.76, 82.43, 28.18, overall 67.12. Compared with Qwen2.5-7B-Instruct, Tool-Zero-7B is reported as +13.32 on Live and +19.32 on Multi-Turn. Among R1-like models, Tool-Zero-7B surpasses DeepSeek-R1 by +8.33 and ToolRL-7B by +6.84.
• SFT generalizes poorly. The preliminary study found SFT models improved sharply on the Live (in-distribution) metric, reported as a gain from 61.1 to 78.6, while showing negligible gains or regressions on Non-Live and Multi-Turn subsets (e.g., 9.6 to 7.8). In cross-dataset evaluation, xLAM-7B-fc led on BFCL but degraded on other benchmarks, producing the lowest overall average (64.12), while foundation models were more consistent.
• Consistent cross-benchmark robustness. In Table 4, Tool-Zero-32B reaches averages of 79.32 across BFCL-v3, API-Bank, SealTool, Tool-Alpaca, and Nexus Raven, compared with 71.98 for ToolACE-8B, 64.12 for xLAM-7B-fc, 68.44 for ToolRL-7B, 73.34 for DeepSeek-R1, and 79.16 for GPT-4o-2024-11-20. Tool-Zero-7B averages 77.65.
• Ablation: every component helps. GG-GRPO showed a +5.26 improvement over GRPO and a +6.8 improvement over SFT in the ablation on BFCL overall performance. The multi-tool and value-error signals contributed more than the tool mask.
• Reward scheduling matters. In the hyperparameter study over transition midpoint t_m in {0, 25, 50, 100} and steepness κ in {0.2, 1.0}, a smaller transition midpoint (t_m = 25) performed best, while larger values (t_m ≥ 50) degraded performance; a lower steepness factor consistently outperformed, indicating gradual reward shaping stabilizes training.
• Base models beat instruct models as starting points. Under GG-GRPO, training from base Qwen2.5-7B outperformed the instruct variant across training datasets (e.g., on ToolACE: 88.98/80.76/25.93 versus 89.39/79.39/21.74), which the authors attribute to the base model's higher plasticity.
• Multi-turn augmentation helps. Removing multi-turn augmentation in xLAM dropped the Multi-Turn result, reported in the text as an increase from 16.18 to 32.28 with the augmentation.
• Scale and algorithm. In the scale study, Qwen2.5-1.5B scored 63.73 average, 3B scored 70.87, 7B scored 77.32, and 32B scored 78.99; the paper describes the 1.5B and 32B results as relatively lower, indicating partial scalability. GG-PPO scored 69.07 average versus 72.38 for GRPO on the same 7B backbone.
• Low tool overlap across benchmarks. The number of unique tools ranges from 41 in Tool-Alpaca to 25,771 in ToolACE, and overlap rates range from 0% (xLAM vs. API-Bank, SealTool, Nexus Raven) to a maximum of 47.1% (ToolACE vs. SealTool), motivating learning generalizable tool-use patterns rather than memorizing tools.

Methodology in Plain English

The authors start from Qwen2.5-7B Base and Qwen2.5-32B Base and never run supervised fine-tuning. Instead, they run GRPO — a policy-optimization algorithm that skips the critic model by comparing a group of sampled completions against each other — directly on the base models.

Their data comes from ToolACE and xLAM, plus an irrelevance-augmented subset from xLAM. Because LLM-generated data can be malformed, they filter out samples with unparsable tool calls or candidate tools. Since xLAM lacks multi-turn trajectories, they synthesize them using four strategies: concatenating related single-turn dialogues, removing a tool and reintroducing it later, masking a parameter to force clarification, and perturbing ground-truth answers to simulate user challenges. They also mask function and parameter names (e.g., calculate_sum becomes func_1) so the model attends to descriptions and semantics instead of surface naming.

The distinctive part is the reward. Each response gets a format reward (0 or 1) for correct use of <think>...</think><answer>...</answer> tags, plus a tool reward. Early in training, the tool reward is lenient: it measures token overlap between the response and the ground truth after splitting on delimiters, rescaled to the range [-0.5, +0.5], giving partial credit for semantically relevant but incomplete answers. As training progresses, a sigmoid function σ(t, m) with steepness κ and transition midpoint m smoothly blends this with a strict AST-based reward that returns 1 only if the generated tool call is structurally equivalent to the reference AST, plus the +0.3 multi-tool bonus and +0.3 penalty per invalid parameter value. The result is a system that first explores broadly and later converges on precise tool use.

Training used the MindSpeed-RL framework on Ascend hardware, completing a run within 28 hours on 5×8 Ascend 910b NPUs, with a global batch size of 128, max prompt and response length of 2048, learning rate 5e-7 with cosine decay, mini batch size 1024, tensor model parallel size 4, no KL loss, ε = 0.2, 8 rollouts, and temperature 0.8. The reported key reward hyperparameters are κ = 0.1 and transition midpoint 25.

Why This Matters

The paper challenges the dominant assumption that strong tool-use ability requires large supervised datasets, showing that a base model can learn generalizable tool calling from rule-based RL alone. It also supplies evidence that SFT improves in-distribution accuracy at the cost of out-of-distribution generalization, which reframes how tool-use training pipelines should be designed.

Real-world applications:

• Agent systems that must call APIs from unfamiliar vendors or new tool sets without retraining, including code transpilation and multi-step workflows.
• Enterprise API orchestration, where the available tool catalog and argument schemas change frequently and fixed-toolset models degrade.
• Multi-turn assistants that need to ask clarifying questions when parameters are missing or ambiguous and to chain dependent tool calls.
• Coding and data-processing assistants that route requests to the correct function or interpreter based on semantic understanding rather than keyword matching.

Industry relevance: because the method trains from base models without a costly supervised data-construction stage, and because the authors report completing a run within 28 hours on 5×8 Ascend 910b NPUs, the recipe is attractive to teams that want scalable, low-label tool-use training. The paper's acknowledgement of industry affiliations (Huawei Technologies, Beijing Academy of Artificial Intelligence, Shanghai Jiao Tong University, Harbin Institute of Technology SCIR Lab) reflects direct commercial interest.

Future Directions

• Reducing the cost of the progressive reward schedule. The authors note that reward switching adds computation during RL, particularly for large models such as Tool-Zero-32B, limiting use on resource-constrained hardware without further optimization.
• Improving extreme-scale behavior. The scale study reports relatively lower results at 1.5B and 32B, and the paper states there is room for optimization at extreme scales.
• Better benchmarks. The limitations section acknowledges that evaluation datasets such as BFCL and API-Bank have known limitations, for example lengthy calling chains arising from design preferences.
• Extending the paradigm further. The conclusion frames the work as a step toward scalable, autonomous tool learning, leaving open how far pure RL can push tool-agnostic generalization beyond the tested benchmarks and tool types.

Target Audience

Researchers and practitioners in LLM post-training, reinforcement learning, and agent/tool-use systems who are weighing supervised fine-tuning against RL for function calling. It is also useful for engineers building production agents on top of base models who need generalization to unseen tools, and for readers interested in reward design and exploration-exploitation trade-offs in rule-based RL.

Authors’ abstract

Training tool-augmented LLMs has emerged as a promising approach to enhancing language models' capabilities for complex tasks. The current supervised fine-tuning paradigm relies on constructing extensive domain-specific datasets to train models. However, this approach often struggles to generalize effectively to unfamiliar or intricate tool-use scenarios. Recently, reinforcement learning (RL) paradigm can endow LLMs with superior reasoning and generalization abilities. In this work, we address a key question: Can the pure RL be used to effectively elicit a model's intrinsic reasoning capabilities and enhance the tool-agnostic generalization? We propose a dynamic generalization-guided reward design for rule-based RL, which progressively shifts rewards from exploratory to exploitative tool-use patterns. Based on this design, we introduce the Tool-Zero series models. These models are trained to enable LLMs to autonomously utilize general tools by directly scaling up RL from Zero models (i.e., base models without post-training). Experimental results demonstrate that our models achieve over 7% performance improvement compared to both SFT and RL-with-SFT models under the same experimental settings. These gains are consistently replicated across cross-dataset and intra-dataset evaluations, validating the effectiveness and robustness of our methods.