ToolForge: A Data Synthesis Pipeline for Multi-Hop Search without Real-World APIs

AI summary

Overview

• Research area: Artificial intelligence / LLM tool learning and synthetic training-data generation for multi-hop search.
• Technical level: Intermediate. The paper assumes familiarity with supervised fine-tuning, retrieval (BM25), tool/function calling, and evaluation metrics such as Exact Match and F1.
• Scope: The paper describes ToolForge, an automated pipeline that synthesizes multi-hop tool-calling training data using 19 virtual tools instead of real API calls, and reports benchmark results for an 8B model fine-tuned on that data.

What This Paper Is About

Training an LLM to call tools and reason over retrieved information normally requires tens of thousands of real API calls, which is expensive and still tends to produce data without multi-hop reasoning or self-reflection. ToolForge aims to produce large-scale, validated tool-calling data from nothing more than a (question, golden context, answer) triple, using a small set of virtual tools so that no real API calls are needed. The goal is a model that generalizes well to real tool-calling and multi-hop search benchmarks despite being trained only on synthetic data.

Key Contributions

1. ToolForge framework. An automated synthesis framework that, given only a (question, golden context, answer) triple, generates large-scale tool-calling data featuring multi-hop reasoning and self-reflection.
2. Virtual tools instead of real APIs. Generalization is pursued through virtual tools and reflection-driven multi-turn interactions, generating diverse reasoning–tool interaction patterns rather than executing real API calls.
3. Extensibility. The pipeline is described as inherently extensible rather than restricted to the 19 virtual tools and 29 interaction patterns instantiated in the paper; additional virtual tools, new noise types, or more complex interaction motifs can be added in a plug-and-play fashion without modifying the core pipeline.
4. Empirical results. ToolForge-8B, produced by fine-tuning Qwen3-8B exclusively on the synthetic data, is reported to outperform strong proprietary models such as GPT-4o across a range of challenging tool-calling benchmarks.

Main Findings

• Synthetic data alone beats larger models. In the Function Call setting, ToolForge-8B surpasses GPT-4o on 8 out of 10 benchmarks and outperforms the much larger Qwen3-32B on 9 out of 10 benchmarks. It is also reported as comparable to Qwen3-235B, surpassing that model's Exact Match scores on 6 of the benchmarks.
• Concrete headline numbers (Table 1, Wikipedia Retriever + Function Call). ToolForge-8B reaches NQ EM 30.25 / F1 40.92, PopQA 35.20 / 41.08, TriviaQA 16.80 / 36.98, SQuAD 24.00 / 32.75, Bamboogle 48.00 / 58.66, MusiQue 37.80 / 45.10, SRST 20.00 / 32.99, SRMT 71.00 / 74.41, MRST 48.00 / 61.55, and MRMT 77.50 / 77.63.
• Function calling is harder than plain search. When models move from Basic Search to Function Call retrieval, GPT-4o and the Qwen3 series show significant performance degradation across multiple benchmarks. ToolForge-8B is reported to mitigate this gap.
• Zero-shot generalization holds on downstream reasoning. ToolForge-8B outperforms pre-fine-tuning Qwen3-8B on six of eight downstream tasks: MATH-500 96.3 versus 97.4, AIME-2025 Pass@64 71.2 versus 67.3, MMLU-Redux 88.4 versus 87.5, GPQA-Diamond 61.3 versus 62.0, IFEval 86.8 versus 85.0, Arena-Hard 87.1 versus 85.8, HumanEval 58.3 versus 57.9, MBPP 73.4 versus 72.1.
• Zero-shot tool-calling benchmarks improve. On BFCL-v3 ToolForge-8B scores 27.3 versus Qwen3-8B 13.9 and Qwen3-32B 16.6; on τ-bench 36.4 versus 17.5 and 31.3; on ToolBench 62.4 versus 36.8 and 49.7; on NFCL 43.2 versus 41.6 and 50.6.
• Single-hop-dominant data mix works best. In the ratio ablation over 4,250 samples, the 9:1 single-hop to multi-hop split gives the highest overall performance across in-domain and out-of-domain benchmarks, while excessive multi-hop data leads to degradation.
• Retrieval quality matters a lot. Moving from the Wikipedia corpus to a controlled smaller-scale corpus of golden and noisy contexts produced average relative improvements of 31.2% in Exact Match and 28.5% in F1. Under BM25 + Function Call, ToolForge-8B reaches SRST 78.3 / 88.2, SRMT 88.5 / 91.4, MRST 75.0 / 81.8, and MRMT 99.5 / 99.5.
• Synthesis routes succeed at high rates. After the Generate and MLV stages, success rates are SRST 91.70%, SRMT 89.50%, MRST 86.30%, and MRMT 83.80% (Table 3).
• Validation is accurate. The MLV module reports Accuracy 98.47%, Precision 96.55%, Recall 98.25%, and F1 97.39% (Table 4). Skipping MLV produces 8.3%–16.2% noisy samples, including formatting errors and logical inconsistencies.
• Both validation layers are needed. In the MLV ablation, using only the Rule Verification Layer or only the Model Verification Layer produces very low scores across all ten benchmarks, while the combined RVL + MVL configuration yields the reported strong results.
• Tool diversity helps. Ablations show that increasing both the number of virtual tool types and the number of behavioral variants per tool consistently improves performance across all benchmarks.

Methodology in Plain English

ToolForge has three modules.

Knowledge Space Preparation (KSP). The authors abstract retrieval domains from intent instances and design 19 domain-specific virtual tools covering areas such as economics, politics, and science. To get diversity without real APIs, they generate tool variants through a "dual-gating mechanism" that checks both semantic similarity (an average cosine similarity against already accepted tools must exceed a threshold) and textual redundancy (average BM25-based textual similarity must stay below a threshold), with a cold-start rule that waves the semantic check when fewer than two tools have been accepted. They also define four tool-calling paradigms of increasing complexity: Single-Round Single-Tool (SRST), Single-Round Multi-Tool (SRMT), Multi-Round Single-Tool (MRST), and Multi-Round Multi-Tool (MRMT). Tool sequence, paradigm, and reasoning rationale are selected sequentially by maximizing conditional probabilities, and all outcomes were cross-validated by several independent annotators.

Generative Interaction Modeling (GIM). Starting from the four paradigms, the authors add three classes of error perturbations — tool misselection, arguments misselection, and tool switching — producing 29 representative reasoning–tool interaction patterns. Dialogue generation runs in four phases: a Planning Phase where an LLM generates an execution trace from the question, answer, and reasoning rationale; an Augmentation Phase that retrieves candidate documents with BM25 and builds paired "good" information (retrieved candidates plus the golden sub-context) and "bad" information (retrieved candidates only); a Generation Phase that synthesizes the multi-turn dialogue; and an Assembly Phase that structures the output. No manually annotated golden context is required — the pipeline reuses supporting evidence in existing QA datasets.

Multi-Layer Validation (MLV). A Rule Verification Layer applies nine static rules across four dimensions: dialogue format and structure consistency, tool protocol correctness, overall dialogue correctness, and traceability of data samples. It checks tool calls without executing them. A Model Verification Layer then judges semantic quality on three principles: correctness of tool-calling, logical soundness of reasoning, and consistency between thought and action, each assessed by an individual expert agent powered by an LLM. Monte Carlo Tree Search is used for hard negative mining. Human experts oversee all results.

Training setup. Data synthesis uses HotpotQA and 2WikiMultiHopQA. The authors design 19 base virtual tools with 20 diverse variants per tool and generate 4,250 training samples, then run full-parameter supervised fine-tuning on Qwen3-8B using ms-Swift, preserving a 9:1 ratio in which SRST accounts for 89.4% and SRMT, MRST, and MRMT each constitute 3.5%. Training uses four NVIDIA A100 80GB GPUs, DeepSpeed ZeRO-2, bfloat16 precision, Flash Attention 2, learning rate 1×10⁻⁵ with a constant schedule and 5% warmup, per-device batch size 2 with gradient accumulation steps of 1 (effective batch size 8), maximum sequence length 12,000 tokens, AdamW with weight decay 0.01, two epochs, completing in approximately 2 hours. Inference uses deterministic decoding at temperature = 0, with the same retriever, number of retrieved documents, knowledge corpus, and prompt across compared methods; the evaluation setting provides the top-10 passages retrieved from Wikipedia.

Why This Matters

Impact on research. The paper argues that existing synthesis pipelines depend on tens of thousands of real API calls, which are costly and hard to scale, and that existing validation is superficial — mainly checking tool-call syntax and final-answer consistency while overlooking the semantic and logical integrity of intermediate steps. ToolForge offers a cheaper path and pairs it with a validation framework whose published accuracy, precision, recall, and F1 are all above 96%. It also shifts multi-hop work from static text-based reasoning toward dynamic interactive reasoning with tools and reflection.

Real-world applications.

• Open-domain question answering systems that must combine retrieved passages across several steps.
• Workflow automation agents that invoke different tools across multiple turns.
• Travel planning assistants that need to sequence and switch between tools as conditions change.
• Customer-facing assistants that must recover from tool-selection or argument errors mid-conversation.

Industry relevance. The result that an 8B model trained on 4,250 synthetic samples can beat GPT-4o on 8 of 10 benchmarks and a Qwen3-32B model on 9 of 10 is directly relevant to teams that cannot afford massive API bills or very large models. The reported extensibility — adding tools, noise types, or interaction motifs plug-and-play — suggests the pipeline can be adapted to proprietary tool inventories without redesigning the core system.

Future Directions

• Expanding the tool and pattern space. The authors frame the 19 virtual tools and 29 interaction patterns as an instantiation, not a limit, and explicitly invite additional virtual tools, new noise types, and more complex interaction motifs.
• Scaling verification-aware synthesis. The conclusion calls for further exploration of scalable, verification-aware data synthesis for training reliable tool-augmented language models.
• Clarifying when multi-hop data helps. The ablation shows a 9:1 single-hop to multi-hop ratio is best under limited data budgets; the boundary conditions under which more multi-hop data begins to hurt remain an open question.
• Reducing reliance on high-quality retrieval. Performance is reported to be sensitive to retrieval quality, with a controlled corpus yielding 31.2% relative EM improvement and 28.5% F1 improvement; how the model behaves under realistic noisy retrieval remains a practical concern.

Target Audience

Researchers and engineers working on LLM tool learning, function calling, and agentic search systems, particularly those building synthetic data pipelines under cost constraints. It is also relevant to practitioners who need a small open-source model to handle multi-hop retrieval tasks, and to teams evaluating validation strategies for automatically generated training data. Readers should be comfortable with supervised fine-tuning, retrieval evaluation metrics (EM and F1), and standard tool-calling terminology.

Note on completeness: the supplied paper content is truncated mid-sentence in Appendix A.2 (the ablation on retrieval top-k selection), so the full top-k results and any remaining appendix material are not reported here.

Authors’ abstract

Training LLMs to invoke tools and leverage retrieved information necessitates high-quality, diverse data. However, existing pipelines for synthetic data generation often rely on tens of thousands of real API calls to enhance generalization, incurring prohibitive costs while lacking multi-hop reasoning and self-reflection. To address these limitations, we introduce ToolForge, an automated synthesis framework that achieves strong real-world tool-calling performance by constructing only a small number of virtual tools, eliminating the need for real API calls. ToolForge leverages a (question, golden context, answer) triple to synthesize large-scale tool-learning data specifically designed for multi-hop search scenarios, further enriching the generated data through multi-hop reasoning and self-reflection mechanisms. To ensure data fidelity, we employ a Multi-Layer Validation Framework that integrates both rule-based and model-based assessments. Empirical results show that a model with only 8B parameters, when trained on our synthesized data, outperforms GPT-4o on multiple benchmarks. Our code and dataset are publicly available at https://github.com/Buycar-arb/ToolForge .