AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning

AI summary

Overview

• Research area: Natural Language Processing / LLM agents — specifically agentic reinforcement learning, tool use and integration, and dynamic tool selection.
• Technical level: Intermediate to Advanced. The paper is readable at a high level, but the core method assumes familiarity with reinforcement learning (policy optimization, KL regularization), Plackett–Luce ranking models, and supervised fine-tuning of LLM backbones.
• Scope: The paper proposes AutoTool, a training framework that teaches LLM agents to dynamically choose tools from large, evolving toolsets during reasoning, backed by a 200k tool-selection dataset and evaluations on ten benchmarks using Qwen3-8B and Qwen2.5-VL-7B backbones.

What This Paper Is About

Most LLM agents that use external tools (search engines, code interpreters, vision modules) are trained under the assumption of a fixed, predefined tool inventory for a single domain, which makes them brittle when the available tools change or when new tools appear at inference time. AutoTool addresses this by training agents to select tools dynamically from an evolving toolset as part of their chain-of-thought trajectory, rather than memorizing a closed set of tool identifiers. The goal is an agent that preserves strong reasoning while adaptively choosing tools it may never have seen during training.

Key Contributions

1. A dual-phase training framework (AutoTool). Phase I performs trajectory stabilization through supervised fine-tuning followed by RL-based policy optimization, teaching the agent stable long-form chain-of-thought reasoning interleaved with tool use. Phase II performs tool-selection refinement, masking out internal reasoning and tool-integration steps and optimizing only the tool-selection segments of trajectories.

2. Embedding-anchored tool selection. Instead of having the model generate tool names directly, AutoTool collects an embedding for each tool from the LLM's own internal embedding layer, using the tool name plus its feature description. The agent generates a selection rationale followed by an anchor token, and the tool is chosen by sampling from a softmax-normalized distance distribution between the predicted anchor embedding and candidate tool embeddings, governed by a skewness parameter gamma. This lets the agent select unseen tools by representation alignment rather than memorized identifiers.

3. A theoretical bridge between Plackett–Luce ranking and policy optimization. The paper frames tool-selection steps as a sequence ranking problem over rollouts, with per-step rewards combining a process reward model score with final-answer accuracy, aggregated into a masked trajectory-level reward. Theorem 2.2 states that a trainable policy equals the optimal KL-regularized policy if and only if their induced Plackett–Luce ranking distributions coincide, which justifies optimizing the policy with a tractable policy-level cross-entropy loss instead of the intractable distribution over N! permutations.

4. A 200k-scale tool-selection dataset and an unseen-tool evaluation setup. The dataset was curated through a three-stage pipeline (toolset and task collection; tool-selection rationale generation; trajectory augmentation) and spans 1,346 tools and 120 task types across mathematics, science, code generation, and multimodal understanding. Toolset construction is deliberately decoupled from training curation: only 460 tools (34.2%) appear in training, leaving 886 tools (65.8%) entirely unseen for inference-time evaluation.

Main Findings

• Balanced cross-domain performance: On Table 1, AutoTool (Qwen3-8B, 8B parameters) reaches AIME24 68.8, AIME25 51.2, HotpotQA 45.1, 2Wiki 48.8, V-Chart 24.7, V-Math 53.0, and V-Code 56.1. AutoTool (Qwen2.5-VL-7B, 7B parameters) reaches AIME24 45.3, AIME25 38.9, HotpotQA 33.2, 2Wiki 36.5, V-Chart 13.2, V-Math 44.3, and V-Code 52.5.

• Domain-specialized baselines are brittle: ReTool (32B) posts the strongest math numbers among baselines (AIME24 70.1, AIME25 47.4) but drops on search (HotpotQA 21.4, 2Wiki 19.7), while Search-R1 (7B) is strong on search (HotpotQA 43.3, 2Wiki 44.5) but weak on math (AIME24 12.1, AIME25 7.3). GPT-4o leads on the multimodal benchmarks in Table 1 (V-Chart 27.3, V-Math 41.4, V-Code 51.2) and is weak on math (AIME24 18.2, AIME25 15.8).

• Gains over standard agentic training: Relative to prior training paradigms (SFT, GRPO, ARPO, and tool-integration methods), AutoTool yields average gains of 6.4% in math and science reasoning, 4.5% in search-based QA, 7.7% in code generation, and 6.9% in multimodal understanding. The paper also reports that AutoTool (Qwen3-8B) delivers an average of 6.7% improvement on AIME24 and 6.5% on HotpotQA compared with GRPO and ARPO.

• Phase II adds complementary gains: Table 2 shows AutoTool outperforming SFT, GRPO, and ARPO for both backbones. For Qwen3-8B: AIME24 66.7, AIME25 53.3, GPQA-Diamond 73.7, HotpotQA 45.1, 2Wiki 48.8, Bamboogle 56.8, V-Chart 24.7, V-Math 53.0, V-Code 56.1. For Qwen2.5-VL-7B: AIME24 46.7, AIME25 40.0, GPQA 38.4, HotpotQA 33.2, 2Wiki 36.5, Bamboogle 48.2, V-Chart 13.2, V-Math 44.3, V-Code 52.5, MMSearch 49.3.

• Generalization to unseen tools: Figure 4 shows that across 7 benchmarks AutoTool (Qwen3-8B) relies largely on unseen tools, particularly on search and multimodal tasks, indicating that performance gains come from generalizing tool selection rather than memorizing training-time tools.

• Robustness under a fully unseen tool pool: In Table 3, restricting inference to the 886 unseen tools, AutoTool scores AIME24 66.7, HotpotQA 43.8, LiveCodeBench 77.9, and MedQA 84.3, versus Search-R1 (13.3, 43.3, 36.2, 73.5) and ReTool (66.7, 21.4, 71.5, 64.7). Restricting to unseen tools causes only a marginal drop of less than 2% on both AIME24 and HotpotQA compared with using the full toolset of 1,346 tools.

• Tool-selection quality tracks accuracy: Higher HitRate (the rate of correct tool selection) consistently correlates with improved downstream accuracy across methods, and Phase II refinement substantially increases AutoTool's HitRate.

• Tool-selection rationales help: In the ablation shown in Figure 6, models trained with tool-selection rationales consistently outperform those trained without them, and rationales increase successful tool executions by 13.5%.

Methodology in Plain English

The researchers start from an existing reasoning model and teach it to alternate between three kinds of steps inside one trajectory: internal chain-of-thought reasoning, a tool-selection step, and a tool-integration step where the chosen tool is actually called and its output fed back into the model.

Training happens in two phases. In Phase I, the model is fine-tuned on curated trajectories and then optimized with RL so it can reliably produce this three-part structure. In Phase II, the focus narrows: the reasoning and tool-call steps are masked out, and only the tool-selection steps are optimized, using a KL-regularized objective that keeps the updated policy close to the previous one.

The distinctive part is how a tool is actually chosen. Rather than emitting a tool name as text, the model emits a rationale and then an "anchor" embedding. Every tool in the library has its own embedding, computed from the model's own internal embedding layer using the tool name plus a description of what the tool does. Selection is a softmax over the negative squared distance between the anchor embedding and each candidate tool embedding. Because the comparison happens in representation space, a tool the model never trained on can still be chosen if its description lands near the anchor.

To train this selection behavior, the team borrowed the Plackett–Luce ranking model from discrete choice theory. For each question they sample multiple trajectories (N = 8 per question), score each tool-selection step with a process reward model plus a final-answer correctness check, and aggregate into one masked trajectory reward. They prove that matching the optimal policy is equivalent to matching its induced Plackett–Luce ranking distribution, which lets them replace an intractable ranking loss with a standard cross-entropy objective.

Supporting all this is a new 200k dataset built in three stages: collecting 1,346 tools (code sandboxes and interpreters, search engines and browser APIs such as Jina Reader, and image modules such as OCR and GroundingDINO) along with 120 task types; prompting an expert reasoning model to write an explicit justification before each tool call; and filtering those rationales with an LLM-as-a-judge before inserting them back into trajectories and having the expert model smooth the full trajectories. Only 460 of the 1,346 tools are used in training, so the rest serve as a genuine unseen-tool test bed. Training used LLaMa-Factory for SFT and VeRL for RL, with Phase II trained for 3 epochs at rollout size N = 8, all on 8xA100-80G GPUs.

Why This Matters

• Research impact: The work reframes tool use from a fixed classification problem into a dynamic selection problem grounded in representation space, and supplies a theoretical equivalence (Theorem 2.2) connecting policy optimization to Plackett–Luce ranking. It also contributes a large dataset and an evaluation protocol that explicitly separates seen from unseen tools, which previous agentic training data and benchmarks did not do.

• Real-world applications:

  • Coding assistants that must route problems among sandboxes, interpreters, linters, and documentation lookup APIs, where the available set of tools changes as platforms evolve.
  • Multimodal question answering and document understanding, where an agent must decide between OCR, grounding, chart-parsing, and search tools on a per-question basis.
  • Retrieval-augmented enterprise assistants whose internal APIs and knowledge connectors are continuously added or retired.
  • Scientific and mathematical research agents that need to combine calculators, symbolic solvers, search, and visualization modules without retraining when a new instrument or library is added.

• Industry relevance: The approach trains on 7B and 8B parameter models and reports outperforming much larger baselines in specific settings, which matters for cost-sensitive deployment. Because the selection step is a lightweight vector comparison over precomputed tool embeddings, the paper argues the tool-selection overhead is negligible relative to LLM inference and scales to toolsets with thousands of tools — a practical property for production systems with large API catalogs.

Future Directions

• Extending the evaluation protocol: The paper tests unseen-tool generalization on a limited set of out-of-domain benchmarks (LiveCodeBench-V2 and MedQA) alongside AIME24, HotpotQA, and V-Chart. Broader coverage of tool ecosystems and task families would test whether the embedding-space alignment generalizes as claimed. The paper does not report a systematic study of how performance degrades as the fraction of unseen tools grows beyond the settings tested.

• Toolset scale and adversarial tools: The method is argued to scale to thousands of tools, but the largest pool evaluated here is 1,346. Behavior with near-duplicate, misleadingly described, or conflicting tools — and the description quality that the embedding depends on — is not characterized in the provided content.

• Reward model dependence: The per-step reward combines a process reward model with final-answer accuracy. Sensitivity to process reward model quality, and whether the masked trajectory reward can be replaced with cheaper signals, is not reported.

• Composition and multi-tool plans: The framework optimizes per-step tool selection within a trajectory. Whether it can be extended to deliberate composition of multiple tools in a single step, or to planning tool sequences before execution, remains an open question. Training compute, wall-clock cost, and the number of SFT/RL steps are also not reported in detail in the provided content.

Target Audience

Researchers and engineers working on LLM agents, agentic reinforcement learning, and tool-augmented language models will benefit most, particularly those building systems where the tool catalog changes over time or where tools must be selected from a large, cross-domain library. The paper is also relevant to practitioners evaluating post-training recipes (SFT, GRPO, ARPO) for agent behavior, and to readers interested in applying ranking-based objectives such as Plackett–Luce to language model policy optimization.

Authors’ abstract

Agentic reinforcement learning has advanced large language models (LLMs) to reason through long chain-of-thought trajectories while interleaving external tool use. Existing approaches assume a fixed inventory of tools, which limits the adaptability of LLM agents to new or evolving toolsets. We present AutoTool, a training framework that equips LLM agents with dynamic tool-selection capabilities throughout their reasoning trajectories. AutoTool employs a dual-phase optimization pipeline: (i) SFT and RL-based trajectory stabilization for coherent reasoning, and (ii) KL-regularized Plackett-Luce Ranking to refine consistent multi-step tool selection. We further build a 200k dataset with explicit tool-selection rationales across 1,000+ tools and 100+ tasks spanning mathematics, science, code generation, and multimodal reasoning. Across ten diverse benchmarks, we train two base models, Qwen3-8B and Qwen2.5-VL-7B, with AutoTool. With fewer parameters, AutoTool consistently outperforms advanced LLM agents and tool-integration methods, yielding average gains of 6.4% in math & science reasoning, 4.5% in search-based QA, 7.7% in code generation, and 6.9% in multimodal understanding. In addition, AutoTool exhibits stronger generalization by dynamically leveraging unseen tools from evolving toolsets during inference.