Don't Adapt Small Language Models for Tools; Adapt Tool Schemas to the Models

AI summary

Overview

Research area: Natural Language Processing, specifically tool-augmented language models and small language model (SLM) agents.

Technical level: Intermediate. The core idea is conceptually simple (renaming tools so models recognize them), but it relies on sampling, edit-distance clustering, and benchmark evaluation protocols that a general ML reader can follow.

Scope: The paper proposes PA-Tool, a training-free method that rewrites tool and parameter names in a schema to match naming patterns the model already encountered during pretraining, and evaluates it on tool-selection and end-to-end agent benchmarks.

What This Paper Is About

Small language models (typically ≤ 8B parameters) are increasingly used as the workers in tool-augmented multi-agent systems, but they frequently fail at tool selection and parameter identification. A common failure mode is schema misalignment: even when the correct tool is present in the provided schema, the model hallucinates a plausible-sounding but non-existent tool name, apparently falling back on naming conventions internalized during pretraining. Rather than fine-tuning the model to fit unfamiliar schemas, the authors instead adapt the schemas to the model.

Key Contributions

1. PA-Tool (Pretraining-Aligned Tool Schema Generation): a training-free schema optimization method that identifies tool component names better aligned with a model's pretrained knowledge by repurposing peakedness — a signal originally used in data contamination detection — as a measure of pretraining familiarity.
2. Empirical gains across SLMs: improvements of up to 17% on MetaTool and RoTBench, with benefits extending from tool selection to parameter identification in both single-turn and multi-turn settings.
3. Diagnosis of why it works: the method primarily attacks schema misalignment, which the error analysis identifies as a dominant failure mode, with those errors reduced by 80.0%.
4. Compatibility and deployment argument: demonstration that the approach is complementary to supervised fine-tuning and to other training-free methods, and that it requires only a one-time name mapping with no model modification, retraining, or risk of catastrophic forgetting.

Main Findings

• MetaTool gains: PA-Tool improves over Base models on most MetaTool subtasks, with the largest gain on Reliability (up to 17.0%, e.g., Llama3.2-3B from 43.6% to 60.6%), followed by Multi-tool (up to 9.6%, Llama3.1-8B from 78.7% to 88.3%) and up to 10.7% on Similar and Scenario.
• RoTBench gains: single-turn tool selection improves across all models by 5.7% to 10.5% (Llama3.1-8B from 58.1% to 68.6%), multi-turn by up to 6%, and parameter identification by up to 4.3% (Qwen2.5-3B multi-turn from 10.0% to 14.3%).
• Beats training-free baselines and matches human renaming: Greedy decoding occasionally underperforms Base (Llama3.2-3B Reliability: 39.8% vs. 43.6%) because it produces only one candidate. MostFreq is sometimes competitive, but PA-Tool is more consistent. PA-Tool also matches or exceeds human-designed schemas (Llama3.2-3B Similar: 65.7% vs. 58.6%), where the human baseline used two PhD-level annotators with software engineering backgrounds and was evaluated only on MetaTool.
• Still a gap to closed-source models, but with exceptions: Claude-Sonnet-4.5 reaches 83.3% tool selection on RoTBench single-turn, yet Llama3.1-8B with PA-Tool reaches 88.3% on MetaTool Multi-tool, surpassing all closed-source models including Claude-Sonnet-4.5 (85.1%), and Qwen2.5-7B with PA-Tool reaches 88.2% on Reliability, approaching Gemini-2.5-Flash (89.2%).
• Error decomposition: for Llama3.1-8B, PA-Tool reduces Schema Misalignment errors by 80.0%, Functional Confusion errors by 24.0%, and Context Understanding errors by 18.8%, suggesting misaligned names also contribute indirectly to other error types.
• Peakedness tracks exposure: when four models (Qwen2.5-3B/7B, Llama3.2-3B, Llama3.1-8B) are trained on tool schemas in a pretraining-style setup (loss computed on all tokens), peakedness measured at epochs 0, 10, 20, 30, 50, and 100 with 256 candidates per component consistently increases, with gains up to +25.8% (Llama3.2-3B).
• Component ablation: tool-name alignment mainly helps tool selection (Llama3.1-8B single-turn: 58.1% to 62.9%), parameter-name alignment mainly helps parameter identification (Llama3.2-3B single-turn: 20.0% to 23.8%), and combining both gives the most balanced result across settings.
• Complementary to fine-tuning: on Llama3.1-8B, PA-Tool alone outperforms both SFT1 (about 2.5K samples) and SFT2 (about 5K samples) on Reliability and Multi-tool without any training; SFT1 + PA-Tool reaches 80.8% on Scenario. SFT2 degrades on RoTBench relative to SFT1 (59.1% vs. 61.0%), which the authors attribute to narrowed generalization from additional domain-specific training.
• Complementary to other training-free methods: retrieval-based correction (BM25 or ToolLLM) yields less than 3% improvement because post-hoc matching often cannot recover the intended tool; constrained generation greatly improves RoTBench (Qwen2.5-7B: 78.1% vs. Base 49.5%) but is mixed on MetaTool; EasyTool-style description enhancement combined with PA-Tool gives the best RoTBench result for most models (Qwen2.5-3B single-turn: 7.6% to 19.1%).
• End-to-end translation: on API-Bank, Qwen2.5-3B improves from 18.0% to 28.5% on the Call task; on τ-Bench (Retail, run 5 times), Qwen2.5-7B improves from 6.8% to 9.7% and Llama3.1-8B from 9.7% to 11.1%, with Llama3.1-8B's schema misalignment cases dropping from 115 to 98.
• Human perception improves, not degrades: three annotators with software development experience rated all 199 MetaTool tools (597 responses). PA-Tool names scored higher on ease of understanding (3.41 vs. 2.72) and match to functionality (3.44 vs. 2.78), both significant at p < 10⁻³⁵, and were preferred in 52.3% and 50.9% of responses versus roughly 10% for originals. Large gains occur for opaque brand-like names (Figlet → ascii_converter, +3.00), while renaming hurts when originals are already intuitive (ProductComparison → compare_options, −1.67).
• Cross-model transfer works partially: schemas aligned to one model generally help another, even across families (Llama3.1-8B using Qwen2.5-7B schemas gains +8.1% on Similar), with self-generated schemas optimal in most settings but by a small margin.
• Generalization beyond the main SLMs: gains appear for Ministral-8B (+8.5% on RoTBench single-turn), GPT-4.1-nano, Gemini-2.5-Flash-Lite, larger models (Llama3.3-70B, GPT-4.1-mini, Gemini-2.5-Flash — e.g., GPT-4.1-mini +12.1% on MetaTool Multi-tool), and reasoning models (Qwen3-1.7B, Qwen3-4B in thinking mode, up to 10.5% on RoTBench single-turn parameter identification).

Methodology in Plain English

The intuition is that a model struggles with a tool called X not because X is hard to use, but because the model has seen names like Y far more often during pretraining. So instead of changing the model, change X to Y.

PA-Tool operates in three stages on each schema component (each tool name and each parameter name):

1. Candidate generation. The model is shown only the component's natural-language description and asked to produce a name. This is repeated 32 times at temperature 0.4, producing a set of candidate names. A separate greedy (temperature 0) pass produces a reference name used only for tie-breaking.
2. Peakedness computation. Borrowed from contamination-detection work, peakedness measures how tightly the model's outputs cluster. For each candidate, the method counts how many other candidates fall within a similarity threshold τ = α · ℓ_max, where ℓ_max is the longest candidate in characters and α = 0.2. Similarity is character-level Levenshtein edit distance.
3. Selection. The candidate with the highest peakedness wins, on the theory that it sits in the densest region of the model's output distribution and therefore reflects the most deeply internalized naming convention. Ties are broken by minimum edit distance to the greedy reference name.

Applying this to every component produces a dictionary mapping original names to pretraining-aligned names, which is then used as a drop-in replacement schema. Name collisions across tools are resolved with a priority-based locking mechanism. Benchmark inference uses temperature 0 for reproducibility, and accuracy is the metric throughout.

A useful illustration from the appendix: for a tool described as simplifying calorie counting, the most frequent sampled name is diet_tracker (5 occurrences), greedy decoding gives nutri_guide, but PA-Tool selects diet_insights (peakedness = 4) — showing that the method deliberately is not a frequency count.

Why This Matters

Impact on research. The paper reframes tool-use failures in small models as an interface-alignment problem rather than a capability deficit. It also repurposes a technique from data contamination detection into a constructive design tool, which is an unusual and transferable move: peakedness becomes a signal for "what does this model already know how to say?" rather than "what has this model memorized?"

Real-world applications.

• Multi-agent systems in which a powerful coordinator orchestrates many small, cheap worker models that call APIs — the exact setting the paper motivates, where one misaligned worker can cause a cascading failure.
• On-device or edge deployments with strict resource constraints, where retraining or hosting larger models is impractical but a one-time schema rename is not.
• Products built around third-party or legacy APIs whose names are arbitrary, brand-specific, or opaque; PA-Tool can map them to descriptive names without touching the backend.
• Agent platforms that integrate large tool catalogs, where renaming can be generated once per model and reused across queries.

Industry relevance. The method is training-free, requires no model modification, and carries no catastrophic-forgetting risk, which lowers deployment cost and operational overhead. Its demonstrated compatibility with JSON-schema constrained generation, retrieval-based correction, and description-rewriting tools means it can be layered onto existing tool-calling pipelines rather than replacing them. The human evaluation matters commercially too: schemas optimized for the model also read better to human developers, so interpretability is not sacrificed.

Future Directions

• Models with very different training distributions. The authors note that the peakedness–familiarity relationship might not hold for models trained on substantially different data, which remains untested.
• Non-English and non-Latin scripts. Evaluation focuses on English schemas, and the authors flag that character-level edit-distance thresholds may behave differently for non-Latin scripts or morphologically complex languages.
• Extending alignment beyond names. PA-Tool leaves descriptions untouched because rewriting them risks semantic drift away from a tool's actual functionality; the authors suggest integration with interaction-driven description refinement methods as a promising direction, building on the complementary EasyTool results.
• Where the ceiling lies. Since gains shrink as model capacity grows because misalignment becomes less severe, the open question is how to identify the settings where schema alignment still compounds — such as multi-tool composition — and how far cross-model schema transfer can substitute for per-model generation.

Target Audience

This paper is most useful to applied NLP and ML engineers building tool-calling or multi-agent systems with small or resource-constrained models; to practitioners deploying agents on edge devices or under strict latency and cost budgets; to API and platform teams who control tool schemas and want a low-cost accuracy improvement; and to researchers working on tool-use evaluation, agent reliability, or training-free adaptation methods. Readers interested in contamination detection will also find the reuse of peakedness as a design signal worth examining.

Authors’ abstract

Small language models (SLMs) enable scalable tool-augmented multi-agent systems where multiple SLMs handle subtasks orchestrated by a powerful coordinator. However, they struggle with tool-use tasks, particularly in selecting appropriate tools and identifying correct parameters. A common failure mode is \textit{schema misalignment}: models hallucinate plausible tool names that are absent from the provided tool schema, due to different naming conventions internalized during pretraining. Rather than training models to adapt to unfamiliar schemas, we propose adapting schemas to align with models' pretrained knowledge. We introduce \textbf{PA-Tool} (Pretraining-Aligned Tool Schema Generation), a training-free method that leverages peakedness, a signal used in contamination detection that indicates pretraining familiarity, to rename tool components. By generating multiple candidates and selecting the candidate with the highest peakedness, PA-Tool identifies pretraining-aligned naming patterns. Experiments on MetaTool and RoTBench show improvements of up to 17\%, with schema misalignment errors reduced by 80\%. PA-Tool enables small models to substantially improve tool-use accuracy without retraining, showing that schema-level interventions can unlock the tool-use potential of resource-efficient models. Our code is available at https://github.com/holi-lab/PA-Tool.