The Pulse
Qwen3-8B Beats 402B Llama on Tool Selection
A post-trained Qwen3 model with 8 billion parameters scored 41.5% on ToolBench G3’s exact-match tool-selection test, compared with 30.6% for Llama-4-Maverick. The paper also reports transfer tests on other tool collections, including one be
AI.info Team · Written with AI · fact-checked ·
A trained Qwen3 model with 8 billion parameters reached 41.5% exact-match success on ToolBench G3, beating the best prompted result from Llama-4-Maverick, a 402-billion-parameter model, which scored 30.6%. The comparison, in a paper submitted to arXiv on October 22, concerns one task: choosing the right set of tools for multi-step instructions. It does not show that the smaller model is better at general language tasks or at using tools once selected.
“On ToolBench, a Qwen3-8B model post-trained with our recipe outperforms Llama-4-Maverick (402B).”
Zhaoyi Zhou and the other authors, authors of Training Small Language Models for Transferable Tool Selection, write
Exact matches, not just promising candidates
The authors test on 2,124 ToolBench G3 instructions. Their metric counts an instruction as successful only when the selected tool set exactly matches the paper’s filtered reference set; selecting an extra tool or missing one counts as failure. That stricter test distinguishes choosing tools from retrieving a list that merely contains the right answer.
The model follows a Plan–Retrieve–Select workflow. It breaks a request into substeps, uses a retriever to find candidate tools for each one, then selects a tool from the candidates. For example, a request to book a flight and find a hotel becomes separate substeps, each with its own retrieval and selection.
Training mattered more than parameter count
The team trained Qwen3 models at 1.7B, 4B and 8B scales, first using supervised fine-tuning on synthetic examples and then reinforcement learning. The examples came from trajectories produced with Llama-4-Maverick as a teacher; the researchers also filtered reference tools to remove ones they judged were not identifiable from the user’s request. On the test set, the 1.7B and 4B models reached 37.6% and 40.3%, respectively, after training.
Llama-4-Maverick’s score depended on how it searched. Its best of four prompted workflows, a Search-R1-style approach, scored 30.6%; ReAct scored 28.4%. The model scored 10.1% when run through the researchers’ own workflow without the specialized training. The results therefore compare trained, task-specific small models with a much larger model prompted to do the job, not matched training regimes.
Transfer works, but not everywhere
The researchers also tested the same trained checkpoints on four tool collections outside their training set. On Gorilla and CRAFT, the 8B model scored 29.9% and 69.0%; on ToolBench G2 it scored 25.8%. The authors say the trained models beat the strongest untrained baseline on G2 and show a transfer advantage on the single-tool Gorilla and CRAFT tests.
TaskBench exposes a limit: Qwen3-8B scored 72.8%, while Llama-4-Maverick with the Search-R1-style workflow reached 80.6%. The authors attribute the gap to the planning stage, which they say can require domain knowledge. Their evaluation also assumes each substep can be handled by one tool and that substeps do not depend on one another.
A new retriever, without retraining
The team trained with BM25, a lexical search method, then swapped in COLT, a dense retriever specialized for ToolBench. They report that the replacement improved success at every tested model scale without additional model training. That finding suggests the selector can work with a different retrieval backend, though it does not remove the need for the retriever to surface a useful candidate in the first place.
The paper’s clearest result is bounded: on ToolBench G3, post-training lifted Qwen3-8B to 41.5% exact-match success, above every prompted Llama-4-Maverick workflow the authors tested. Its performance on TaskBench, where the largest model’s prompted search workflow scored higher, shows why the headline comparison should stay tied to that benchmark and metric.