Benchmarking LLM Tool-Use in the Wild

AI summary

Overview

• Research area: Large language model agents and tool-use evaluation, with an emphasis on human-computer interaction (the paper is filed under cs.HC).
• Technical level: Intermediate. Readers need a working familiarity with LLM function calling, multi-turn dialogue, and agent benchmarking, but the paper is written around behavioral challenges rather than model internals.
• Scope: The paper introduces WildToolBench, a human-verified benchmark of 256 scenarios and 1,024 tasks built from real user behavior patterns, and reports results for 57 LLMs.

What This Paper Is About

Existing tool-use benchmarks for LLMs are idealized: tasks are well-specified, semantically independent, and often synthesized by LLMs with a single annotated optimal path. From large-scale analysis of real user logs, the authors argue that real interactions are "wild" — users bundle many small requirements into one instruction, spread their intent across several turns, and switch freely between task-giving, clarifying, and casual chatting.

The goal of the paper is to build a benchmark that reflects these three user behaviors and to measure how badly current models degrade when those behaviors are present, rather than when tool-call procedures are made artificially complex.

Key Contributions

1. A characterization of three real user behaviors. From real user logs, the authors identify compositional tasks (multiple simple requirements in one instruction requiring tool-call topology orchestration), implicit intent (information omitted or referenced across dialogue turns), and instruction transition (mixing task queries, clarifications, explanations, and casual conversation within one conversation).
2. WildToolBench itself. A benchmark of 256 scenarios and 1,024 tasks, built with a pipeline combining real-log-seeded scenarios, more than 1,600 publicly available APIs (drawn from 400 of the tool lists in the public-apis repository, following ToolAlpaca), controlled generation across four task types, and human verification and annotation by five human experts.
3. A fine-grained tool-orchestration evaluation protocol. An enumerate-match-score pipeline that uses depth-first topological sorting to enumerate all legal tool execution paths from human-labeled adjacent dependencies, then computes Optimal Path Rate (OP Rate) and Accomplish Progress Rate (AP Rate) in addition to task accuracy.
4. A large-scale evaluation of 57 LLMs spanning proprietary general, open-source general, and open-source specialized models, showing that no model exceeds 15% session accuracy on the benchmark.

Main Findings

• No model is close to solving the benchmark. None of the 57 evaluated models achieves a session accuracy higher than 15%, and most fall below 60% in task accuracy. The highest reported session accuracy is 14.45% (Gemini-2.0-Thinking), followed by 14.06% (Gemini-2.5-Pro) and 12.50% (Claude-4-Sonnet).
• Proprietary and reasoning models lead. Proprietary LLMs generally outperform open-source ones, and reasoning-oriented variants consistently surpass non-reasoning variants within the same series. GLM-4.5 and Kimi K2 reach performance comparable to the top three proprietary models.
• Task type matters enormously. Casual chat and tool-free queries are handled reliably by most models, but tasks requiring clarification frequently trigger spurious function calls. Multi-step tool use is substantially less accurate than single-step invocation.
• Later tasks in a dialogue are harder. Task accuracy declines from task 1 through task 4 within a session, reflecting increasing dependence on preceding dialogue context.
• Tool orchestration is weak. For compositional tasks, the highest task accuracy is 43.75%, dropping to 25% for mixed sequential-and-parallel tasks. The peak Optimal Path Rate is only 42.74%. Specialized tool-use models perform significantly worse than general-purpose models, with xLAM-2-70B, ToolACE2-8B, Watt-8B, and Hammer2.1-7B all trailing the leading general models.
• Hidden intent across turns is hard, and long-range dependency is hardest. No model exceeds 50% accuracy on long-range dependency tasks; those scores cluster between 30 and 45 and show the largest performance gap among models (17.3). Reasoning models such as o1 and Gemini-2.0-Thinking excel at partial-information tasks, while Claude-4-Sonnet leads on coreferential reference tasks.
• Frequent instruction transitions degrade performance. Across all model families, task accuracy drops as the number of instruction transitions in a dialogue increases, with drops reaching as much as 30% in some cases. The authors attribute this to self-conditioning (previous responses biasing later decisions) and to long context diluting attention to the current task.
• Errors have shifted from syntax to semantics. Wrong tool name / missing info and redundant calls are the most prevalent errors. Gemini-2.0-Thinking shows a "cautious" profile with a 24.56% refusal rate and 8.02% wrong-name rate, while Grok-4 shows an "eager" profile with 3.72% refusal but 24.07% wrong-name error. Specialized models xLAM-2-70B and Watt-8B have wrong-name errors exceeding 30%. Parameter-level errors such as parameter type error and parameter hallucination are consistently lower across all models.
• Benchmark comparison. In the comparison table, WildToolBench is reported at 100% for hidden information in context and 100% for user instruction transition, versus 15.7% and 39.7% for BFCL v3, and 0.0% for both on BFCL v1, BFCL v2, ToolBench, AnyToolBench, T-EVAL, and UltraTool.
• Dialogue complexity. The average dialogue length in WildToolBench is 5.27 turns and the average number of tool-call steps is 1.92, compared to BFCL at 3.75 turns and 1.68 steps. Tool types span 8 major categories and 24 subcategories.

Methodology in Plain English

The authors started from real user logs. They sampled interactions from those logs and summarized the recurring patterns into three challenges, using the samples as few-shot examples so generated scenarios would follow the real distribution without leaking actual user data.

They then assembled a tool set from public APIs, following the ToolAlpaca approach, and had five human experts inspect and refine tool combinations and parameter specifications to make the tools logically coherent. For each of 256 scenarios they selected a matching tool subset, generated four tasks covering single-tool, multi-tool, clarification, and chat types, and varied the tasks across sentence structure, linguistic style, task background, task length, and task difficulty. Later turns were generated to embody hidden-intent patterns such as partial information, coreferential reference, and long-range dependency (where the missing information lies more than two turns back).

A multi-agent setup with simulated user and assistant roles produced initial trajectories. Every tool call in those trajectories was automatically checked for function hallucination, parameter hallucination, type errors, and redundant calls, and then manually inspected and corrected by human experts, who also annotated tool-call dependencies. These dependencies feed a depth-first topological sort that enumerates every legal execution path, including branching and parallel ones. At evaluation time, each model tool call is matched incrementally against that decision tree set; a call either terminates the path or descends into a sub-tree. Scoring then checks whether the completed path has the minimum depth among the enumerated trees (Optimal Path Rate) and what fraction of nodes were successfully completed (Accomplish Progress Rate).

Quality control involved repeated rounds in which human experts sampled 20% of the data and held discussion-based review sessions. A major issue was that synthetic dialogue was "too well-organised," so experts rewrote user utterances to be more natural — for example, replacing a repeated full weather question with a short follow-up like "How about Shanghai?" The process was repeated with different pairs of experts and different 20% samples until the detected error rate dropped to zero. The paper reports that after four such iterations data quality improved, though the specific sequential figures are cut off in the provided text.

Why This Matters

The paper argues that apparent progress on tool-use benchmarks is spurious, because those benchmarks omit the behaviors that dominate real usage. It shifts evaluation emphasis away from artificially complex tool procedures and toward the interaction between LLMs, users, and tools.

Real-world applications affected:

• Agentic assistants in productivity tools (for instance document operations or slide generation), where users issue bundled instructions like searching for popular movies and generating a survey slide in one request.
• Travel and information services, where multi-city weather inquiries are interleaved with trip planning and later turns refer back with pronouns rather than restating details.
• Customer-facing chat systems, where a single session mixes task requests, clarification questions, and casual conversation, requiring the model to switch policy rather than default to a tool call.
• Function-calling model development, where the reported error profile (wrong tool name, redundant calls, over-refusal) points to planning and context management as the bottleneck rather than parameter formatting.

Industry relevance: the results indicate that specialized tool-use models do not generalize better than general-purpose models, and that reasoning-enabled variants do better at orchestration. The paper explicitly counters prior work (Zhou et al., 2025) claiming reasoning models do not outperform non-reasoning models on tool use. The benchmark is positioned as a structured rubric, not just a leaderboard, for guiding model iteration. All datasets, evaluation scripts, and trajectories for all 57 LLMs are released for what the authors describe as 100% reproducibility.

Future Directions

• Scaling the data pipeline. The paper's own limitations section notes that human annotation limits data size, and that maintaining quality while covering all policy transition types limits feasible task length. The authors state they are working on combining human-annotated rubrics with a fully automated synthetic environment scaling pipeline for both training and evaluation.
• Closing the long-range dependency gap. Long-range dependency tasks are the weakest dimension, with no model above 50% and the largest inter-model spread, making them an obvious target for improvement.
• Mitigating self-conditioning. The observed bias toward repeating a previous response style (tool call, parallel call) raises the question of how to prevent long conversational context from diluting attention to the current task.
• Transferring benchmark results into training. The paper frames WildToolBench as providing structured rubrics to help developers interpret user behavior from multiple perspectives, leaving open how those rubrics translate into targeted model iteration.

Target Audience

This paper is most useful for researchers and engineers building or evaluating tool-using LLM agents, benchmark designers working on multi-turn and multi-step evaluation, and product teams deploying assistant-style agents where real users issue compositional, vague, and rapidly shifting instructions. It is also relevant to model developers deciding between general-purpose, specialized, and reasoning-oriented architectures, and to HCI researchers interested in how real user behavior diverges from simulated users.

Authors’ abstract

Fulfilling user needs through Large Language Model multi-turn, multi-step tool-use is rarely a straightforward process. Real user interactions are inherently wild, being intricate, messy, and flexible. We identify three key challenges from user behaviour: compositional tasks that demand efficient orchestration of tool-call topologies, implicit intent spread across dialogue turns that require contextual inference, and instruction transition, which mixes task queries, clarifications, and casual conversation, forcing LLMs to adjust their policies on the fly. Existing benchmarks overlook these behaviors, making the apparent progress of LLMs on tool-use spurious. To address this, we introduce WildToolBench, an LLM tool-use benchmark grounded in real-world user behavior patterns. Comprehensive evaluations of 57 LLMs reveal that no model achieves an accuracy of more than 15%, indicating a substantial gap in the robustness of LLMs' agentic ability. Controlled experiments and in-depth analyses further indicate that the real challenge for LLM tool-use lies not in artificially complex tasks, but in the wild nature of user behavior, emphasizing the need to reconsider the interactions among LLMs, users, and tools.