The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution

AI summary

Overview

Research area: Natural Language Processing — evaluation of tool-using language agents (agentic benchmarks built on the Model Context Protocol).

Technical level: Intermediate. The paper is readable without deep systems knowledge, though familiarity with language agents, tool calling, and benchmarks helps.

Scope: The paper introduces Toolathlon, a benchmark of 108 tasks spanning 32 real software applications and 604 tools, designed to measure how well language agents handle diverse, realistically initialized, long-horizon, multi-application workflows.

What This Paper Is About

Existing benchmarks for tool-using language agents tend to cover narrow domains, simplified tasks, or artificial environment states, so they do not capture the diversity, realism, and long-horizon complexity of real work. The authors build Toolathlon to close that gap: an agent might need to monitor a production database like BigQuery, find a PDF operations manual, and send emails based on what that manual says. The goal is a benchmark where tasks are drawn from realistic user demands, start from realistic software states, and are graded by deterministic execution-based scripts rather than LLM judges.

Key Contributions

1. A diverse, multi-application benchmark. Toolathlon contains 108 tasks across 32 software applications and 604 tools, spanning domains from daily affairs and education to technology and finance, including systems such as Google Calendar, Notion, WooCommerce, Kubernetes, Snowflake, and BigQuery.

2. Realistic environment construction, including initial states. Rather than starting from empty or synthetic states, 72 of the 108 tasks (67%) come with state initialization scripts or pre-set workspace files. Because simulating states like a Canvas course with dozens of students requires scalable control, the authors deploy open-source software locally in containers — Poste.io for email, Canvas for course administration, Kubernetes for cluster orchestration, and WooCommerce for e-commerce.

3. Reliable, execution-based, containerized evaluation. Each task ships with a hand-crafted deterministic evaluation script that checks final environment states against static snapshots or dynamically retrieved ground truth. Tasks run isolated in separate containers, enabling parallel evaluation — running Claude-4.5-Sonnet on all 108 tasks took about 70 minutes of wall time on a standard Ubuntu 24.04 cluster with 16 CPUs and 64 GB of memory using 10 parallel processes.

4. Fuzzy, realistic task prompts. Instructions mirror authentic user queries, which are concise and ambiguous; the intended behavior must be inferred from the environment state, so models must plan rather than follow a step-by-step script. Contrast this with benchmarks that spell out detailed plans, which reduce the planning role of the agent.

5. A comprehensive model evaluation with failure-mode analysis. The paper benchmarks proprietary models (GPT-5 family, o3/o4-mini, Claude-4-Sonnet, Claude-4.5-Sonnet, Claude-4.5-Haiku, Gemini-2.5-Pro/Flash, Grok-4, Grok-4-Fast, Grok-Code-Fast-1) and open-weight models (Qwen-3-Coder, DeepSeek-V3.2-Exp, Kimi-K2-0905, GLM-4.6), and analyzes tool-calling errors, long-context behavior, cost, and output token usage.

Main Findings

• The best model scores below 40%. Claude-4.5-Sonnet ranks first with a pass@1 success rate of 38.6% (± 2.7), using 20.2 tool calling turns on average. The paper's comparison table reports an average of 26.8 turns per task when measured with Claude-4-Sonnet as a proxy for task complexity.

• Open-weight models lag clearly. DeepSeek-V3.2-Exp is the top open-weight model at 20.1% (± 1.2). GLM-4.6 reaches 18.8% (± 2.2), Qwen-3-Coder 14.5% (± 1.9), and Kimi-k2-0905 13.0% (± 2.0).

• A clear second tier exists among proprietary models. GPT-5 (30.6% ± 1.5), Claude-4-Sonnet (29.9% ± 1.6), GPT-5-high (29.0% ± 3.1), Grok-4 (27.5% ± 1.7), and Claude-4.5-Haiku (26.2% ± 1.9) all sit above 26% but below 30%; every other model is at or below 20%.

• More internal reasoning does not uniformly help. Increasing reasoning effort (GPT-5 vs. GPT-5-high) shows no benefit, which the authors interpret as evidence that exploring new observations matters more than extended internal reasoning in agentic tasks.

• Reliability differs sharply from capability coverage. Large gaps between pass@3 and pass^3 across models indicate that many models can occasionally solve tasks but cannot do so consistently.

• Domain strengths are uneven. Claude-4.5-Sonnet performs strongly across almost all domains and especially on Campus and E-Commerce tasks; GPT-5 performs notably well on Daily tasks; Grok-4 stands out on Tech tasks.

• Longer tasks are harder, but not monotonically. Splitting tasks into equally sized Easy, Medium, and Hard groups by average turns, success rates generally drop as turns increase. However, both Claude-4.5-Sonnet and GPT-5 score higher on the Hard subset than on the Medium one, which the authors suggest points to premature task termination rather than pure multi-step execution length as a difficulty source.

• Overlong tool outputs hurt most models. Across models, roughly 15% to 35% of trajectories encountered overlong tool outputs, and most models showed a decline in success rate when that happened — even on logically straightforward tasks such as price comparison or data extraction.

• Tool errors matter differently by type. All models produced tool execution errors to varying degrees, with no significant correlation between those errors and overall success. Hallucinating non-existent tools (wrong tool names), by contrast, is more likely to affect final scores, and leading models produce few such errors. Error messages themselves can help models adjust in later turns.

• Agentic ability is not proportional to spend. Claude-4-Sonnet and Grok-4 incur relatively high expenses, while most other models stay under $1 per task with prompt caching enabled. Claude-4.5-Sonnet has the highest performance but ranks third in cost. Grok-4-Fast, Grok-Code-Fast-1, and DeepSeek-V3.2-Exp cost little, making them plausible choices under tight budgets.

• Output token profiles vary. Most models cluster between 5K and 10K output tokens. Reasoning-focused models such as o4-mini and GPT-5(-high) generate more, while the Claude series and Grok-4 achieve strong results with fewer tokens, suggesting heavier reliance on environment observation than on internal reasoning.

• Gemini-2.5 models struggle most. Gemini-2.5-Pro reaches 10.5% (± 1.9) and Gemini-2.5-Flash 3.7% (± 1.5) with the lowest output token counts; the authors attribute this to insufficient requirement understanding and proactive exploration, with models neglecting requirements or giving up early.

Methodology in Plain English

The authors treat each task as a partially observable Markov decision process: the agent sees observations, takes actions by calling tools, and the tool implementations define how the environment state changes. Rewards are not judgments of the agent's reasoning — they come from checking the resulting environment state.

To build the benchmark, the authors, who are computer science researchers and senior undergraduates, first picked a list of valuable, common real applications and looked for open-source MCP servers for them. Where none existed, or where existing servers had bugs and missing tools, they implemented or revised servers themselves, ending with 32 MCP servers. Task sourcing happened in two stages: an open round across an initial server list covering more than 50 applications, then a targeted round for important but underrepresented applications. Each task went through multiple rounds of quality checking by 5–6 experienced authors across several weeks.

The environments mix real remote services (Google Sheets, Google Calendar, Notion, Gmail) with locally containerized open-source software that permits scalable account and state setup. Implementing a single task took, on average, 4–6 hours of work by a research graduate student in computer science, and each round of quality checking took roughly 5 hours of labor per task.

For evaluation, each task specifies only the necessary MCP servers (fewer than 10) rather than all 32, but all tools within a selected server are loaded — matching mainstream agent frameworks and forcing the agent to ignore distracting tools. Each task is preconfigured with an average of 69.9 tools (minimum 28, maximum 128). The agent framework is built on the OpenAI Agents SDK (v0.0.15) with added handling for tool errors (errors are returned as observations instead of breaking the loop), overlong tool responses (truncated to 100K characters, with cached raw output searchable via paging at a 10K character page size), and context history management (models can inspect token counts and drop turns; as a safeguard, everything except the last 10 turns' preview and the initial user input is cleared on overflow). The framework also adds common local tools: Python execution, Web Search, Done, and Sleep, alongside 7 local toolkits containing 16 tools. The maximum number of turns allowed is 100, and each model is run three times.

Why This Matters

Impact on research. Toolathlon argues that benchmark realism should extend beyond tool interfaces to environment state diversity and prompt fuzziness, and it provides an execution-based, container-isolated evaluation harness. Its reported ceiling of 38.6% gives the community a concrete, difficult target and a set of diagnosed failure modes (context handling, wrong tool names, premature termination) to work against.

Real-world applications (as described in the paper):

• Enterprise operations monitoring: an administrative agent monitoring a real Snowflake or BigQuery database for customer tickets, locating the correct PDF operation manual, and emailing managers and customers according to that manual.
• Personal and team coordination: managing emails while coordinating with calendars and file systems.
• Education administration: operating on a Canvas course with dozens of students, including managing course state.
• E-commerce management: handling online storefront operations through WooCommerce.
• HR and recruitment: updating candidate information in a Notion HR record based on a set of resumes, checking whether applied-for positions are open.
• Technical operations: cluster management via Kubernetes and developer workflows via Git and GitHub.
• Financial analysis: working with real-world financial spreadsheets and live market information such as shareholder data.

Industry relevance. The benchmark is built on Model Context Protocol servers, the emerging standard for connecting agents to applications, so results speak directly to the ecosystem practitioners are building against. The paper also reports cost per task and token usage, which matters for teams choosing models under budget constraints — the finding that several low-cost models are competitive is relevant to production deployment decisions. The authors state that the benchmark and environment are fully open-sourced, and that they will release the MCP servers they developed.

Future Directions

• Improve long-context and overlong-output handling. Most models degrade when tool responses are overlong, and the paper identifies long-context modeling as a major challenge across the board; better retrieval, paging, and summarization strategies are open problems.
• Reduce premature task termination. The higher scores on the Hard subset than the Medium one suggests models stop before sufficiently exploring available observations; teaching agents to persist and explore could raise success rates.
• Make agent behavior consistent, not just occasionally correct. The gap between pass@3 and pass^3 indicates reliability, not raw capability, is a bottleneck for real-world deployment.
• Investigate tool-name hallucination. Incorrect tool names correlate with lower scores more than execution errors do, despite execution errors being common.
• Expand environment and application coverage. The authors note that current MCP server coverage is still insufficient and that they developed new servers themselves to extend into more domains, implying further expansion is expected as MCP adoption grows.
• Close the proprietary/open-weight gap. Open-weight models top out at 20.1%, leaving headroom for open-model agent development.

Target Audience

This paper is most useful to researchers and engineers building or evaluating tool-using language agents, particularly those working with MCP-based architectures. It also serves model developers seeking a difficult, execution-verified benchmark for agentic capability, and product teams deciding which models to deploy under cost and reliability constraints. Readers interested in benchmark design — especially realistic state initialization and deterministic, script-based evaluation — will find the environment construction approach instructive.

Authors’ abstract

Real-world language agents must handle complex, multi-step workflows across diverse Apps. For instance, an agent may manage emails by coordinating with calendars and file systems, or monitor a production database to detect anomalies and generate reports following an operating manual. However, existing language agent benchmarks often focus on narrow domains or simplified tasks that lack the diversity, realism, and long-horizon complexity required to evaluate agents' real-world performance. To address this gap, we introduce the Tool Decathlon (dubbed as Toolathlon), a benchmark for language agents offering diverse Apps and tools, realistic environment setup, and reliable execution-based evaluation. Toolathlon spans 32 software applications and 604 tools, ranging from everyday platforms such as Google Calendar and Notion to professional ones like WooCommerce, Kubernetes, and BigQuery. Most of the tools are based on a high-quality set of Model Context Protocol (MCP) servers that we may have revised or implemented ourselves. Unlike prior works, which primarily ensure functional realism but offer limited environment state diversity, we provide realistic initial environment states from real software, such as Canvas courses with dozens of students or real financial spreadsheets. This benchmark includes 108 manually sourced or crafted tasks in total, requiring interacting with multiple Apps over around 20 turns on average to complete. Each task is strictly verifiable through dedicated evaluation scripts. Comprehensive evaluation of SOTA models highlights their significant shortcomings: the best-performing model, Claude-4.5-Sonnet, achieves only a 38.6% success rate with 20.2 tool calling turns on average, while the top open-weights model DeepSeek-V3.2-Exp reaches 20.1%. We expect Toolathlon to drive the development of more capable language agents for real-world, long-horizon task execution.