MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use

AI summary

Overview

Research area: Evaluation benchmarks for LLM-based agents, specifically tool invocation via the Model Context Protocol (MCP).

Technical level: Intermediate. Readers should know roughly what an LLM agent is and what a tool call looks like, but the paper requires no specialized knowledge of protocol internals or training methods.

Scope: This paper introduces MCPAgentBench, a locally deployed, reproducibility-focused benchmark that measures not just whether LLM agents complete MCP tool-use tasks, but how efficiently they do so in the presence of distractor tools.

What This Paper Is About

Existing benchmarks for testing whether LLM agents can use MCP tools depend on live remote MCP servers, which makes results unstable and hard to reproduce, and they mostly measure only whether a task succeeded. The authors argue this misses a critical dimension: a model that eventually completes a task by brute-forcing many tool calls is far less useful in practice than one that plans the right calls in the right order with minimal tokens and time. MCPAgentBench addresses both problems by building a self-contained sandbox with locally hosted MCP servers and by adding efficiency-aware scoring metrics to the evaluation.

Key Contributions

1. Reproducible local MCP infrastructure. The authors collected over 20,000 MCP tools from public marketplaces and the HuggingFace MCP hackathon, then reconstructed 141 selected tools into 141 locally maintained MCP servers. This removes dependence on remote service uptime while preserving authentic tool definitions and protocol-compliant behavior.

2. A structured, difficulty-graded task suite. Starting from 841 raw tasks sourced from Infinity-Instruct and the Schema-Guided Dialogue dataset, and after LLM-assisted annotation plus manual curation, the authors produced 178 high-quality tasks with unique solutions. Tasks are organized across two domains (Daily, Professional) and four invocation-complexity levels (Single-tool, Dual-tool Serial, Dual-tool Parallel, Multi-tool).

3. New efficiency-oriented metrics. The benchmark introduces Task Finish Score (TFS), Task Efficiency Finish Score (TEFS), Token Efficiency, and Time Efficiency. TFS checks whether the correct set of tool calls was made, ignoring order. TEFS additionally requires the execution order—including serial versus parallel structure—to match the golden solution.

4. Distractor-injected evaluation. Each task ships with a dynamically generated candidate list of 20–30 tools that mixes the correct tools with functionally unrelated or easily confused ones, testing the agent's tool discrimination in a "needle in a haystack" setting.

Main Findings

• Completion and efficiency are not the same skill. Claude Sonnet 4.5, o3, and glm-4.6 led on TFS (71.6, 66.0, 65.1), but under the stricter TEFS metric the leaders shifted to Claude Sonnet 4.5, glm-4.6, and qwen3-235b-a22b-instruct-2507 (57.7, 54.4, 51.8). Nearly every model lost more than 10 points going from TFS to TEFS, and o3 dropped 28.5 points—the largest gap observed.

• Parallel tool calls are a widespread weak point. Dual Parallel tasks scored higher than Dual Serial tasks under TFS, suggesting serial logic is harder to get right. But under TEFS, Dual Parallel scores collapsed across the board, exposing that models frequently fail to execute genuinely parallel invocations. All OpenAI-series models tested, including gpt-5, o3, and o4-mini, scored exactly 0 on Dual Parallel TEFS.

• Models adopt opposite and equally flawed execution strategies. OpenAI models default to strict serialization, which is why they score zero on parallel tasks. Claude Sonnet 4.5 goes the other way, parallelizing aggressively—this earns it top Time Efficiency but causes it to over-parallelize Dual Serial tasks, producing an anomalous TEFS drop relative to its TFS.

• Task difficulty scales as expected. In both domains, average scores decline as the number of required tool invocations grows from Single to Multi. Professional tasks are consistently harder than Daily tasks.

• Token efficiency favors non-thinking models. qwen3-235b-a22b-instruct-2507 achieved the highest Token Efficiency, helped by its NoThinking design. gpt-5 recorded the lowest, indicating its extensive reasoning tokens did not translate into proportional score gains.

• Time efficiency favors parallelization. Claude Sonnet 4.5 ranked first, followed by glm-4.6 and qwen3-235b-a22b-instruct-2507, with gpt-5 last. The authors caution these results are sensitive to network latency and region and recommend using official APIs for reproduction.

• Scale helps; distractors hurt. Across the Qwen2.5 and Qwen3 model families, TEFS generally rises with model size (with an unexplained dip at Qwen2.5 32B). As the number of distractor tools increases from 10 to 20 to 30, TEFS declines modestly for all tested models.

Methodology in Plain English

The authors built the benchmark in four stages. First, they scraped real MCP tool definitions from public repositories and marketplaces, and collected real user requests from existing datasets. Second, they ran an LLM annotation pass to generate descriptive tags for tools and tasks, manually merged and cleaned those tags into a controlled vocabulary, then ran the LLM again constrained to that vocabulary so every item got consistent labels. Third, they matched tasks to tools by shared tags and manually curated each pair until the task had one unambiguous solution using only the intended tools. Fourth, they had an LLM generate Python stub functions implementing each tool, then had human experts review every function for correct signatures, correct logic, and sandbox safety.

At evaluation time, an Autogen-based sandbox loads each task along with a candidate tool list. That list contains the correct tools plus K−n distractor tools drawn from the main library, for a total of K tools (typically 20). The tested agent receives the task and the list, chooses tools, and emits call parameters. The sandbox records every call, then compares the recorded sequence against the stored golden solution. TFS asks whether the right calls were made; TEFS additionally asks whether they were made in the right order with the right serial/parallel structure. Efficiency metrics divide the earned score by tokens consumed or time elapsed. Each model was run four times per task and averaged (avg@4).

Why This Matters

Impact on research. The paper reframes MCP tool-use evaluation as a two-axis problem—correctness and efficiency—rather than a single correctness score. This matters because it reveals that models which look nearly equivalent on completion rates diverge sharply once you account for wasted calls, wasted tokens, and misused parallelism. It also provides a stable, local, protocol-faithful testbed, which lets other researchers reproduce and extend results without depending on third-party server availability.

Real-world applications:

• Agent framework selection. Teams choosing a backbone model for a production agent now have evidence that the top correctness model is not automatically the best efficiency choice, and that some widely used models cannot do parallel tool calls at all.

• Cost forecasting for agentic workloads. Token and time efficiency metrics translate directly into API billing and latency budgets, which is what actually constrains deployed agents.

• Robustness testing against tool confusion. The distractor mechanism mirrors real MCP registries, where an agent must pick from dozens of plausibly relevant tools it did not author.

• Regression testing during model upgrades. A stable local benchmark makes it practical to detect whether a new model version silently regresses on serial ordering or parallel execution.

Industry relevance. MCP is positioned as a standard interface between agents and external services, and vendors such as Anthropic and OpenAI are shipping it in production tooling. A benchmark that quantifies the failure modes of that interface—particularly the near-total failure of parallel invocation in several flagship models—gives platform vendors concrete targets and gives enterprises a way to validate claims before committing to an agent stack.

Future Directions

• Diagnosing the parallel-call deficit. The zero scores on Dual Parallel tasks for an entire model family suggest either a training-data gap or a protocol-handling limitation. Determining which would inform both model developers and MCP implementers.

• Reducing sensitivity to external conditions. The authors acknowledge that Time Efficiency varies with network latency and region. A fully offline or latency-normalized timing methodology would strengthen cross-laboratory comparisons.

• Richer golden solutions. Currently each task has one unique solution with a fixed serial/parallel structure. Real tasks often admit several valid plans with different efficiency profiles, so scoring would benefit from accepting equivalent alternative plans.

• Expanding scale and coverage. With 178 tasks and 141 tools, the benchmark is deliberately curated but small relative to the 20,000-tool corpus collected. Growing the suite—and adding adversarial or multi-turn tasks—would test whether the observed efficiency gaps persist.

• Aligning efficiency with task value. TEFS treats all extra or reordered calls as failures, but some redundancy is legitimate. Future metrics could weight redundant calls by actual cost rather than binary penalty.

Target Audience

This paper is most useful to AI engineers building or deploying tool-using agents, to researchers working on agent evaluation and benchmarking methodology, and to model developers who need diagnostic signal on tool-selection, planning order, and parallel-execution behavior. Product and infrastructure teams evaluating which LLM to put behind an MCP-enabled agent will also find the comparative results directly actionable. It assumes familiarity with agent terminology but not with the paper's internals.

Authors’ abstract

Large Language Models (LLMs) are increasingly serving as autonomous agents, and their utilization of external tools via the Model Context Protocol (MCP) is considered a future trend. Current MCP evaluation sets suffer from issues such as reliance on external MCP services and a lack of difficulty awareness. To address these limitations, we propose MCPAgentBench, a benchmark based on real-world MCP definitions designed to evaluate the tool-use capabilities of agents. We construct a dataset containing authentic tasks and simulated MCP tools. The evaluation employs a dynamic sandbox environment that presents agents with candidate tool lists containing distractors, thereby testing their tool selection and discrimination abilities. Furthermore, we introduce comprehensive metrics to measure both task completion rates and execution efficiency. Experiments conducted on state-of-the-art LLMs reveal significant performance differences in handling complex, multi-step tool invocations. All code is open-source at https://github.com/Brunestuder/MCPAgentBench.