MCP-Flow: Facilitating LLM Agents to Master Real-World, Diverse and Scaling MCP Tools

AI summary

Overview

Research area: Large language model (LLM) agents that use external tools via the Model Contextual Protocol (MCP) — specifically, automated construction of training data so agents can operate real-world MCP servers.

Technical level: Intermediate. The paper assumes familiarity with LLM agents, function calling, fine-tuning (LoRA), and retrieval augmentation, but the pipeline itself is described conceptually.

Scope (one sentence): MCP-Flow is an automated web-agent-driven pipeline that discovers MCP servers, synthesizes and filters instruction–function-call data from them, and uses that data to train and augment LLM agents for real-world MCP tool use.

What This Paper Is About

LLM agents increasingly depend on external tools, and the MCP ecosystem of servers is growing quickly, but existing MCP research covers only a handful of servers, relies on costly manual curation, and provides no data for training models. MCP-Flow addresses this by automatically collecting servers and tools from MCP marketplaces and generating a large-scale, filtered dataset of instructions and function calls. The goal is to let both small fine-tuned models and large non-trainable models use real-world MCP tools more accurately and at lower cost.

Key Contributions

1. An automated pipeline for server and tool collection. MCP-Flow uses a web agent (Playwright) to navigate MCP marketplaces and retrieve server configuration files without site-specific parsing code. The authors state this design generalizes across platforms and requires only incremental updates for newly released servers rather than rerunning the whole process.

2. A large-scale, high-quality dataset. The released dataset covers 1,166 servers and 11,536 tools, containing 68,733 instruction–function-call pairs and 6,439 trajectories, with 356 servers producing valid responses and trajectories. The authors report this exceeds the scale of all previous MCP-related work combined, and the dataset supports training, evaluation, and retrieval augmentation.

3. A compact fine-tuned model suite. Three backbones (Qwen3-0.6B, Qwen3-4B, Llama3.1-8B) are fine-tuned with LoRA, and the authors report they outperform substantially larger models on MCP tool selection and function-call formatting.

4. Retrieval augmentation and agentic-task evaluation. The dataset also serves as a retrieval database (top-k set to 5) for closed-ended models such as GPT-4o, and is used to generate the initial function call for the GAIA agentic benchmark.

Main Findings

• SOTA models underperform on real-world MCP tools. In the 10-tool setting (Table 2, seen test), GPT-4o scores 88.6 Tool / 68.2 Param / 58.8 AST, Claude-4-Sonnet scores 85.8 / 68.6 / 56.6, and Gemini-2.5-Pro scores 54.2 / 42.8 / 36.8. The paper states current SOTA LLMs have less than 60% AST accuracy on real-world MCP tools.

• Performance collapses as the candidate tool set grows. With 100 tools (Table 3, averaged over three test splits), GPT-4o reaches 72.3 Tool / 66.9 Param / 53.8 AST, Claude-4-Sonnet reaches 68.3 / 63.3 / 51.6, and Groq-8B-Tool-Use reaches only 2.9 / 1.4 / 1.3. Groq-8B-Tool-Use is noted to often predict "I can't help."

• Small fine-tuned MCP-Flow models outperform much larger models. With 10 tools, MCP-Flow-Qwen3-0.6B scores 96.8 / 87.2 / 75.4 on seen test, and MCP-Flow-Qwen3-4B scores 99.2 / 91.8 / 81.2. MCP-Flow-Llama3.1-8B scores 98.6 / 91.0 / 81.6 on seen test and 99.4 / 77.0 / 65.2 on unseen server. With 100 tools, MCP-Flow-Qwen-4B reaches 81.7 / 82.1 / 67.0 versus 61.7 / 59.8 / 49.0 for the Qwen3-4B baseline.

• The unseen-server split is harder than the unseen-tool split. Unseen-tool subsets share servers with the seen test and show similar performance, while nearly all models drop on unseen-server. For example, Claude-4-Sonnet falls from 85.8 / 68.6 / 56.6 (seen test) to 72.6 / 56.0 / 48.4 (unseen server).

• Dataset scale matters. Models fine-tuned on MCPToolBench++ (e.g., Qwen3-4B at 91.4 / 77.2 / 62.2 on seen test) are described as limited in scale and coverage, yielding limited improvement relative to MCP-Flow.

• Retrieval augmentation helps closed-ended models. Adding retrieved exemplars from MCP-Flow consistently improves model performance; Claude-4-Sonnet gains more than GPT-4o. Despite this, the paper reports these large models still underperform MCP-Flow, especially on unseen-server.

• MCP-Flow improves agentic task performance on GAIA. On GAIA, Qwen3-4B's success rate rises from 10.68 to 21.36 with MCP-Flow (reported +100%), GPT-4o from 29.13 to 33.98 (+17%), and Claude-4-Sonnet from 55.34 to 57.28 (+4%). Weighted Step values reported are 1.88 to 2.01 (–7%) for Qwen3-4B, 3.07 to 1.92 (+32%) for GPT-4o, and 6.01 to 5.29 (+12%) for Claude-4-Sonnet.

• Larger training tool sizes improve robustness. Models trained with larger tool sizes perform better when evaluated with more candidate tools, though performance decreases on test sets with more candidate tools overall.

• Scaling behavior differs by metric. Tool accuracy quickly plateaus near 100% as data scales, while AST accuracy still shows room for improvement.

• Human annotation confirms low label noise. Over 300 randomly sampled instances (100 per test split) reviewed by two annotators, correctness was 99.00 (seen test), 96.00 (unseen tool), and 97.25 (unseen server), with correlation error of 2.00, 2.50, and 2.00 respectively — averaging over 96% correctness and noise at or below 2.5%.

• Servers targeting the same task differ widely. In a weather-task case study across six weather-related MCP servers with 13 test instructions and GPT-4o as both execution agent and judge, success rates ranged from 23.1 to 84.6, with differing coverage, token use, and monthly call counts (recorded from Smithery as of September 16, 2025).

Methodology in Plain English

MCP-Flow has two stages.

Server discovery and tool collection. A web agent (Playwright) navigates six MCP marketplaces — Smithery, Glama, MCP.so, MCPHub, PipeDream, and PulseMCP — and pulls each server's JSON configuration from page snapshots. Because the agent follows high-level instructions instead of site-specific HTML parsing rules, it adapts to new marketplaces with minimal human modification, and the pipeline can be run incrementally for newly released servers. Duplicate servers are removed by comparing tool descriptions rather than names or providers: two servers with an identical list of tool descriptions are treated as the same entity. Servers are then deployed locally through an MCP client (built on the dolphin-mcp repository), using npm and uvx for stdio-based servers and URLs for SSE servers. Servers requiring API keys or proprietary software are excluded.

Data synthesis. For each tool, the model generates five distinct instructions from human-curated examples. A slot-filling step ensures every parameter required by the tool is populated with a valid value, and regular-expression rules replace any remaining placeholders. WizardLM evolution then rewrites instructions to add complexity and diversity, with evolution depth set to 2. The ground-truth tool, its input schema, and the instruction are given to GPT-4o to produce the formalized function call, and tool responses are collected by communicating with the locally deployed server to form complete trajectories.

Filtration. Three filters are applied: (1) instructions whose embedding similarity to the tool description exceeds 0.8 are discarded, since they make tool selection trivial; (2) GPT-4o and DeepSeek-V3 must both identify the correct tool from the labeled tool plus two random candidates, otherwise the sample is discarded; (3) DeepSeek-V3 scores instruction and function-call quality, and anything below 6/10 is discarded.

Training and evaluation. Data is split per marketplace: servers at 12:1 (held-out = unseen-server), tools within seen servers at 11:1 (held-out = unseen-tool), and remaining samples at 10:1 (train versus seen test), giving 6 marketplaces × 4 splits = 24 subsets. Dataset statistics list 1,166 servers, 11,536 tools, 52,169 training samples, 5,216 seen-test samples, 5,249 unseen-tool samples, 6,099 unseen-server samples, and 6,439 trajectories. Fine-tuning uses LoRA via LLaMA-Factory, mostly with training tool size 10 and one epoch. Metrics are tool accuracy (Tool), parameter accuracy (Param), and abstract syntax tree accuracy (AST), plus GPT-4o as judge for task success rate (SR) and efficiency metrics.

Why This Matters

Impact on research. The paper argues that no large-scale, high-quality MCP dataset previously existed, and that prior MCP studies mostly functioned only as evaluation platforms. MCP-Flow reframes data construction as a first-class step, providing training, evaluation, and retrieval resources in one pipeline, and it introduces a web-agent-based collection approach that removes the dependency on human-curated crawling code.

Real-world applications (drawn from the paper's own use cases):

• Agents that must choose the correct tool from a large candidate pool of MCP tools, including tools they were never trained on.
• Retrieval augmentation for closed-source models, letting models that cannot be fine-tuned improve by retrieving similar examples at inference time.
• Multi-step agentic workflows such as GAIA tasks, where MCP-Flow generates the initial function call and steers the agent toward tool use rather than answering from internal knowledge.
• Comparative evaluation of MCP servers that serve the same purpose, illustrated by the weather case study, where servers differ in response quality, capability coverage, efficiency, and popularity.

Industry relevance. MCP-Flow is a collaboration involving TikTok and Shanghai Jiao Tong University, and it targets practical deployment concerns: lowering inference cost, reducing wasted attempts on unavailable servers, and producing compact models that match or exceed much larger commercial models on MCP-specific tasks. The paper also notes that MCP marketplaces currently lack mechanisms to detect malicious servers, which is a direct operational concern for anyone deploying agents against public tool ecosystems.

Future Directions

1. Extending coverage to API-key-gated and software-specific servers. The authors identify these two categories as the most significant challenge for automated construction, since deployment and key-provisioning procedures vary widely across providers and are hard to standardize.

2. Detecting and defending against adversarial MCP servers. The paper warns that malicious actors can upload or update servers, inject misleading information, or return manipulated outputs, and states that markets do not currently implement explicit detection mechanisms. It proposes using MCP-Flow's platform for attack and defense research.

3. Systematic evaluation of MCP servers and tools themselves. Suggested directions include unified benchmarks that test the same task across functionally similar tools, automated stress tests for stability, reliability, and latency, richer metadata schemas such as structured capability statements, and reinforcement learning or multi-agent comparison to rank tools by observed performance rather than static descriptions.

4. Addressing long-term reliability and the multi-turn gap. The limitations section notes that servers may fail, degrade, or be deprecated, that a gap remains between synthesized tasks and messy real-world user behavior, and that MCP-Flow currently prioritizes single-turn tool invocation rather than iterative, context-aware multi-turn workflows.

Target Audience

Researchers and engineers working on LLM agents and tool use, particularly those building or evaluating MCP-based systems. It is also relevant to practitioners who need to fine-tune small open models for tool calling, teams designing retrieval-augmented tool-use pipelines for closed-source models, and anyone comparing MCP servers for reliability, quality, and cost. Readers focused on agentic benchmarks will find the GAIA results useful, while those focused on datasets will find the scale and split design most relevant.

Authors’ abstract

Large Language Models (LLMs) increasingly rely on external tools to perform complex, realistic tasks, yet their ability to utilize the rapidly expanding Model Contextual Protocol (MCP) ecosystem remains limited. Existing MCP research covers few servers, depends on costly manual curation, and lacks training support, hindering progress toward real-world deployment. To overcome these limitations, we introduce MCP-Flow, an automated web-agent-driven pipeline for large-scale server discovery, data synthesis, and model training. MCP-Flow collects and filters data from 1166 servers and 11536 tools, producing 68733 high-quality instruction-function call pairs and 6439 trajectories, far exceeding prior work in scale and diversity. Extensive experiments demonstrate MCP-Flow's effectiveness in driving superior MCP tool selection, function-call generation, and enhanced agentic task performance. MCP-Flow thus provides a scalable foundation for advancing LLM agents' proficiency in real-world MCP environments. MCP-Flow is publicly available at https://github.com/wwh0411/MCP-Flow.