MedMCP-Calc: Benchmarking LLMs for Realistic Medical Calculator Scenarios via MCP Integration

AI summary

Overview

Research area: Evaluation of large language model (LLM) agents on medical calculator workflows, using the Model Context Protocol (MCP) to connect models to real clinical tools and databases.

Technical level: Advanced — assumes familiarity with LLM agents, tool calling, MCP, SQL, POMDP formalisms, and benchmark design.

Scope: The paper introduces MedMCP-Calc, the first MCP-based benchmark for realistic, multi-step medical calculator scenarios, evaluates 23 models on it, and presents a fine-tuned model (CalcMate) that improves performance through scenario planning and tool augmentation.

What This Paper Is About

Medical calculators turn patient data into risk scores and treatment thresholds, but in real practice clinicians do not receive clean, pre-packaged case text with instructions on which calculator to use. They must pull fragmented data from electronic health records (EHRs), decide which calculators fit an evolving clinical picture, and chain several calculations together. Existing medical calculator benchmarks test models on static, single-step computations with explicit instructions. MedMCP-Calc closes this gap by building 118 fuzzy, workflow-level tasks over an interactive EHR database, web search, and a Python execution environment, all delivered through MCP servers.

Key Contributions

1. MedMCP-Calc benchmark. The first MCP-integrated benchmark for medical calculator workflows, containing 118 scenario tasks across 4 clinical domains, with fuzzy task descriptions, an interactive PostgreSQL EHR database built from MIMIC-IV data, Google Search and Python Executor servers, and process-level evaluation.

2. Evaluation of 23 leading models. A comprehensive study spanning proprietary LLMs (Claude Opus 4.5, Claude Sonnet 4.5, GPT-5, Gemini-3-Pro-Preview, Gemini-2.5-Pro), open-source LLMs (GLM-4.7, DeepSeek-V3.1, Kimi-K2-Instruct, the Qwen3 series, Llama-3.3, Llama-3.1), and medical models (Baichuan-M2-32B, MedGemma-27B-IT), revealing systematic failures in calculator selection, SQL-based data retrieval, and tool use.

3. CalcMate. A fine-tuned model built from Qwen3-4B-Instruct-2507 that combines scenario-planning training with aggressive tool augmentation, achieving state-of-the-art performance among open-source models and surpassing both models used to generate its training data.

4. Evidence that tool avoidance, not just knowledge gaps, drives failure. A tool-usage analysis showing that most models invoke SQL only 3 to 5 times and call Python fewer than once on average, while CalcMate's heavier tool engagement correlates with large metric gains.

Main Findings

• Top proprietary models lead but remain far from solved. Claude Opus 4.5 achieves the best overall scores (Task Fulfillment 100, Calculator Selection 66.66, Evidence Acquisition 71.08, Quantitative Precision 33.92), followed by Claude Sonnet 4.5 (CS 65.60, EA 53.86, QP 27.80), GPT-5 (CS 59.81, EA 53.12, QP 26.70), Gemini-3-Pro (CS 54.05, EA 55.45, QP 26.49), and Gemini-2.5-Pro (CS 55.96, EA 40.45, QP 20.77). The paper states that even Claude Opus 4.5 reaches only 33.92% calculation accuracy.

• Most models struggle to pick the right calculators from fuzzy queries. Apart from Claude Opus 4.5 (66.66), Claude Sonnet 4.5 (65.6), and GPT-5 (59.81), most models — including medical-specialized ones — cluster around 30 on Calculator Selection. MedGemma-27B-IT scores 30.33 CS with 3.61 QP; Baichuan-M2-32B scores 33.95 CS but has a low Task Fulfillment of 79.66.

• Iterative SQL interaction is a weak point. Models plan poorly and are not robust when composing queries for complex task intents. Without explicit schema inspection, models hallucinate table structures and persist in erroneous queries despite failure signals. The authors added list_tables and describe_tables tools to improve robustness.

• Models avoid tools for numerical computation. Python usage averages below one call for most models; GPT-5 records 0.06 Python calls per task, and CalcMate-4B's source model Qwen3-4B-Instruct-2507 records 0.13. Claude Opus 4.5 is a notable exception at 3.93. Search and Fetch usage approaches zero for several models, with Qwen3-235B-Instruct-2507 recording zero for both. Computational hallucinations and arithmetic errors appear across small and medium-sized models.

• Extended thinking improves calculation accuracy but can hurt evidence gathering. All thinking models achieve higher Quantitative Precision than their non-thinking counterparts. However, thinking mode degrades Evidence Acquisition — for example, Qwen3-235B drops from 38.26 to 30.60 — which the authors attribute to over-reliance on internal knowledge rather than external tools.

• Performance varies sharply by clinical domain. All models score higher in Critical Care & Perioperative Medicine and Internal Medicine & Organ Systems than in Neurology & Psychiatry and Special Populations & Universal Tools. The latter involve complex scoring systems and cognitive scales, where QP scores often fall below 10%.

• Task Fulfillment is near-saturated. Mainstream models show near-perfect Task Fulfillment after minimal post-processing, indicating that multi-turn ReAct-style interaction in clinical scenarios is no longer the bottleneck.

• CalcMate delivers broad gains. CalcMate-30B reaches Task Fulfillment 100, CS 65.66, EA 75.31, and QP 20.07, while CalcMate-4B reaches TF 100, CS 56.97, EA 71.06, and QP 15.02 — up from an EA of 9.34 for its base model. CalcMate also invokes tools far more often: CalcMate-4B averages 7.97 SQL, 7.32 Python, 8.16 Search, and 8.07 Fetch calls, versus Claude Opus 4.5 at 5.19 SQL, 3.93 Python, 0.08 Search, and 0.08 Fetch.

Methodology in Plain English

The researchers first worked with physicians to define a taxonomy of 4 clinical domains covering 125 distinct clinical use categories for medical calculators. They collected 1,970 calculators from MDCalc, Medscape, and QxMD, used GPT-OSS-120B for two rounds of semantic similarity screening plus manual filtering to reduce this to 1,173 unique calculators, and then used DeepSeek-V3.1 to assign calculators to each of the 125 scenarios.

For each scenario, GPT-5 proposed a coherent sequence of calculators representing the clinical workflow; these sequences were manually reviewed. GPT-5 then converted each workflow into a naturalistic, underspecified task prompt — for example, asking for risk stratification rather than naming the HEART score. One task was created per scenario.

To create the data environment, the team modeled a relational schema based on real hospital EHR systems with nine tables, extracted and filtered records from MIMIC-IV, and selected patients with complete records across all tables, yielding a cohort of 49,419 patients. GPT-5 synthesized task-specific patient data, and rule-based code generation produced SQL insert statements. Patient information was stripped from the task questions so models must retrieve it themselves.

Quality verification involved manual checks of task questions against real clinical workflows and item-by-item verification of calculator names, outputs, and input parameters against the three calculator websites as ground truth. After excluding inadequate data, 118 tasks remained.

Evaluation uses the ReAct agent framework with DeepSeek-V3.1 for structured result extraction. GPT-OSS and MedGemma-4B-IT were excluded because of poor instruction following. Four metrics are computed: Task Fulfillment, Calculator Selection, Quantitative Precision, and Evidence Acquisition. The task is formalized as a POMDP with planning actions and tool actions over MCP servers.

CalcMate was trained using 1,000 scenario tasks generated by GPT-5 along with global planning reasoning, plus an aggressive tool augmentation strategy where each calculator computation follows the ReAct format with reference lookup, database exploration and querying, and code-assisted computation. The run_sql training data came from GPT-5; all other steps were constructed using Qwen3-235B-A22B-Instruct-2507-FP8.

Why This Matters

Impact on research. The paper shifts medical calculator evaluation from static reading comprehension toward agentic, tool-driven clinical workflows. It shows that near-perfect instruction following does not imply competence at planning, evidence acquisition, or numerical accuracy — and it provides process-level metrics that separate these failure modes.

Real-world applications:

• Emergency triage and risk stratification. The paper's motivating example chains initial stability screening such as NEWS2, risk stratification via the HEART Pathway, mortality prediction via GRACE, and procedure-risk assessment via Mehran before angiography.
• Automated EHR data gathering for scoring. Agents that query hospital databases directly, rather than requiring a clinician to assemble inputs by hand.
• Real-time reference retrieval. Using search tools to pull current calculator definitions and clinical guidelines instead of relying on memorized formulas.
• Precise dosage and quantitative computation. Offloading arithmetic to a Python executor to reduce computational hallucination.

Industry relevance. The findings argue against deploying general-purpose clinical agents without safeguard design. The paper explicitly states that MedMCP-Calc is an academic evaluation framework, not a deployable clinical decision support system, and that decision-making authority must remain with qualified healthcare professionals. The persistent tool avoidance and calculation errors documented here set a concrete bar that clinical AI products should be measured against before deployment.

Future Directions

• Reducing region-specific bias and EHR heterogeneity. The authors note that current data association patterns may reflect region-specific biases and cannot capture the diversity of EHR systems across global healthcare settings.
• Lowering the cost of tool augmentation. CalcMate's proactive tool augmentation significantly increases context length, raising inference latency and computational overhead. The paper suggests knowledge distillation or long-context compression, particularly for resource-constrained clinical environments.
• Domain-specific optimization. Neurology & Psychiatry and Special Populations & Universal Tools lag substantially behind the other two domains, and QP scores often fall below 10% there, pointing to a need for targeted work on complex scoring systems and cognitive assessment scales.
• Improving SQL robustness and tool adherence. Models still hallucinate schemas, persist with failed queries, and avoid external computation tools; the paper treats these as open challenges rather than solved problems.

Target Audience

Researchers building or evaluating LLM agents for clinical reasoning and tool use; benchmark designers working on MCP-based or multi-step agent evaluation; developers of clinical decision support and hospital EHR integration systems; and medical AI teams interested in fine-tuning for workflow planning and tool adoption. Clinicians involved in validating AI-assisted calculation tools will also find the failure-mode analysis relevant.

Authors’ abstract

Medical calculators are fundamental to quantitative, evidence-based clinical practice. However, their real-world use is an adaptive, multi-stage process, requiring proactive EHR data acquisition, scenario-dependent calculator selection, and multi-step computation, whereas current benchmarks focus only on static single-step calculations with explicit instructions. To address these limitations, we introduce MedMCP-Calc, the first benchmark for evaluating LLMs in realistic medical calculator scenarios through Model Context Protocol (MCP) integration. MedMCP-Calc comprises 118 scenario tasks across 4 clinical domains, featuring fuzzy task descriptions mimicking natural queries, structured EHR database interaction, external reference retrieval, and process-level evaluation. Our evaluation of 23 leading models reveals critical limitations: even top performers like Claude Opus 4.5 exhibit substantial gaps, including difficulty selecting appropriate calculators for end-to-end workflows given fuzzy queries, poor performance in iterative SQL-based database interactions, and marked reluctance to leverage external tools for numerical computation. Performance also varies considerably across clinical domains. Building on these findings, we develop CalcMate, a fine-tuned model incorporating scenario planning and tool augmentation, achieving state-of-the-art performance among open-source models. Benchmark and Codes are available in https://github.com/SPIRAL-MED/MedMCP-Calc.