MMedAgent-Plus: Scaling Multimodal Medical Agents across Modalities, Tasks and Workflows

AI summary

Overview

Research area: Multimodal medical AI, vision-language agents, and tool orchestration (Natural Language Processing / clinical machine learning). Technical level: Intermediate — readers should know roughly what vision-language models and tool-calling agents are, but the paper's core idea is intuitive. Scope: The paper builds and evaluates MMedAgent-Plus, a vision-language planner that coordinates 18 specialist medical models across 12 tasks and 8 modalities, and reports separate measurements of tool selection, response synthesis, and end-to-end workflow execution.

What This Paper Is About

Medical AI is split between specialist models that are excellent at one narrow task and generalist vision-language models that cover many tasks but perform worse on each. Neither kind is built to decide which analyses a case needs, run them in sequence, and hand intermediate results from one step to the next. The authors build MMedAgent-Plus, which pairs a vision-language planner with a library of 18 leading publicly available specialist models and trains the planner to call the right specialists in the right order and write a final answer grounded in their outputs.

Key Contributions

1. A tool-orchestrating medical agent. MMedAgent-Plus couples a vision-language planner with 18 specialist models spanning 12 medical tasks and 8 modalities (text, CT, MRI, X-ray, ultrasound, retinal fundus, OCT, and pathology), each chosen as the leading publicly available model for its task based on reported benchmark performance. All tools share a unified interface declaring inputs, parameters, and outputs, so architecturally different models are invoked through one call format.
2. A large instruction-tuning corpus with three workflow patterns. The planner is tuned on 208,928 samples drawn from 32 public medical benchmarks, organized into single-tool multi-round, multi-tool multi-round, and multi-tool single-round dialogues.
3. A staged evaluation design. Rather than reporting one aggregate score, the authors evaluate tool orchestration on 1,500 held-out dialogues, response synthesis on 600 cases with correct specialist outputs supplied, and end-to-end execution on 350 cases — separating failures in tool selection from failures in response generation.
4. An empirical demonstration that orchestration preserves specialist performance. In end-to-end execution the system invokes the required specialist in 90.6% of 350 cases and reaches a final-answer accuracy of 0.98, matching the dedicated specialist and far exceeding the strongest direct-answer baseline (0.28).

Main Findings

• Tool-call accuracy varies sharply by dialogue setting. MMedAgent-Plus selects the correct specialist for 91.0% of individual tool calls in single-tool multi-round, 98.1% in multi-tool multi-round, and 80.7% in multi-tool single-round dialogues.
• Whole-case accuracy follows the same ordering. Complete correct tool sequences are produced in 91.0% of single-tool multi-round dialogues (95% CI 88.2–93.2%), 95.8% of multi-tool multi-round dialogues (93.7–97.2%), and 77.1% of multi-tool single-round dialogues (73.2–80.6%).
• Multi-round context helps. The multi-tool multi-round setting is strongest (98.1% tool-call accuracy over 1,251 tool calls), with micro-averaged precision, recall, and F1 of 98.2%, 98.1%, and 98.1% and macro-averaged scores of 98.3%, 97.7%, and 97.9% — close agreement suggesting performance is spread across the tool library rather than driven by frequently used tools.
• Single-round multi-tool planning is the hard case. In that setting micro-averaged precision is 87.5% but recall is 80.7%, meaning missed tool calls are more common than unnecessary ones; the planner must commit to the full sequence before any intermediate result exists.
• Errors concentrate in a few ambiguous tools. All but three tools achieve recall above 0.90 and nine reach 1.00; the lowest are UniGradICON (0.36), CONCH (0.64), and MedSAM (0.76). 36% of CONCH cases are routed to LLaVA-Med, and 64% of UniGradICON cases and 24% of MedSAM cases are answered with no tool at all. At task level, most tasks reach recall of at least 0.94 except Registration (0.36) and Segmentation (0.88).
• Response synthesis is strong when tool outputs are correct. On 600 held-out cases with the correct specialist outputs supplied, MMedAgent-Plus scores 14.4–14.8 out of 15 on faithfulness, completeness, and conciseness. Totals are 14.77 versus 10.89 for Qwen3-VL-8B-Instruct in multi-tool single-round, 14.52 versus 13.27 in single-tool multi-round, and 14.36 versus 12.86 in multi-tool multi-round; it reaches 5.00 on both faithfulness and completeness in the multi-tool single-round setting.
• End-to-end invocation is reliable across seven tasks. Across 350 held-out cases, 317 correct invocations give 90.6% accuracy (95% CI 87.1–93.2%). QA/rationale generation, named entity recognition, and multimodal classification each reach 50/50; general medical image segmentation reaches 92.0%, report generation 90.0%, fundus segmentation 82.0%, and pathology classification is lowest at 70.0%.
• End-to-end response quality beats direct-answer VLMs on all five text tasks judged. Largest margins appear in radiology report generation (4.40 versus 2.41 for MedGemma-1.5-4B-IT), named entity recognition (4.96 versus 2.95), and multimodal classification (4.93 versus 3.21 for Qwen3-VL-8B), with smaller gaps in QA/rationale generation (4.53 versus 4.12) and pathology classification (3.57 versus 2.29). Conciseness is the one criterion where baselines occasionally lead — MedGemma-1.5-4B-IT scores 5.00 versus 4.14 for MMedAgent-Plus in QA/rationale generation.
• Specialist performance is largely preserved end to end. MMedAgent-Plus matches the specialist accuracy of 0.98 in QA/rationale generation, preserves the specialist RadGraph F1 of 0.64 in named entity recognition while reaching a higher entity-level F1 (0.90 versus 0.84), and stays closest to specialist outputs for report generation and multimodal classification. The exception is ROUGE-L in QA/rationale generation, where MedGemma-1.5-4B-IT scores 0.23 versus 0.20.
• Pathology classification loses the most. Micro-F1 drops from 0.72 for the specialist to 0.38 across all 50 cases; among the 35 cases with correct invocation the scores are 0.69 and 0.51, showing both routing failures and post-invocation synthesis contribute.
• The margin is largest in retinal imaging. For fundus images all baselines score below 50% while MMedAgent-Plus approaches 100%, with a similarly large gap for retinal OCT; averaged across eight modality axes the system reaches 96.1% versus 78.2% for the strongest baseline (Qwen3-VL-8B).
• Broader coverage than the base system. MMedAgent-Plus extends MMedAgent from 5 modalities and 6 tasks to 8 modalities and 12 tasks, while prior systems such as WSI-Agents, RadAgents, MARCH, and ADAgent remain confined to a single modality or disease domain.

Methodology in Plain English

The authors do not try to compress expert medical ability into one model. Instead they keep the experts and teach a planner to use them. Each of the 18 specialists is wrapped in a common interface that states what it accepts, what settings it takes, and what it returns, so the planner can call any of them the same way.

Training data comes from 32 public medical benchmarks, converted into 208,928 example dialogues. Crucially, these are written in three shapes that mirror real workflows: one tool used over several turns, several tools used over several turns, and several tools composed inside a single request. In multi-tool examples, earlier tool outputs are passed forward to later steps, so the planner learns to carry intermediate evidence rather than just pick tools independently.

Evaluation is deliberately split into three stages. First, on 1,500 held-out dialogues (500 per setting), the authors check whether the right tools were called, scoring both per-call accuracy and whether the entire ordered sequence matched. Second, on 600 disjoint cases, they hand the model the correct specialist outputs and ask an LLM judge to rate the resulting answer for faithfulness, completeness, and conciseness on a 1–5 scale each. Third, on 350 cases across seven tasks they remove the supplied outputs entirely and measure whether the system selects, runs, and integrates a specialist on its own. Approximately 10% of dialogues per setting were held out before training, with no overlap in dialogues or source images; the 95% Wilson intervals for tool-orchestration accuracy have margins of error within 4 percentage points across all three settings.

Why This Matters

The paper argues for a different scaling strategy for medical AI: instead of one ever-larger generalist model, coordinate validated specialists behind a learned planner. That reframes an architectural question as an orchestration question, and the staged evaluation gives other researchers a template for attributing failures to selection versus synthesis rather than reporting a single opaque number.

Real-world applications:

• Radiology reporting: automatically routing a chest X-ray through the appropriate specialist before drafting a report, where the largest response-quality margins appeared.
• Retinal screening: fundus and OCT analysis, the modality with the biggest gap over direct-answer baselines.
• Pathology triage: whole-slide classification and cell segmentation, where the paper also documents the hardest failure mode.
• Multi-step diagnostic workups: chaining classification, segmentation, and retrieval steps so that one result informs the next.

Industry relevance: the unified tool interface is directly relevant to hospital and vendor environments where many single-task models already exist and need to be composed without retraining each one. A planner that emits an explicit tool-call sequence also produces an auditable record of which analyses were run and in what order, which matters for clinical governance. The results also warn deployment teams where to focus validation: ambiguous tool pairs and implicit intermediate steps are where orchestration silently fails.

Future Directions

• Longer and adaptive workflows. The current multi-tool chains involve only two to three tools built from public benchmarks. The authors explicitly flag as untested longer sequences, workflows where a later tool depends on the specific output of an earlier one, and requests with several valid tool orderings.
• Resolving tool ambiguity and implicit steps. Confusion between CONCH and LLaVA-Med, and omitted intermediate steps such as report generation before a downstream call, suggest targeted instruction tuning, explicit planning, and tool-selection verification as remedies.
• Handling specialist error propagation. Segmentation output quality remains limited even with correct invocation, so how planner-level checks should catch and compensate for weak specialist outputs is an open problem.
• Extending the library safely. Adding new capabilities requires the planner to learn when and how to invoke them; prior MMedAgent work suggests incremental training can add tools without major degradation, but this is unverified for the larger, more heterogeneous 18-tool library. Prospective clinical evaluation is also not reported.

Target Audience

Medical AI and clinical informatics researchers; agent and tool-use researchers interested in orchestration rather than single-model scaling; ML engineers building systems on top of existing specialist models; and clinical stakeholders evaluating whether multi-model pipelines can be trusted for multi-step diagnostic workflows.

Authors’ abstract

Medical workflows often require multiple interdependent analyses, but existing specialist and generalist AI models are not designed to coordinate them. We introduce MMedAgent-Plus, a multimodal medical tool-use agent that combines a vision-language planner with 18 leading specialist models spanning 12 tasks and 8 modalities. Through a unified tool interface, the planner selects specialists, executes them in sequence, and synthesizes responses grounded in their outputs. The planner is instruction-tuned on 208,928 samples from 32 public medical benchmarks, covering single- and multi-tool interactions across single and multiple rounds. We separately evaluate tool orchestration, response synthesis, and end-to-end execution on held-out samples. MMedAgent-Plus achieves tool-call accuracies of 91.0%, 98.1%, and 80.7% in single-tool multi-round, multi-tool multi-round, and multi-tool single-round settings, respectively. Given correct tool outputs, MMedAgent-Plus achieves scores of 14.4-14.8 out of 15 for faithfulness, completeness, and conciseness across 600 evaluation samples. In end-to-end execution without pre-supplied tool outputs, MMedAgent-Plus correctly invokes the required specialist in 90.6% of 350 evaluation cases spanning seven tasks. Its final-answer accuracy reaches 0.98, matching the dedicated leading specialist and exceeding the strongest baseline (0.28).