MTBBench: A Multimodal Sequential Clinical Decision-Making Benchmark in Oncology

AI summary

Overview

Research area: Machine learning for clinical decision support, with a focus on multimodal and longitudinal reasoning in precision oncology.

Technical level: Intermediate. The paper is readable by anyone comfortable with LLM benchmarks and evaluation metrics, but some familiarity with vision-language models, foundation models, and agentic tool-use helps.

Scope: MTBBench is a new benchmark and agent framework that evaluates how well AI models can reason over multimodal, time-evolving cancer patient data the way a real Molecular Tumor Board (MTB) would.

What This Paper Is About

Current medical AI benchmarks test models on static, single-modality, single-turn questions where all the needed information is handed to the model at once. Real clinical decision-making, especially in oncology tumor boards, works very differently: experts gather images, lab results, pathology reports, and genomic data across multiple timepoints, request information as needed, and revise their conclusions as new evidence arrives. This paper builds MTBBench to close that gap, providing a benchmark and an agent framework where models must actively retrieve files, integrate H&E and IHC pathology images, hematology panels, and genomics, and answer sequential clinical questions.

Key Contributions

1. A longitudinal, multimodal clinical reasoning benchmark. MTBBench simulates MTB-style workflows using two tracks: MTBBench-Multimodal (26 head-and-neck cancer cases from HANCOCK, 390 QA pairs) and MTBBench-Longitudinal (40 cases from MSK-CHORD, 183 QA pairs) covering diagnosis, biomarker interpretation, progression, and outcome prediction.

2. Systematic evaluation of open and closed-source models. A broad set of vision-language and text-only models (Gemma-3, GPT-4o, o4-mini, InternVL3, Llama, Mistral, Qwen) is benchmarked on both tracks, with 95% confidence intervals computed via bootstrap resampling.

3. An agent framework that exposes foundation models and biomedical tools as callable resources. Agents can invoke CONCH for H&E image-text matching, a custom UNI2 + ABMIL tool for IHC marker quantification, PubMed for literature retrieval, and DrugBank for drug metadata.

4. Clinician-validated data and released tooling. A custom companion web app was built to let domain experts review each question, inspect slides, and provide corrections. Ten external experts participated in validation, and the benchmark, annotations, agent logs, and tools are publicly released.

Main Findings

• Multimodal benchmark performance is uneven and often weak. The best overall multimodal model, InternVL3-78B, reached only 69.1% accuracy, slightly ahead of GPT-4o (66.7%). Outcome and recurrence prediction hovered near chance (roughly 42–66%), even for frontier models.

• Bigger models do not automatically win. In digital pathology, Gemma-3-12B outperformed its larger 27B sibling, and overall performance correlated better with how many files an agent accessed than with parameter count.

• Hematology is the easiest task. Structured lab values were the most tractable input type, with models like Llama-90B reaching 82.8% accuracy, likely because the data is tabular and interpretable.

• Longitudinal reasoning exposes sharp weaknesses. Qwen3-32B hit 83.0% on outcome prediction, but progression and recurrence tasks remained near or slightly above chance across all models, indicating models can detect coarse survival signals but not nuanced temporal dependencies.

• Tool augmentation consistently helps. Access to foundation-model tools improved multimodal tasks by up to 9.0%, with digital pathology benefiting most. Especially notable is that hematology accuracy improved even though no dedicated hematology tool exists, suggesting better contextual grounding from related modalities.

• Longitudinal gains from tools are real but modest (up to 11.2%). The authors attribute this to the absence of a specialized foundation model for longitudinal clinical reasoning; agents relied on general tools like PubMed and DrugBank instead.

• Agents fail in characteristic ways. The paper reports frequent hallucination, difficulty reconciling conflicting evidence across modalities, and poor handling of time-resolved data when tools are unavailable.

Methodology in Plain English

The researchers assembled two datasets of real cancer patient records. One came from HANCOCK (head-and-neck cancer with pathology slides and lab work), and the other from MSK-CHORD (genomic and clinical timelines). From these, they created clinically meaningful multiple-choice and true/false questions: some generated with GPT-4o and then reviewed by clinicians, others written directly by the research team.

To make the benchmark realistic, they built an agentic setup where the model cannot see all files at once. At each turn, the model is told what files exist but must explicitly request the ones it wants. Files do not persist across turns, so the agent has to re-request anything it needs later, mimicking how a clinician must go back to the chart. Patient history is presented as an evolving timeline.

They also gave agents optional access to specialized tools. For H&E slides, CONCH compares an image to candidate text descriptions. For IHC slides, a custom pipeline uses UNI2 embeddings plus attention-based multiple-instance learning to estimate the percentage of stained cells for markers like CD3 or CD163, trained on QuPath-annotated images. For literature and drugs, PubMed retrieval is reranked with a BGE reranker returning the top three abstracts, and DrugBank entries are linked to any drug mentioned in the timeline.

All questions were reviewed by clinicians through a Streamlit web app that shows the clinical context, thumbnail grids organized by tissue region and marker, full-resolution slide viewers, and feedback fields. Ten external experts across multiple hospitals and countries participated. Evaluation reported mean accuracy per task with 95% confidence intervals from 1,000 bootstrap iterations, and the team also measured how many files each model accessed per question.

Why This Matters

Impact on research: MTBBench moves clinical AI evaluation away from static question-answering and toward dynamic, tool-augmented, decision-centric assessment. It establishes that model scale alone does not predict clinical reasoning ability, and that information-gathering behavior is a stronger signal of success than parameter count. It also provides a reusable framework for evaluating foundation models as components within agent systems, rather than in isolation.

Real-world applications:

• Tumor board decision support, where an AI assistant helps oncologists synthesize pathology, labs, and genomics for a specific patient case.
• Pathology triage and biomarker quantification, since the IHC tooling quantifies marker staining without per-cell labels.
• Clinical trial matching and treatment sequencing, aided by PubMed retrieval and DrugBank integration over patient timelines.
• Prognostic modeling that updates as new tests and events arrive, rather than producing a one-shot prediction.

Industry relevance: Pharmaceutical companies, diagnostics developers, hospital systems, and clinical AI vendors all depend on benchmarks that reflect real workflows. MTBBench provides a more honest signal of whether a model is ready for clinical-adjacent deployment, and it highlights the value of composing general LLMs with domain-specific foundation models in a tool-calling architecture.

Future Directions

• Specialized foundation models for longitudinal reasoning. The paper explicitly notes that no such model exists today, which explains why longitudinal tool gains are modest. Building one is an open problem.

• Extending to more clinical domains. MTBBench currently covers head-and-neck cancer and clinicogenomic timelines. Broader oncology subspecialties and other specialties could follow.

• Interactive evaluation with incomplete or ambiguous inputs. The current benchmark is still offline. Real clinicians ask clarifying questions and cope with missing data; testing agents in that regime is a natural next step.

• Adding interpretability and oversight mechanisms. When agents lean heavily on tools, transparency decreases, which creates regulatory and safety concerns that future work needs to address.

Target Audience

This paper is most valuable to machine learning researchers working on clinical agents, multimodal reasoning, or benchmark design; medical AI engineers building tool-augmented systems; oncologists and clinical informaticists interested in how current models fail on tumor board tasks; and pharmaceutical or diagnostics teams evaluating whether LLM-based decision support is mature enough for their workflows. Readers looking for immediately deployable clinical tools will not find them here, but anyone designing rigorous evaluations of clinical AI will find the benchmark design and failure analysis directly useful.

Authors’ abstract

Multimodal Large Language Models (LLMs) hold promise for biomedical reasoning, but current benchmarks fail to capture the complexity of real-world clinical workflows. Existing evaluations primarily assess unimodal, decontextualized question-answering, overlooking multi-agent decision-making environments such as Molecular Tumor Boards (MTBs). MTBs bring together diverse experts in oncology, where diagnostic and prognostic tasks require integrating heterogeneous data and evolving insights over time. Current benchmarks lack this longitudinal and multimodal complexity. We introduce MTBBench, an agentic benchmark simulating MTB-style decision-making through clinically challenging, multimodal, and longitudinal oncology questions. Ground truth annotations are validated by clinicians via a co-developed app, ensuring clinical relevance. We benchmark multiple open and closed-source LLMs and show that, even at scale, they lack reliability -- frequently hallucinating, struggling with reasoning from time-resolved data, and failing to reconcile conflicting evidence or different modalities. To address these limitations, MTBBench goes beyond benchmarking by providing an agentic framework with foundation model-based tools that enhance multi-modal and longitudinal reasoning, leading to task-level performance gains of up to 9.0% and 11.2%, respectively. Overall, MTBBench offers a challenging and realistic testbed for advancing multimodal LLM reasoning, reliability, and tool-use with a focus on MTB environments in precision oncology.