NPCBench: A Clinical Apprenticeship Benchmark for Multimodal Care-Pathway Reasoning in Nasopharyngeal Carcinoma

AI summary

Overview

Research area: Evaluation of medical multimodal large language models, specifically oncology care-pathway reasoning in nasopharyngeal carcinoma (NPC).

Technical level: Advanced. The paper assumes familiarity with benchmark design, LLM/MLLM evaluation, staging systems, and rubric-based scoring, though the core argument is stated in plain terms.

Scope (one sentence): NPCBench is a six-level plus consultation benchmark built from NPC guidelines, pathology and multi-sequence MRI, and 100 real patient trajectories, used to test whether 30 models that answer individual clinical questions well can also make jointly correct diagnostic and treatment decisions for the same patient across a full course of care.

What This Paper Is About

Most medical AI benchmarks score isolated factual questions, single images, or static case vignettes, so a high score says little about whether a model can combine specialty guidelines with imaging evidence and follow a patient's condition as it changes. This paper builds NPCBench, a benchmark for nasopharyngeal carcinoma that is structured like specialist apprenticeship training: guideline recall first, then atomic guideline application, image evidence grounding, single-state case planning, longitudinal revision, and finally linked patient CarePath episodes with an open-ended consultation suite. The goal is to measure whether correct answers hold together for the same patient, not just whether they are correct one at a time.

Key Contributions

1. A clinical apprenticeship evaluation framework. The benchmark shifts the measurement target from static answer correctness to pathway correctness, asking whether a model makes consistent decisions across patient-state transitions while respecting guideline constraints, multimodal evidence, and disease evolution. It is organized as levels L1–L6 plus an open-ended consultation suite (O1).

2. A specialist-curated multimodal and longitudinal dataset. 4,777 component and consultation items are released alongside 100 CarePath episodes containing 417 scored diagnostic and care tasks, unifying guideline rules, pathology slides, multi-sequence MRI, longitudinal electronic health record trajectories, and consultation rubrics in one oncology pathway. Data and code are released on Hugging Face and GitHub (code under MIT, original data and annotations under CC BY 4.0).

3. Diagnostic dissection of a "composition gap." Across 30 models the authors combine capability profiling, controlled interventions (matched-history, order-only shuffle, MCQ-versus-short-answer, reference-evidence substitution), and physician-calibrated judging to locate where visual grounding separates from treatment decisions, how history changes answers, and where high local scores coexist with patient-level failure.

4. Physician calibration of automated consultation judging. Two audit sets (548 and 1,013 criterion-level labels) compare automated rubric judges against physician annotations, exposing sensitivity–specificity imbalances.

Main Findings

• Guideline recall does not transfer to case planning. Gemini-3.1-Pro-Preview leads both guideline memory (94.0%) and atomic application (92.5%), but falls to 71.9% on case-level planning (L4). GPT-5.1 and DeepSeek-V3.1 have nearly identical L1 and L2 scores yet diverge sharply on L4 (63.5% vs. 51.8%). On 71 physician-audited L1 items, removing multiple-choice options while holding the target knowledge fixed lowers accuracy by 7.0 percentage points for Gemini-3.1-Pro-Preview, 12.7 for GPT-4.1, and 15.5 for GPT-5.1.

• Stage labels often lack anatomical support. The best T-stage accuracy is 43.5% against a 37.5% majority-class baseline across 400 patients; N-stage accuracy reaches 42.8% against a 40.0% N1 majority baseline. On the 100-patient anatomical panel, direct T-stage accuracy is 43.0%, yet complete 18-structure invasion correctness and evidence-grounded T-stage success both fall below 5.0%.

• The staging logic itself is sound; the evidence reading is not. With reference findings, both a deterministic staging rule and a fixed GPT-OSS-120B stager reach 99.0% accuracy. Gemini-3.1 drops to 49.0% when given predicted invasion with known anatomy and 36.0% with fully predicted evidence, indicating invasion recognition is the bottleneck.

• Class imbalance inflates apparent staging skill. Qwen3.6-27B has the highest overall N accuracy (42.8%) but only 31.9% balanced accuracy. In CarePath, constant T3 predictions score 58%, above the best model's 52%; the highest balanced T accuracy is 30.5% versus 25% for constant predictions.

• History both corrects and introduces errors. Gemini-3.1-Pro-Preview achieves the highest L5 turn accuracy (87.9%) and all-turn success (71.0%), followed by Claude-Sonnet-4.6 (85.2% and 63.0%). In paired controls on the same 305 questions and 100 patients, full history gives higher turn-accuracy point estimates for 14 of 20 models, but 19 of 20 confidence intervals span zero (DeepSeek-V3.1 gains 4.9 pp, Gemini-3.1 3.3 pp). GPT-5.1 is correct on 22 questions only with full history and 19 only with the snapshot, so a 1.0 pp turn-accuracy gain coexists with a 1.0 pp decline in all-turn success; three of 20 models show opposite directions on the two endpoints. An order-only shuffle yields intervals spanning zero for 19 of 20 models, with Gemini-3.1 the sole exception (+3.0 pp; 95% CI 0.6–5.7 pp).

• The composition gap caps patient-level success. Gemini-3.1 achieves 89.9% Care accuracy (195/217 decisions) and 83.0% all-care success but only 11.0% Strict success. Claude-Sonnet-4.6 has lower Care (80.6%) and all-care (64.0%) but the highest Strict success at 23.0% (95% CI 15.2–32.5%). Across 21 multimodal models, Strict success tops out at that 23.0%.

• Errors split by model type. For Gemini-3.1, 72 of 100 patients have all care decisions correct despite at least one incorrect pathology or MRI T-stage label; the figure is 41 for Claude-Sonnet-4.6. Conversely, Hulu-Med-32B has 38 patients with both diagnostic labels correct but at least one incorrect care decision.

• Better evidence does not reliably improve care choices. In 25 CarePath patients, 14 local models answer the same 146 care questions with predicted versus reference pathology, MRI T-stage, and invasion findings; patient-averaged care accuracy changes by −3.2 to +4.1 pp, and 12 of 14 intervals include zero.

• Episode composition shifts the ranking. Nine of 21 models have lower decision-weighted Care but higher patient-weighted all-care in the 75 assigned-pathology episodes than in the 25 patient-matched episodes; Claude-Sonnet-4.6 scores 76.5% vs. 84.3% on Care but 68.0% vs. 52.0% on all-care, across 102 and 115 decisions respectively.

• High consultation scores coexist with safety violations. GPT-5.1 has the highest consultation score (58.4 on a 0–100 scale), yet the primary GPT-OSS-120B judge flags major-error criteria in 14.6% and minor-error criteria in 40.8% of its responses. Re-weighting the 103 consultations by their 42 underlying cases shifts scores by only −2.0 to +0.1 points, leaving GPT-5.1 highest.

• Automated judges are only partly reliable. GPT-OSS-120B reaches 81.4% agreement (κ = 0.610) on the 548-label Targeted Audit but 70.5% (κ = 0.387) on the 1,013-label Expanded Audit; Qwen3.6-27B leads the Expanded Audit (77.8%, κ = 0.544). The primary judge's agreement in the Expanded Audit comes with 51.8% sensitivity and 85.8% specificity, meaning it misses satisfied requirements more often than it misses errors.

Methodology in Plain English

The authors chose nasopharyngeal carcinoma because it is a disease where imaging, pathology, a blood biomarker (plasma EBV DNA), and guideline-defined staging all drive the same decisions, and where later choices depend on earlier responses and toxicities. They turned this into a ladder of tasks, each reviewed by oncologists, pathologists, and radiologists.

• L1 (250 questions) tests guideline memory, drawn from CSCO and NCCN guidelines: 153 items are supported by both sources, 80 by CSCO only, and 17 by NCCN only.
• L2 (185 short vignettes) tests applying a single guideline rule, scored as an average across four equally weighted groups: initial definitive management (84 items), post-treatment recurrence (30), metastatic/systemic management (44), and toxicity or contraindication management (27).
• L3 tests image reading: 69 pathology overview cases (39 NPC-positive, 30 negative), 200 MRI sequence-recognition questions, 400 T-stage cases, and 400 N-stage cases on patient-disjoint cohorts, plus a 100-patient anatomical panel with 2,700 structure cards over 18 base structures that checks both structure identification and tumor invasion.
• L4 (165 questions over 84 patients) asks for the next plan from a frozen clinical snapshot, scored with patient-macro averaging.
• L5 (305 questions, 100 trajectories, one to five checkpoints each) tests revising decisions as new findings arrive.
• L6 (100 patients, 417 tasks: 100 pathology, 100 MRI T-stage, 217 care decisions) is the pathway test. "Strict" success requires the pathology label, the MRI T stage, and every care answer to be correct for the same patient.
• O1 (103 consultation items, 42 patients, 1,726 rubric criteria: 1,069 positive requirements and 657 negative criteria) scores free-text advice, adding weighted credit for satisfied requirements and deducting two points for major errors and one for minor errors, with a floor at zero and normalization to 0–100.

Four design rules govern construction: each level must isolate a distinct competence; only evidence available at that decision point may be shown, so future outcomes that would leak the answer are stripped; reference answers must be determinable from guidelines or expert consensus; and open-ended scoring must be auditable. Controlled comparisons hold patients and questions fixed while changing one factor at a time (full history vs. a snapshot with time markers removed, shuffled line order, answer options removed, predicted vs. reference imaging findings). Automated judges are checked against physician labels on two audit sets, and all local models run on four NVIDIA A800-SXM4-80GB GPUs.

Why This Matters

Impact on research. NPCBench reframes medical AI evaluation around whether answers compose into a coherent care pathway. Its central construct, the gap between decision-level accuracy and strict patient-level success, gives the field a measurable target that exam-style benchmarks cannot express, and its paired controls separate visual grounding, temporal information, and answer format as distinct sources of model behavior. The physician-calibrated judge audits also show why rubric scores from automated judges need explicit sensitivity and specificity reporting.

Real-world applications (as the authors frame the intended use):

• Model comparison and procurement decisions for clinical AI, using strict patient-level success rather than average question accuracy.
• Clinical error analysis, for example spotting models that name the right treatment while misreading the underlying MRI.
• Development of evaluation methodology: paired controls, episode-level endpoints, and criterion-level consultation rubrics.
• Clinician training and quality review, where care-pathway checkpoints mirror how trainees are assessed. The paper explicitly states that out-of-scope use includes direct diagnosis, treatment recommendation, triage, or patient-specific decision-making without physician supervision.

Industry relevance. The results give concrete failure signatures for vendors and hospital systems: staging accuracy near majority-class baselines, near-total collapse on evidence-grounded anatomical reasoning (under 5.0%), high consultation scores alongside 14.6% major-error rates, and strict care-pathway success capped at 23.0% even for the strongest multimodal models. Because the benchmark and code are released openly, it can serve as a shared acceptance test for NPC-focused clinical products.

Future Directions

• Multi-center data and fully patient-matched modalities. The authors list primarily single-center records, assigned pathology in 75 model-selected CarePath episodes, and a six-case Expanded Audit as limitations, and call for multi-center cohorts and prospective studies to test whether benchmark gains improve actual care.
• Closing the evidence-grounding gap. Accuracy with reference findings reaches 99.0% while predicted evidence drives Gemini-3.1 to 36.0%, so improving invasion recognition and anatomical interpretation is the clear lever for raising strict pathway success.
• Stabilizing longitudinal reasoning. Full history helps some models and hurts others, with 19 of 20 confidence intervals spanning zero and one model showing opposite directions on turn accuracy and all-turn success, leaving open how history should be presented and how errors distribute across trajectories.
• Trustworthy automated judging. The primary judge's 51.8% sensitivity and 85.8% specificity, plus disagreement between judges across the two audit sets, point to a need for better-calibrated or human-in-the-loop consultation grading, and for scoring that is less sensitive to class balance and episode length.

Target Audience

Researchers and engineers building or evaluating medical LLMs and multimodal models, especially those working on oncology, radiology, or pathology applications. It is also relevant to clinical informatics teams and hospital review committees choosing models for deployment, to benchmark designers interested in episode-level and rubric-based evaluation methods, and to clinicians in head and neck oncology who want to understand where current AI systems fail. Readers need some background in model evaluation or clinical workflows; the paper's core argument about composition is accessible without it.

Authors’ abstract

Current medical benchmarks typically break down clinical competence into isolated factual questions, single-image tasks, or static case vignettes. These tests score individual answers, but reveal less about whether a model can integrate subspecialty knowledge with multimodal evidence and track a patient's condition throughout a course of care. To close this gap, we introduce NPCBench, a clinical apprenticeship benchmark for nasopharyngeal carcinoma (NPC). NPCBench mirrors specialist training, from guideline and foundational knowledge to case planning, longitudinal review, CarePath integration, and open-ended consultation. It contains 4,777 component and consultation items and a 100-patient CarePath cohort with 417 scored diagnostic and care tasks. The episode-level evaluation tests whether diagnostic and care answers are jointly correct within a care pathway. By linking stepwise tasks to patient-level care trajectories, NPCBench evaluates whether models ground decisions in visual evidence, interpret changes in patient state, and meet clinical requirements in consultation. Across 30 evaluated models, we observe a composition gap: strong performance on individual tasks does not ensure jointly correct diagnostic and care answers for the same patient. Among 21 multimodal models, strict CarePath success reaches at most 23%. Paired controls and physician calibration further examine evidence grounding, longitudinal revision, and consultation judging. Overall, NPCBench broadens the evaluation of medical artificial intelligence beyond static question-answering paradigms to an apprenticeship-oriented assessment of clinical workflows, thereby offering a testbed for evidence-based and temporally coherent reasoning in oncology care.