The Pulse
Microsoft AI Scores 83% on Complex Cases, Versus 36% for Doctors
Microsoft says its MAI-DxO diagnostic system scored 83% on complex New England Journal of Medicine cases, compared with 36% for physicians using reference tools. The Science study tested the system on a step-by-step benchmark, while a plann
AI.info Team · Written with AI · fact-checked ·
83% on complex cases, against 36% for physicians
Microsoft’s AI diagnostic system reached 83% accuracy on complex medical cases, compared with 36% for physicians who could consult textbooks and the internet, the company said October 22. The results appeared in Science in a study of sequential diagnosis: a clinician or AI system starts with limited information, then decides what to ask or test before settling on a diagnosis. Microsoft’s system, called MAI-DxO, also scored 80% on more typical emergency-department cases, compared with 47% for physicians.
The comparisons are striking, but they describe performance on a benchmark, not outcomes for patients receiving care. Microsoft says MAI-DxO was about 2.3 times as accurate as physicians with reference tools on the complex cases, and about 3.7 times as accurate as physicians working without them. On the emergency-department cases, its accuracy was more than 1.7 times the physician result.
MAI-DxO asks what to do next
The benchmark, called the Sequential Diagnosis Benchmark, includes 304 complex cases drawn from The New England Journal of Medicine and a second set of more typical emergency-department presentations from MIMIC-IV-ED. Rather than show all case details at once, it gives the system an initial summary and lets it request additional information in stages. Microsoft says the evaluation tracks accuracy, evidence requested and estimated testing costs, so it measures the route to a diagnosis as well as the final answer.
MAI-DxO organizes multiple rounds of proposed diagnoses, questions and tests. The system weighs each test’s likely value against its cost and updates its assessment as new information arrives. Microsoft describes the benchmark this way: “Each becomes an interactive case in which AI models and/or physicians start with limited information and choose what to request next.”
The company also reports that a cost-focused configuration reached 81% accuracy on the complex cases at less than half the cost of comparative systems. Its broader claim is that no other tested human or AI approach was both more accurate and less costly across the two datasets. Those cost estimates are part of the benchmark, not evidence of savings already achieved in a hospital.
The benchmark has limits built into its design
SDBench models a step-by-step workup, but its cases are drawn from existing records rather than conversations with patients in live clinical settings. The benchmark’s gatekeeper returns a real finding when one is in the case record; when it is absent, the gatekeeper can provide a synthetic result. That design choice is documented on the SDBench site and matters when interpreting performance: some requested information may be generated for the simulation rather than drawn from the original case.
Access to the underlying case sets also has conditions. The NEJM cases must be licensed from NEJM, while the MIMIC dataset requires credentialing, training and acceptance of its data-use agreement. Microsoft says it is releasing benchmark and evaluation code for select NEJM cases so researchers can reproduce results and test other models; the benchmark site distinguishes those licensed clinical cases from five synthetic cases provided for open, access-free evaluation.
The study therefore supports a narrow conclusion: MAI-DxO performed better than the physician comparison groups on the cases and under the conditions tested. It does not establish that the system outperforms specialists, clinical teams or doctors working with patients in ordinary practice. Nor does a higher benchmark score by itself show that using the tool improves patient health.
A hospital study is the next test
Microsoft says a new partnership with Beth Israel Deaconess Medical Center in Boston will evaluate clinical reasoning across hospital settings. In the planned prospective study, patients will interact directly with the system, and its performance will be measured against physicians’ independent assessments. That test addresses a different question from SDBench: whether the approach works when people, clinical workflows and real-world care are involved.
For now, the 83% result belongs to a controlled evaluation of written cases. The next concrete measure will come from the hospital study Microsoft says it plans to conduct, where the system’s recommendations can be assessed alongside independent physician judgments.