The Pulse
OpenAI’s o1 Outperformed Doctors on Some Diagnostic Tests
A study in Science found that OpenAI’s o1 outperformed physician comparison groups on several text-based clinical reasoning tasks. Its strongest reported gap was on a management-reasoning exercise, while a separate emergency-department comp
AI.info Team · Written with AI · fact-checked ·
Science study tested OpenAI’s o1, not a Microsoft-built system
A study published in Science on April 30, 2026, reports that OpenAI’s o1 language model outperformed physician comparison groups across several clinical reasoning tests. The paper is not an evaluation of a Microsoft-built diagnostic tool, despite the framing of some coverage. Its authors include researchers affiliated with Beth Israel Deaconess Medical Center, Harvard, Stanford and Microsoft Research.
The researchers assessed the model across five experiments using challenging clinical cases and physician baselines drawn from hundreds of doctors. They also compared AI-generated second opinions with doctors’ assessments in 76 randomly selected emergency-department cases.
“In all experiments, the LLM outperformed physician baselines and displayed continued improvement from prior generations of AI clinical decision support.”
Peter G. Brodeur et al., the authors of the paper
The largest gap was on management reasoning
On a management-reasoning exercise, o1-preview’s median score was 89%, compared with 34% for physicians using conventional resources. The difference was larger than the gaps reported in the emergency-department comparison, where o1’s triage diagnosis was rated correct or very close in 67% of cases, compared with 55% and 50% for the two attending physicians, according to the Science Media Centre’s summary.
The emergency-department cases came from patient records at one academic medical center, rather than a live test in which the AI examined patients or directed their treatment. The task was to generate a second-opinion differential diagnosis from clinical information. That distinction matters: the comparison measured whether a model’s written suggestions matched clinicians’ assessments, not whether using the model improved care or patient outcomes.
Strong scores do not establish clinical readiness
Researchers also compared the model with physicians on structured tasks involving diagnoses, management decisions and probability estimates. In the management-reasoning exercise, physicians with access to GPT-4 scored 41%, while GPT-4 alone scored 42%.
Those results come from controlled, text-based evaluations. In emergency care, doctors also examine patients, interpret images and other non-text information, weigh uncertainty and coordinate treatment. The study did not test whether the model can perform those broader responsibilities or whether clinicians working with it make better decisions.
The authors call for prospective trials
The paper’s abstract says the findings motivate the urgent need for prospective trials. Such trials would evaluate AI in real clinical workflows and measure more than diagnostic agreement, including safety and effects on patient care. Independent experts responding to the paper likewise cautioned against reading the emergency-department comparison as a test of real-time decision-making or patient management.
The study makes a case for testing language models as potential diagnostic second opinions, not for replacing doctors or advising patients to rely on consumer chatbots. Its real-world result is limited to a retrospective comparison: among 76 emergency-department records, o1’s suggestions were rated exact or very close to the final diagnosis in 67% of cases at triage.