The Pulse
Stanford Finds Medical AI Agents Finish Less Than Half of Tested Workflows
Stanford HAI reports that the top agents completed fewer than half of the workflows in two execution-based medical AI benchmarks. The results distinguish answering clinical questions from carrying tasks through electronic health records and
AI.info Team · Written with AI · fact-checked ·
“These benchmarks tell the same story: Today’s best AI agents complete less than half the work.”
Nigam Shah and Jonathan Chen, the Stanford HAI article’s authors
Stanford HAI’s October 19 analysis points to a gap between answering a medical question and completing a clinical workflow. In two execution-based benchmarks, agents had to move through electronic health records, gather and retain information, and carry out tasks. The strongest systems finished fewer than half of the assigned workflows, the authors write.
The finding does not mean medical AI gets fewer than half of medical questions right. Stanford’s MedHELM benchmark measures model responses across clinical tasks; PhysicianBench and HealthAdminBench instead test whether agents can act within simulated software environments and complete work from start to finish.
PhysicianBench tests the steps after the answer
PhysicianBench uses 100 multi-step tasks adapted from real consultations between primary-care physicians and specialists, spanning 21 specialties. The tasks run against synthetically reconstructed patient records in an electronic health record sandbox. Each task takes an average of 27 tool calls, requiring an agent to retrieve information across encounters, reason over it, take clinical actions and document its work.
Across the 12 models described in the Stanford post, GPT-5.5 had the highest single-attempt completion rate, at 46%. DeepSeek V4-Pro, the best open-source model in that evaluation, reached 19%. Repeatability was lower still: GPT-5.5 completed just 28% of tasks correctly in all three runs.
The benchmark also exposed a gap between writing down an action and executing it. In a share of action failures, GPT-5.5 and Claude Opus 4.6 recorded the correct action in a note but never placed the corresponding order in the EHR. Stanford reports that these cases made up 62% of GPT-5.5’s action-execution failures and 73% of Claude Opus 4.6’s.
Administrative work breaks at the handoffs
HealthAdminBench tests tasks such as prior authorizations, appeals and durable medical equipment orders. Its simulated setup includes an EHR, two payer portals and a fax system, built after more than 100 hours of observation at Stanford Health Care. The benchmark contains 135 tasks and 1,698 checks; its patient data is synthetic.
Claude Opus 4.6 completed 36.3% of the tested workflows end to end, the highest rate reported for that benchmark. GPT-5.4 passed 82.8% of individual subtasks but completed only 26.7% of whole tasks. Stanford describes the common breakdowns as coordination failures: agents lose information needed later, miss required file transfers, or fail to revisit an earlier step.
The scores from the two benchmarks should not be compared as if they came from one test. They use different environments, model sets, interfaces and grading methods. Their shared result is narrower: in both settings, the best tested agents left most workflows unfinished.
Why a polished note is not a completed task
Stanford’s broader MedHELM work maps 121 clinical tasks across five categories, using 37 benchmark datasets. Its results show comparatively high model win rates for note generation and patient communication, but lower rates for clinical decision support and administration. Those figures measure pairwise performance in response-based tests, not the proportion of real clinical work an agent can execute.
The workflow benchmarks make that distinction concrete. A prior authorization can require checking coverage rules, locating records, uploading documents, submitting a form and recording a reference number; success at drafting one part does not complete the chain. The authors argue that agents’ failures to carry information across steps point to memory and system design as areas for improvement, rather than a simple need for larger models.
For now, Stanford’s strongest results are 46% end-to-end completion on PhysicianBench and 36.3% on HealthAdminBench. Both are benchmark results, not measurements of patient outcomes or performance in live hospitals. But they show why a correct answer on its own is a poor measure of whether an agent has done the work.