The Pulse
ChatGPT-5 Detected 51% of Distal Radius Fractures in Study
In a study of wrist X-rays, ChatGPT-5 missed 83 of 170 confirmed distal radius fractures when asked a single yes-or-no question. The results apply to one model version, prompt, anatomical region and dataset.
AI.info Team · Written with AI · fact-checked ·
ChatGPT-5 identified just 51.2% of distal radius fractures in a study of wrist X-rays, missing 83 of the 170 confirmed fractures in the main analysis. The researchers tested the model with one question—“Is there a fracture in this X-ray? Answer with yes or no.”—and published their results October 26 in Emergency Radiology. The findings concern a narrow use case: a general-purpose model, a single prompt and anteroposterior images of the distal radius, the end of the forearm bone nearest the wrist.
A single question, 396 X-rays
The team submitted 396 radiographs from Stanford’s MURA dataset, each in a separate temporary chat session. Two orthopedic reviewers—a senior resident and a board-certified surgeon—agreed on whether each image showed a fracture; the original MURA labels were used to select candidates, not as the study’s final diagnostic standard. The researchers excluded pediatric images, images with hardware and cases the reviewers could not classify with confidence.
Twenty-six images, or 6.6%, produced a hedged or otherwise non-binary answer and were left out of the primary analysis. That left 370 images: 170 with fractures and 200 without. Among those, ChatGPT-5’s overall accuracy was 61.6%, and it incorrectly flagged 59 fracture-free images.
The missed cases matter more than the score
The missed fractures outnumbered the false alarms, a difference the authors found statistically significant. In a separate projection using a hypothetical group of 1,000 X-rays with a 20% fracture rate, the model would miss 98 of 200 fractures and produce 236 false alarms. That projection is a calculation from the study’s measured sensitivity and specificity, not an observed clinical trial outcome.
The authors also tested how the excluded, non-binary answers could affect the result. Counting each as an error, or treating them all as “no,” brought sensitivity to 47.3%; treating every one as “yes,” the most favorable assumption for the model, raised it to 54.9%. The study’s primary sensitivity estimate therefore did not depend on one particular way of handling those responses.
Not a direct test against fracture-detection software
The paper contrasts ChatGPT-5’s result with published performance for dedicated fracture-detection systems, but the researchers did not run those systems on the same images. They describe the comparison as indirect because studies can differ in patient groups, datasets, reference standards and imaging protocols. A general-purpose chatbot’s score in this experiment should not be read as a measure of all medical AI.
The evaluation also covered only one model version, one anatomical region and one view of each X-ray. The images came from a selected dataset rather than a consecutive series of patients, and the model was tested without added clinical history, examples or task-specific training. No clinical decision relied on its answers.
The authors limit what the result can show
“Results are specific to this model version, this prompt, this anatomical region, and this dataset; they constrain claims about autonomous, minimal-prompt use rather than the broader potential of engineered LLM-based radiological workflows.”
Study authors Bana Awad, Yaniv Keren and Dror Ronel write
The study does not establish how ChatGPT-5 would perform with a different prompt or in a clinician-designed workflow. It does show that, in this test, a yes-or-no answer from the model was not reliable enough to rule out a distal radius fracture: it missed nearly half of the fractures in the primary analysis.