The Pulse
ChatGPT and Claude Show No Consistent Bias in Abstract Reviews
A study of ChatGPT and Claude found no consistent gender or geographic bias when the models scored 50 medical research abstracts under fictional author identities. The researchers caution that the controlled test does not show whether AI re
AI.info Team · Written with AI ·
Concerns that AI reviewers could carry gender or geographic bias into scientific publishing meet a qualified counterpoint in a study published October 2: ChatGPT and Claude showed no consistent scoring differences when the same medical abstracts appeared under different author identities. But the experiment tested a tightly controlled scoring task—not full manuscripts, real editorial decisions or whether the models’ judgments were sound.
Four fictional identities, 50 medical abstracts
Paul Sebo and Ting Wang selected 50 research abstracts from 10 general internal medicine journals, then presented each abstract as the work of four fictional authors: a woman and a man from the United States, and a woman and a man from Côte d’Ivoire. The researchers kept the abstract and prompt constant while changing the author identity.
ChatGPT, identified in the study as GPT-5.3, and Claude, identified as Claude Sonnet 4.6, each scored the abstracts twice under each identity. They rated scientific quality, novelty and likelihood of acceptance on a 0-to-10 scale, producing 800 evaluations. The study appeared in the Journal of General Internal Medicine.
“In this controlled setting, we detected no consistent evidence of gender or geographic bias, and reproducibility was high, although the restricted score range may have contributed to the observed agreement,” the authors write.
Similar scores, with a few exceptions
Median scores were alike across the four identities. ChatGPT gave identical median quality and novelty scores; its acceptance medians varied slightly, from 6 to 7. Claude’s median scores for all three measures were identical across identities.
Statistical analyses did not find an overall association between identity and any score for ChatGPT, or between identity and quality or novelty scores for Claude. Claude did show a global association with acceptance scores, but no individual identity comparisons were statistically significant. ChatGPT also gave lower novelty and acceptance scores in some individual comparisons when an abstract carried the American male identity; the authors caution that these isolated differences appeared without a significant overall association.
Repeated scores were highly consistent: agreement exceeded 98% for both models across all three measures. Fleiss’ kappa, a statistic that accounts for agreement expected by chance, ranged from 0.88 to 0.89 for ChatGPT and 0.74 to 0.80 for Claude.
Consistency does not establish accuracy
The results do not show that either model can judge research quality correctly. The researchers had no independent human assessment or other reference standard to compare against, and publication of an abstract did not establish what score it should receive. As the paper stresses, repeated agreement measures consistency, not whether an evaluation is accurate or useful to editors.
The scope was also narrow: 50 abstracts from one medical field, four fictional identities based on only two countries and two genders, and prompts demanding numerical scores without written feedback. The researchers note that this standardized format may have limited opportunities for more subtle or narrative bias to appear. The models used default settings, and the study was not preregistered.
The next test is real peer review
The findings cannot establish that AI reviewers are less biased than human reviewers, or that they are ready to assess submissions independently. Full peer review calls for scrutiny of study design, statistical methods, results and interpretation—work that an abstract-only scoring exercise did not test.
Sebo and Wang point to comparisons using manuscripts with existing human review reports and editorial decisions as a way to examine AI and human judgments side by side. Their study identifies a limited result: under a fixed prompt, on published medical abstracts, the two models generally assigned similar scores despite changes to the fictional author identities.