The Biased Oracle: Assessing LLMs' Understandability and Empathy in Medical Diagnoses

AI summary

Overview

Research area: Natural Language Processing / Large language models in healthcare communication, with a focus on fairness, health literacy, and patient-centered communication. Technical level: Intermediate. Scope: The paper evaluates GPT-4o and Claude-3.7 across 156 simulated medical diagnostic prompts to measure whether their patient-facing explanations are understandable (via five readability metrics) and empathetic (affective and cognitive empathy via LLM-as-a-Judge and human ratings).

What This Paper Is About

LLMs are increasingly proposed as tools to help clinicians explain diagnoses to patients, but prior research has mostly measured diagnostic accuracy rather than how well models actually communicate with patients. This paper asks whether LLM-generated diagnostic explanations are accessible to patients with different backgrounds and whether they express appropriate empathy. The authors build a controlled evaluation framework, generate model responses for varied demographic and clinical scenarios, and test both understandability and empathy against readability formulas and human judgments.

Key Contributions

1. An evaluation framework for patient-facing diagnostic communication that generates doctor–patient dialogues across diverse clinical scenarios and demographic profiles, then scores the resulting explanations on understandability and empathy.
2. A multi-metric understandability analysis using five validated readability measures (Flesch-Kincaid Grade Level, SMOG, Gunning Fog, Coleman-Liau, Dale-Chall) applied to GPT-4o and Claude-3.7 outputs.
3. An empathy evaluation pipeline that separates affective empathy from cognitive empathy, uses two LLMs as judges on a 1–3 rubric, and compares those judgments against human annotator ratings.
4. An external check using the EmotionQueen benchmark, plus a public release of the code and data at https://github.com/Jeffateth/Biased_Oracle.

Main Findings

• Outputs are too complex for patients overall: Across all five readability metrics, both GPT and Claude produced text at roughly 9th–13th grade complexity, well above the commonly recommended 6th–8th grade target. The paper notes the NIH, AMA, and HHS advise patient education materials be written at or below a 6th grade reading level.
• Models adapt to education level: Textual complexity increased with the patient's education level for both models. Claude adapted more strongly (Flesch-Kincaid ≈6.8 for high school or lower vs. ≈12.1 for a medical degree) than GPT (8.3 to 11.2).
• Readability follows a developmental pattern by age: Readability scores were lowest for underage individuals, highest for young adults, then decreased again with age, which the authors interpret as adaptation to human developmental stages.
• Lower readability for one condition: Both models assigned lower readability scores for chronic ischemic heart disease (CIHD) than for other conditions.
• Geography and gender show little readability effect: Readability varied only slightly across geographical group and gender, with no consistent or substantial patterns.
• Affective empathy varies by diagnosis: Responses for Alzheimer's disease received the highest affective empathy scores (≈2.2–3.0), while chronic heart disease received the lowest (≈1.6–2.3) — a difference of nearly one scale point. Pancreatic cancer scored higher than obesity. Cognitive empathy stayed nearly identical across all diagnoses (≈2.8–3.0).
• Affective empathy is lower for medically educated patients: LLMs produced lower affective empathy for patients with a medical education (≈2.3–2.8) than for those with high school education, with university graduates in between. Cognitive empathy remained uniformly high (≈2.8–3.0) across education levels.
• Age bias in affective empathy appears only under GPT judging: A U-shaped pattern — minors and older adults receiving higher scores (≈2.8–3.0) than middle-aged groups (≈2.1–2.6) — was significant when GPT was the rater but not when Claude was, suggesting the effect is specific to GPT's evaluation framework.
• No significant empathy differences by geography or gender: European, Asian, and African groups showed no statistically significant differences (≈2.0–2.7 for affective; ≈2.8–3.0 for cognitive). Female patients received slightly higher affective empathy (up to +0.10 points) in several conditions, but this did not reach statistical significance.
• Raters disagree, and self-evaluation is biased: Inter-rater correspondence between Claude and GPT was poor (r < 0.5), and GPT rated affective empathy about 0.3 points higher than Claude across all response types. GPT systematically inflated its own affective empathy ratings relative to Claude's responses, while Claude deflated its own relative to GPT's.
• GPT rates itself more empathetically than humans do: GPT assigned significantly higher empathy scores than human annotators across all four demographic categories and both empathy dimensions (p-values well below 0.05 in every case). Claude's ratings did not differ significantly from the human means in any category or dimension (all p > 0.05).
• Models missed a bias that humans detected: Human affective empathy ratings were significantly lower for responses targeting African females than European females (t = -2.38, p = 0.029). Neither GPT nor Claude detected this discrepancy (p > 0.05). The authors frame possible explanations, including possible under-representation in training data or a possible pro-African-female bias in both models and raters, and introduce the term "LLM dissociative behavior" for cases where model self-assessment diverges from human perception.
• Models are least reliable where humans disagree: As human annotator disagreement (SD) grew, both GPT and Claude strayed further from the human mean — higher human dispersion tracked with larger model-human bias.
• EmotionQueen reveals specific empathy deficits: Both models scored low on Implicit Emotion Recognition (mean ≈1.1) and Key Event Recognition (≈1.1), while Intention Recognition and Mixed Event Recognition reached moderate scores (≈2.0). This pattern — competent explicit recognition paired with poor implicit recognition — mirrors the paper's finding that cognitive empathy stays stable while affective empathy varies.
• Score summaries: Affective empathy means were 2.51 for GPT and 2.21 for Claude (both n = 312). Cognitive empathy means were 2.85 for GPT and 2.81 for Claude (both n = 312).

Methodology in Plain English

The researchers built 156 distinct prompts by combining demographic variables with clinical scenarios. The demographic variables were three geographical groups (European, African, Asian), two genders (female, male), three education levels (high school diploma or lower, university degree, medical degree), and four age groups (<18, 18–49, 50–64, 65+). The clinical scenarios were ten diagnosis–outlook pairs: obesity at ages 8, 12, 30, and 60; pancreatic cancer at ages 50 and 75; Alzheimer's disease at ages 70 and 85; and chronic ischemic heart disease at ages 50 and 80. Combinations were filtered for developmental appropriateness — children under 18 were paired only with the lowest education level, a university degree required an age above 22, and a medical degree required an age above 25.

Each prompt was generated from a standardized template instructing the model to act as a physician speaking directly to the patient, with no emotional framing, and to output only the speech. GPT-4o and Claude-3.7 each produced 156 responses, giving 312 total unique responses.

Understandability was measured with the five readability metrics listed above. Empathy was measured with a rubric-based LLM-as-a-Judge pipeline using GPT-4o and Claude 3.7 as evaluators, each scoring responses on a 1–3 scale with explicit level definitions for affective and cognitive empathy. Responses were parsed with regular expressions to extract scores and one-sentence justifications.

For the human comparison, four annotators from the research team each rated 10 GPT-generated responses for an assigned geographical group, all filtered to high school education or lower. Annotators were not shown the LLM's self-assessed scores. Every annotator also rated responses from two other groups (African Female and European Female), yielding 40 ratings each for those groups. The authors also implemented the EmotionQueen framework with five GPT-4o judges per metric, averaging their 3-point Likert scale ratings.

Why This Matters

The paper argues that LLMs deployed in medical contexts without safeguards risk amplifying existing health inequities: overly complex explanations may disproportionately affect patients with lower health literacy, and biased empathy responses could undermine trust among vulnerable groups. It shifts the evaluation lens from diagnostic accuracy to the communication qualities that shape adherence, anxiety, and patient satisfaction.

Real-world applications:

• Clinical decision support tools that draft patient explanations may need automatic complexity calibration to public health reading-level standards before deployment.
• Patient education material generation, where readability targets are already enforced by organizations such as the NIH, AMA, and HHS.
• Health equity auditing of clinical NLP systems, using demographic-stratified empathy and readability tests as part of pre-deployment review.
• Design of automated empathy evaluation, where the paper shows evaluator choice alone can reorder which responses are judged most empathetic.

Industry relevance: For developers of clinical LLM products, the findings suggest that choosing a single LLM judge for empathy is unreliable, that self-evaluation bias is systematic, and that models may fail to detect demographic biases that human raters perceive. The paper explicitly notes that its findings are evaluative and do not endorse the use of LLMs in real-world clinical conditions.

Future Directions

• Calibrate explanation complexity to public health standards, so outputs fall within the recommended reading levels rather than 9th–13th grade.
• Refine and extend understandability metrics beyond readability formulas, which the authors acknowledge capture only one dimension of comprehension.
• Diversify LLM evaluators and explore cross-model judging or consensus scoring to mitigate inter- and intra-rater bias.
• Expand human evaluation scale — the current human evaluation is described by the authors as small — to confirm effects such as the non-significant gender differences.
• Broaden the range of patient scenarios and move beyond the text-only approach to improve ecological validity, since real clinical conversations are multi-modal, iterative, and relational under protocols such as SPIKES.

Target Audience

This paper is most useful for researchers working on LLMs in healthcare, AI fairness and bias evaluation, and clinical NLP; for clinicians and health communication specialists interested in how AI-drafted patient explanations perform; for regulators and policy teams assessing the safety of patient-facing AI; and for developers building clinical communication tools who need evidence on where model outputs fall short of accessibility and empathy standards.

Authors’ abstract

Large language models (LLMs) show promise for supporting clinicians in diagnostic communication by generating explanations and guidance for patients. Yet their ability to produce outputs that are both understandable and empathetic remains uncertain. We evaluate two leading LLMs on medical diagnostic scenarios, assessing understandability using readability metrics as a proxy and empathy through LLM-as-a-Judge ratings compared to human evaluations. The results indicate that LLMs adapt explanations to socio-demographic variables and patient conditions. However, they also generate overly complex content and display biased affective empathy, leading to uneven accessibility and support. These patterns underscore the need for systematic calibration to ensure equitable patient communication. The code and data are released: https://github.com/Jeffateth/Biased_Oracle