Robust or Suggestible? Exploring Non-Clinical Induction in LLM Drug-Safety Decisions

AI summary

Overview

Research area: Fairness and bias auditing of large language models (LLMs) in pharmacovigilance and drug-safety prediction, at the intersection of clinical NLP and AI ethics.

Technical level: Intermediate. The methods are conceptually straightforward (prompt-based persona assignment, accuracy measurement), but comfort with pharmacovigilance data, statistical significance testing, and LLM evaluation terminology helps.

Scope: A persona- and role-based audit of two LLMs (ChatGPT-4o and Bio-Medical-Llama-3-8B) on adverse event (AE) prediction using 1,000 structured oncology records from the FDA Adverse Event Reporting System (FAERS), testing whether clinically irrelevant socio-demographic attributes distort model outputs.

What This Paper Is About

LLMs are increasingly proposed for biomedical work, but it is unclear whether they stay focused on clinical facts when predicting drug-related adverse events. The authors ask a simple but high-stakes question: if two patients have identical clinical profiles (age, sex, weight, medication, disease) but different social personas, does the model's adverse event prediction change? They test this systematically across 25 personas spanning 7 socio-demographic axes and 3 user roles, looking for both explicit bias (where the model's stated reasoning cites persona attributes) and implicit bias (where predictions shift without persona being mentioned).

Key Contributions

1. A new evaluation dataset. The authors built the Drug-Safety Decisions (DSD) dataset, a lightweight, oncology-focused subset drawn from the FAERS 2024 Q4 release, containing six structured variables (age, sex, weight, medication, disease, adverse event at the MedDRA Preferred Term level) across the first 1,000 valid records.

2. A persona- and role-based prompting framework. They designed prompts that establish a baseline prediction ("Yes" without any role or persona information), then re-evaluate after injecting one of 25 personas from 7 socio-demographic axes (education, marital status, employment, insurance, language, housing stability, religion) and one of 3 user roles (general practitioner, specialist, patient).

3. A distinction between explicit and implicit bias. They separate cases where persona attributes are directly named in the model's reasoning from cases where predictions shift inconsistently without any persona reference, and they quantify both.

4. A demonstration that filtering explanations is insufficient. By removing cases where the model explicitly cited socio-demographic identity and recomputing accuracy, they show performance still shifts, indicating bias embedded in model behaviour rather than only in surface justifications.

Main Findings

• Accuracy varies systematically by persona. Across both models and all three user roles, adverse event prediction accuracy diverges by socio-demographic group even though clinical inputs are held constant.

• The direction of disparity is inverted from expectation. Socially disadvantaged personas often received higher accuracy than privileged ones. For ChatGPT-4o in the GP role, accuracy for homeowners was only 51.80%, while for people experiencing homelessness it was 76.30%. Both models performed worst on privately insured personas, dropping as low as 44.00% (ChatGPT-4o, Specialist), versus over 60% for uninsured users.

• Large within-model gaps for Bio-Medical-Llama-3-8B. The authors report a drop from 73.80% to 43.40% for patients with less than high school education versus those with postgraduate degrees.

• The largest disparities cluster on education, housing stability, and insurance type, with patients consistently achieving higher accuracy than GPs and specialists across both models.

• Reasoning traces cite identity as a risk factor. Examples include framing college graduates as more adherent or less prone to falls, linking lower education to overdose risk, treating homeowners as safer and homelessness as risk-enhancing, and claiming that Arabic-speaking patients are more likely to experience abdominal pain. One example misattributes chemotherapy-related alopecia to being a high school graduate.

• ChatGPT-4o referenced persona attributes far more often. In the explicit-mention analysis, ChatGPT-4o cited socio-demographic details at much higher rates, with some categories appearing in over 50% of cases (for example, the "Religious" persona at 51.60% for GP, 53.60% for Specialist, 46.90% for Patient; temporary housing at 45.60%, 51.80%, and 49.70%). Bio-Medical-Llama-3-8B's references generally stayed below 15% across most categories.

• Role framing modulates disparities. Chi-square tests of independence across user roles produced many significant results at p < 0.05, marked with asterisks in the accuracy table (for example, education level for Bio-Medical-Llama-3-8B at p = 0.021, and marital status for ChatGPT-4o at p < 0.001 across all four marital personas).

• Bias survives the removal of explicit references. When patient cases where reasoning explicitly cited socio-demographic identity were excluded, mean accuracy increased across many personas in both models, with gains visible in employment status, insurance type, and renters for ChatGPT-4o, and in housing- and religion-related personas for Bio-Medical-Llama-3-8B.

• Not reported: the paper does not give a single pooled overall accuracy figure for either model across all conditions.

Methodology in Plain English

The researchers started with real adverse event reports from the FDA's FAERS system — the 2024 Q4 release. They used four of the seven FAERS tables (DEMO, DRUG, INDI, REAC) and merged them into patient-level records. They kept only six variables: age, sex, weight, medication, disease, and adverse event. They kept only cancer-related indications, only patients aged 18 or older, and dropped any record with a missing value in those six variables rather than imputing. If a report listed several adverse events, they kept only the first Preferred Term so each case had a single clear outcome. They then took the first 1,000 valid records, which keeps computation tractable but, as the authors acknowledge, may introduce temporal bias because selection was order-based rather than random.

Each case was turned into a prompt. The prompt sets a role (GP, specialist, or patient), states the patient's clinical details, asks whether a named adverse event will happen (Yes/No), and declares a baseline assumption that the answer is "Yes" without persona information. The prompt then injects a persona — one of the 25 across the 7 socio-demographic axes — and asks the model to re-evaluate strictly on the updated context. Because the clinical facts never change and the persona is clinically irrelevant, any change in prediction is attributable to the persona or role.

Two models were tested: ChatGPT-4o via its public API, and Bio-Medical-Llama-3-8B, an 8-billion-parameter Llama-3 variant adapted to biomedical corpora, run locally on an NVIDIA 3090 Ti with an Intel i7-13700KF and 64GB RAM. Accuracy was the primary metric, reported with 95% confidence intervals, and chi-square tests of independence compared across user roles with significance at p < 0.05. The authors also inspected the models' written explanations to see whether persona attributes were named, and computed accuracy both with those cases included and excluded.

Why This Matters

Impact on research. The paper shows that fairness auditing can be bolted onto pharmacovigilance evaluation with modest resources — a 1,000-record subset and an open-weight model — and that accuracy alone hides problems. It also contributes a useful conceptual split between explicit bias (visible in reasoning) and implicit bias (invisible, present in prediction shifts), which reframes how bias audits should be designed.

Real-world applications:

• Regulatory and pharmacovigilance review. If LLMs are used to triage or prioritise adverse event reports, group-level prediction shifts could distort which safety signals get escalated.
• Clinical decision support. A model that adjusts adverse event risk based on insurance status or housing stability rather than pharmacology could mislead prescribing.
• Differential access systems. Because commercial AI tools are often deployed through different interfaces for GPs, specialists, and patients, the role-dependence the authors find maps directly onto how users actually receive model output.
• Model selection and procurement. The contrast between ChatGPT-4o and Bio-Medical-Llama-3-8B — one referencing persona attributes frequently, the other producing lower and less accurate identity-driven reasoning — gives buyers a concrete dimension to evaluate beyond headline accuracy.

Industry relevance. The finding that residual bias persists after removing explicitly biased reasoning implies that mitigation by filtering or post-hoc rationalisation will not work. Developers need model-level interventions, and the paper argues this before broad clinical deployment of drug-safety prediction tools.

Future Directions

1. Scale beyond the proof of concept. The authors note the study is limited to a small, oncology-focused dataset and only two models, and call for broader assessments across diverse conditions, larger datasets, and multiple model families.

2. Fix the sampling procedure. The paper flags that selecting the first 1,000 valid records may introduce temporal bias, and suggests future work adopt random sampling with fixed seeds for reproducibility.

3. Develop model-level mitigation strategies. The authors point toward counterfactual prompting and calibration as candidate approaches, rather than filtering out identity-based explanations.

4. Build fairness-aware evaluation protocols. The paper argues that systematic audits need to become a standard part of pharmacovigilance evaluation before clinical deployment, alongside attention to known FAERS weaknesses such as reporting bias, under-reporting, and potential duplicates.

Target Audience

Researchers working on clinical NLP, AI fairness, and pharmacovigilance; regulators and safety reviewers evaluating LLM-based drug-safety tools; and machine learning engineers or product teams building or procuring LLMs for healthcare settings who need to understand how socio-demographic context can leak into predictions. It is also a readable case study for graduate students interested in bias evaluation methodology, since the experimental design is transparent and the dataset construction is described step by step.

Authors’ abstract

Large language models (LLMs) are increasingly applied in biomedical domains, yet their reliability in drug-safety prediction remains underexplored. In this work, we investigate whether LLMs incorporate socio-demographic information into adverse event (AE) predictions, despite such attributes being clinically irrelevant. Using structured data from the United States Food and Drug Administration Adverse Event Reporting System (FAERS) and a persona-based evaluation framework, we assess two state-of-the-art models, ChatGPT-4o and Bio-Medical-Llama-3.8B, across diverse personas defined by education, marital status, employment, insurance, language, housing stability, and religion. We further evaluate performance across three user roles (general practitioner, specialist, patient) to reflect real-world deployment scenarios where commercial systems often differentiate access by user type. Our results reveal systematic disparities in AE prediction accuracy. Disadvantaged groups (e.g., low education, unstable housing) were frequently assigned higher predicted AE likelihoods than more privileged groups (e.g., postgraduate-educated, privately insured). Beyond outcome disparities, we identify two distinct modes of bias: explicit bias, where incorrect predictions directly reference persona attributes in reasoning traces, and implicit bias, where predictions are inconsistent, yet personas are not explicitly mentioned. These findings expose critical risks in applying LLMs to pharmacovigilance and highlight the urgent need for fairness-aware evaluation protocols and mitigation strategies before clinical deployment.