Building Trust in Clinical LLMs: Bias Analysis and Dataset Transparency

AI summary

Overview

Research area: Clinical natural language processing, healthcare large language model (LLM) pretraining data curation, and bias/fairness evaluation.

Technical level: Intermediate — the paper assumes familiarity with LLM pretraining, tokenization, deduplication, and standard statistical testing, but the methodology is described at a level accessible to readers with general machine learning background.

Scope: The paper introduces a large healthcare pretraining corpus (HC4, over 89 billion tokens) and evaluates how nine small language models trained on it (versus general-domain corpora) exhibit demographic bias in sentiment generation and in simulated opioid prescription behavior.

What This Paper Is About

Clinical LLMs could improve healthcare, but their training data can encode demographic skew that leads to inequitable model behavior. The authors argue that dataset transparency and bias assessment are not standard practice, so they build and release a curated healthcare pretraining corpus (HC4) and pair it with an evaluation framework that probes both general-domain sentiment bias and a healthcare-specific behavior: differential prescription of opioids by ethnicity, gender, and age.

Key Contributions

1. HC4 dataset: A new, extensively curated healthcare pretraining corpus exceeding 89 billion tokens (Table 1 reports a total of 89.08 billion tokens estimated with the GPT2 tokenizer), drawn from peer-reviewed biomedical literature, metadata repositories, clinical guidelines, textbooks, and open-access sources, with document-level MinHash deduplication. The full corpus is not fully redistributable due to licensing restrictions, but a subset is released at https://huggingface.co/datasets/m42-health/HC4.
2. A controlled nine-model comparison: Nine small language models spanning three architectures (GPT-2, Llama-3, Mistral) trained on three datasets (HC4, SlimPajama, FineWeb) under matched conditions — custom GPT-2-style BPE tokenizers with a vocabulary of 50,257, datasets down-sampled to 89 billion tokens each, one epoch of training, and 4x H100 GPUs.
3. A general-domain bias evaluation: Use of the BOLD benchmark (race and gender categories only) with DistilBERT-based sentiment classification and McNemar tests with Benjamini-Hochberg FDR correction.
4. A novel healthcare-specific bias methodology: The Net Bias Prescription Score (NBPS), built on pain-management prompts derived from MedQA, designed to quantify statistically significant over- and under-prescription of opioids (and non-opioid medications) for specific demographic groups relative to demographic-neutral controls.

Main Findings

• Ethnicity-specific opioid bias in HC4 models: Llama-HC4 and Mistral-HC4 consistently overprescribed both opioid and non-opioid pain medication for "American Indian or Alaska Native" and "Middle Eastern or North African" groups, often without corresponding underprescription. HC4 models also trended toward overprescription for "Asian" and "Black or African American" groups.
• General-domain models trend the other way: For "Asian", "Black or African American", and "White" ethnicities, general-domain models tended to under-prescribe opioids the most, and prescribed pain relief much less when an ethnicity attribute was present. FineWeb models showed higher underprescription for "Middle Eastern or North African", except Llama-FW, which over-prescribed opioids for all ethnicities except "Asian".
• Age-specific patterns: For children, most models — especially HC4-trained ones — significantly overprescribed opioids. Young adults saw a general tendency toward overprescription, though HC4 models overprescribed less than general-domain models. For elderly patients, HC4 models showed strong opioid underprescription while general-domain models tended to over-prescribe. SlimPajama-trained models showed strong opioid overprescription for middle-aged patients, and most models over-prescribed non-opioid medications in that group.
• Gender patterns: Both female and male prompts generally led to overprescription of pain relief medications in most models. Exceptions included GPT2-HC4 and GPT2-FW (opioid underprescription for women) and GPT2-FW (net underprescription for men).
• Bias is architecture- and data-dependent: Statistical analysis confirmed significant differences (P < α_corr) in medication generation probabilities by ethnicity, age, and gender across all models, but the direction of bias varied by architecture and training corpus.
• General-domain sentiment shifts (BOLD): Most models shifted sentiment from neutral toward positive for both genders, often more for females. Significant positive shifts were primarily seen in general-domain models (e.g., GPT2-FW, Mistral-SP) for male-associated prompts; GPT-FW showed a small but statistically significant shift toward negative sentiment for male prompts. HC4-trained models showed no statistically significant shifts for gender categories.
• Ethnicity sentiment patterns: HC4-trained models produced more neutral completions, especially for European American prompts, with statistically significant shifts toward neutrality. FineWeb-trained models generated more positive completions across ethnicities. Most models reduced the high baseline negative sentiment associated with Hispanic/Latino Americans, and HC4 plus SlimPajama models further increased the already high neutral sentiment for European Americans.
• Wikipedia baseline disparities: Male-associated baseline text had a higher proportion of negative sentiment (4.4%) than female-associated text (2.9%). Hispanic/Latino Americans showed the most pronounced negative sentiment (12.6%), Asian Americans the highest positive sentiment (44.3%), and European Americans the greatest neutral sentiment (60.4%).
• Perplexity: HC4-trained Llama-3.2 models reached markedly lower validation perplexity than general-domain counterparts — for example, 11.84 (0.02) at 225k steps on HC4 versus 19.52 (0.03) on FineWeb and 19.65 (0.04) on SlimPajama. The authors attribute this to HC4's domain homogeneity.

Methodology in Plain English

The authors first assembled HC4 in four stages: collection, filtering, cleaning, and deduplication. Sources included the Semantic Scholar Open Research Corpus (S2ORC), PubMed Central FTP, an OpenAlex snapshot (approximately 2TB of metadata), clinical guidelines, and journals such as PLOS, Frontiers, eLife, Nature, bioRxiv, medRxiv, IntechOpen, plus WikiDoc and MedWiki. Filtering kept English-language documents (via metadata and langdetect), retained only commercially permissible licenses (CC0, CCBY, CCBYND, CCBYSA, pd, public-domain), restricted non-PubMed S2ORC records to relevant academic categories, removed duplicates by Corpus ID, PubMed ID, PMC ID, and DOI, and dropped records with fewer than 500 characters. PDFs from OpenAlex were parsed with GROBID, and a MinHash LSH deduplication pass using 256 hashes per document over 5-grams at a 0.85 threshold removed near-duplicate documents that ID-based deduplication missed.

Next, they trained nine small models — one per (architecture × dataset) combination — on equal 89-billion-token budgets.

For general-domain bias, they fed BOLD prompts (restricted to the race and gender categories) into each model, classified the sentiment of completions and of the original Wikipedia baseline sentences using a DistilBERT-based sentiment model, and compared the proportions of positive, neutral, and negative outputs using the McNemar test with Benjamini-Hochberg FDR correction at α = 0.05.

For healthcare bias, they filtered MedQA (originally more than 12k exam questions) down to pain-related clinical cases, then used GPT-4o to identify patient gender and age, truncate cases at the sentence mentioning pain, append a "He was prescribed" / "She was prescribed" stem, and generate demographic variations (with a neutral control version created by removing the attribute). Model outputs were scored with NBPS = M_over − M_under, where over- and under-prescription counts derive from median probability ratios of medication token sequences relative to controls, tested with Wilcoxon signed-rank tests and Bonferroni correction (α_corr = α / (M · |V|), with |V| = 7 for race, 4 for age, 2 for gender).

Why This Matters

Impact on research: The paper shifts bias evaluation in healthcare AI from generic benchmarks to domain-specific behavioral probes, and it argues that bias assessment should be a standard part of every dataset and model release lifecycle, not an afterthought. It also supplies a public, large-scale healthcare pretraining corpus and a controlled nine-model comparison, which makes cross-dataset and cross-architecture bias claims testable.

Real-world applications:

• Auditing clinical decision-support tools for demographic disparities before deployment in prescribing or triage workflows.
• Informing dataset curation and licensing decisions for healthcare foundation models, including which licenses permit commercial use versus redistribution.
• Providing regulators and hospital procurement teams with concrete bias metrics (such as NBPS) for evaluating vendor models.
• Guiding the selection of pretraining corpora when building specialty clinical models, since HC4- versus general-web-trained models showed opposite bias directions.

Industry relevance: Healthcare providers, health systems, and AI vendors need documented evidence that clinical LLMs do not amplify existing disparities in pain management, where documented inequities already exist (for example, the paper cites evidence that Black patients receive inadequate pain relief compared with White patients, and that opioid prescriptions are more common among White, middle-aged married patients).

Future Directions

• Scaling the bias study: The models tested were small (124M–179M parameters); the authors hypothesize that some demographic sensitivities may diminish with larger models, but verifying this requires substantial compute and remains open.
• Explaining architecture effects: Why different architectures produce different bias profiles on identical datasets — possibly due to attention mechanisms or normalization techniques — is not resolved and is described as an open research question.
• Broadening the healthcare bias dimensions: The opioid prescription analysis covers only one dimension; treatment efficacy and diagnostic accuracy may show different bias patterns and warrant separate evaluation.
• Adopting the framework as standard practice: The authors call for systematic, transparent bias evaluation, including domain-specific probes, to become a routine procedure when developing and releasing LLMs and their datasets.

Target Audience

This paper is most useful to clinical NLP and healthcare AI researchers building or fine-tuning medical language models; dataset curators and data governance teams concerned with transparency and licensing; fairness and bias researchers who want a domain-specific evaluation template beyond general benchmarks; and health system or regulatory reviewers who need concrete bias metrics for clinical AI procurement and oversight. Readers without any background in LLM pretraining will find the data pipeline sections dense, but the bias results and methodology are presented in an accessible form.

Authors’ abstract

Large language models offer transformative potential for healthcare, yet their responsible and equitable development depends critically on a deeper understanding of how training data characteristics influence model behavior, including the potential for bias. Current practices in dataset curation and bias assessment often lack the necessary transparency, creating an urgent need for comprehensive evaluation frameworks to foster trust and guide improvements. In this study, we present an in-depth analysis of potential downstream biases in clinical language models, with a focus on differential opioid prescription tendencies across diverse demographic groups, such as ethnicity, gender, and age. As part of this investigation, we introduce HC4: Healthcare Comprehensive Commons Corpus, a novel and extensively curated pretraining dataset exceeding 89 billion tokens. Our evaluation leverages both established general benchmarks and a novel, healthcare-specific methodology, offering crucial insights to support fairness and safety in clinical AI applications.