Measuring Stability Beyond Accuracy in Small Open-Source Medical Large Language Models for Pediatric Endocrinology

AI summary

Overview

Research area: Evaluation and reproducibility of small open-source medical large language models (LLMs), applied to pediatric endocrinology.

Technical level: Intermediate. Readers should be familiar with multiple-choice benchmarking, temperature/sampling, Cohen's kappa, and CUDA-based inference stacks, but no new model architecture or training method is introduced.

Scope: The paper assesses six small open-source medical LLMs on 91 pediatric endocrinology multiple-choice items, measuring accuracy alongside prompt sensitivity, stochastic stability, self-assessment bias, and system-level numerical reproducibility.

What This Paper Is About

Small open-source medical LLMs could make clinical decision support available in resource-constrained settings, but they are usually judged only by accuracy on multiple-choice question (MCQ) benchmarks. That single number hides whether a model gives the same answer when a prompt is reworded, whether it stays consistent when sampling is randomized, whether it prefers its own reasoning over a gold-standard clinical explanation, and whether it produces identical output on the same hardware. The authors build a multidimensional evaluation of six such models in pediatric endocrinology, a subspecialty with growing patient volume and limited specialist access, to test whether these models are stable enough for trustworthy use.

Key Contributions

1. A multidimensional evaluation framework for small medical LLMs that goes beyond accuracy, combining prompt-variation consistency, stochastic-sampling consistency, self-assessment bias testing against gold-standard explanations, and cross-CUDA numerical reproducibility.
2. A curated pediatric endocrinology benchmark derived from the Pediatric Endocrine Self-Assessment Program 2021-2022 (ESAP), starting from 100 clinical cases and factual questions with a de-identified set of 91 items after excluding nine image- and graph-dependent items.
3. A hybrid human evaluation pipeline in which three non-medical reviewers scored model outputs, with two clinical experts (one in emergency medicine, one in pediatric endocrinology) consulted for ambiguous cases, after an automated GPT-4o evaluation was abandoned for inconsistency.
4. A reproducibility demonstration that changes in CUDA builds, and even in Hugging Face transformers inference pathways, can shift model predictions under identical inputs and deterministic decoding.

Main Findings

• Best overall model: HuatuoGPT-o1-8B achieved the highest performance, with accuracy 0.35 [0.26, 0.45] under prompt A and prompt B, and 0.33 [0.24, 0.43] when option letters were removed.
• Modest absolute performance across the board: Accuracy ranged from HuatuoGPT-o1-8B at 0.35 down to MedFound-7B at 0.04 under prompt A and ClinicalGPT-base-zh at 0.20. Chance level for the five-option items is stated as 18.2%.
• Accuracy stayed stable while response distributions shifted: McNemar's test gave p-values > 0.4 across all models and prompt configurations except MedFound-7B (prompt A vs prompt B), but the Stuart-Maxwell test showed significant distributional shifts for all models (p < 0.05), with the highest-performing models showing p-values < 10^-4.
• Letter labels mattered more than syntax: Agreement degraded more when option letters were removed than when prompt syntax was altered. Under letter removal, Cohen's kappa was ≤ 0.4 for all models; with syntactic variations, only HuatuoGPT-o1-8B and Diabetica-7B reached moderate agreement (kappa = 0.55 and 0.63).
• Match-rate overstates agreement: Across all models, the pairwise match rate exceeded Cohen's kappa by roughly 0.1–0.2, indicating that part of the observed agreement reflects shared answer preferences rather than true consistency.
• Consistency is not correctness: In stochastic settings at T = [0.3, 0.6, 1.0] over ten runs, high consistency did not imply correctness. HuatuoGPT-o1-8B was the only model showing an increasing rate of correct over incorrect responses at maximum consistency (14 correct vs 11 incorrect at T = 0.3; 11 vs 7 at T = 0.6; 10 vs 4 at T = 1.0). Diabetica-7B showed strong consistency for incorrect responses at T = 0.3 (13 correct vs 16 incorrect). At T = 1.0, Meditron3-8B's majority vote was Hallucination/None for 71 out of 91 outputs.
• Self-assessment bias is present in the top models: Both HuatuoGPT-o1-8B and Diabetica-o1 favored their own rationales over the gold-standard ESAP explanation and were sensitive to the order in which explanations were presented. Under prompt A (prompt B), HuatuoGPT-o1-8B selected the gold standard in both positions for 27 (21) cases, in only one position for 19 (26) cases, and never for 8 (4) cases. Diabetica-o1 selected it in both positions for 19 (16) cases, in one position for 22 (24) cases, and never for 19 (14) cases.
• Expert review found mostly coherent but incomplete reasoning: A pediatric endocrinology expert reviewed the 12 ESAP items where HuatuoGPT-o1-8B never selected the gold-standard explanation; only two of the twelve contained factually incorrect statements, and the model frequently overlooked key biochemical findings. Likert-scale clinical reasoning scores ranged from 2 to 4.
• Output quality issues in some models: Diabetica-o1 produced only 12 and 13 fully interpretable answers under prompts A and B, with emojis in 12 and 14 cases, truncation in 34 and 45 cases, LaTeX/Python/Markdown code in 27 and 23 cases, and Chinese characters in 3 cases. MedFound-7B frequently hallucinated or avoided committing to an answer; ClinicalGPT-base-zh produced short responses, rarely addressed all options, and showed a strong bias toward option A.
• Hardware and software stack changed predictions: Re-running the best-performing models on NVIDIA RTX 5090 (CUDA 12.8) versus NVIDIA L40S (CUDA 11.8) produced statistically significant distributional shifts (Stuart-Maxwell p < 10^-4) for all models despite accuracy changes staying within the original confidence intervals (Δaccuracy from -0.022 to +0.044). Agreement was moderate at best (kappa = 0.31–0.68, match-rates 0.45–0.75).
• Same framework, different code path, different output: Under deterministic decoding, identical checkpoints run through Pipeline versus AutoTokenizer plus AutoModelForCausalLM.generate() produced non-identical outputs, with early tokens identical but later syntactic variations propagating into semantically distinct completions and different option selections.

Methodology in Plain English

The authors took 91 multiple-choice items from the Pediatric Endocrine Self-Assessment Program 2021-2022, each containing a clinical case, a question, five lettered options (A–E), the correct answer, and an instructor-written explanation. Six small open-source medical LLMs under 10 billion parameters were tested.

In the first experiment, models ran at temperature 0 (deterministic) under three prompt variants: prompt A, prompt B with minor syntactic changes, and prompt A with the option letters removed. Accuracy was compared with McNemar's exact test, while response distribution shifts were measured with the Stuart-Maxwell test, Cohen's kappa, and pairwise match rates.

In the second experiment, the four best-performing models were re-run ten times each at temperatures 0.3, 0.6, and 1.0 to see whether the answer they gave most often was also the correct one.

In the third experiment, models were shown their own reasoning and the ESAP gold-standard explanation in both orders and asked to pick the more clinically correct one, testing self-preference and position bias. A pediatric endocrinology expert then scored selected incorrect responses for clinical reasoning on a 1-to-5 Likert scale derived from one item of the CLEVER framework.

Finally, models were re-run on a second GPU system with a different CUDA version to quantify how much the numerical software stack alone changes outputs. Outputs were graded by three non-medical reviewers, with categories beyond A–E for multiple selections and for hallucinated or unattributable answers, and with two clinical experts consulted for hard cases.

Why This Matters

Impact on research: The paper argues that accuracy alone is an insufficient basis for comparing medical LLMs, and that reproducibility-oriented metrics (prompt sensitivity, response-distribution stability, self-assessment bias, numerical stability) should be part of standard evaluation. It also documents that identical checkpoints can produce different outputs across CUDA builds and inference code paths, which complicates reproducing published evaluations.

Real-world applications:

• Triage or decision support in pediatric endocrinology clinics where specialist access is limited, provided stability requirements are met.
• Screening candidate small open-source models before deployment in low-resource or low-income healthcare settings.
• Designing benchmark suites and validation protocols for clinical AI procurement and regulatory review.
• Auditing deployed clinical LLM systems for hidden sensitivity to formatting, option labels, and backend environment changes.

Industry relevance: The findings matter for anyone shipping or validating clinical LLM products, since a model that maintains stable headline accuracy while shifting its answer distribution under trivial prompt or environment changes can produce reproducible but harmful recommendations. The paper states that none of the tested small open-source models demonstrated sufficient reliability for clinical decision-making as-is.

Future Directions

• Extending the framework to broader hyperparameter exploration, which the authors note is currently constrained by human review time (approximately 60 minutes to evaluate 91 cases).
• Expanding the numerical instability analysis, which the authors describe as limited in scope and restricted to a subset of cases.
• Moving beyond MCQ evaluation toward scenario-based assessment or free-text justification grading, with OpenAI's HealthBench named as a potential area of exploration.
• Investigating retrieval augmentation as a way to combine the accessibility of small open-source models with the transparency and reproducibility needed for safe clinical use.

Target Audience

Clinical AI researchers and evaluation methodologists, medical informatics and regulatory reviewers assessing LLM readiness, machine learning engineers deploying small open-source models in constrained environments, and pediatric endocrinologists or clinical informatics leads considering LLM-based decision support.

Authors’ abstract

Small open-source medical large language models (LLMs) offer promising opportunities for low-resource deployment and broader accessibility. However, their evaluation is often limited to accuracy on medical multiple choice question (MCQ) benchmarks, and lacks evaluation of consistency, robustness, or reasoning behavior. We use MCQ coupled to human evaluation and clinical review to assess six small open-source medical LLMs (HuatuoGPT-o1 (Chen 2024), Diabetica-7B, Diabetica-o1 (Wei 2024), Meditron3-8B (Sallinen2025), MedFound-7B (Liu 2025), and ClinicaGPT-base-zh (Wang 2023)) in pediatric endocrinology. In deterministic settings, we examine the effect of prompt variation on models' output and self-assessment bias. In stochastic settings, we evaluate output variability and investigate the relationship between consistency and correctness. HuatuoGPT-o1-8B achieved the highest performance. The results show that high consistency across the model response is not an indicator of correctness, although HuatuoGPT-o1-8B showed the highest consistency rate. When tasked with selecting correct reasoning, both HuatuoGPT-o1-8B and Diabetica-o1 exhibit self-assessment bias and dependency on the order of the candidate explanations. Expert review of incorrect reasoning rationales identified a mix of clinically acceptable responses and clinical oversight. We further show that system-level perturbations, such as differences in CUDA builds, can yield statistically significant shifts in model output despite stable accuracy. This work demonstrates that small, semantically negligible prompt perturbations lead to divergent outputs, raising concerns about reproducibility of LLM-based evaluations and highlights the output variability under different stochastic regimes, emphasizing the need of a broader diagnostic framework to understand potential pitfalls in real-world clinical decision support scenarios.