From Knowledge to Inference: Formalizing Specialized Public Health Reasoning on GlobalHealthAtlas

AI summary

Overview

Research area: Natural Language Processing / domain-specific large language model (LLM) training and evaluation, focused on public health reasoning.

Technical level: Intermediate. The paper assumes familiarity with supervised fine-tuning, LoRA, LLM-as-a-judge evaluation, and data-leakage testing, though its core argument is accessible.

One-sentence scope: The paper introduces GlobalHealthAtlas, a 280,210-instance multilingual public health dataset with 15 domains and 17 languages, plus a six-dimension evaluator and a fine-tuned Public-Model, to formalize population-level public health reasoning as a measurable machine learning problem.

What This Paper Is About

Existing medical AI benchmarks focus on clinical tasks such as diagnosis support and exam-style question answering, and they leave public health reasoning — population-level inference grounded in scientific evidence, expert consensus, and safety constraints — largely unmeasured. The authors argue this gap comes from two things: a lack of high-quality public health data with narrow and imbalanced topic coverage, and the absence of evaluation frameworks aligned to public health concerns like policy context and intervention effectiveness.

The paper's goal is to build a large, authoritative, multilingual public health dataset, an evaluator that scores model outputs along public-health-specific dimensions, and a fine-tuned model that demonstrates the dataset's value.

Key Contributions

1. GlobalHealthAtlas dataset. A multilingual public health dataset of 280,210 instances with complete Chain-of-Thought reasoning, synthesized from over 35,500 sources from WHO IRIS, spanning 15 public health domains and 17 languages, with three difficulty levels and both Question-Answer and Single-Choice formats.

2. A generalizable AI4S data construction paradigm. A reproducible, model-driven pipeline combining prompt-driven synthesis, multi-stage quality gating with weighted scoring, evidence-grounding checks, deduplication, leakage detection, and expert-calibrated thresholds.

3. A domain-aligned evaluator with six orthogonal dimensions. A fine-tuned scorer (Public-Evaluator, based on qwen3-8b with LoRA) that rates outputs on Accuracy, Reasoning, Completeness, Consensus Alignment, Terminology Norms, and Insightfulness.

4. Public-Model, a public-health-tailored LLM. A model built on the dataset and evaluator that shows consistent gains in public health reasoning, and the accompanying benchmarking, ablation, data-scaling, transfer, and robustness experiments.

Main Findings

• Clinical models do not transfer to public health. In a domain performance gap analysis, Medical_Model(8B) scored 5.250 on GlobalHealthAtlas and 4.697 on Instruction-public-health-dataset, while scoring 7.184 on Medical_test_modified. Larger models did not saturate the benchmark either: AntAngelMed reached 6.449 and BAICHUAN-M3 reached 5.970 on GlobalHealthAtlas. Public-Model scored 6.532.

• Domain fine-tuning closes much of the gap. Public-Model, based on qwen3-8b, reached 6.619 on the representative benchmark subset, a +0.685 gain over the vanilla qwen3-8b score of 5.934, and placed third overall in that comparison. GLM-4.7 led at 7.326, followed by kimi-k2-thinking at 6.930.

• Reasoning-oriented models dominate. Strong reasoning models outperformed standard instruction-following baselines. Beyond GLM-4.7 and kimi-k2-thinking, claude-sonnet-4-5-20250929-thinking scored 6.534, grok-3-mini 6.352, and deepseek-v3.2 6.365.

• Models score higher on harder-profiled tasks. Accuracy was higher on Difficulty A (Academic/Professional) than Difficulty C (Popular Science). The authors attribute this to construction: Level C is curated from specific regulations, while Level A emphasizes methodological inference, so models can reason logically even without specific parametric knowledge.

• Supervised fine-tuning helps across model families. In the ablation on the same 247,599-instance training set, qwen3-8b improved on GlobalHealthAtlas from 6.490 to 6.532, qwen3-14b from 6.134 to 6.565, and Llama-3.1-8B-Instruct from 5.461 to 6.425. Gains also appeared on general medical benchmarks, for example qwen3-14b on medical-o1-reasoning-SFT improved from 5.906 to 6.643.

• Data scaling plateaus. Across qwen3-4b, qwen3-8b, and qwen3-14b at 0%, 10%, 30%, 60%, and 100% of data, performance generally rose but plateaued or slightly degraded beyond the 60% mark: qwen3-8b went 6.4896 (0%) to 6.5279 (60%) to 6.5316 (100%), and qwen3-14b peaked at 6.5748 at 60% before dipping to 6.5648 at 100%.

• Cross-domain transfer is mixed but recoverable. SFT improved GPQA for all three qwen3 sizes (4B: 4.948 to 5.342; 8B: 5.342 to 5.904; 14B: 5.277 to 5.723) and MMLU-pro for 4B (6.762 to 7.043) and 8B (7.139 to 7.405), but slightly reduced MMLU-pro for 14B (7.426 to 7.309). Weight-space interpolation at 0.85 recovered MMLU-pro to 7.545 while keeping GlobalHealthAtlas at 6.512 versus 6.5648 for pure SFT.

• Domain-aligned models are more robust to perturbations. On 1,000 multiple-choice questions, Public-Model retained 86.00% accuracy under linguistic reformulation and 84.40% under cross-lingual translation. Under noise injection, the hardest setting, Public-Model and qwen3-14b retained around 77% (77.34% and 77.10%), while Llama-3.1-8B-Instruct dropped to 50.00%.

• The evaluator matches experts better than baselines. Public-Evaluator achieved the lowest MAE (1.4259) and highest ICC (0.9735) among compared models on 100 independent human-annotated scores, plus the highest IdenticalRate (0.5533) and lowest StdDev (0.2772) across ten inference runs. On out-of-distribution MedQA data, it showed no over-penalization (0.00% over-penalization rate) and correlated with a general medical baseline at r = 0.83 for Reasoning and r = 0.78 for Terminology Norms.

• Leakage is minimal. In 5-gram and 10-gram continuation tests, exact matches were rare: most tested models showed 0% leaked data proportion in both settings, with the highest at 0.97% (qwen3-max, 5-gram).

• Human quality check. A 5% sample (14,010 examples: 7,077 single-choice and 6,933 question-answer) covering all 15 domains and 17 languages was scored by 14 experts on the same rubric; the average human score was 4.503.

Methodology in Plain English

The authors start from authoritative documents — primarily WHO IRIS — and convert heterogeneous PDFs into structured Markdown while preserving document hierarchy. The text is chunked into coherent evidence segments, then a prompt-driven engine generates either question-answer or single-choice items along with a detailed reasoning trace. Each item is tagged with metadata: domain, language, difficulty, and label.

A quality gate then scores every candidate on four weighted dimensions — Question/Option Quality, Answer Quality, Text Relevance, and Overall Consistency — using different weights for single-choice items (0.45/0.25/0.20/0.10) and question-answer items (0.25/0.35/0.25/0.15). Seven senior global health experts averaging over 8 years of experience, including WHO officials, finalized the weights by structured voting. Items with a score below 4.5, or with domain irrelevance, language inconsistency, or missing materials, were rejected.

For the evaluator, the authors used multi-source sampling (responses from diverse LLMs such as qwen, deepseek, and GPT) to avoid style overfitting, hierarchical consensus filtering to produce high-confidence labels, and adversarial samples for robustness. The consensus procedure compares scores from qwen3-Max and deepseek-V3.2; if they differ by more than 2, gpt-5 adjudicates, and if the adjudicated score is not one of the two, the item goes to human review. The evaluator itself was trained with LoRA supervised fine-tuning on qwen3-8b, and the Public-Model used the same technique on the 247,599-instance training set.

Why This Matters

Public health decisions affect whole populations, so models used in this space need to be evaluated on evidence grounding, consensus alignment, and safety — not just exam-style accuracy. This paper provides an open dataset, evaluator, and model so that public health reasoning can be trained and measured reproducibly rather than treated as a byproduct of clinical benchmarks.

Real-world applications:

• Epidemic surveillance and response support: Models that reason about infectious disease prevention, surveillance, and response, one of the high-priority domains in the dataset.
• Health policy and governance analysis: Evaluating whether model outputs align with authoritative scientific consensus and policy context, rather than only factual recall.
• Vaccination and immunization communication: Producing immunization guidance that respects terminology norms and safety constraints across 17 languages.
• Multilingual public health information delivery: Cross-lingual performance results (Public-Model at 84.40% under translation perturbation) indicate viability for non-English health communication, including child and adolescent health.

Industry relevance: Organizations deploying LLMs in health-adjacent products now have a domain-specific scorer (Public-Evaluator) and a reference tuned model (Public-Model) that can be used as quality gates, plus a construction paradigm that can be adapted to other data-scarce scientific domains. The finding that data quality can partially compensate for model scale is directly relevant to teams choosing between larger general models and smaller fine-tuned ones.

Future Directions

• Explaining the data-scaling plateau. Performance improved up to roughly 60% of the training data and then flattened or slightly declined, which raises the question of whether the limit is data volume, data diversity, or optimization.
• Reducing source-domain bias in larger models. qwen3-14b lost MMLU-pro performance after fine-tuning on GlobalHealthAtlas, and the paper only partially addresses this with weight interpolation; a more principled multi-domain training approach remains open.
• Extending beyond the current domain and language coverage. The dataset covers 15 domains and 17 languages, and the paper reports a "long-tail" of complementary fields, leaving room to expand coverage and test whether the construction pipeline generalizes to other scientific fields.
• Understanding the Difficulty A versus C inversion. The counterintuitive result that models score higher on Academic/Professional items than Popular Science items suggests task construction, not just model capability, shapes measured performance — a target for further diagnostic work.

Target Audience

• Public health researchers and epidemiologists who need machine-readable, multilingual evaluation resources grounded in authoritative sources.
• ML engineers and applied NLP practitioners building or fine-tuning LLMs for health domains, who can reuse the dataset, evaluator, and construction pipeline.
• Benchmark and evaluation researchers interested in LLM-as-a-judge design, multi-dimensional rubrics, consensus filtering, and leakage detection.
• Policy and health-informatics teams assessing whether model outputs align with scientific consensus and safety guidelines before deployment.

The paper is not reported to include a formal limitations section in the provided content, and several details (full rubric definitions, prompt sets, and hyperparameters) are referenced only as being in the appendix.

Authors’ abstract

Public health reasoning requires population level inference grounded in scientific evidence, expert consensus, and safety constraints. However, it remains underexplored as a structured machine learning problem with limited supervised signals and benchmarks. We introduce GlobalHealthAtlas, a large scale multilingual dataset of 280,210 instances spanning 15 public health domains and 17 languages. We further propose a large language model (LLM) assisted construction and quality control pipeline with retrieval, deduplication, evidence grounding checks, and label validation to improve consistency at scale. Finally, we present a domain aligned evaluator distilled from high confidence judgments of diverse LLMs to assess outputs along six dimensions: Accuracy, Reasoning, Completeness, Consensus Alignment, Terminology Norms, and Insightfulness. Together, these contributions enable reproducible training and evaluation of LLMs for safety critical public health reasoning beyond conventional QA benchmarks. We publicly release project codebase, evaluator, and model at:: https://github.com/Jan8217/GlobalHealthAtlas, https://huggingface.co/aerovane0/GlobalHealthAtlas_Public_Evaluator and https://huggingface.co/aerovane0/GlobalHealthAtlas_Public_Model