SAGE: Semantic Anchor-Guided Evolution for Grounded Medical QA Data Synthesis

AI summary

Overview

Research area: Natural Language Processing, specifically synthetic training-data generation for medical question answering (clinical LLMs).

Technical level: Advanced. The paper mixes a formal neural-symbolic formulation (taxonomy graph, anchor populations, mutation operators) with a large empirical study across medical QA benchmarks.

Scope: The paper introduces SAGE, a framework that lets small, locally deployed language models synthesize grounded medical QA training data using publicly available medical taxonomies (MeSH) as "semantic anchors," instead of relying on large document corpora or external proprietary APIs.

What This Paper Is About

Building reliable medical question-answering models is bottlenecked by the scarcity of high-quality, expert-annotated clinical data, which is expensive to produce and constrained by privacy rules; hospitals also often lack the resources to run large proprietary LLMs or cloud APIs. Existing synthetic-data routes are flawed: "self-derived" methods generate diverse data from a model's own parametric knowledge but hallucinate facts, while "document-based" methods are factually grounded but need big curated corpora and tend to teach memorization rather than reasoning. SAGE's goal is to synthesize high-quality, grounded medical QA data from a small local model and a small seed set, with no external API and no large document collection.

Key Contributions

1. SAGE framework: A Semantic Anchor-Guided Evolution method for synthesizing grounded medical QA data that uses small models and public taxonomies (MeSH), avoiding large corpora and external APIs.
2. Parallel iterative synthesis strategy: A dual-track design that combines atomic generation (single-concept, definition-oriented) for factual coverage with associative generation (multi-concept) for reasoning complexity, balancing exploitation and exploration.
3. Grounded, evolving data quality: SAGE-grounded data is reported to substantially improve factual reliability and consistency, with fine-tuned models consistently beating strong baselines in resource-constrained settings.
4. Empirical validation across medical QA: Extensive experiments over multiple medical QA benchmarks plus ablations of all four evolutionary components, scaling analysis, and out-of-distribution evaluation.

Main Findings

• Average accuracy of 68.12%: SAGE's full configuration reaches 68.12% average accuracy across five medical QA benchmarks in the main table — MedMCQA 64.98, MedQA 70.07, PubMedQA 80.80, MMLU-Pro (medical) 63.19, and GPQA (medical) 61.54.
• Beats self-derived methods at equal scale: All self-derived baselines were generated at 10k samples with the same seed set; the best of them, CoT-Self-Instruct, reached 63.71 average, versus SAGE's 68.12.
• Beats document-based methods: Document-based baselines used MeSH ontology scope notes as a unified reference corpus; Easy-Dataset averaged 49.11 (28k), Genie 49.98 (30k), Self-QA 46.34 (230k), and Wrap 50.20 (98K).
• Rivals or exceeds open-source medical datasets: m1k reached 63.40 (1k), m23k 67.39 (23k), Huatuo-SFT 52.80 (20k), and MedReason 58.60 (33k).
• Every component of the evolutionary loop helps: Removing iteration drops the average to 66.45, removing the parallel design gives 66.81, removing atomic generation gives 66.21, and removing associative generation gives 66.13 — all below the full 68.12.
• Generalizes beyond QA: On MedNLI (Natural Language Inference) SAGE reaches 87.27 versus CoT-Self-Instruct 84.60 and Wrap 72.08; on MTS-Dialog (medical summarization, measured with BERTScore) it reaches 85.02 versus 83.16 and 83.62.
• Cross-scale efficiency: SAGE-4B's 68.12 average matches HuatuoGPT-o1-8B at 68.14, is above UltraMedical-8B at 62.56, m1-7B at 63.55, and MedReason-8B at 65.42, and approaches Qwen2.5-72B-Instruct at 68.03.
• Balanced factuality and reasoning: A judging study over 1,000 generated samples using DeepSeek-V3.2 found document-based Wrap high on factuality but weak on reasoning depth, self-derived ScaleQuest moderate on reasoning but with factual hallucinations, while SAGE balanced both. SAGE also minimized confabulations on the TruthfulQA and MedHALT hallucination benchmarks against all baselines.
• Semantic depth and diversity hold up as data scale grows: Over 10,000 generated points, evaluated per batch of 1,000 samples (diversity measured by discovery rate R = N_new / B), SAGE maintained high semantic depth and combinatorial diversity, whereas KPDDS and MathScale collapsed toward broad, shallow, recurring concepts.

Methodology in Plain English

SAGE treats a medical taxonomy — MeSH — as a fixed map of medical concepts. Each concept, or group of concepts, becomes a "semantic anchor" that defines what a generation step should be about.

Starting from a tiny seed set of QA examples (m1k, 1,000 instances), the system maps each question to MeSH concepts to build an initial pool of anchors. Anchors with a single concept go to an atomic track: the model is given that concept's official definition and a clinical qualifier such as pathology or adverse effects, and writes a definition-centered question. Anchors with multiple concepts go to an associative track: the model is given a few examples and asked to build a clinical scenario that links those concepts, requiring multi-step reasoning.

Generated candidates are filtered by self-consistency — the model produces K auxiliary answers and a candidate survives only if its answer matches the majority vote — plus diversity controls that retire anchor configurations whose expansion budgets are used up. Surviving questions are mapped back onto MeSH. Any new concepts that appeared during generation (for instance, a side effect mentioned while discussing a drug) are added as fresh single-concept anchors for the next round, while accepted multi-concept configurations are carried forward. Repeating this loop grows the explored region of the concept space from minimal seeds.

The experiments use Qwen3-4B with thinking disabled as the data-synthesis backbone for all methods, then fine-tune a fresh Qwen3-4B-Base on each resulting dataset with an identical protocol: 5 epochs, learning rate 1×10⁻⁵, global batch size 16. Generation settings are temperature 0.7, top-p 0.95, maximum length 4,096 tokens, N=2 candidates per anchor, n_few = 2 few-shot examples, and K=3 auxiliary answers for consistency checking.

Why This Matters

Research impact. The paper positions structured medical taxonomies as a third path between ungrounded self-generation and corpus-heavy document grounding. It provides evidence that a neuro-symbolic scaffold (fixed ontology plus a language model acting as a variation operator) can produce training data that competes with much larger datasets and larger models, which reframes how data efficiency in high-stakes domains can be measured.

Real-world applications.

• Hospitals and clinics with limited compute could build medical QA training data on-premises from a small local model and a public taxonomy, without sending data to external APIs.
• Medical education and exam preparation, since SAGE-generated items follow clinical examination perspectives and the target benchmarks include entrance-exam-derived sets.
• Quality improvement for clinical reasoning assistants trained on synthetic questions rather than extracted text.
• Low-resource or privacy-sensitive healthcare settings in general, where large annotated corpora and proprietary cloud models are unavailable.

Industry relevance. The approach targets the practical constraint that proprietary LLM APIs and large proprietary corpora are often off-limits in healthcare. A 4B model's synthesized data matching an 8B medical model's performance (68.12 versus 68.14 for HuatuoGPT-o1-8B) is directly relevant to cost, deployment footprint, and data-governance decisions in medical AI products. The authors also note that the work was supported by the Guizhou Provincial Program on Commercialization of Scientific and Technological Achievements (Qiankehezhongyindi [2025] No. 006).

Future Directions

1. Reduce dependence on mature taxonomies. SAGE's effectiveness is partly tied to the scope and organization of external resources like MeSH; applying it to domains without such semantic resources may require building task-specific knowledge schemas.
2. Move beyond combinatorial coverage. The authors state that generated scenarios are shaped by anchor-concept combinations and may not fully capture the nuanced, open-ended reasoning found in authentic clinical narratives.
3. Add clinical expert verification. Because of the scale of the synthesized data, no comprehensive clinician double-checking of all samples was performed, so subtle clinical inaccuracies may remain unexamined.
4. Broaden evaluation targets. The current evaluation centers on question-answering benchmarks; interactive clinical tasks such as patient dialogue simulation and diagnostic decision support would clarify practical utility. The authors also list dynamically expanding semantic anchors, integrating multimodal medical knowledge such as imaging data, and adapting SAGE to other high-stakes knowledge-intensive domains.

Target Audience

Researchers and practitioners working on medical LLMs, synthetic data generation, or domain adaptation under privacy and compute constraints; NLP scientists interested in neuro-symbolic and taxonomy-grounded generation; and engineers or clinical informatics teams in hospitals and health-tech organizations that need to build training data locally without proprietary APIs or large document corpora. Readers should be comfortable with machine learning experimentation and benchmark evaluation; the paper's formal notation makes it most accessible to an advanced audience.

Authors’ abstract

Developing reliable models for clinical tasks, such as Medical Question Answering (QA), is severely constrained by the limited availability of high-quality, expert-annotated training data. This challenge is exacerbated by stringent privacy requirements and the impracticality of utilizing large open-source corpora or proprietary cloud APIs within resource-limited clinical settings. To address these obstacles, we introduce SAGE (\textit{Semantic Anchor-Guided Evolution}), a novel data synthesis framework that enables small, locally deployed models to generate high-quality medical training data. SAGE leverages lightweight, publicly available taxonomies such as MeSH as semantic anchors, imposing a structured prior to effectively guide and ground the data generation process. At its core, SAGE iteratively interleaves atomic (individual concept-based) and associative (relation-based) synthesis, bootstrapping training data from minimal seeds. This approach eliminates the need for large collections of medical documents or reliance on external APIs, providing a practical solution for on-premises data creation. Extensive experiments across multiple medical question-answering benchmarks demonstrate that models fine-tuned with SAGE-synthesized data consistently outperform those trained using self-derived or conventional document-based paradigms, highlighting tangible improvements in data efficiency and resource utilization for medical LLM development. Code is available at https://github.com/DIaacKr/SAGE.