Demo: Guide-RAG: Evidence-Driven Corpus Curation for Retrieval-Augmented Generation in Long COVID

AI summary

Overview

Research area: Clinical artificial intelligence, specifically retrieval-augmented generation (RAG) for medical question answering, applied to Long COVID (LC) clinical decision support.

Technical level: Intermediate. The paper assumes familiarity with RAG pipelines (retrieval, chunking, embeddings, LLM-as-a-judge evaluation) but explains its corpus design decisions in accessible terms.

Scope: A demo paper that builds and evaluates six RAG corpus configurations for answering clinician-authored Long COVID questions, and proposes Guide-RAG, a chatbot and evaluation framework combining curated expert guidance with literature databases.

What This Paper Is About

Long COVID affects an estimated 7% of U.S. adults (approximately 18 million people), is characterized by more than 200 reported symptoms, and lacks a standardized diagnostic biomarker or evidence-based treatment. Clinicians therefore face a hard question when designing AI decision support: should a chatbot retrieve from a narrow, consensus-based clinical guideline, or from a huge, unfiltered literature database like PubMed?

The authors test both extremes plus a middle option—a guideline combined with high-quality systematic reviews—to find which grounding strategy produces answers that clinicians can actually trust and use.

Key Contributions

1. Expert corpus curation. Targeted curation of a clinical guideline supplemented by three high-quality systematic reviews outperformed both large-scale literature databases and narrow single-guideline approaches for LC question answering using RAG.
2. Evaluation metrics. The authors adopted faithfulness, relevance, and comprehensiveness metrics specifically for LC clinical applications using an LLM-as-a-judge framework, plus an overall metric that combines the three with equal weighting.
3. LongCOVID-CQ. A specialized evaluation dataset of 20 expert-generated Long COVID clinical questions, written by a physician-scientist specializing in LC care, covering diagnosis, management strategies, and mechanisms.
4. Guide-RAG. A proposed chatbot system and accompanying evaluation framework that integrates both curated expert knowledge and comprehensive literature databases.

Main Findings

• GS-4 wins overall. The configuration combining the AAPM&R guideline with three systematic reviews (GS-4) consistently outperformed competing approaches, achieving win rates of 57.5–65% in pairwise comparisons, and ranked highest in both faithfulness and comprehensiveness.
• Small corpus, broad coverage. Despite using only 4 sources, GS-4 achieved greater comprehensiveness than both R-110 (the guideline's cited references) and PM (PubMed), with a 60% win rate in both comparisons.
• PubMed wins on relevance. PM demonstrated slight superiority in relevance evaluation, achieving win rates of 50–52.5% against other conditions. The authors hypothesize that PM's access to expanded information enables this.
• Guidelines beat their own references. G-1 (guideline only) achieved a 57.5% win rate over R-110 in both faithfulness and overall evaluation. R-110 maintained an advantage in comprehensiveness (55% win rate) and had comparable relevance performance (50% win rate).
• Uncurated corpora can be misleading. Broader corpora (R-110 and PM) produced answers that appeared superficially plausible but leaned on single studies and overemphasized speculative mechanisms. R-110 attributed autonomic dysfunction solely to vagus nerve imbalance and recommended poorly researched interventions such as stellate ganglion block; PM endorsed exercise training based on a review that extrapolated from other diseases—advice LC experts explicitly caution against because standard exercise regimens may worsen symptoms.
• Curated output acknowledged uncertainty. The GS-4 response framed autonomic dysfunction as one part of a systemic illness, offered practical management guidance, acknowledged evidence gaps, suggested specialist involvement, and noted that physical activity must be carefully tailored to the patient.
• Not all configurations were fully comparable. The heatmaps did not include NR-0 (no retrieval) or WS (web search) for the faithfulness or overall comparisons because retrieved text chunks were required to make those comparisons.

Methodology in Plain English

The researchers set up six ways for the same chatbot to find its evidence, then had the chatbot answer the same clinical questions under each configuration and compared the answers head-to-head.

• No retrieval (NR-0): plain GPT-4o, used as the control.
• Guideline only (G-1): the single AAPM&R "Multidisciplinary collaborative guidance on the assessment and treatment of patients with Long COVID: A compendium statement," representing current U.S. consensus guidance.
• Guideline + systematic reviews (GS-4): the guideline plus three high-quality systematic reviews chosen for comprehensiveness and recency.
• References of the guideline (R-110): the 110 references cited within the AAPM&R guideline. One overlapping citation with GS-4 was excluded for independent comparison, and two references were omitted because they could not be preprocessed (one was only available in German; the other was a website compiling social security regulation links).
• PubMed corpus (PM): the National Library of Medicine database encompassing MEDLINE (>39M citations), PubMed Central, and Bookshelf, using a hybrid sparse-dense retrieval approach.
• Web search (WS): GPT-4o's web search with constraints to retrieve only peer-reviewed medical publications with source attribution.

For retrieval, expert-curated documents were converted to plaintext, split into 1200-character chunks with 600-character overlap, embedded with OpenAI's text-embedding-3-small model (dimensionality 1536), and indexed in a FAISS vector database; the top-25 nearest neighbors by cosine similarity were retrieved. For PubMed, GPT-4o first generated a PubMed search query, PubMed's Best Match algorithm returned candidates, and query generalization was applied iteratively if fewer than 25 results came back.

Evaluation used GPT-4o as an LLM judge over pairwise comparisons across faithfulness (is the answer supported by its context?), relevance (does it address the question without digression?), comprehensiveness (does it cover all aspects?), and an equally weighted overall score. Prompts included explicit instructions to ignore response length and presentation order, with randomized ordering and tie handling. Win rates were computed by assigning 100 to the winner, 0 to the loser, and 50 to each in a tie, then averaging over the questions.

Why This Matters

Research impact. Existing medical question-answering benchmarks predominantly use multiple-choice formats testing factual recall, and medical RAG chatbots are evaluated primarily on accuracy. This paper argues for and demonstrates multi-dimensional evaluation—faithfulness, relevance, comprehensiveness—that better reflects clinical decision support needs. It also provides the first evaluation, to the authors' knowledge, of a RAG framework grounding answers jointly in clinical guidelines and systematic reviews for an emerging disease.

Real-world applications:

• Clinical decision support at the point of care: a chatbot that cites source documents inline lets clinicians verify claims rather than trusting an opaque answer.
• Emerging-disease response: the paper cites the early COVID-19 pandemic, when more than 23,000 COVID-19 articles were indexed in the first six months alone, many offering little new or meaningful information; a curated corpus helps avoid flooding clinicians during future outbreaks.
• Clinician education and onboarding: guideline-anchored answers that acknowledge evidence gaps model appropriate uncertainty for providers unfamiliar with LC.
• Patient counseling: GS-4's response correctly flagged that physical activity must be tailored to the patient, an area where uncurated sources gave potentially harmful advice.

Industry relevance. Commercial clinical AI products—including resources like OpenEvidence that synthesize literature algorithmically—face the same corpus-curation tradeoff this paper measures. The results suggest vendors can achieve better overall quality with a small, well-chosen corpus of secondary literature than by indexing a massive primary-literature database, while still keeping a broad database available for relevance on specific queries.

Future Directions

• Human and multi-model evaluation. The authors call for human expert ratings and evaluation with multiple models, since the study used a single LLM (GPT-4o) as judge without validation against human or model-diverse raters.
• Retrieval ablations. The study did not systematically explore retrieval hyperparameters including chunk size, embedding model selection, and reranking thresholds.
• Broader domain testing. The framework should be extended to additional high-complexity, high-uncertainty medical domains beyond Long COVID.
• Larger question sets and controlled retrieval. The small, expert-generated question set limited statistical robustness, and differing retrieval strategies across corpora (dense retrieval for smaller datasets versus hybrid sparse-dense for PubMed) may have introduced bias in recall quality—both are flagged as constraints on generalizability and reproducibility.

Target Audience

Clinicians and clinical informaticists designing AI decision-support tools; AI/NLP researchers working on RAG, medical question answering, or evaluation methodology; health system and health-tech product teams choosing what documents to index; and policymakers or researchers concerned with trustworthiness and evidence grounding in medical chatbots. Readers focused on emerging or high-uncertainty diseases will find the corpus-curation framing particularly useful, while those seeking benchmark-scale quantitative validation should note the study's 20-question, single-judge design.

Authors’ abstract

As AI chatbots gain adoption in clinical medicine, developing effective frameworks for complex, emerging diseases presents significant challenges. We developed and evaluated six Retrieval-Augmented Generation (RAG) corpus configurations for Long COVID (LC) clinical question answering, ranging from expert-curated sources to large-scale literature databases. Our evaluation employed an LLM-as-a-judge framework across faithfulness, relevance, and comprehensiveness metrics using LongCOVID-CQ, a novel dataset of expert-generated clinical questions. Our RAG corpus configuration combining clinical guidelines with high-quality systematic reviews consistently outperformed both narrow single-guideline approaches and large-scale literature databases. Our findings suggest that for emerging diseases, retrieval grounded in curated secondary reviews provides an optimal balance between narrow consensus documents and unfiltered primary literature, supporting clinical decision-making while avoiding information overload and oversimplified guidance. We propose Guide-RAG, a chatbot system and accompanying evaluation framework that integrates both curated expert knowledge and comprehensive literature databases to effectively answer LC clinical questions.