MedBioRAG: Semantic Search and Retrieval-Augmented Generation with Large Language Models for Medical and Biological QA

AI summary

Overview

Research area: Biomedical natural language processing, specifically question answering (QA) and information retrieval, combining dense/semantic search, lexical search (BM25), retrieval-augmented generation (RAG), and supervised fine-tuning of large language models (LLMs).

Technical level: Intermediate. The paper assumes familiarity with retrieval metrics (NDCG, MRR, MAP), RAG pipelines, dense embeddings, and LLM fine-tuning, but presents the pipeline and formulas at an accessible level.

Scope in one sentence: The paper introduces MedBioRAG, a retrieval-augmented framework that pairs semantic and lexical document retrieval with a fine-tuned LLM (all fine-tuning uses GPT-4o as the base model) and evaluates it on biomedical text retrieval, close-ended QA, and long-form QA benchmarks.

What This Paper Is About

Biomedical question answering is harder than general QA because answers must be precise, current, and factually grounded, yet LLMs like GPT-4o rely on static pre-training data and can hallucinate or go stale. MedBioRAG addresses this by retrieving relevant biomedical documents with a hybrid search mechanism (with semantic search playing the dominant role), re-ranking them, and feeding them to a fine-tuned LLM that generates either a short answer choice or a structured long-form explanation. The goal is to improve retrieval quality and answer accuracy simultaneously across medical and biological QA tasks.

Key Contributions

1. A retrieval-augmented biomedical QA framework (MedBioRAG) that integrates semantic search, lexical search, document ranking, and fine-tuned LLM response synthesis. Semantic search is the primary retrieval mechanism, with BM25-based lexical search as a complementary component.

2. A demonstration that fine-tuning GPT-4o improves biomedical QA. Fine-tuned GPT-4o combined with retrieval augmentation is reported to outperform zero-shot GPT-4o and other fine-tuned LLMs across multiple benchmarks.

3. A three-part evaluation spanning retrieval performance, close-ended reasoning, and long-form synthesis, using NFCorpus, TREC-COVID, MedQA, PubMedQA, BioASQ, LiveQA, and MedicationQA, with NDCG, MRR, Precision@10, Recall@10, F1-score@10, MAP, accuracy, ROUGE, BLEU, BERTScore, and BLEURT as metrics.

4. Reported state-of-the-art results on PubMedQA and BioASQ, claimed to surpass previous benchmarks and the GPT-4o base model in all evaluated tasks.

Main Findings

• Semantic search beats lexical search on every retrieval metric and dataset. On NFCorpus, semantic search reaches NDCG@10 of 37.91 versus 31.34 for lexical, and MRR@10 of 64.29 versus 51.63. On TREC-COVID, semantic search reaches NDCG@10 of 61.02 versus 48.35, and MRR@10 of 89.17 versus 82.50. Semantic search also leads on DCG@10, Precision@10, Recall@10, F1-score@10, and MAP@10 for both datasets (for example, NFCorpus MAP@10 of 56.15 versus 46.01, and TREC-COVID MAP@10 of 82.19 versus 72.31).

• Close-ended QA improves when retrieval is added to a fine-tuned model. Fine-tuned GPT-4o plus MedBioRAG records 89.47 on MedQA, 85.00 on PubMedQA, and 98.32 on BioASQ, versus 87.88, 80.70, and 97.06 for fine-tuned GPT-4o without MedBioRAG. The same pattern holds for GPT-4o, GPT-4o-mini, and GPT-4, though GPT-3.5 plus MedBioRAG scores lower than GPT-3.5 alone on all three datasets (45.36 versus 51.52 on MedQA, 38.60 versus 19.30 on PubMedQA is the one exception where it is higher, and 66.91 versus 88.24 on BioASQ).

• Long-form QA shows the largest and most consistent gains on PubMedQA. Fine-tuned GPT-4o plus MedBioRAG reaches ROUGE-1 of 37.49, ROUGE-2 of 14.78, ROUGE-L of 27.89, BLEU of 6.11, and BERTScore of 37.02, all above the fine-tuned GPT-4o baseline (35.82, 13.55, 26.09, 4.34, 35.33). On BioASQ, fine-tuned GPT-4o plus MedBioRAG reaches ROUGE-1 of 34.30, ROUGE-2 of 18.81, ROUGE-L of 27.74, and BERTScore of 35.43.

• The long-form gains are not uniform across datasets. On LiveQA, adding MedBioRAG to fine-tuned GPT-4o lowers every ROUGE and BLEU number relative to fine-tuned GPT-4o alone (ROUGE-1 15.73 versus 24.12, ROUGE-2 4.58 versus 6.18, ROUGE-L 10.74 versus 13.31, BLEU 1.20 versus 1.63). On MedicationQA, GPT-4o plus MedBioRAG scores below GPT-4o alone on ROUGE-1, ROUGE-2, ROUGE-L, BLEU, and BERTScore. Adding MedBioRAG to GPT-4o on LiveQA does help (ROUGE-1 27.33 versus 26.96; BLEU 15.29 versus 1.41). All BLEURT values in Table 1 are negative.

• Retrieval depth has an optimal range. Figure 4 examines increasing Top-K on MedQA and reports that performance initially improves but deteriorates beyond an optimal threshold because of noise and conflicting information; the figure caption states that as the number of retrieved documents increases, performance on all metrics decreases for this short-form task.

• Reported discrepancies between the narrative and the tables. The close-ended QA text states MedBioRAG improves MedQA to 90 percent (previous state of the art 82 percent), PubMedQA to 85 percent (previous state of the art 82 percent), and BioASQ to 96 percent (previous state of the art 94 percent), while Table 2 lists 89.47, 85.00, and 98.32. The text also states MedBioRAG attains the highest BLEU score of 7.89 on MedicationQA, but 7.89 is GPT-4o's value in Table 1 and the highest MedBioRAG BLEU there is 7.24; and it states BioASQ achieves the highest BLEURT score among all models, while Table 1 shows BioASQ's best BLEURT as -2.41 for fine-tuned GPT-4o without MedBioRAG. The paper does not reconcile these differences.

• Not reported. The paper does not report the embedding model used for semantic search, the specific values of Top-K, the confidence threshold used in content filtering, hardware or compute specifications, statistical significance tests, human evaluation results, or whether code, model checkpoints, or data are released.

Methodology in Plain English

MedBioRAG works in three stages.

First, a hybrid retrieval module finds relevant documents. The lexical half uses BM25, a classic keyword-ranking function that scores documents by term frequency and inverse document frequency. The semantic half converts the query and each document into dense vectors with an encoder and ranks them by cosine similarity, selecting the top k. Semantic search is treated as the dominant component because it can match conceptually related text even without shared keywords.

Second, a fine-tuned LLM generates the answer. The model is trained with a standard next-token prediction loss on (input, answer) pairs, where the input includes the query plus the retrieved document context, so it learns biomedical terminology and grounded response generation.

Third, prompt engineering and content filtering shape the output. Different system messages and decoding parameters are used for each task type: closed-ended QA gets a two-token limit, temperature 0.1, Top P 0.7, frequency penalty 0.5, presence penalty 0.1, and a newline stop sequence; long-form QA gets a 300-token limit, temperature 0.2, Top P 0.8; short-form QA gets a 50-token limit, temperature 0.2, Top P 0.85, frequency penalty 0.2. Generated responses also receive a softmax confidence score, and low-confidence outputs are discarded or revised.

Fine-tuning details are given per task: base models were GPT-4o throughout. Closed-ended QA trained on MedQA (10,178 samples, 2 epochs, batch size 13), PubMedQA PQA-L (552 samples, 3 epochs, batch size 1), and BioSQA (5,049 samples, 3 epochs, batch size 2). Long-form QA trained on PubMedQA PQA-A (196,144 samples, 1 epoch, batch size 64), MedicationQA (551 samples, 3 epochs, batch size 1), LiveQA (500 samples, 3 epochs, batch size 1), BioSQA (5,049 samples, 3 epochs, batch size 10), and a Combined Custom Dataset (6,652 samples, 3 epochs, batch size 13). Short-form QA trained on MedQA (10,178 samples, 2 epochs, batch size 13).

Why This Matters

The work contributes to the argument that domain-specific retrieval plus domain-specific fine-tuning is more effective than prompting a general-purpose LLM alone for medical text, and it provides a head-to-head comparison of semantic versus lexical retrieval across two biomedical IR benchmarks.

Real-world applications suggested by the design:

• Clinical decision support tools that ground answers in current literature rather than static training data.
• Biomedical literature search engines and research assistants that need concept-level matching when exact keywords differ (for example, "heart attack" versus "myocardial infarction").
• Patient- or clinician-facing medical QA assistants that produce both multiple-choice answers and structured explanatory responses.
• Evidence synthesis and drug or treatment information lookups where retrieved source documents must accompany the answer.

Industry relevance: the pipeline points toward a practical division of labor for healthcare AI products, where retrieval infrastructure, ranking quality, and prompt/decoding configuration are optimized separately from the underlying model. The stated cost is real-time retrieval overhead and dependence on retrieval quality, both of which matter for latency-sensitive clinical settings.

Future Directions

1. Expert validation. The paper's own limitations section names the absence of validation by medical professionals as its key limitation, leaving open how well the model aligns with expert reasoning.

2. Handling contradictory retrieved evidence. Unresolved contradictions among retrieved documents are identified as a concern for factual accuracy, which suggests better ranking, filtering, or conflict-resolution mechanisms.

3. Reducing computational overhead. Real-time retrieval is noted as increasing cost and limiting applicability in time-sensitive settings, so inference speed optimization is listed as future work.

4. Specialized and real-world domains. The authors call for further fine-tuning for areas such as clinical diagnosis and for broader evaluation on real-world data including clinical case reports and electronic health records (EHRs).

5. Retrieval ranking refinement. The conclusion also lists refining retrieval ranking as an explicit next step, which connects to the observed Top-K degradation on short-form QA.

Target Audience

This paper is most useful to applied NLP and information-retrieval researchers working on biomedical or domain-specific RAG systems, and to engineers building clinical or life-science QA products who need a concrete reference configuration of retrieval methods, fine-tuning settings, prompt templates, and decoding parameters. It is also relevant to medical informatics researchers evaluating whether semantic search or lexical search is the better fit for biomedical corpora. Readers seeking statistical rigor, human evaluation, or released artifacts will not find them here, since the paper reports single-run benchmark numbers, no significance testing, and does not state whether code or checkpoints are available.

Authors’ abstract

Recent advancements in retrieval-augmented generation (RAG) have significantly enhanced the ability of large language models (LLMs) to perform complex question-answering (QA) tasks. In this paper, we introduce MedBioRAG, a retrieval-augmented model designed to improve biomedical QA performance through a combination of semantic and lexical search, document retrieval, and supervised fine-tuning. MedBioRAG efficiently retrieves and ranks relevant biomedical documents, enabling precise and context-aware response generation. We evaluate MedBioRAG across text retrieval, close-ended QA, and long-form QA tasks using benchmark datasets such as NFCorpus, TREC-COVID, MedQA, PubMedQA, and BioASQ. Experimental results demonstrate that MedBioRAG outperforms previous state-of-the-art (SoTA) models and the GPT-4o base model in all evaluated tasks. Notably, our approach improves NDCG and MRR scores for document retrieval, while achieving higher accuracy in close-ended QA and ROUGE scores in long-form QA. Our findings highlight the effectiveness of semantic search-based retrieval and LLM fine-tuning in biomedical applications.