MedTutor: A Retrieval-Augmented LLM System for Case-Based Medical Education

AI summary

Overview

Research area: Medical NLP / clinical education — retrieval-augmented generation (RAG) applied to radiology residency training.

Technical level: Intermediate. The paper is readable without deep ML background, but familiarity with retrieval, embeddings, and LLM serving helps.

Scope: The paper describes MedTutor, an open-source, locally deployed RAG pipeline that turns a single clinical radiology report into an evidence-grounded educational module plus multiple-choice questions, and reports a multi-radiologist evaluation of its output quality.

What This Paper Is About

Medical residents learn by reading and discussing case reports, but finding the textbook passages and current literature relevant to a specific case is slow and inconsistent. MedTutor automates that step: it takes a clinical report as input and generates a concise study module and self-test questions, grounded in retrieved textbook excerpts and live-academic-search abstracts rather than the LLM's internal memory. The paper's goal is both to build this tool and to rigorously test whether its output is clinically and educationally sound.

Key Contributions

1. A complete RAG system architecture for case-based medical education. MedTutor decomposes a radiology report into search queries, retrieves evidence from two complementary sources simultaneously (an indexed radiology textbook and the PubMed / Semantic Scholar APIs), reranks it, and synthesizes it into educational material with MCQs.

2. A scalable, privacy-preserving engineering design. The system runs entirely on local open-source models with no cloud LLM APIs, and uses an asyncio event loop for I/O-bound API calls alongside multi-GPU vLLM workers with continuous batching, so hundreds of long-context reports can be processed efficiently.

3. A new expert-annotated evaluation benchmark. Three board-certified radiologists scored system outputs on a 5-point Likert scale across upstream (query quality, paper relevance) and downstream (textbook summary, educational material, MCQ quality) tasks, over 50 densely annotated cases.

4. A large-scale LLM-as-a-Judge study with agreement analysis. Four judge models scored outputs from six generator models across 2,000 reports from five radiology datasets (Yale Internal, MIMIC-CXR, MIMIC-IV-note, CheXpert Plus, ReXGradient-160K), and the authors measured correlation with human experts and inter-annotator agreement. A 144K-example dataset is planned for public release.

Main Findings

• Expert-rated output quality is solid but not exceptional. Llama-3.3-70B-Instruct averaged 3.44/5 with human raters; MedGemma-27B, a medically specialized model, averaged 3.65/5. Both were judged to produce clinically valuable content.

• Medical specialization helps, especially for MCQs. MedGemma-27B's multiple-choice questions scored 3.53 with humans versus 3.11 for Llama-3.3-70B. Radiologists noted MedGemma generated more plausible distractors — a genuinely hard sub-task.

• Retrieval is the weak link upstream. Humans rated extracted search queries highly (3.73/5) but retrieved academic papers only 2.88/5. The system finds the right topic but the live literature search surfaces articles that are not tightly matched to the specific clinical context.

• LLM judges inflate scores but preserve ranking. LLM-as-a-Judge averaged 4.20 for paper relevance and 3.91–4.16 overall, well above human scores, yet they correctly ranked MedGemma-27B above Llama-3.3-70B. This is a moderate alignment — useful for relative comparison, unreliable as an absolute quality measure.

• Inter-annotator agreement varies sharply by task. Krippendorff's alpha was good for MedGemma's textbook summaries (0.627–0.661) and fair for paper relevance (0.474–0.493), but near zero or negative for MCQ evaluation (−0.16 to 0.11). Structured summarization is reliably gradeable; subjective quality of generated questions is not, at least under the current guidelines.

• Full local deployment is feasible. The system uses open-source models throughout, keeping protected health information on-premises — a practical requirement for hospital adoption.

Methodology in Plain English

MedTutor runs a case report through four stages.

First, decomposition: a Llama-3.3-70B-Instruct model reads the report and extracts keyword-based diagnostic queries, such as "acute appendicitis" or "colonic diverticulosis."

Second, hybrid retrieval, done in parallel for each keyword. One branch searches a vector database built from an OCR'd radiology textbook (Dähnert), indexed page-by-page with Qwen3-Embedding-8B embeddings and matched by cosine similarity. The other branch issues live queries to PubMed and Semantic Scholar to pull recent literature, with semaphore-controlled rate limiting to avoid API throttling.

Third, evidence processing. A Qwen3-Reranker-8B model scores retrieved documents against a contextualized query that combines the original report text with the specific keyword, keeping only the top two papers. Simultaneously, a generator LLM distills the retrieved textbook pages into a query-focused summary.

Fourth, generation. The report, keywords, top paper abstracts, and textbook summaries are assembled into roughly 3,500-token prompts, batched, and sent concurrently to a vLLM engine to produce two outputs: a long-form educational module and a set of MCQs with answers and explanations.

Architecturally, the system is an asynchronous orchestrator talking to per-GPU ModelWorker processes over multiprocessing queues. The main event loop awaits futures keyed to request IDs; workers resolve them when inference completes. This bridges network-bound API calls and compute-bound GPU work without either blocking the other.

For evaluation, three radiologists scored 50 cases on a 1–5 scale with detailed guidelines, and four LLM judges scored the larger 10,000-report corpus (2,000 reports × 5 datasets × 6 generators). Agreement was measured with Krippendorff's alpha andCohen's kappa.

Why This Matters

Impact on research. The paper provides a concrete demonstration that RAG can be operationalized for education rather than question answering, and — more valuably — it quantifies where LLM-as-a-Judge diverges from expert radiologists. That calibration gap is directly useful to anyone building automated evaluation for high-stakes medical content. The planned 144K-example benchmark with paired human and LLM judgments fills a real gap, since most medical RAG benchmarks focus on QA accuracy rather than pedagogical quality.

Real-world applications:

• Radiology residency programs can deploy MedTutor locally to auto-generate study material from the teaching files residents already review, supplementing rather than replacing attending feedback.
• Board exam preparation — the MCQ generation, when paired with MedGemma-27B, produces usable self-assessment items grounded in cited literature.
• Continuing medical education and credentialing for practicing radiologists who need rapid grounding in unfamiliar case types.
• Hospital knowledge management — the same retrieve-rerank-synthesize architecture can be repurposed for clinical documentation support, protocol lookup, or onboarding materials, since the pipeline is domain-agnostic.

Industry relevance. The design choices — no cloud LLM APIs, vLLM continuous batching, multiprocessing GPU workers, asynchronous API orchestration — are the engineering patterns health-system IT and medical AI vendors actually need. The paper is as much an infrastructure template as a model contribution, and the negative finding that MCQ evaluation has near-zero inter-rater agreement is a warning worth heeding for anyone selling AI-generated assessment content.

Future Directions

1. Extend beyond radiology. The pipeline is claimed to be generalizable, but evaluation covers only radiology. Pathology, cardiology, and dermatology have different report structures and knowledge sources, and performance there is untested.

2. Improve retrieval precision. The 2.88/5 human rating on paper relevance is the clearest technical deficiency. Better query expansion, structured PubMed filters (study type, recency, MeSH terms), or a stronger reranking stage could close the gap between topic relevance and clinical relevance.

3. Fix MCQ evaluation and generation. Near-zero annotator agreement means nobody can reliably say what makes a generated question good. This calls for sharper rubric development, fine-tuning on human-rated items, and possibly multi-dimensional scoring (distractor plausibility, answer correctness, difficulty calibration) instead of a single Likert rating.

4. Scale and diversify human evaluation. Fifty cases with three radiologists limits statistical power, particularly for subgroup comparisons across datasets and generator models. A larger, multi-site study would also test whether the LLM-judge calibration gap is stable across institutions.

5. Measure downstream learning outcomes. The paper establishes content quality but not whether residents actually learn more or faster. A controlled study comparing MedTutor-augmented study against standard case review would be the strongest validation.

Target Audience

Medical AI researchers and clinical NLP engineers will find the most value, particularly those working on RAG systems, medical LLM evaluation, or LLM-as-a-Judge methodology. Radiology educators and residency program directors should read it to understand both the capability and the current limits of AI-generated teaching material. Health-system informatics teams evaluating on-premises deployment options will find the architecture section directly actionable. The evaluation methodology sections — particularly the inter-annotator agreement analysis — are also useful to anyone designing expert-annotation studies for subjective generative outputs.

Authors’ abstract

The learning process for medical residents presents significant challenges, demanding both the ability to interpret complex case reports and the rapid acquisition of accurate medical knowledge from reliable sources. Residents typically study case reports and engage in discussions with peers and mentors, but finding relevant educational materials and evidence to support their learning from these cases is often time-consuming and challenging. To address this, we introduce MedTutor, a novel system designed to augment resident training by automatically generating evidence-based educational content and multiple-choice questions from clinical case reports. MedTutor leverages a Retrieval-Augmented Generation (RAG) pipeline that takes clinical case reports as input and produces targeted educational materials. The system's architecture features a hybrid retrieval mechanism that synergistically queries a local knowledge base of medical textbooks and academic literature (using PubMed, Semantic Scholar APIs) for the latest related research, ensuring the generated content is both foundationally sound and current. The retrieved evidence is filtered and ordered using a state-of-the-art reranking model and then an LLM generates the final long-form output describing the main educational content regarding the case-report. We conduct a rigorous evaluation of the system. First, three radiologists assessed the quality of outputs, finding them to be of high clinical and educational value. Second, we perform a large scale evaluation using an LLM-as-a Judge to understand if LLMs can be used to evaluate the output of the system. Our analysis using correlation between LLMs outputs and human expert judgments reveals a moderate alignment and highlights the continued necessity of expert oversight.