Multilingual Medical Reasoning for Question Answering with Large Language Models

AI summary

Overview

Research area: Natural Language Processing / multilingual medical Question Answering with Large Language Models (LLMs), combining retrieval-augmented generation, knowledge distillation of reasoning traces, and supervised fine-tuning.

Technical level: Advanced. The paper assumes familiarity with chain-of-thought reasoning, retrieval-augmented generation (RAG), few-shot in-context learning, supervised fine-tuning (SFT), and automatic translation metrics.

Scope: The paper describes a pipeline that builds multilingual medical reasoning traces grounded in Wikipedia medical knowledge for English, Italian, and Spanish, and evaluates whether those traces improve multiple-choice medical QA in both in-context learning and fine-tuning settings.

What This Paper Is About

Reasoning-capable LLMs have shown strong results on English medical QA, largely by distilling reasoning traces from stronger models, but almost all of this work is monolingual and mostly ungrounded in external medical knowledge. The authors build a retrieval-augmented pipeline over Wikipedia's WikiProject Medicine to generate roughly 500k multilingual reasoning traces in English, Italian, and Spanish for questions drawn from MedQA and MedMCQA (which they also translate into Italian and Spanish). They then test whether exposing models to these traces — at inference time or during fine-tuning — improves medical QA accuracy across languages and across in-domain and out-of-domain benchmarks.

Key Contributions

1. The first dataset of medical reasoning traces covering Italian, Spanish, and English, generated by a pipeline that can in principle be extended to any language present in Wikipedia.
2. Comprehensive experiments showing that the traces improve multilingual multiple-choice medical QA both through in-context learning and supervised fine-tuning.
3. A released multilingual reasoning model reported to achieve state-of-the-art results in Italian, English, and Spanish medical QA among 8B-parameter models across multiple model families.
4. Two additional released resources: a collection of all medicine-related Wikipedia pages in English, Italian, and Spanish, and translated versions of MedMCQA and MedQA in Italian and Spanish, plus a manual evaluation of trace quality by medical doctors.

Main Findings

• Traces help at inference time: In a 2-shot in-context learning setup where each example is a question–trace–answer triplet, all evaluated models produced more accurate answers than baselines prompted with question–answer examples only. For MedMCQA and MedQA the average accuracy increase was +7 to +10 points across languages.
• Smaller gain out-of-domain: On MedExpQA — the manually validated out-of-distribution dataset — the average increase was +1.8 points, and Gemma-3-27B and Qwen-3-32B received no benefit there.
• Statistical significance for ICL: A paired t-test on the performance deltas gave a p-value of 0.02, below the α = 0.05 threshold. A Bonferroni correction was applied because multiple tests were run on the same data.
• Reasoning helps even without relevant retrieval: When the two in-context examples were chosen randomly rather than by similarity, randomly picked examples still helped 8 out of 9 models, with an average impact of +2.5 points on MedExpQA and +4.7 on MedQA.
• Traces help during fine-tuning too: Fine-tuning Llama-3.1-8B-Instruct and Qwen3-8B on the traces gave an average accuracy improvement of +3.9 points over baseline (p < 0.005). Gains were smaller than ICL on in-distribution datasets but larger out-of-distribution: +4.8 points on MedExpQA for SFT versus +1.8 for ICL.
• Better than prior trace sets: In fine-tuning comparisons against traces from MedReason, m1, and Huatuo, the authors' traces were superior across all datasets and languages except for Huatuo traces applied to Qwen on MedMCQA and MedQA.
• State-of-the-art claim: Their fine-tuned Qwen3-8B outperformed the best prior model, Huatuo, by an average of +2.9 points; adding two of their traces as inference-time examples raised this to a +7 point average advantage, and +1.7 points on the English portion of the datasets alone.
• Ablation results: Training on reasoning traces gave +2.7 accuracy points over standard question–answer SFT, and grounding those traces in Medical-Wikipedia added a further +3.3 points.
• Expert error analysis: The main bottleneck is the model's limited medical knowledge rather than faulty reasoning. Salient inconsistencies came from integrating clinical information, especially the interplay between patient characteristics and underlying conditions — failing to use the full patient history, misapplying standard diagnostic protocols, and limited nuanced clinical judgment.

Methodology in Plain English

The pipeline has four stages. First, the authors take medical questions from MedQA and MedMCQA and translate each question–options pair into Italian and Spanish using Qwen-2.5-72B with a 5-shot prompt. Translation quality is checked by back-translation: the text is translated back to English and compared to the original using COMET, chrF, chrF++, and BERTScore. Back-translation quality was high — BERTScore 96.6, COMET 88.9, chrF 78.6, chrF++ 77.9. As a further check, they translated the manually revised MedExpQA and compared it to the human revision, obtaining BERTScore 97, COMET 88, chrF 78, chrF++ 75.

Second, they build a knowledge base. They collect all WikiProject Medicine pages in English, extract main text, infoboxes, and interlanguage links, and use those links to fetch the corresponding Italian and Spanish pages. After de-duplication and restricting to pages available in all three languages, the knowledge base contains 8.577 English pages (20 million words), 8.492 Italian pages (8 million words), and 8.513 Spanish pages (11 million words). Pages are split into section-level chunks; sections over 5,000 words are discarded and sections under 250 words are merged into the preceding one, with irrelevant sections like "Bibliography" filtered out.

Third, they retrieve context. Chunk embeddings are pre-computed with Qwen3-Embedding-8B, and the top five most similar chunks to each question–options pair are selected. Qwen3-32B rewrites the retrieved chunks to reduce redundancy and duplication.

Fourth, they generate reasoning traces with Qwen3-32B, given the formatted context, the question, the options, and the correct answer, with prompts designed to encourage exploring candidate answers before concluding. Traces whose extracted final answer is wrong are discarded; fewer than 5% were discarded. The resulting dataset contains more than 500k traces: Italian has 166.257 from MedMCQA and 9.468 from MedQA; Spanish has 168.771 from MedMCQA and 9.584 from MedQA; English has 169.098 from MedMCQA and 9.520 from MedQA.

Evaluation used downstream accuracy. Baseline prompts used 2-shot examples selected by embedding similarity. SFT used 5,837 MedMCQA traces and 5,594 MedQA traces per language, totaling 34,293 examples, trained for 3 epochs on two H200 GPUs with per-device batch size 32, AdamW, learning rate 5e-6, cosine scheduler, and 0.1 warm-up ratio, taking 1.5 hours per model. For comparisons against MedReason, m1, and Huatuo, training ran on two H200 GPUs for approximately 18 hours.

Why This Matters

The work pushes medical QA beyond English by treating factual grounding and multilinguality as first-class design choices rather than afterthoughts, and it releases the traces, translated datasets, Wikipedia-derived knowledge base, and models for reuse. It also argues that knowledge, not reasoning, is the limiting factor for current 8B-scale medical models — a useful diagnostic for where future effort should go.

Real-world applications:

• Multilingual clinical decision-support tools for clinicians working in Italian, Spanish, or English.
• Medical education and exam preparation, since MedQA and MedMCQA are drawn from Indian and US medical exams.
• Cross-lingual transfer of medical QA resources to languages with less training data, using Wikipedia's interlanguage links as the bridge.
• Benchmark datasets for evaluating multilingual medical model safety and translation quality.

Industry relevance: The paper targets small (8B-parameter) open models that are cheaper to serve than frontier models, and it reports that combining a fine-tuned 8B model with two retrieved reasoning traces at inference gave the highest overall accuracy in their comparison — a practical deployment pattern. The released datasets and knowledge bases also give companies a starting point for localized medical assistants without building a medical corpus from scratch.

Future Directions

• Extending beyond the three languages studied, since the authors note the pipeline could apply to any language in Wikipedia.
• Replacing Wikipedia with more authoritative medical sources to reduce Wikipedia-derived bias and strengthen grounding.
• Scaling the fine-tuning experiments beyond 8B-parameter models, which the authors left unexplored due to computational constraints.
• Moving from multiple-choice QA to open-ended clinical reasoning tasks.
• Addressing the selection bias introduced by filtering traces solely on whether the final answer was correct, and improving integration of patient history and clinical context, which the expert analysis identified as the dominant error source.

Target Audience

Researchers and engineers working on multilingual NLP, medical QA, or reasoning-model training; practitioners building clinical decision-support systems who need open, small-model alternatives; and dataset builders interested in retrieval-augmented, knowledge-grounded synthetic data generation. Clinicians and medical informatics specialists will find the expert error analysis and the ethical caveats about factual inaccuracies and the non-faithfulness of reasoning traces most relevant.

Authors’ abstract

Large Language Models (LLMs) with reasoning capabilities have recently demonstrated strong potential in medical Question Answering (QA). Existing approaches are largely English-focused and primarily rely on distillation from general-purpose LLMs, raising concerns about the reliability of their medical knowledge. In this work, we present a method to generate multilingual reasoning traces based on medical knowledge extracted from Wikipedia. We produce 500k traces in English, Italian, and Spanish, using a retrieval-augmented generation approach over medical information from Wikipedia. The traces are generated to solve medical questions drawn from MedQA and MedMCQA, which we extend to Italian and Spanish. We test our pipeline in both in-domain and out-of-domain settings across Medical QA benchmarks, and demonstrate that our reasoning traces improve performance both when utilized via in-context learning (few-shot) and supervised fine-tuning, yielding state-of-the-art results among 8B-parameter LLMs. We believe that these resources can support the development of more transparent clinical decision-support tools in multilingual settings. We release the full suite of resources: reasoning traces, translated QA datasets, Medical-Wikipedia, and fine-tuned models.