SOLVE-Med: Specialized Orchestration for Leading Vertical Experts across Medical Specialties

AI summary

Overview

• Research area: Natural Language Processing, specifically medical question answering via multi-agent systems and small language models (SLMs).
• Technical level: Intermediate. The paper is accessible to readers familiar with basic NLP concepts (fine-tuning, ROUGE/BERTScore, routing/classification), but the architecture details assume some background.
• Scope: The paper describes SOLVE-Med, a multi-agent system that routes Italian medical questions to ten domain-specialized 1B-parameter models and synthesizes their answers with a larger orchestrator model, reporting evaluation against standalone models up to 14B parameters.

What This Paper Is About

Deploying large language models in clinical settings is difficult because of hallucinations, bias, weak interpretability, high computational cost, and privacy risks tied to cloud-based closed-source services. The authors propose SOLVE-Med, a multi-agent system that instead relies on small, locally deployable models: a router that picks which medical specialties a question belongs to, ten specialists each fine-tuned on one medical domain, and an orchestrator that merges their answers into one coherent response. The goal is to match or beat much larger general-purpose models on Italian medical question answering while keeping computation and data handling local.

Key Contributions

1. A three-part multi-agent architecture for medical QA. SOLVE-Med combines a Router Agent (a fine-tuned DistilBERT-base-multilingual-cased multi-label classifier), a pool of ten Medical Specialists (each a LLaMA-3.2-1B-Instruct model fine-tuned on one medical domain), and an Orchestrator Agent (a quantized Gemma-2-9B-IT model that synthesizes specialist outputs).
2. A curated Italian medical dataset pipeline. Starting from roughly 700,000 question–answer pairs annotated with one of 102 platform-defined categories drawn from the Italian forums Medicitalia and Dica33, the authors manually consolidated labels into 10 macro-categories and built disjoint training subsets for the router and the specialists to prevent data leakage.
3. Two routing strategies with an explicit precision–recall trade-off. The Router Agent selects specialists either by a confidence threshold tuned to maximize the F-beta score (beta = 2 and beta = 3) or by top-n selection (top-2 and top-3), deliberately favoring recall so that the Orchestrator can filter irrelevant contributions.
4. An empirical comparison against larger standalone models. SOLVE-Med is benchmarked against LLaMA-3.1-8B-Instruct, Gemma-2-9B-IT, and the Italian LLM Velvet-14B, plus an interactive demo with a multi-turn chat interface that exposes individual specialist outputs and router confidence scores, and allows users to query specific agents directly.

Main Findings

• SOLVE-Med outperforms larger standalone baselines. On the 100-question evaluation set, SOLVE-Med (Top-3) reached ROUGE-1 0.3010, ROUGE-2 0.0748, ROUGE-L 0.1881, ROUGE-Lsum 0.2197, BLEU 0.0260, METEOR 0.2655, BERT-P 0.6720, BERT-R 0.7256, and BERT-F1 0.6974. The comparison models scored lower on ROUGE-1 and BERTScore F1: LLaMA-3.1-8B-Instruct (0.2183 and 0.6657), Gemma-2-9B-IT (0.2546 and 0.6641), and Velvet-14B (0.2501 and 0.6775).
• All four SOLVE-Med configurations beat the baselines. SOLVE-Med (Top-2) scored ROUGE-1 0.2974 and BERT-F1 0.6953; SOLVE-Med (F2) scored 0.2920 and 0.6928; SOLVE-Med (F3) scored 0.2975 and 0.6939.
• More specialists selected tends to mean better answers. The authors report a consistent trend that strategies increasing the average number of selected specialists yield improved outcomes, which they attribute to greater diversity of clinical perspectives.
• Router precision and recall trade off as expected. Top-2 selection gave precision 0.4020 and recall 0.8040 with 2.000 specialists per query; Top-3 gave precision 0.2960 and recall 0.8870 with 3.000 specialists. The F2 threshold gave precision 0.5699, recall 0.8290 and 1.528 specialists; the F3 threshold gave precision 0.4157, recall 0.8900 and 2.277 specialists. Higher beta raised recall at the cost of precision and more predicted labels.
• The system supports local deployment. Because the specialists are 1B-parameter quantized models and the orchestrator is a quantized Gemma-2-9B-IT, the authors state the architecture reduces memory consumption and inference latency relative to cloud-dependent alternatives, while preserving acceptable response quality.
• The evaluation set is small and manually curated. Results rest on 100 question–answer pairs (10 per macro-category), sampled from forum content not used in training; the authors note the limited size reflects manual curation effort and a focus on high-quality evaluation.
• Human evaluation is not reported. The conclusion lists human evaluations as future work, so no human-assessment results appear in this paper.

Methodology in Plain English

The authors began with Italian medical forum data from Medicitalia and Dica33: about 700,000 question–answer pairs, each tagged with one of 102 platform categories. They manually merged those fine-grained tags into 10 broad medical domains — Cardiology and Hematology, Dermatology and Aesthetics, Gastroenterology, Gynecology, General Medicine and Surgery, Neurology, Eye, ENT and Pulmonology, Orthopedics, Mental Health, and Urology and Andrology.

To prevent the router and the specialists from training on the same inputs, they built separate training subsets. For each specialist they selected 10,000 training, 500 validation, and 500 test samples by computing text embeddings for all question–answer pairs, reducing dimensionality with UMAP, then clustering with HDBSCAN to pick the most representative pairs in each macro-category. For the router they sampled 5,000 training, 1,000 validation, and 1,000 test examples of queries paired with macro-category labels, evenly distributed across the ten categories.

The router was built by fine-tuning DistilBERT-base-multilingual-cased as a multi-label classifier. When a question arrives, the router either assigns every specialty whose label score exceeds a fixed threshold (0.15 for beta = 2, 0.10 for beta = 3) or takes the top-2 or top-3 highest-scoring labels. The ten specialists were each fine-tuned from LLaMA-3.2-1B-Instruct using Unsloth, with parameter-efficient fine-tuning via Low-Rank Adaptation (LoRA), and a quantized version was used at runtime. The orchestrator is a quantized Gemma-2-9B-IT with a structured prompt that frames it as a professional medical assistant instructed to merge specialist outputs into one evidence-based answer.

Evaluation used a held-out test set of 100 question–answer pairs (10 per macro-category) with ROUGE, BLEU, METEOR, and BERTScore for answer quality, and precision and recall for the router.

Why This Matters

Impact on research. The paper provides evidence that a collection of small, domain-specialized models coordinated by a router and an orchestrator can beat single general-purpose models up to 14B parameters on Italian medical QA, at least on the metrics and test set used here. It also offers a reusable template for multi-agent medical QA: explicit routing, disjoint training data across components, and a deliberate recall-first routing policy that offloads precision to the aggregation stage.

Real-world applications:

• Local clinical question answering where patient data cannot leave the premises, since the whole pipeline runs on compact quantized models.
• Triage-style decision support that returns multiple specialty perspectives on a single query rather than one opaque answer.
• Multilingual or non-English medical QA, with this work demonstrating an Italian-language system built from Italian forum data.
• Transparent assistant tooling for clinicians, since the demo surfaces each specialist's output and the router's confidence scores alongside the final answer.

Industry relevance. The Healthcare Chatbots Market is projected to surpass US$11.8 billion by 2033 according to the market figure cited in the paper. For vendors, the appeal is cost: ten 1B-parameter specialists plus a quantized 9B orchestrator may be cheaper to train and serve than a single much larger model, while avoiding dependence on external cloud infrastructure. The authors also frame the system as a support tool intended to assist, not replace, medical judgment.

Future Directions

• Human evaluation. The authors explicitly list human evaluations as future work; no clinician or user study is reported here.
• Improved context handling. Better handling of conversational context and multi-turn state is named as a next step, extending the existing multi-turn chat demo.
• Scaling and generalizing the evaluation. The current test set is 100 manually curated questions; expanding it and testing across more settings would strengthen the claims.
• Reconciling single-label annotations with multi-label routing. The source data carries single-label annotations while SOLVE-Med is designed to route to all potentially relevant specialists, a mismatch that future work could address, for example through multi-label annotated data.

Target Audience

This paper is most useful to NLP and clinical-AI researchers working on multi-agent systems, model routing, or domain-specialized small language models; to engineers building privacy-preserving or on-premises medical QA systems; and to healthcare informatics teams evaluating whether collections of small fine-tuned models can substitute for large general-purpose LLMs. It is also relevant to readers interested in non-English (Italian) medical NLP resources and in practical precision–recall trade-offs for specialist routing.

Authors’ abstract

Medical question answering systems face deployment challenges including hallucinations, bias, computational demands, privacy concerns, and the need for specialized expertise across diverse domains. Here, we present SOLVE-Med, a multi-agent architecture combining domain-specialized small language models for complex medical queries. The system employs a Router Agent for dynamic specialist selection, ten specialized models (1B parameters each) fine-tuned on specific medical domains, and an Orchestrator Agent that synthesizes responses. Evaluated on Italian medical forum data across ten specialties, SOLVE-Med achieves superior performance with ROUGE-1 of 0.301 and BERTScore F1 of 0.697, outperforming standalone models up to 14B parameters while enabling local deployment. Our code is publicly available on GitHub: https://github.com/PRAISELab-PicusLab/SOLVE-Med.