Modular Expert Merging for Biomedical Retrieval

AI summary

Overview

Research area: Natural Language Processing / Information Retrieval — specifically domain-specialized dense retrieval (embedding models) built from decoder-only LLMs, and parameter-space model merging.

Technical level: Intermediate. The paper assumes familiarity with dense retrieval, contrastive training (InfoNCE), LoRA adapters, and model-merging concepts such as task vectors, TIES, and DARE.

Scope: The paper argues that merging separately trained domain-specialized retrieval experts outperforms training one model on mixed-domain data, and introduces the Synthesize-Train-Merge (STM) recipe (LLM-synthesized hard negatives, LoRA fine-tuning, merging) validated across four decoder-only LLM families, four merging methods, and twelve MTEB retrieval tasks.

What This Paper Is About

Adapting general-purpose LLMs into domain-specialized dense retrievers has typically required large-scale training on mixed-domain data, which is computationally expensive and tends to average out domain-specific strengths. The authors test whether this approach can be replaced entirely by training separate, smaller domain experts and combining them in parameter space. They also introduce STM, a recipe that uses a top-tier LLM (GPT-5) to synthesize hard negatives, fine-tunes domain experts with LoRA, and then merges them — without any continual pre-training.

Key Contributions

1. Merging beats mixed-domain training. The authors demonstrate that parameter-space merging of dense retrieval experts consistently performs above large-scale mixed-domain fine-tuning across four decoder-only LLM families (0.6B–7B), four merging methods, and twelve retrieval tasks from MTEB.
2. The STM framework. They introduce Synthesize-Train-Merge, a modular framework for building domain-specialized dense retrievers without continual pre-training, consisting of synthetic hard-negative generation, LoRA fine-tuning of experts, and merging.
3. LLM-synthesized hard negatives. They show that prompting a top-tier LLM to generate hard negatives is practically effective for retrieval expert training across backbone families, with especially strong benefits for smaller models.
4. Released artifacts. They release code (github.com/TIO-IKIM/STM), an improved retrieval dataset, and STM model checkpoints (huggingface.co/collections/ikim-uk-essen/stm).

Main Findings

• Best overall result: STM_Mistral (7B) records the highest mean NDCG@10 across the twelve tasks at 0.671, which is 0.021 above the strongest baseline, Promptriever 7B (0.650), and 0.049 above E5 Large V2 (0.622).
• Small models are competitive: STM_Qwen3 (0.6B) records a higher mean NDCG@10 (0.610) than GTR T5 XL (1.2B, 0.583), and Recall@100 results are consistent with the NDCG@10 findings.
• Balanced domain profile: BMRetriever scores higher on medical than general (0.645/0.560), indicating specialization at the cost of general-domain coverage, while STM models show a more balanced profile. STM_Gemma records a higher general score than BMRetriever (0.573 vs. 0.560) without sacrificing medical performance (0.648 vs. 0.645), despite sharing the same 2B base model.
• Merging beats full-data fine-tuning in every family: In Table 5, the best STM variant exceeds both the strongest individual expert and full-dataset fine-tuning for Qwen3 (0.610 linear vs. FT-all HNM 0.591), Gemma (0.617 TIES/TA vs. 0.600), Phi4 (0.636 DARE-TIES vs. 0.621), and Mistral (0.671 TIES vs. 0.655).
• No single merging method dominates: TIES and Task Arithmetic achieve the same average NDCG@10 across the four families (0.632). TIES leads for Mistral (0.671), Task Arithmetic ties with TIES for Gemma (0.617), and DARE-TIES leads for Phi4 (0.636). Interference-aware methods (TIES, DARE-TIES) are recommended when task interference is non-negligible, consistent with a measured sign-conflict rate of ~87% across expert task vectors for all four backbones.
• Mechanistic explanation: Full-data fine-tuning produces larger per-layer weight deviations from the base model than any individual expert, suggesting it overwrites rather than preserves domain-specific structure; the best merged model instead stays closer to its expert parents in parameter space.
• Synthetic hard negatives help most smaller models: SHN yields average NDCG@10 gains of +4.2% (Qwen3-0.6B), +2.0% (Gemma-2B), and +2.4% (Phi4-3.8B) over standard hard-negative mining, driven mainly by the general-domain Search split (+10.1% for Qwen3, +18.9% for Phi4). Mistral-7B is the sole exception with a slight average decline of −0.8%, an inverse scaling trend where smaller models benefit most.
• Synthetic negatives are genuinely "hard": GPT-5 negatives have a mean query–passage cosine similarity of 0.56, comparable to mined negatives (0.54) and well below positives (0.73), while random negatives fall to 0.20. A two-judge false-negative audit with two LLMs distinct from the generator confirms 95–96% of GPT-5 negatives as topically related but not correct answers, with 96% raw inter-judge agreement.
• Curated expert pools matter: A curated mix of standard and SHN adapters (best pool) records the highest mean NDCG@10 for Gemma (0.617), Phi4 (0.634), and Mistral (0.668) under Task Arithmetic merging — notably for Phi4 and Mistral, where SHN gave little or no single-expert gain. For Qwen3, all pools fall within 0.003 points of each other.
• Cheap coefficient search is enough: Random search with N=10 samples is competitive with CMA-ES, and for Phi4 random search records higher mean NDCG@10 than evolve on three of four methods (e.g., DARE-TIES: 0.636 vs. 0.624 for evolve); for Mistral, random search is higher for Task Arithmetic (0.668 vs. 0.653) and TIES (0.671 vs. 0.665).
• Medical experts dominate the merge: Medical experts receive the highest weights across all families, with MedReal consistently dominant. Several optimal configurations assign near-zero weight to at least one expert — NLU down-weighted to 0.01 in Gemma's best Task Arithmetic configuration (MedReal 0.84, Search 0.41); Search to 0.06 in Phi4's Task Arithmetic base configuration (MedReal 0.43); Search to 0.01 in Mistral's Task Arithmetic hn configuration (MedReal 0.41, MedSynth 0.44).
• Pre-training is not necessary: Pre-training on 10M unlabeled pairs underperforms models fine-tuned only on 1.4M pairs. Dropping pre-training reduces the training compute of a single fine-tuning run by approximately 88% relative to a PT+FT pipeline.
• The merging gain is not from the synthetic data: Merging base experts trained only on mined negatives still records higher mean NDCG@10 than full-data fine-tuning across all four backbones (Qwen3 0.591 → 0.610; Gemma 0.600 → 0.615; Phi4 0.621 → 0.636; Mistral 0.655 → 0.667).
• Dev-split selection is reliable: The frozen development split correlates strongly with full test benchmarks (NFCorpus: Pearson r = +0.93, Spearman ρ = +0.91; FiQA2018: Pearson r = +0.89, Spearman ρ = +0.89).

Methodology in Plain English

The authors start from the BMRetriever fine-tuning data mixture, which originally contained 14 data sources, and reorganize it into four coherent subsets balanced by size, task, and domain: Med-Synth (431,000 pairs), Med-Real (306,000 pairs), Search (438,000 pairs), and NLU (251,000 pairs) — roughly 1.4M pairs total.

For the synthetic negatives, the authors prompt GPT-5 with a query, a positive passage, and an existing mined negative, asking it to produce a new negative that stays lexically and topically aligned with the query but is semantically irrelevant or contradictory. Then they fine-tune one expert per data split using LoRA adapters applied to all linear layers, with an InfoNCE contrastive loss, EOS-token pooling (chosen over mean pooling after preliminary experiments showed it was slightly stronger), and a retrieval prompt prepended to each query.

Merging is performed with MergeKit on HuggingFace-compatible models using four strategies: linear interpolation, task arithmetic, TIES merging, and DARE-Ties. For linear and task arithmetic the weight coefficients α are swept over {0, 0.1, …, 0.9}; for TIES and DARE-Ties the same weight range is swept along with a density parameter ρ over {0.1, 0.2, …, 0.9}. Coefficients are selected either by random search over N sampling rounds or by the CMA-ES evolutionary algorithm at N evaluations. All merged models are evaluated without any training on a frozen development split of four retrieval tasks (NFCorpus, FiQA-2018, FEVER, CLUE-IR) using unweighted mean NDCG@10.

The backbones are Qwen3 (0.6B, 1,024 hidden dimensions), Gemma (2B, 2,048), Phi-4 (3.8B, 3,072), and Mistral (7B, 4,096). Evaluation covers seven English medical MTEB tasks (TREC-COVID, SciFact, NFCorpus, CUREv1, PublicHealthQA, FeedbackQA, MedicalQA) and five general-domain datasets (FiQA, ArguAna, SciDocs, NanoFEVER, NanoQuora), reporting NDCG@10 and Recall@100 with a shared retrieval prompt. Baselines include BM25, Contriever, E5-v2, GTR, LLM2Vec 3B, BMRetriever 2B, and Promptriever 7B.

Why This Matters

Impact on research. The paper challenges a common assumption that domain adaptation requires large-scale mixed-domain or continual pre-training. If parameter-space composition of small, cheap-to-train experts can match or beat monolithic training, it reframes how specialized retrieval models are built and suggests an efficient data-ablation and modular-development pathway.

Real-world applications:

• Biomedical literature search and evidence retrieval, where general-purpose retrievers underperform on specialized terminology.
• Retrieval-augmented generation for clinical question answering, grounding LLM outputs in medical corpora such as MEDIQA, medical StackExchange QA, and medical dialogue data.
• Building systems that need both specialist accuracy and general-domain coverage (e.g., a single retriever serving medical and open-domain queries).
• Low-resource or cost-constrained deployments, since merging runs offline on CPU, requires no additional training, and produces a single model rather than an ensemble.

Industry relevance. Merging produces one model at inference time, unlike rank-based fusion or ensembling, which must run multiple models — a better efficiency–performance trade-off. The training

Authors’ abstract

Adapting general-purpose LLMs into domain-specialized dense retrievers typically requires large-scale training on mixed-domain data. We show that merging independently trained domain-specialized experts consistently exceeds this approach across four decoder-only LLM families (0.6B-7B), four merging methods, and twelve medical and general retrieval tasks from MTEB, suggesting that parameter-space composition captures complementary domain strengths that large-scale mixed-domain training averages out. To further maximize expert quality, we introduce Synthesize-Train-Merge (STM), a modular framework that synthesizes hard negatives with a top-tier LLM and fine-tunes domain-specialized experts via LoRA before merging them, without continual pre-training. Synthesized hard negatives yield the largest gains for smaller models, and STM achieves strong performance on biomedical retrieval tasks while maintaining competitive general-domain results across all four backbone families.