To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Models

AI summary

Overview

Research area: Post-training of large language models, specifically Reinforcement Learning with Verifiable Rewards (RLVR) across multiple task domains, and model merging as an alternative to joint training.

Technical level: Advanced. The paper assumes familiarity with RLVR, GRPO, KL divergence, on-policy versus off-policy distillation, and weight-merging techniques such as TIES and DARE.

One-sentence scope: A controlled empirical comparison of two paradigms for building multi-domain expert LLMs — mixed multi-task RLVR versus separate per-domain RLVR followed by weight merging or distillation — across math, coding, science, instruction following, and agent tasks, using Qwen3-4B-Base and open-source Nemotron 3 Nano datasets.

What This Paper Is About

Teams building general expert-level models must decide whether to train on several domains at once (mixing) or to train separate specialists and combine them afterwards (merging). Most prior state-of-the-art models adopt one of these two paradigms but do not publish a detailed head-to-head comparison or an analysis of why each works. This paper fills that gap with extensive qualitative and quantitative experiments on five domains, plus internal-mechanism analysis from the perspectives of information constraints, model prediction behavior, and self-verification.

Key Contributions

1. A controlled head-to-head benchmark of the two multi-domain RLVR paradigms. The authors train five single-domain RLVR experts, one mixed multi-task RLVR model, and multiple merged models (average merging, task arithmetic, TIES-merging, SCE, with and without DARE, plus multi-teacher off-policy and on-policy distillation), and evaluate all of them on 9 benchmarks using the Avg@K metric.

2. Evidence that cross-domain interference is small and that reasoning domains are mutually synergistic. They report that the three reasoning domains (math, coding, science) improve each other, and that instruction following improves the reasoning domains, while mixed multi-task RLVR reaches comparable performance to separate-then-merge at 58.3% of the GPU hours.

3. The "policy neighborhood" concept for explaining why weight merging works. Using cross-domain KL divergence between expert policies and the merged model, they define a domain ℬ as a policy neighborhood of domain 𝒜 when a KL threshold (ε = KL(π_E𝒜 ∥ π_merge) + 0.02) is satisfied, and show through pairwise merging ablations that merging with neighborhood experts yields extra gains while merging with non-neighbors does not. They also show the relation is asymmetric.

4. An analysis of emergent self-verification, its task sensitivity, and the robustness-efficiency trade-off. Models are used as generative reward models on their own trajectories, comparing outcome-based versus process-based verification, and showing that RL-Multi is efficient but degrades process verification sharply.

Main Findings

• Mixing is far cheaper. Mixed multi-task RLVR achieves comparable performance with separate RLVR followed by model merging using 58.3% of the GPU hours. Total GPU hours reported are 4,782.3 (math), 6,404.4 (coding), 1,140.8 (science), 1,180.0 (instruction following), 2,271.0 (agent), 1,451.9 (multi-teacher on-policy distillation), and 10,050.6 (multi-task RL).

• Under matched budget, mixing also wins on quality. With five 200-step single-domain RLVR runs versus a 1,000-step multi-task RLVR run, multi-task RLVR achieves an average improvement of 4.15 points across the 9 benchmarks (average 61.18 versus 57.03 for separate-then-merge).

• Interference is small; reasoning domains are synergistic. Gradient interference among domains is not significant, and the reasoning domains (math, coding, science) can improve each other's performance. Instruction following also improves the reasoning domains of the model.

• Agent data behaves asymmetrically. When each domain is trained independently, the other four domains do not improve agent performance, because those rollouts involve no tool calls or multi-round interaction. However, in joint training the other domains do help the agent: both RL-Multi and Merging outperform individually trained RL-Agent on BFCL v3, and pairwise merges such as Agent+Math and Agent+Coding improve agent performance.

• Merging method comparison. Average merging scores 53.74 average; SCE 60.70; TIES 61.00; TIES+DARE 60.75; task arithmetic 60.22; task arithmetic+DARE 60.99; multi-teacher on-policy distillation 61.57; multi-teacher off-policy distillation 60.53. Different methods show a seesaw effect across benchmarks. MT-OPD is only slightly better than weight merging but requires additional GPU hours.

• Merging can exceed the individual experts. Model merging preserves most domain performance and can improve on it, for example on AIME'24, AIME'25, HLE, IFEval, and BFCL v3. Pairwise merging examples include Math+Coding reaching 80.29 on AIME'24 (up 2.63) and Coding+Agent reaching 62.86 on LCB v6 (up 4.00).

• Policy neighborhoods drive merging gains. For a given domain, merging with policy-neighborhood experts improves that domain, while merging non-neighboring experts does not. Coding is a neighbor of math in the math domain; agent is a neighbor of coding in the coding domain; all other domains are neighbors of science in the science domain; for instruction following, all other domains are neighborhoods; for agent, math, coding, and instruction following are neighborhoods. The neighborhood relation is asymmetric, which the authors attribute to the asymmetry of KL divergence.

• Capability acquisition differs by paradigm. All three multi-domain models (RL-Multi, TIES-merging, MT-OPD) overlap substantially with single-task models in gained capabilities. Math shows the highest gain consistency across the 5 benchmarks, suggesting math skills are more homogeneous and resistant to inter-task interference. Weight merging (TIES) shows higher gain consistency than RL-Multi and MT-OPD on most domains, meaning weight merging primarily inherits original single-task capabilities, whereas multi-task training and on-policy distillation learn capabilities that diverge more from single-task training.

• Self-verification emerges and is task-sensitive. Outcome-based judging correlates positively with generation performance (Pearson Correlation Coefficient r = 0.63), while process-based judging correlates negatively (r = -0.53), averaged over math, coding, science, and instruction following using AIME'24, LCB v5, GPQA-D, and IFEval. For logic-intensive tasks, process verification beats outcome verification; for constraint-intensive tasks like instruction following, process verification severely underperforms because of an intent-execution gap that produces false positives.

• The agent expert is the best process verifier. RL-Agent achieves the highest process judging scores across domains: 95.3 on AIME, 88.5 on IFEval, 99.4 on LCB, and 81.1 on GPQA (average 91.1).

• The robustness-efficiency trade-off. RL-Multi matches merging performance at lower GPU cost but its average process verification score collapses to 58.1, far behind the merging methods. Weight merging achieves the highest average outcome judgment (80.6) and acts as an effective regularizer. MT-OPD integrates expert knowledge in behavior space and secures a robust, balanced foundation for both outcome and process verification.

• The approach is competitive with the official Qwen3-4B model. Their best open-source-data model averages 61.18 across the 9 benchmarks, versus 52.94 for Qwen3-4B (Thinking).

Methodology in Plain English

The authors pick five domains representative of different demands: math, coding, science, instruction following, and agent (tool use). They use Qwen3-4B-Base as the starting model and apply a simplified, controllable pipeline of supervised fine-tuning followed by reinforcement learning, rather than the multi-stage alternating SFT/RL schedule used by many production models. SFT uses about 14M samples for one epoch, blended to follow the Nemotron 3 Nano technical report proportions (math formal proofs 2.37%, math 20.89%, science 16.04%, code 27.81%, chat 30.52%, conversational agent 2.37%).

For reinforcement learning they use GRPO with verifiable (binary or rule-based) rewards. Single-domain runs use a batch size of 128, 16 rollouts per prompt, 400 steps, a maximum generation length of 32k tokens, sampling temperature 1.0, a constant learning rate of 2×10⁻⁶, and one gradient update per 2,048 rollouts. Math answer verification uses the evaluator from Qwen QwQ-32B. The multi-task run trains for 1,000 steps on directly mixed data; multi-teacher on-policy distillation trains for 300 steps with batch size 256 and 4 rollouts, distilling from routed teachers with the supervised fine-tuned model as student.

Merging is done either by blending weights (average merging, task arithmetic, TIES-merging, SCE, optionally combined with DARE at a mask ratio of 0.8, using the SFT model as the anchor) or by distilling from the five expert teachers (off-policy or on-policy). Evaluation covers 9 benchmarks — AIME'24, AIME'25, LiveCodeBench v5 and v6, HLE, GPQA-Diamond, IFEval strict prompt, IFBench, and BFCL v3 — reported with Avg@K.

For mechanism analysis, the authors compute cross-domain KL divergences between each expert and the merged policy on trajectories generated by each expert, define policy neighborhoods via a threshold, and run pairwise merging ablations under TIES. They measure gain consistency by constructing per-sample gain vectors relative to the SFT baseline, taking a union gain vector over the five single-task models, and computing cosine similarity with the multi-domain models. They also use each model as a generative reward model on its own trajectories, scoring outcome-based and process-based verification against ground-truth answers.

Why This Matters

Impact on research. The paper supplies the missing controlled comparison behind two widely used but under-documented training recipes, and introduces the policy neighborhood framing as a mechanistic explanation for weight merging. It also reframes evaluation: frozen benchmark accuracy can hide a collapse in process-level verification ability, which suggests multi-domain RLVR evaluation should include self-verification metrics.

Real-world applications:

• Selecting a post-training strategy for a fixed compute budget when building a general expert assistant, since the mixed paradigm reportedly reaches comparable performance at 58.3% of GPU hours.
• Designing single-domain expert models, for example adding coding and STEM data to strengthen a math expert, an approach the synergy findings support.
• Choosing reward design per task type: process-based verification for logic-intensive tasks such as math and coding, outcome-based verification for constraint-intensive tasks such as instruction following.
• Developing verifier or critic models, where the finding that agentic multi-turn training produces the strongest process judge (average 91.1) informs how to build robust trajectory checkers.

Industry relevance. The work directly addresses decisions made by teams reproducing models such as DeepSeek-R1, Qwen3, GLM-4.5, and MiMo-V2-Flash, which split between mixed multi-task RL and separate-RL-then-merge. GPU-hour accounting (for example 10,050.6 GPU hours for multi-task versus 4,782.3 to 6,404.4 for individual math or coding runs) and the observation that model merging is more predictable while multi-task training yields emergent cross-domain strategies give practitioners concrete grounds for choosing a pipeline.

Future Directions

• Refining the policy neighborhood threshold. The authors state that the choice of ε is relatively complex and not naively +0.02, and that it requires future exploration.
• Building more reliable process-level verifiers. Since multi-task RLVR collapses average process verification to 58.1, work is needed on training paradigms that keep efficiency without degrading verification.
• Exploiting synergy for single-domain experts. The mutual gains among reasoning domains raise the question of how to systematically select beneficial auxiliary domain data for a target domain, and how to merge other experts into a target expert using neighborhood relations.
• Clarifying when emergent versus inherited capabilities are preferable. The paper distinguishes weight merging (inheriting single-task capabilities predictably) from multi-task training (producing emergent strategies), leaving open how to deliberately schedule training to obtain one or the other.

Target Audience

Machine learning researchers and engineers working on LLM post-training, RLHF/RLVR practitioners, and infrastructure or applied teams deciding between joint multi-domain reinforcement learning and specialist training plus model merging. Readers need working familiarity with reinforcement learning for language models, KL divergence, and weight-merging methods to follow the mechanism analysis; the benchmark comparison sections are accessible to a broader applied audience.

Authors’ abstract

Reinforcement Learning with Verifiable Rewards (RLVR) plays a key role in stimulating the explicit reasoning capability of Large Language Models (LLMs). We can achieve expert-level performance in some specific domains via RLVR, such as coding or math. When a general multi-domain expert-level model is required, we need to carefully consider the collaboration of RLVR across different domains. The current state-of-the-art models mainly employ two different training paradigms for multi-domain RLVR: mixed multi-task RLVR and separate RLVR followed by model merging. However, most of the works did not provide a detailed comparison and analysis about these paradigms. To this end, we choose multiple commonly used high-level tasks (e.g., math, coding, science, instruction following, and agent) as our target domains and design extensive qualitative and quantitative experiments using open-source datasets. We find the RLVR across domains exhibits small mutual interferences, and reasoning-intensive domains have mutually synergistic effects. Furthermore, we analyze the internal mechanisms from the perspectives of information constraints, model prediction behavior and self-verification. Our homepage is at https://github.com/Mosi-AI/M2RL.