Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillation

AI summary

Overview

• Research area: Machine learning / large language model distillation and synthetic data generation, specifically multi-teacher knowledge distillation for training small student LLMs.
• Technical level: Intermediate. Readers should be comfortable with supervised fine-tuning, reward models, and preference-model concepts such as the Bradley-Terry model, but the paper's central idea can be understood without deep mathematical background.
• Scope: The paper proposes PerSyn (Personalized data Synthesis), a router-guided strategy that assigns each prompt to its individually optimal teacher model and evaluates it across instruction tuning and math reasoning with student models from several families.

What This Paper Is About

Distillation usually assumes that the strongest available teacher produces the best training data for a small student model, but prior work has shown that strong teachers can generate outputs that are too complex for a student to learn from. Existing fixes, such as mixing strong and weak teacher data (Mix) or selecting one balanced teacher per student (CAR), follow a costly "Generate then Select" pattern in which every teacher must answer every prompt. PerSyn instead routes each prompt to its own optimal teacher using both the student's learnability and the teacher's response quality, then only that teacher generates the response.

Key Contributions

1. A new synthesis paradigm. PerSyn shifts distillation data construction from "Generate then Select" to "Route then Generate," where a router first assigns each prompt to its optimal teacher based on learnability and quality, and each teacher synthesizes only its assigned prompts.
2. Strong empirical results across models and tasks. Experiments cover five student models and two distillation settings, with reported gains such as 8.7% on IFEval and 7.5% on MATH.
3. A new public artifact. The authors construct PerSyn-Math, a math dataset containing parallel responses from 15 teacher models over 10K sampled queries, to support future research.
4. Analysis of routing behavior. Ablations and allocation studies show that both quality and learnability matter (quality more so), that smaller teachers are frequently selected, and that Long-CoT teachers are still needed for some complex prompts.

Main Findings

• PerSyn beats all baselines. On Qwen2.5-3B, PerSyn surpasses the Strong baseline by 2.9% on SVAMP, 7.6% on TruthfulQA, and 8.7% on IFEval. Relative to CAR on Llama-3.2-3B, PerSyn gains 4.1% on TruthfulQA, 5.8% on IFEval, and 7.5% on MATH. Across the five main student models, PerSyn's average score exceeds CAR's (for example, 34.13 vs. 32.77 on Qwen2.5-0.5B and 50.63 vs. 49.21 on Qwen2.5-1.5B).
• Stronger teachers are not always better teachers. The paper reports that over 95% of prompts are routed to smaller teacher models rather than the single super-sized teacher used by the Strong baseline. Small teachers such as Qwen2.5-3B-Instruct receive higher allocation than Qwen2.5-7B/14B/32B-Instruct and even Llama-3.1-405B-Instruct for a Qwen2.5-3B student.
• Quality matters more than learnability. Ablations dropping either reward term degrade performance, but removing quality hurts more. Consistent with this, sweeping the balance parameter α from 0.1 to 0.9 shows a peak at α = 0.4, so 0.4 is the default.
• The router generalizes to larger students. In instruction tuning, PerSyn improves over CAR by an average of 3.4%, 3.6%, 3.1%, and 2.7% on Qwen2.5-7B, Llama-3.1-8B, Gemma-2-9B, and Qwen2.5-14B respectively.
• A small amount of parallel data suffices. 500K pairwise training samples, constructible from roughly 2.5K prompts with parallel teacher responses, are enough for an effective router in instruction tuning; 250K pairwise samples from about 2.5K prompts suffice in math reasoning. The 1.5B router backbone performs comparably to the 3B backbone, and PerSyn router results are comparable to or exceed an Oracle router that uses ground-truth rewards.
• Long-CoT teachers remain necessary for some prompts. Forcibly reassigning prompts originally given to Long-CoT teachers to a strong Short-CoT teacher (Qwen2.5-Math-7B-Instruct) caused a 1.3% average performance drop, and that Short-CoT teacher answered only 7.4% of those prompts correctly. Training entirely on Long-CoT data (the Strong baseline) degrades performance and produces repetitive, non-terminating reasoning.
• Qwen2.5-72B-Instruct is a consistently favored teacher. It receives high allocation across student models in both settings, described as a strong and versatile teacher.

Methodology in Plain English

For each prompt, the method needs to know which teacher is best for a given student. It scores every candidate teacher response with two signals:

1. Learnability reward — the student model's own average log-probability over the response tokens. A higher value means the response fits what the student already knows.
2. Quality reward — a score from a reward model (Skywork-Reward-Llama-3.1-8B in instruction tuning; a binary correct/incorrect signal in math).

These are combined as (1 − α) × quality + α × learnability after normalization across teachers, with α set to 0.4.

To avoid generating responses from every teacher for every prompt, the authors train a lightweight router (a Qwen2.5-1.5B model whose language modeling head is replaced by a linear "coefficient head" with one output per teacher). The router is trained on pairwise preferences built from a small subset of prompts — around 2.5K — where all teachers did answer in parallel. It uses the Bradley-Terry formulation, where the probability that teacher B is preferred over teacher A is the sigmoid of the difference in router scores, trained with binary cross-entropy. At synthesis time, each prompt goes to the highest-scoring teacher, and that teacher alone produces the response. The resulting dataset is used to fine-tune students with LLaMA-Factory (full-parameter up to 14B, LoRA above that).

Evaluation uses TruthfulQA, LiveBench, and IFEval for instruction tuning, and SVAMP, MATH, and GSM8K for math reasoning, in zero-shot settings except GSM8K, which is 5-shot with greedy decoding.

Why This Matters

If the best teacher for a small model depends on the prompt, then distillation pipelines can get better results and spend less compute at the same time — the paper's own example notes that generating a 100K-prompt dataset with 20 teachers requires 2,000K generations under the old paradigm. PerSyn shifts this cost because each teacher only handles its assigned share, and most prompts go to smaller, cheaper teachers.

Real-world applications:

• On-device and edge assistants. Producing compact models such as Qwen2.5-0.5B or Gemma-2-2B that run under hardware constraints while retaining instruction-following and math ability.
• Cost-efficient fine-tuning pipelines. Teams maintaining a pool of open teacher models can synthesize targeted data without paying for every teacher to answer every prompt.
• Educational and math tutoring systems. The math reasoning results and PerSyn-Math dataset directly target competition-level and grade-school math problem solving.
• Model portfolio management. The finding that Qwen2.5-72B-Instruct is a consistently strong teacher, while Long-CoT models are only needed for certain complex prompts, informs how organizations allocate teacher resources.

Industry relevance: the work reduces the compute bill of synthetic data generation and offers a recipe for matching student models to teacher pools, which matters for anyone distilling open-source models instead of training from scratch.

Future Directions

• Generalization beyond the two tested settings. The authors state it is unclear whether PerSyn transfers to code generation, multi-modal understanding, and other specialized domains.
• Scaling the student side. Experiments only cover student models up to 14B parameters; 32B and 70B students were not evaluated due to computational constraints.
• Better understanding of routing behavior. Why certain complex prompts require Long-CoT teachers, and how to identify them earlier, remains open.
• Reward model choices. The paper's appendix includes experiments on weak versus strong reward models, and the sensitivity of PerSyn to reward quality is a natural follow-up question.

Target Audience

Researchers and engineers working on LLM distillation, synthetic data generation, or small-model training, plus practitioners who maintain pools of open-weight teacher models and want to cut data-synthesis cost. It is also useful for readers tracking the "stronger is not always better teacher" line of work, since it directly extends the Mix and CAR baselines. Some familiarity with reward models and preference learning will help, but the core routing idea is accessible to anyone who has fine-tuned a language model.

Authors’ abstract

Training student models on synthetic data generated by strong teacher models is a promising way to distilling the capabilities of teachers. However, recent studies show that stronger models are not always optimal teachers, revealing a mismatch between teacher outputs and student learnability. To address this issue, we propose PerSyn (Personalized data Synthesis), a novel synthesis strategy that operates under a new ``Route then Generate'' paradigm to create data tailored to each student model, enabling it to learn more effectively. Specifically, PerSyn first assigns each prompt to its optimal teacher via a query-level router that jointly considers student learnability and teacher response quality. Each teacher then synthesizes data only for its assigned prompts, making the process more efficient than the conventional ``Generate then Select'' paradigm, where all teachers must generate parallel responses for the entire prompt set before constructing the final dataset. Extensive experiments across different model families and scales demonstrate that PerSyn consistently achieves superior or comparable performance to all baselines in instruct tuning and math reasoning settings. Further analysis verifies the effectiveness of PerSyn and offers extra insights to propel future research.