Leave It to the Experts: Detecting Knowledge Distillation via MoE Expert Signatures

AI summary

Overview

Research area: Machine learning security and model provenance, specifically detecting knowledge distillation (KD) between large language models using Mixture-of-Experts (MoE) routing behavior.

Technical level: Advanced. The paper assumes familiarity with Mixture-of-Experts architectures, sparse routing, knowledge distillation, and permutation-invariant distance measures (Wasserstein-1).

Scope: The paper proposes a framework for detecting whether a suspected student model was distilled from a suspected teacher model, using expert specialization and expert collaboration signatures extracted from MoE routing, plus a black-box extension called Shadow-MoE that builds proxy MoE models for models whose internals are inaccessible.

What This Paper Is About

Knowledge distillation lets smaller "student" models inherit capabilities from larger "teacher" models, but it also enables unauthorized copying of proprietary models and risks making the model ecosystem homogeneous. Existing detection methods rely on self-identity probing or output-similarity comparisons, and the authors argue these are easy to defeat with prompt engineering or produce false positives because models trained on similar data behave alike anyway. This paper asks whether distillation leaves a structural fingerprint in how MoE experts specialize and collaborate, and whether that fingerprint can be detected reliably — even when only black-box text access is available.

Key Contributions

1. Formalizing KD detection with MoE Expert Signatures. The authors cast distillation detection as a binary hypothesis test over a "Knowledge Distillation Set," and introduce two routing-based signatures: expert specialization (which experts activate for which task domains) and expert collaboration (which experts co-activate). They report accuracy up to 94% using these signatures.

2. Shadow-MoE, a black-box extension. For models that are dense, non-MoE, or accessible only through an API, the authors train proxy sparse MoE models ("Shadow-MoE" proxies) via text-level distillation to mimic each target model's input-output behavior, exposing analyzable routing patterns. They report this raises accuracy to 100%.

3. A new reproducible benchmark. The authors describe their work as the first to provide a benchmark for distillation detection with reproducible experimental protocols and diverse distilled checkpoints, including 18 student checkpoints produced from 9 domain-specific datasets under 2 training conditions.

4. Robustness and ablation evidence. The paper reports robustness to prompt-based evasion and analyzes which MoE layer and which calibration prompts yield the strongest detection signal.

Main Findings

• Routing signatures separate distilled from non-distilled students. In the setting with a black-box teacher and white-box student MoE models, distilled students consistently showed lower Wasserstein distance to the teacher's proxy than their non-distilled counterparts, with reductions of 4% to 20% for Expert Specialization and 2% to 19% for Expert Collaboration. The one exception was Code Contest, where the non-distilled model showed 5% and 2% lower distance.

• Semi-black-box accuracy of 94%. With a black-box teacher (DeepSeek-R1) and white-box student models (OLMoE-1B-7B checkpoints), Shadow-MoE averaged 94% accuracy across the task sets, versus 46% for the Linear baseline, 54% for BERT, 88% for Idiosyncrasies, and 0% for Model self-identity. It reached 100% on Math, Science, and Puzzle, and 75% on Code.

• Perfect accuracy in the pure black-box setting. When proxies were built for both teacher and student, the method reached 100% average accuracy, with distance reductions of 11% to 62% for Expert Specialization and 11% to 46% for Expert Collaboration. The previously difficult Code Contest task showed 11% and 12% lower distances for the distilled model.

• Identity probing fails completely. The Model self-identity baseline, which uses jailbreaking via GPTFuzz to probe identity consistency, scored 0% across all task sets in both settings.

• Idiosyncrasies is the strongest prior baseline. At 88% average accuracy in both settings, the Idiosyncrasies baseline (which fine-tunes a text embedding model, LLM2vec, on teacher-generated responses) was the best-performing existing method, though it dropped to 50% on Code tasks.

• Deeper MoE layers carry the strongest signal. Extracting signatures from the first, median, and last MoE layer gave average accuracies of 46%, 85%, and 94% respectively in the semi-black-box setting. The last layer, which the authors use throughout, performed best.

• General instruction-following calibration prompts beat domain-specific ones. Testing 9 training tasks against 28 diverse calibration subsets, specialized math and code calibration sets failed to capture significant routing differences, while general instruction-following calibration sets achieved reductions of -60% to -100%. The authors suggest the most informative routing changes caused by distillation appear when processing instruction-related tokens rather than domain-specific content.

• Ablation on the published abstract claim. The abstract states over 94% detection accuracy across scenarios; Table 2 reports exactly 94% average in the semi-black-box setting, and Table 3 reports 100% in the pure black-box setting.

Methodology in Plain English

The core idea is that distillation copies more than outputs — it copies "habits" in how a model computes. In an MoE model, those habits show up as routing decisions: which experts fire for a given kind of input, and which experts tend to fire together.

The authors first define what it means for a model to belong to a teacher's "distillation set" and frame detection as a yes/no test: was this student distilled from this teacher, or not?

To capture routing behavior, they compute two profiles from the model's last MoE layer. The specialization profile records how often each expert is selected for each task domain (math, code, science, puzzle), normalized so it reads like a probability distribution over experts. The collaboration matrix records how often pairs of experts are active on the same input. Because expert index labels are arbitrary — one model might number its experts differently while behaving identically — they compare profiles using permutation-invariant Wasserstein-1 distances, searching over all possible expert re-labelings. The final detection score is the negative average of the two distances, so a higher score means the student's routing looks more like the teacher's.

When a model is not an MoE, or is only reachable through an API, they cannot read routing directly. Their fix is to train a sparse MoE "shadow" proxy to imitate that model's input-output behavior using a small calibration set of prompts, adding a load-balancing regularizer so that experts do not collapse into only a few being used. Detection then becomes a comparison between shadow proxies.

In experiments, DeepSeek-R1 serves as the black-box teacher and Moonlight-16B-A3B is trained as the shadow proxy; OLMoE-1B-7B is the student architecture, trained both with and without distillation on 9 domain datasets spanning Code, Math, Science, and Puzzle. The calibration set contains 280 prompts sampled from allenai/tulu-3-sft-mixture, and proxies are trained for 3 epochs at a learning rate of 5×10⁻⁶. Each detector must pick which of two student checkpoints from the same dataset was distilled, and accuracy is averaged across the 9 domains.

Why This Matters

Impact on research. The paper reframes distillation detection from surface behavior to internal structure. Instead of judging a model by what it says or claims to be, it looks at how the model organizes computation. It also introduces a benchmark with paired distilled and scratch-trained checkpoints across domains, addressing a gap the authors identify in reproducible evaluation for this problem.

Real-world applications:

• Model provenance auditing: A lab or regulator could test whether a released model was derived from a specific proprietary teacher, supporting licensing and provenance claims.
• Intellectual property protection: Model developers could investigate suspected unauthorized distillation of their models, which the paper frames as a central motivation.
• API-only investigation: Because Shadow-MoE works without access to weights, providers who only expose text endpoints could still make detection attempts on third-party models.
• Ecosystem diversity monitoring: The paper raises concern that over-reliance on a few teachers homogenizes the model landscape; this method could help measure how widespread distillation from dominant teachers has become.

Industry relevance. Companies that license model weights or serve models through APIs have a direct interest in detecting copying. The paper reports that a lightweight self-identity check — one plausible low-cost approach — scored 0% in their tests, suggesting that robust detection requires more than prompt probing. The Shadow-MoE route does, however, require training proxy models, which is a nontrivial computational cost that the paper does not quantify.

Future Directions

• Reducing proxy training cost. Shadow-MoE requires training a Moonlight-16B-A3B proxy for the teacher in the semi-black-box setting and for both sides in the pure black-box setting. Whether smaller or cheaper proxies can preserve 94% to 100% accuracy is not reported.

• Extending beyond expert routing. The paper focuses on output text-level distillation, one of three granularity levels it identifies (hidden-state-level and logits-level being the others). Whether these signatures detect other distillation variants remains open.

• Understanding the calibration prompt effect. The finding that general instruction-following calibration sets outperform domain-specific ones by a wide margin (down to -60% to -100% reductions) is described as counterintuitive and is not fully explained.

• Robustness to deliberate evasion beyond prompting. The paper reports robustness to prompt-based evasion, but does not report tests against adversaries who know about routing-signature detection and modify training to obscure it.

• Determining distillation direction and strength. The framework tests whether a specific student came from a specific teacher; the paper does not report methods for establishing direction among many candidate models or for quantifying how much distillation occurred.

Target Audience

Researchers in machine learning security, model provenance, and intellectual property protection for AI systems; engineers who build or audit large language models and need practical tooling to identify copied models; and practitioners studying Mixture-of-Experts architectures who want to understand what routing patterns reveal about how a model was trained. Readers need prior exposure to MoE routing and distillation to follow the formal definitions, though the high-level intuition — that distillation leaves structural fingerprints — is accessible to a broader audience.

Authors’ abstract

Knowledge Distillation (KD) accelerates training of large language models (LLMs) but poses intellectual property protection and LLM diversity risks. Existing KD detection methods based on self-identity or output similarity can be easily evaded through prompt engineering. We present a KD detection framework effective in both white-box and black-box settings by exploiting an overlooked signal: the transfer of MoE "structural habits", especially internal routing patterns. Our approach analyzes how different experts specialize and collaborate across various inputs, creating distinctive fingerprints that persist through the distillation process. To extend beyond the white-box setup and MoE architectures, we further propose Shadow-MoE, a black-box method that constructs proxy MoE representations via auxiliary distillation to compare these patterns between arbitrary model pairs. We establish a comprehensive, reproducible benchmark that offers diverse distilled checkpoints and an extensible framework to facilitate future research. Extensive experiments demonstrate >94% detection accuracy across various scenarios and strong robustness to prompt-based evasion, outperforming existing baselines while highlighting the structural habits transfer in LLMs.