Antidistillation Fingerprinting

AI summary

Overview

Research area: Machine Learning — specifically LLM watermarking, model attribution, and intellectual property protection for foundation models.

Technical level: Advanced. The paper assumes familiarity with autoregressive language models, softmax/Jacobian derivations, fine-tuning dynamics, and hypothesis testing.

Scope: A method for embedding statistically detectable fingerprints into a teacher LLM's outputs so that third-party distillation (fine-tuning a student on those outputs) can later be proven with a rigorous p-value, without degrading the teacher's generation quality.

What This Paper Is About

Frontier LLM developers want to know when a competitor has quietly distilled their model by fine-tuning a student on its outputs. Existing watermark-based detection methods statically bias outputs toward a random "green list" of tokens, which forces a bad trade-off: the watermark must be strong enough for the student to learn it, but strong enough to visibly hurt generation quality. ADFP reframes fingerprinting as an optimization problem aligned with how the student actually learns, targeting tokens that will most efficiently drive the student toward the fingerprint while leaving normal text quality largely intact.

Key Contributions

1. Antidistillation Fingerprinting (ADFP): A principled fingerprinting scheme that replaces heuristic uniform green-list boosts with a gradient-derived logit perturbation, Δ^ADS_t = q_t · (I[t ∈ S] − L), where q_t is a proxy model's token probability and L is its total green-list mass.

2. Theory linking fingerprinting to learning dynamics: A derivation showing the perturbation approximates the inner product between the gradient of the token log-probability and the gradient of a per-step green-list loss, justified by an isotropic Gram-matrix approximation that holds exactly when only a final linear layer is trained.

3. Statistically grounded detection: A detection procedure based on the average green-list token probability (GTP) with a Hoeffding-derived conservative p-value, p = exp(−2n(g_obs − γ)²), working for both open-weight and closed-weight students.

4. Empirical Pareto improvement across three domains: Demonstrations on GSM8K (math reasoning), OASST1 (dialogue), and MBPP (code) that ADFP dominates the red-and-green-list baseline in the quality-versus-fingerprinting trade-off, including when the student architecture differs from the proxy.

Main Findings

• Roughly an order-of-magnitude better detection at equal quality: At matched teacher accuracy on GSM8K, ADFP reduces the expected false positive rate from p ≈ 0.09 (baseline) to p ≈ 0.01 — i.e., stronger statistical evidence of distillation for the same utility cost.

• Consistent Pareto improvement across all three domains: The quality–p-value trade-off curves on GSM8K, OASST1, and MBPP sit strictly better than the red-and-green-list baseline, whether quality is measured by answer-forced accuracy, NLL on the original teacher, or code execution pass rate plus syntax validity.

• Works when the student architecture is unknown: ADFP was tested with students either matching the proxy (Qwen2.5-3B, Qwen2.5-Coder-3B) or differing from it (Llama-3.2-3B), and the advantage held in both regimes.

• Minimal fine-tuning degradation: When the proxy equals the student, ADFP causes only negligible loss in downstream fine-tuning quality even at strong fingerprinting strength — unlike the baseline, which visibly harms the student.

• Robust to partial data contamination: When only a fraction α of the student's fine-tuning data is fingerprinted, ADFP still detects distillation more effectively than the baseline at comparable teacher quality, degrading gracefully as α shrinks.

• More coherent text qualitatively: ADFP outputs are observed to be less repetitive and more coherent than the baseline's at comparable fingerprinting strength.

Methodology in Plain English

The teacher model owner holds a secret key that, given the last couple of tokens, deterministically produces a random "green list" of half the vocabulary. During generation, the teacher slightly reweights its next-token distribution.

The baseline approach simply adds a fixed bonus to every green-list token. ADFP does something smarter. It runs a small proxy model (a stand-in for the unknown student) on the same context to see which tokens the proxy considers likely. It then perturbs the teacher's logits by an amount proportional to each token's proxy probability, but with a twist: green-list tokens get boosted, red-list tokens get suppressed, and the whole thing is centered around the proxy's own green-list mass L. This comes out of a gradient calculation — it's the direction that most increases the proxy's expected green-list probability if we were to fine-tune it on the chosen token.

For detection, the owner takes an evaluation dataset distinct from (but related to) the student's training data, computes the fraction/probability of green-list tokens the student would produce, and turns that into a p-value using Hoeffding's inequality. A small p-value (say, 0.01) is strong evidence that the student was trained on the teacher's fingerprinted outputs; a p-value near γ ≈ 0.5 means no evidence.

The experimental pipeline is three stages: (1) the teacher generates fingerprinted outputs, (2) a student is LoRA fine-tuned on those outputs, (3) the owner evaluates the fine-tuned student with the secret key and reports p-values alongside utility metrics.

Why This Matters

Impact on research: The paper provides a theoretical explanation for why watermarks propagate during distillation — they are absorbed proportionally to a student's learning signal on the specific tokens they bias — and shows how to design them deliberately rather than incidentally. It also connects watermarking, membership inference, and distillation detection into a single statistical framework.

Real-world applications:

• API terms enforcement: A model provider can statistically evidence that a third party's model was trained on their API outputs, in violation of terms of service.
• Licensing audits: Companies selling model weights or API access can embed fingerprints to verify downstream compliance with license terms.
• Benchmark contamination detection: Marked samples can be planted proactively in public datasets, and downstream models screened for absorption.
• Model provenance and attribution: Fingerprints offer a forensic trail for determining whether a suspicious model is derivative of a particular teacher.

Industry relevance: Distillation is both widespread and legally ambiguous. Model providers currently lack statistically defensible evidence of infringement. ADFP gives them a low-cost mechanism (a per-token logit perturbation) that does not degrade normal service quality, making it practically deployable. It also reduces the false-accusation risk that would otherwise make watermark-based accusations legally dangerous.

Future Directions

• Robustness to adversarial student-side defenses: Can a determined distiller detect and strip the fingerprint — for example, by paraphrasing, token-filtering, or distilling through an intermediate model?
• Stronger fingerprint identity: Extending the scheme from binary detection ("was this student trained on the teacher?") to attribution ("which of N teachers was this student trained on?") via multi-key or multi-signature designs.
• Reward-model or preference-tuning regimes: The current framework assumes supervised fine-tuning on teacher text; whether ADFP carries over to RLHF, DPO, or preference-based distillation is untested.
• Tighter statistical bounds and calibration: The Hoeffding p-value is conservative; sharper concentration bounds or likelihood-ratio tests could yield more detection power without additional utility cost.

Target Audience

Researchers and practitioners in LLM security, watermarking, and intellectual property protection, plus applied scientists at frontier labs or model-serving companies who need to enforce API or license terms. The paper will also interest statisticians and machine learning theorists working on membership inference and negative-result attribution. Readers should be comfortable with transformer fine-tuning, softmax gradients, and basic hypothesis testing; the math is self-contained but non-trivial.

Authors’ abstract

Model distillation enables efficient emulation of frontier large language models (LLMs), creating a need for robust mechanisms to detect when a third-party student model has trained on a teacher model's outputs. However, existing fingerprinting techniques that could be used to detect such distillation rely on heuristic perturbations that impose a steep trade-off between generation quality and fingerprinting strength, often requiring significant degradation of utility to ensure the fingerprint is effectively internalized by the student. We introduce antidistillation fingerprinting (ADFP), a principled approach that aligns the fingerprinting objective with the student's learning dynamics. Building upon the gradient-based framework of antidistillation sampling, ADFP utilizes a proxy model to identify and sample tokens that directly maximize the expected detectability of the fingerprint in the student after fine-tuning, rather than relying on the incidental absorption of the un-targeted biases of a more naive watermark. Experiments on GSM8K, OASST1, and MBPP demonstrate that ADFP achieves a significant Pareto improvement over state-of-the-art baselines, yielding stronger detection confidence with minimal impact on utility across mathematical reasoning, dialogue, and code generation, even when the student model's architecture is unknown.