AWM: Accurate Weight-Matrix Fingerprint for Large Language Models

AI summary

Overview

• Research area: Natural Language Processing; specifically intellectual property protection and model lineage verification for large language models (LLM fingerprinting / provenance detection).
• Technical level: Advanced. The paper relies on linear algebra and kernel methods (Linear Assignment Problem, Centered Kernel Alignment, Hilbert-Schmidt Independence Criterion) and on Transformer internals (RMSNorm, RoPE, attention scores, residual connections). The experimental results, however, are readable on their own.
• Scope: A training-free, weight-matrix-based fingerprinting method that decides whether a suspect LLM is derived from a base LLM, tested on 150 model pairs across six categories of post-training modification.

What This Paper Is About

Training an LLM from scratch is expensive, so some developers may take an existing open-source base model, apply post-training or repackaging, and falsely claim they trained it themselves. The paper asks how a model owner or a third party can tell whether a suspect model is genuinely trained from scratch or derived from an existing base model, even after heavy post-training and deliberate weight manipulation. The proposed answer is a training-free fingerprint computed directly from weight matrices, called AWM (the paper's repository is at https://github.com/LUMIA-Group/AWM).

Key Contributions

1. A theoretical analysis of admissible weight manipulations. The authors formalize LLM weight manipulations (Definition 4.1) and show, via residual connections (Proposition 4.2), RMSNorm (Theorem 4.3), and RoPE plus attention scores (Theorem 4.4), that manipulations compatible with preserving model behavior are heavily constrained to constant scaling, permutation, sign flips, and special orthogonal transforms, applied to word embeddings and the Q, K matrices.
2. A training-free fingerprinting algorithm (AWM). A two-stage procedure combines the Linear Assignment Problem (LAP, solved with the Hungarian algorithm) to recover a permutation and signature matrix from word embeddings, with an unbiased Centered Kernel Alignment (UCKA) similarity over Q and K matrices.
3. A comprehensive testbed of 150 model pairs. 60 positive (base–offspring) pairs, 10 for each of six post-training categories, plus 90 negative (independent) pairs, built from 10 independent 7B and 10 independent 13B LLMs.
4. A performance and cost claim. Perfect scores on all classification metrics on this testbed (AUC, pAUC for FPR < 0.05, and TPR@1%FPR all 1.0), with the entire computation completing within 30 seconds on a single NVIDIA 3090 GPU.

Main Findings

• Clear separation on the effectiveness test. Offspring models of LLaMA2-7B and LLaMA2-13B scored very high similarity to their base models (for example 99.99 for WizardMath-7b and Selfrag_llama2_13b, and 96.60 for Llama2-13b-Estopia), while independent models scored near zero (for example 0.58 for Baichuan-7b, 0.04 for MPT-7b, 0.03 for Jais-13b).
• Near-zero false-positive risk on 90 unrelated pairs. Mean similarity for independent pairs was 0.49 (7B) and 0.26 (13B), compared with HuRef means of 3.56 and 2.17 and REEF means of 42.47 and 47.44. Maximum similarity for the proposed method was 1.5 (7B) and 0.8 (13B). REEF frequently produced scores above 80 and sometimes above 95 for unrelated models.
• Robustness across all six post-training categories. The method reported high absolute Z-scores on all 60 positive pairs, including continued pretraining runs up to 5.5T tokens (Qwen2.5-Coder). HuRef's |Z| fell as low as 0.57 for Qwen2.5-coder and 0.28 in one continual-pretraining entry, while REEF consistently produced |Z| often below 2.0.
• Pruning is the hardest case. Within the pruning group, the Llama3-1B entry had the lowest reported |Z| for the proposed method (12.14), with Llama3-3B at 106.29; the remaining pruning entries ranged from 297.50 to 344.07.
• Perfect overall classification metrics. Across the full 60 positive / 90 negative testbed, the method achieved AUC = 1.0, pAUC = 1.0 (for FPR < 0.05), and TPR@1%FPR = 1.0 in every post-training category, versus HuRef AUC = 0.957 and REEF AUC = 0.778 overall.
• Baseline collapse cases. REEF's pAUC and TPR@1%FPR fell to 0.0 for Upcycling (UP) and Multi-modal (MM); HuRef's TPR@1%FPR dropped to 0.500 under Continual Pre-Training (CPT).
• Largest separation margin. Average absolute Z-score overall was 302.936 for the proposed method, versus 28.142 for HuRef, 1.285 for REEF, 1.392 for Intrinsic Fingerprint, and 57.457 for PCS.
• Efficiency. The full similarity computation finishes within 30 seconds on one NVIDIA 3090 GPU, and the method is training-free, so it does not degrade model performance.

Methodology in Plain English

The authors first reason about what an attacker can actually do to a model's weights without breaking its behavior. Because Transformer blocks use residual connections, RMSNorm, and attention with RoPE, most arbitrary edits to the weights would change the model's outputs. The analysis shows that the edits which survive are essentially: multiplying by a constant, permuting rows/columns, flipping signs, and applying special orthogonal transforms — concentrated in the word embedding matrix and the Q and K projection matrices.

Given that, detection works in two steps. First, the method takes the two models' embedding matrices, restricts to the vocabulary they share, and builds a matrix of absolute cosine similarities between embedding columns. The Hungarian algorithm solves the Linear Assignment Problem to find the best one-to-one matching (the permutation), and the signs of the matched similarities give the signature matrix. This recovers the scaling, permutation, and sign-flip manipulations.

Second, the method compares the Q and K matrices layer by layer using Centered Kernel Alignment, which is invariant to orthogonal transformations and constant rescaling. The authors use the unbiased variant (UCKA) to avoid finite-sample bias. The final score averages these per-layer similarities over the Q and K matrices across all layers. For models with different layer counts — for instance after layer pruning — the method solves another assignment problem on the layer-wise similarity matrix to pair layers up before computing the overall score. Everything is done directly on weights, with no extra training and no access to model outputs.

Why This Matters

Research impact. The paper reframes LLM provenance detection as a problem about which weight manipulations are even possible under behavioral preservation, rather than about training a detector or embedding a watermark. Its combination of LAP-based recovery and unbiased CKA gives a parameter-free similarity metric, and its 150-pair testbed spanning SFT, continued pretraining, RL, multi-modal extension, pruning, and upcycling provides a harder benchmark than prior work that mainly tested fine-tuning.

Real-world applications.

• License enforcement: verifying whether a released model was built on a base model whose license restricts derivative use.
• Due diligence in model acquisition: a buyer or auditor checking a vendor's claim that a model was trained from scratch.
• Dispute resolution in model-theft controversies, where a high-fidelity, low-false-positive test is needed before making an accusation.
• Redistribution monitoring: detecting models that have been re-released under a new name after only light modification.

Industry relevance. Model-as-a-Service providers, licensing teams, and regulators all need cheap, reliable provenance checks. The claimed under-30-second, single-GPU cost and the absence of any training or performance penalty are the properties that make deployment plausible in a commercial pipeline; the near-zero false-positive behavior matters because wrongly accusing a legitimate model developer is the costlier error.

Future Directions

• Improving detection for aggressive pruning: the Llama3-1B pruning case had the lowest reported |Z| (12.14), well below the other pruning entries, suggesting structure-altering compression is the weakest point.
• Handling vocabulary divergence: the method relies on a shared vocabulary intersected between the two models, so tokenizer changes or vocabulary extension are not directly addressed in the reported experiments.
• Extending the analysis beyond word embeddings and Q/K matrices: the paper notes that remaining avenues targeting the other weights are deferred to Appendix B.5, and that the detection focuses on the partial weight set.
• Further reducing the parameter burden of semi-orthogonal transformations induced by pruning, where CKA's invariance is only partial, and testing whether the perfect classification scores on this 150-pair testbed hold on larger and more diverse model populations.

Target Audience

Researchers and engineers working on LLM security, model provenance, watermarking, and intellectual property protection; machine learning practitioners who need to audit model lineage; and legal, licensing, or compliance teams evaluating claims about how a model was built. Readers without a linear algebra or Transformer-internals background will find the motivation and experimental tables accessible, but the theoretical sections (Section 3 and Section 4) require an advanced mathematical background.

Authors’ abstract

Protecting the intellectual property of large language models (LLMs) is crucial, given the substantial resources required for their training. Consequently, there is an urgent need for both model owners and third parties to determine whether a suspect LLM is trained from scratch or derived from an existing base model. However, the intensive post-training processes that models typically undergo-such as supervised fine-tuning, extensive continued pretraining, reinforcement learning, multi-modal extension, pruning, and upcycling-pose significant challenges to reliable identification. In this work, we propose a training-free fingerprinting method based on weight matrices. We leverage the Linear Assignment Problem (LAP) and an unbiased Centered Kernel Alignment (CKA) similarity to neutralize the effects of parameter manipulations, yielding a highly robust and high-fidelity similarity metric. On a comprehensive testbed of 60 positive and 90 negative model pairs, our method demonstrates exceptional robustness against all six aforementioned post-training categories while exhibiting a near-zero risk of false positives. By achieving perfect scores on all classification metrics, our approach establishes a strong basis for reliable model lineage verification. Moreover, the entire computation completes within 30s on an NVIDIA 3090 GPU. The code is available at https://github.com/LUMIA-Group/AWM.