Perturb Your Data: Paraphrase-Guided Training Data Watermarking

AI summary

Overview

• Research area: Natural Language Processing / LLM data governance — specifically training data watermarking and membership detection.
• Technical level: Intermediate. The paper assumes familiarity with autoregressive language models, token log probabilities, p-values, and membership inference, but explains its own scoring method in detail.
• Scope: The paper introduces SPECTRA (Score sampled rePhrasing to detECt TRAining data), a watermarking-and-verification pipeline that lets content creators determine whether their text was used to train a large language model, without needing decoding-layer access or a matched held-out dataset.

What This Paper Is About

Large language models are pretrained on massive internet-scraped corpora, which may include copyrighted or license-restricted text, and model developers often do not disclose what they trained on. Existing membership inference attack (MIA) methods try to detect training data from model token probabilities but are fragile to distribution shifts and have been shown to perform no better than random once temporal artifacts are removed. The paper's goal is a watermark that a content creator applies before publishing, which remains statistically detectable after large-scale continued pretraining even when the watermarked data is a vanishingly small fraction (less than 0.001%) of the training corpus.

Key Contributions

1. A deploy-before-release watermark that survives large-scale training. The authors show SPECTRA can watermark pretraining data such that the watermark is still measurable after continued pretraining with 5 billion tokens, with each watermarked dataset constituting less than 0.001% of the training corpus.

2. A paraphrase-sampling strategy designed to avoid false positives. SPECTRA samples a paraphrase whose Min-K%++ score is close to the original document's score, using weights proportional to exp(−α|r_ij − 1|) with α = 100, and balances selection across the above-original and below-original sides in inverse proportion to how often each type appears globally. The paper reports this sampling strategy outperforms random sampling and selecting paraphrases with the maximum Min-K%++ score.

3. A statistical test that needs no non-member dataset. Verification compares the ratio of watermarked-to-original Min-K%++ scores under the target model against the same ratio under a separate scoring model, tested with a 1-sided paired t-test.

4. A benchmark against existing training-data detection methods. SPECTRA is reported as the only method yielding a statistically significant result for identifying membership across all datasets tested under a threshold of p < 10⁻⁴, with the largest member-versus-non-member p-value separation.

Main Findings

• Min-K%++ wins at small scale but collapses at large scale. In Table 1, after continued pretraining of Pythia 410m with 500 million tokens, Min-K%++ achieves ROC-AUC of 0.85 (Wiki), 0.84 (HN) and 0.72 (PubMed), the best among Loss, DC-PDD and Min-K%. After 5 billion tokens, all methods fall near chance: Loss 0.55 / 0.54 / 0.52, DC-PDD 0.55 / 0.52 / 0.50, Min-K% 0.56 / 0.55 / 0.52, Min-K%++ 0.55 / 0.55 / 0.51.

• SPECTRA detects membership on every dataset tested. Member p-values under Pythia 410m trained on the watermarked data, against a threshold of p < 10⁻⁴: 1E-17 (PubMed), 4E-19 (Wiki), 3E-60 (HN) and 2E-12 (PeS2o).

• SPECTRA produces no false positives at that threshold. Non-member p-values (from the original Pythia 410m) were 0.02 (PubMed), 0.02 (Wiki), 0.59 (HN) and 3E-3 (PeS2o), all above 10⁻⁴.

• The member/non-member p-value gaps exceed nine orders of magnitude. Reported Member-to-Non-member p-value ratios were 2E15 (PubMed), 5E16 (Wiki), 2E59 (HN) and 2E9 (PeS2o). The paper states SPECTRA consistently achieves a ratio greater than 10⁹, higher than the baselines at 500 samples each.

• Baselines fail under the strict threshold. LLM-DI member p-values were 0.06 (PubMed), 0.02 (Wiki), 0.49 (HN) and 0.02 (PeS2o). STAMP member p-values were 0.01, 0.17, 7E-4 and 0.15 respectively. The "Maximum" paraphrase-selection baseline produced non-member p-values of 1.00 on PubMed, Wiki and HN and 1.00 on PeS2o, and did not detect the HN member (3E-6 ratio of 3E5 — the p-value 3E-6 is below 10⁻⁴ for HN, but its non-member p-value of 1.00 is also above threshold). The Random baseline produced a false positive on Wiki (non-member 0.10) and failed to detect membership on PeS2o.

• Paraphrase quality is largely preserved. P-SP scores were 0.88 (PubMed), 0.93 (Wiki), 0.76 (HN) and 0.93 (PeS2o), against 0.78 reported for a human-generated paraphrase. In human evaluation, 54 randomly selected watermarked documents were distributed among four evaluators, with each evaluator reviewing 27 documents and each document assessed by two different evaluators on a 1–5 Likert scale. Mean scores for all three criteria (meaning, structure, author tone) exceeded 4, with structure preservation scoring comparatively lower, especially for conversational text such as Hackernews.

• Around 100–150 samples suffice. The p-value for every dataset tested drops below the threshold after 100–150 samples, while non-member p-values stay above it.

• SPECTRA is robust to the choice of scoring model. Against the original OLMo-1b scoring model on PeS2o paraphrases, Spearman ρ / Kendall τ were 0.826 / 0.639 (OLMo-7b), 0.824 / 0.635 (Pythia-2.8b-deduped), 0.699 / 0.514 (Pythia-160m-deduped) and 0.818 / 0.631 (Pythia-6.9b). The paper notes correlation is above 0.8 for models with more than 2.8b parameters and that τ > 0.6 is considered strong agreement.

• Cross-architecture detection works. SPECTRA correctly detected membership on PeS2o even though the scoring model (OLMo-1b) and target model (Pythia 410m) differ in architecture.

Methodology in Plain English

A content creator who wants to publish text first generates multiple paraphrases of each document with a large language model — in this study, Llama 3.1-405b, with 10 paraphrases per document, each sampled at a different temperature. Each paraphrase is scored with Min-K%++, which averages the normalized log probabilities over the highest-surprisal K% of tokens (K = 20%) under a separate scoring model that has never seen the data. That scoring model stands in for the target model's pre-training state; it is easy to choose because the data is still unpublished.

Rather than keep the highest-scoring paraphrase, SPECTRA samples one paraphrase whose score sits close to the original document's score, using an exponential weighting that favors closeness, and balances selection between paraphrases scoring above and below the original so the watermark does not shift the score distribution. Only the paraphrase is published; the originals stay with the creator.

To test a suspect model, the creator computes the ratio of the watermarked document's Min-K%++ score to the original document's score under both the suspect model and the scoring model. Because training raises Min-K%++ scores, the ratio under a model that saw the watermarked data should be lower than the ratio under the scoring model. A 1-sided paired t-test over all document pairs yields a p-value; a low p-value rejects the null hypothesis that the two ratios are equal. The authors test on 500 documents per Pile domain (Wiki, Hackernews, PubMed) and 216 watermarked documents for the PeS2o subset of Dolma, with matching held-out sets, and continue-pretrain Pythia 410m on the watermarked data plus 5 billion tokens sampled from the Common Pile dataset (USPTO, USGPO, ArXiv, LibreText, Doab domains, restricted to documents published after December 2020). Training used AdamW with learning rate 10⁻⁴, (β₁, β₂) = (0.99, 0.999), cosine decay, batch size 40, 0.5% warmup, no weight decay, on an L40S Tensor Core GPU with Transformers v4.43 and seed 1234.

Why This Matters

Training data detection matters because several lawsuits have concerned unauthorized use of pay-walled data for model training, and because undisclosed training data leaves open-weight model users exposed to liability. The paper frames the broader risk as an "extractive dead end," where content creators lack incentive to keep producing content that LLM providers scrape without permission.

Real-world applications:

• Copyright and licensing enforcement: content owners can test whether a suspect model was trained on their pay-walled or license-restricted text.
• Benchmark integrity: publishers of benchmarking datasets can check whether their data leaked into pretraining.
• Model release due diligence: open-weight model developers and their users can verify or dispute claims about training data provenance.
• Third-party arbitration: the paper notes that for closed-source models, testing could be handled by a neutral or court-appointed arbiter with grey-box access, since SPECTRA needs token log probabilities but not model weights or architecture.

Industry relevance: the work comes from JPMorgan AI Research, and it addresses a grey-box setting typical of open-weight models. Its practical advantages over MIA are that it requires no held-out dataset from the same domain, and over STAMP that it needs no decoding-layer access (which can be prohibitively GPU-expensive) and no large store of private rephrases. The authors argue the three-orders-of-magnitude p-value gap they attribute to STAMP is insufficient for a decision with significant legal implications, and adopt a stricter p < 10⁻⁴ threshold accordingly.

Future Directions

• Verification by third parties, not just the creator. The authors call for watermarking techniques that allow interested third parties, not only the content owner, to verify membership.
• Validation at true pretraining scale. They note that additional verification at the scale of pretraining a model from scratch, rather than continued pretraining, would inspire greater confidence for legal use.
• Detecting already-published content. The approach only helps going forward, since the watermark must be applied before publication; content already released cannot be protected this way.
• Better handling of structured or conversational text. Paraphrasing quality was weakest for Hackernews-style conversational text (P-SP 0.76, lower human structure-preservation scores), and the authors flag this as an open weakness, alongside the assumption of log-probability access for proprietary models.

Target Audience

Content creators and publishers enforcing data-use policies; legal and policy professionals working on AI training data disputes; LLM developers and auditors concerned with data provenance and benchmark contamination; and NLP researchers working on membership inference, dataset inference, and watermarking, who will find the comparison against Min-K%++, Min-K%, Loss, DC-PDD, STAMP and LLM-DI directly relevant.

Authors’ abstract

Training data detection is critical for enforcing copyright and data licensing, as Large Language Models (LLM) are trained on massive text corpora scraped from the internet. We present SPECTRA, a watermarking approach that makes training data reliably detectable even when it comprises less than 0.001% of the training corpus. SPECTRA works by paraphrasing text using an LLM and assigning a score based on how likely each paraphrase is, according to a separate scoring model. A paraphrase is chosen so that its score closely matches that of the original text, to avoid introducing any distribution shifts. To test whether a suspect model has been trained on the watermarked data, we compare its token probabilities against those of the scoring model. We demonstrate that SPECTRA achieves a consistent p-value gap of over nine orders of magnitude when detecting data used for training versus data not used for training, which is greater than all baselines tested. SPECTRA equips data owners with a scalable, deploy-before-release watermark that survives even large-scale LLM training.