Source Identification Is Not Fitness Testing: Measuring the Limits of Synthetic-Data Attribution

AI summary

Overview

• Research area: Machine learning — provenance and attribution of model-generated text, and recursive (self-consuming) training.
• Technical level: Intermediate. The paper uses standard classification metrics, conformal prediction and permutation tests, but explains each in plain terms.
• Scope: One sentence — this paper measures how well the source of generated financial-risk text can be recovered before and after rewriting, and tests separately whether that source information helps select generated examples that produce better models over three rounds of retraining.

What This Paper Is About

Repeatedly training a model on data generated by earlier models can degrade the later models, so provenance — knowing where a generated passage came from — looks like a natural way to decide which generated examples to reuse. This paper tests two things that are often conflated: whether provenance can actually be recovered reliably, and whether provenance is useful for choosing training data. The answer it reports is that identification works well under the original test conditions but weakens sharply after rewriting, and that a provenance-based selection rule and a reference-model-perplexity rule pick different examples yet show no detected difference in downstream degradation over three recursive rounds.

Key Contributions

1. It shows that generated-versus-human detection and exact generator attribution are distinct source-identification tasks, and that exact generator identification degrades substantially after paraphrasing and style rewriting.
2. It tests two further robustness questions: whether detectors trained on known rewriting methods transfer to other rewrites in the same family, and whether uncertain source assignments are rejected rather than returned as one wrong label.
3. It compares two selection rules in recursive training — a source-detector-confidence rule and a reference-model-perplexity rule — and reports that the rules select different examples while the planned comparison does not detect a stable difference in downstream degradation.
4. It explicitly separates source identity, criterion-facing selection, and recursive training outcome, and states that this separation does not establish equivalence of the rules or superiority of provenance.

Main Findings

• Attribution is strong before rewriting: In the initial ten-generator experiment, the mean pairwise AUROC was 0.9995 with a minimum of 0.9811, and the ten-way classifier reached 98.15% accuracy. The later seven-generator study's word-TFIDF classifier identified the generating model with 98.7% accuracy on original passages.
• Attribution collapses after rewriting: Averaged over the seven source models, word-TFIDF accuracy fell from 98.7% to 96.7% after roundtrip translation, 53.1% after paraphrasing, 29.0% after style transfer, and 46.9% after generic evasion. A separately trained DistilRoBERTa classifier scored 96.0% before rewriting and 9.4% after style transfer, so the failure was not specific to the word-based classifier.
• Generated-versus-human detection holds up differently: Against the tested human comparison corpus, the binary TF-IDF AUROC stayed above 0.9999 under every rewrite. In a separate test, the minimum per-generator AUROC exceeded 0.9999. The paper stresses these results apply only to the register-matched but not task-matched human baseline.
• An eight-class classifier was highly accurate on originals: Its overall accuracy was 98.81%, and every human paragraph in that test set was assigned to the human class.
• Training on known rewrites helps only within that family: When instances of the tested rewrite methods were included in detector training, a character-TFIDF classifier reached 99.3% to 99.8% accuracy on those transformations. Holding one rewrite type out gave mean accuracy of 95.7%, with the lowest value, 83.4%, on held-out style transfer.
• Abstention does not remove every error: At alpha = 0.05, paraphrasing and generic evasion usually produced non-singleton prediction sets, with wrong singleton rates of 0.0027 and 0.0032. Style transfer behaved differently: mean singleton rate 0.5467 and wrong singleton rate 0.4083, reaching 0.8567 for the worst source model, meaning the detector often returned one wrong source label instead of abstaining.
• Unseen-source recognition depends on which source is held out: When one of the seven reference generators was excluded from training, unseen-source AUROC ranged from 0.7700 to 0.9923, mean 0.9261. For the five additional test-only generators (600 items each), AUROC ranged from 0.9714 to 0.9929, mean 0.9822. The paper notes the two settings train on six and seven generators respectively, so the AUROCs are not directly comparable.
• Different selection rules, no detected recursive difference: The provenance rule and the perplexity rule retained substantially different examples — mean overlap of 36.0 of 80 at G1, 37.1 at G2, and 36.2 at G3, against a descriptive reference of 40 for two independent uniform 80-of-160 selections. The maximum absolute correlation between source-detector confidence and reference-model perplexity was 0.255.
• The primary comparison: Mean held-out-perplexity slope was 7.543 points per generation under the perplexity rule and 7.367 under the provenance rule, a difference of +0.176. The exact two-sided permutation test over 252 assignments did not reject the null (p = 0.746). The random-rule mean slope was 7.390.
• Uncertainty remains wide: The test-inversion 95% interval was [-0.794, +1.267] perplexity points per generation, and a Welch interval was [-0.959, +1.311], so the experiment does not support an equivalence claim.
• The design had limited power: Using the observed pooled lineage standard deviation of 0.716 in 200,000 balanced five-versus-five simulations, estimated power was 0.048 for a true difference of 0.00, 0.157 for 0.5 points, 0.476 for 1.0 point, 0.799 for 1.48 points, and 0.964 for 2.0 points per generation.
• The setup could register recursive change: Held-out perplexity increased from G0 to G3 in every fixed-draw lineage, and at G1 the perplexity rule produced lower held-out perplexity than the provenance rule for all five shared seeds — a direction that did not persist across later generations.

Methodology in Plain English

The work has two parts.

Part one — can we tell where text came from? The researchers used a checksum-frozen corpus of financial-risk text generated by ten models under a common prompt structure, with an initial attribution sample using responses of at least 80 words, capped deterministically at 400 items per model and condition. The robustness study used the seven source models for which the full sweep was complete, with 16,384 records per source. Human comparison text came from 10-K Item 1A financial-risk paragraphs filed in 2019 or earlier, length-matched by token stratum, with a deterministic cap of 20 paragraphs per ticker retaining 879 paragraphs. They trained linear classifiers on word and character TF-IDF features (unigrams and bigrams, sublinear scaling, minimum document frequency 3, logistic regression) plus a fine-tuned DistilRoBERTa transformer, training and evaluating each independently.

They then rewrote the generated text four ways meant to preserve content while changing surface form: roundtrip translation through a pivot language, cross-family paraphrasing, style transfer to the human 10-K register, and generic evasion. The threat model was grey-box — the rewriting model knew stylometric identification was in use but had no detector parameters, training data or query access, and produced one rewrite per item. For each condition, the rewriting model was excluded from detector training. Secondary diagnostics trained character-TFIDF models with examples of the tested rewrite methods, measured transfer between them, and held one rewrite type out at a time.

To test abstention, they used a split-conformal predictor with least-ambiguous-classification scores, producing an eight-way prediction set at alpha = 0.05. An empty or multi-element set counted as an abstention; a singleton set counted as a source assignment, and a wrong singleton was a one-label set containing the wrong source. They also tested unseen sources two ways: holding one of the seven roster models out of training, and using five test-only generators of 600 items each kept outside detector development.

Part two — does source information help select training data? Each independent run, called a lineage, started from the same Qwen2.5-3B-Instruct checkpoint and went through three generate-select-train transitions, G0 to G3. At each transition the current checkpoint generated one financial-risk response for each of 160 fixed prompts, using target windows of 120–160 and 100–140 words, temperature 1.0, top-p 1.0, top-k 0, no truncation and a preassigned lineage seed. Every candidate in this part was generated by the current model; the human corpus was used only for downstream evaluation.

Four rules each retained 80 of the 160 generated items under the same fine-tuning budget: provenance (the 80 items with the lowest top-two posterior margins from a frozen eight-way generator-plus-human source classifier), perplexity (the 80 with the lowest perplexity under a frozen Qwen2.5-1.5B-Instruct reference model, called "Criterion" in the frozen records), random (a content-blind draw redrawn per cell), and a fixed-draw control (the same content-blind 80-of-160 draw, seed 7, across all lineages and generations). The source classifier and reference model were fixed before the run and never refitted, and no selection rule was evaluated using the score that selected its own training data. Notably, Qwen2.5-3B-Instruct was not in the seven source models, though two related Qwen-family models (Qwen2.5-14B-1M and Qwen2.5-32B) were present.

Provenance and perplexity used five prime-number seeds (2, 3, 5, 7, 11); the random and fixed-draw controls used three (2, 3, 5). Since selection starts at G1, this yields 48 evaluated rule-generation cells after G0. The primary outcome was corpus perplexity on a frozen set of 200 pre-2020 human financial-risk paragraphs excluded from source-detector training. For each lineage an OLS slope of held-out perplexity against generation was fitted, and the primary test compared mean slopes between the perplexity and provenance rules with an exact two-sided permutation over all 252 assignments at alpha = 0.05. The same test was inverted under a constant additive-shift model for the primary 95% interval, with a Welch interval as a robustness check, and a simulation estimated detectable slope differences.

Why This Matters

Impact on research. The paper separates questions that are often reported as one: detecting that text is generated, naming the generator, recognising a source never seen in training, and selecting which generated examples to reuse. It reports that neither the provenance score nor the tested criterion-facing proxy was established as sufficient for future recursive behaviour, and it explicitly frames the next problem as determining what information a curation rule must preserve for a declared downstream use. It also links this to a separate theoretical treatment of task-relative information contracts.

Real-world applications

• Auditing and traceability in managed machine-learning pipelines, where provenance can support audit and policy enforcement even if it does not predict training value.
• Data curation and filtering decisions in self-improving or continuously adapted systems that mix generated and real data.
• Content and authorship forensics, where the paper's finding that stylometric attribution weakens sharply under paraphrasing and style transfer matters for tools that claim to name a generating model.
• Quality and compliance workflows that rely on generated summaries, simulation output, testing material or documentation entering a managed pipeline, where the paper notes provenance alone does not indicate whether retaining a given example helps a later model.

Industry relevance. The paper is authored at Ericsson and motivates the problem through standards describing managed machine-learning pipelines, orchestration and data-handling functions for future networks. The experiment itself uses financial-risk text as a controlled setting; the authors state it is not a deployment experiment on telecommunications data, and its communications relevance concerns the selection problem in managed training pipelines rather than performance on network data.

Future Directions

• Determine what information a curation rule must preserve for a declared downstream use, and whether that information requirement still holds after training changes the learner — the paper names this as the unresolved problem.
• Test whether the recursive result transfers beyond a single model scale, since the experiment used Qwen2.5-3B-Instruct only.
• Extend beyond three recursive updates (G0 to G3) to capture longer-horizon effects, and beyond five lineages per primary rule, which limited power for small differences.
• Broaden beyond the four tested rewriting transformations, since holding one out does not approximate the full range of future transformations, and beyond one domain and task.
• Address the human-comparison mismatch: the human corpus matches the financial-risk register but not the exact generation task, so every generated-versus-human result is bounded by that comparison.
• Revisit the fact that the recursive source model (Qwen2.5-3B-Instruct) was absent from the seven-model source-classifier roster, meaning its confidence should not be read as a calibrated probability of true source membership.

Target Audience

Researchers and practitioners working on synthetic data, provenance and data curation for training pipelines, and generated-text detection or attribution; engineers designing managed machine-learning pipelines who need to know what provenance does and does not establish; and readers interested in recursive or self-consuming training loops and in how to report source-identification results without conflating detection, attribution and selection.

Authors’ abstract

Repeated training on model-generated data can degrade later models. One possible response is to use provenance when deciding which generated examples to reuse. We test both how reliably that provenance can be recovered and whether it helps identify better training data. Using financial-risk text, we first identify the source of generated passages and then repeat the test after rewriting them. Generator attribution is 98.7% accurate on the original passages but falls to 53.1% after paraphrasing and 29.0% after style rewriting. Generated-versus-human detection remains close to perfect against the tested human comparison set. We then compare two ways of selecting generated examples over three rounds of generation and retraining. One uses source information. The other uses a score from a separate reference model. The two rules select different examples, but the planned comparison does not detect a stable difference in the degradation of the resulting models. The results show that identifying where data came from and identifying which data are useful for training are separate problems. The experiment therefore separates source identity, criterion-facing selection, and recursive training outcome: neither the provenance score nor the tested criterion-facing proxy is established as sufficient for future recursive behaviour.