Future Horizons

The Model Collapse Crisis: How AI Is Eating Its Own Training Data

Model collapse is real but conditional: it needs replacement, not accumulation. What has changed is the setting — the EU's marking duty applies since August 2026, and the data wall's near end is now.

Gabriele Masetti · 2026-08-28

A Feedback Loop With No Off Switch

In July 2024, Nature published a paper with a title blunt enough to function as a warning label: "AI models collapse when trained on recursively generated data." The authors — Ilia Shumailov and Yarin Gal of the University of Oxford, Zakhar Shumaylov and Ross Anderson of the University of Cambridge, Yiren Zhao of Imperial College London, and Nicolas Papernot of the University of Toronto and the Vector Institute — gave a name to something practitioners had been muttering about for years: train a generative model on its own outputs, generation after generation, and the model does not merely stagnate.

It actively forgets. Rare events vanish first, variance narrows, and eventually the system converges toward a degenerate, repetitive core that no longer resembles the data distribution it started with.

The finding matters because of timing, not novelty. The same team had already sketched the mechanism a year earlier in a 2023 preprint, "The Curse of Recursion: Training on Generated Data Makes Models Forget," and mathematicians had long understood that sampling error compounds across iterated processes. What changed in 2024 was the scale of the crisis it described.

By the time the Nature paper appeared, the open web — the substrate every foundation model is trained on — had already been running for nearly two years as a mixing chamber of human and machine-written text, with no reliable way to tell which was which. Model collapse stopped being a theoretical curiosity about recursive loops and became a live question about whether the entire pipeline that produces large language models has started poisoning itself.

That is worth being precise about, because the popular version of the story has drifted from the paper's actual claims. Model collapse is real, well-specified, and reproducible. It is not, however, evidence that synthetic data is categorically toxic, nor that the industry is racing toward an inevitable collapse of capability. The more interesting and more defensible claim is narrower: unfiltered, replacement-style recursive training degrades models in a predictable way, the mechanism generalizes across architectures, and the corporate and infrastructural response to that risk is currently inadequate to the scale of the problem — not because the science is unclear, but because the incentives to fix it are weak.

What Collapse Actually Is

Shumailov and colleagues describe model collapse as a degenerative process that unfolds across generations of models, where "generation" means training run rather than architecture version. Take a model, sample outputs from it, and use those outputs — either exclusively or predominantly — as training data for the next model. Repeat. The Nature paper identifies two compounding sources of error in this loop.

Statistical approximation error arises because any finite sample from a model's output distribution under-represents the tails — the rare word choices, unusual syntactic constructions, and low-probability but valid completions that make a distribution rich rather than flat. Functional approximation error arises because the model doing the approximating is never a perfect representation of the true distribution to begin with; every architecture has its own biases and blind spots, and those biases get reinforced rather than corrected when a model learns from another model's guesses.

Run this process enough times and the two errors compound. The tails of the distribution — the low-frequency but real patterns that give language its diversity — are the first casualties, because each generation samples them less often than the generation before, and the training signal has no path to bringing them back once they are gone. The paper demonstrated this pattern in variational autoencoders, in Gaussian mixture models, and in language models, which is precisely why the finding was hard to dismiss as an artifact of one architecture.

The mechanism is structural: it emerges from what iterated resampling does to a distribution, independent of the specific tokens or pixels involved. Later generations do not become adversarial or overtly broken in ways a casual reader would spot instantly — they become bland. Outputs cluster around the model's own most likely completions, variance collapses, and the model's grip on rare but valid modes of the distribution deteriorates until what remains is a narrow, repetitive residue of the original.

The Real Escape Hatch: Accumulation, Not Purity

The most important corrective to the collapse narrative did not come from critics disputing the Nature result, but from a 2024 paper that tested a different assumption embedded in the original experiments. Matthias Gerstgrasser, Rylan Schaeffer, and their coauthors, in "Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data," pointed out that Shumailov et al.'s core experiments modeled a world in which each generation's synthetic data replaces the previous training set rather than adding to it.

That is a strong and somewhat artificial assumption. When Gerstgrasser and colleagues instead simulated accumulation — each generation's synthetic output added on top of the original human-generated corpus rather than substituted for it — the test error no longer grew without bound across iterations. It leveled off at a finite ceiling instead.

That is not a minor technical footnote; it reframes the entire practical stakes of the problem. If replacement is the mechanism that drives collapse, then the question facing the industry is not "can synthetic data ever be used safely" but "is anyone actually discarding the accumulated human-generated corpus in favor of pure synthetic replacement."

In production settings, almost no one does this deliberately — labs mix synthetic and human data, and the training sets that matter, like those behind GPT, Claude, and Gemini-class models, still anchor on large accumulated human corpora rather than resetting to synthetic-only pools each generation. The danger the Nature paper describes is real but conditional, and the condition — full replacement rather than accumulation — is one that careful practice can avoid. The failure mode is not synthetic data itself; it is losing track of provenance so thoroughly that accumulation becomes impossible to guarantee.

Distillation Is the Counter-Example the Narrative Ignores

The strongest evidence against a blanket "synthetic data is dangerous" reading is that some of the most capable models in production today were substantially trained using synthetic data on purpose, and it worked. Distillation — training a smaller "student" model on outputs generated by a larger, more capable "teacher" model — is standard practice precisely because carefully generated synthetic data from a stronger source can transmit capability efficiently.

That is not recursive self-training in the collapse sense: the data flows downhill, from a more capable model to a less capable one, rather than a model consuming its own undifferentiated output in a closed loop. The distinction that determines whether synthetic data helps or hurts is not synthetic-versus-real; it is whether the process has a quality gradient and a mechanism — human curation, verification against ground truth, filtering for correctness — that prevents errors from accumulating unchecked across generations. Distillation has both. Naive scrape-and-retrain loops on unfiltered web text typically have neither.

The Data Wall Makes the Incentive Problem Worse

Model collapse would be a containable, well-understood risk if labs had an unlimited supply of clean human-generated text to fall back on. They do not, and that scarcity is what turns a manageable technical risk into a structural one. Epoch AI's research on data scaling, most thoroughly laid out in "Will we run out of data? Limits of LLM scaling based on human-generated data," estimates the effective stock of quality, repetition-adjusted human-generated public text available for training at roughly 300 trillion tokens.

Epoch's own projections for when that stock gets fully utilized have moved over time as understanding of multi-epoch training improved — the group's 80% confidence interval for exhaustion spans from the mid-2020s out toward 2032, with more recent revisions pushing the near end of that window toward 2028 as evidence accumulated that models tolerate several training epochs over the same data without severe degradation.

The exact date is less important than the direction: the supply of fresh, human-authored text was never going to keep pace with compute scaling, and every lab in the field knows it. The near end of that interval is no longer a forecast. It is the current year.

That scarcity creates exactly the wrong incentive at exactly the wrong moment. When human data is finite and expensive to source, curate, and license, synthetic data becomes an economically attractive way to keep scaling — not because anyone believes naive recursive training is safe, but because the alternative is slowing down. The technical fix that Gerstgrasser and colleagues identified — accumulate rather than replace, and keep the human-generated anchor intact — requires exactly the kind of disciplined data governance that a supply squeeze makes harder to maintain, since it means labs cannot simply substitute cheap synthetic tokens for expensive human ones; they have to keep both and know which is which.

The Web Itself Is the Uncontrolled Experiment

While labs debate accumulation strategies inside their own training pipelines, a much less controlled version of the same experiment is running on the open internet, which is the substrate every future model's pretraining corpus will be scraped from. Multiple independent measurement efforts — using different methodologies and arriving at different point estimates, which is itself informative about how unsettled this measurement problem is — converge on the same qualitative picture: the share of newly published web text that is AI-generated or AI-assisted rose sharply after the end of 2022 and now represents a substantial fraction of new content, with estimates for recent web pages ranging from roughly a third to a large majority depending on the detection method and the definition of "AI-generated" used.

Whatever the precise number, the direction is unambiguous, and it means that any model trained on a fresh web crawl today is training on a corpus that already contains an unknown and growing quantity of previous models' output, unlabeled and unfiltered.

That is the scenario closest to the Nature paper's actual experimental setup, except distributed across the entire industry rather than confined to one lab's training loop, and with no single actor able to enforce an accumulation discipline over data nobody controls. A lab that carefully curates its own training pipeline to avoid replacement-style collapse still has to contend with a shared, uncurated commons where that discipline does not apply.

Provenance Infrastructure Is the Real Bottleneck

The technically sound response to this problem is not to ban synthetic data — that would forfeit distillation's real benefits and would not stop anyone else from using it — but to build infrastructure that lets anyone, at scrape time, tell human-authored content apart from machine-generated content. That is the explicit goal of the Coalition for Content Provenance and Authenticity, an industry standards effort founded by Adobe, Arm, Intel, Microsoft, and Truepic that attaches a cryptographically signed manifest to a piece of content recording its origin and edit history.

Version 2.1 of the C2PA specification extended this approach by binding Content Credentials to durable digital watermarks rather than relying solely on file metadata, which matters because metadata-only provenance is trivially destroyed by re-saving or re-uploading a file, while a watermark embedded in the underlying pixel or token data is designed to survive those transformations. The published specification has since reached version 2.4. The standard has not been the bottleneck for some time.

Regulation has since caught up with the same goal from a different direction. Article 50 of the EU AI Act requires providers of systems that generate synthetic audio, image, video or text to mark their outputs as artificially generated in a machine-readable form. It stopped being a forthcoming obligation on 2 August 2026, when the remaining provisions of the Act became applicable and the Commission began enforcing the transparency rules; providers whose systems were already on the market that day have until 2 December 2026 to comply with Article 50(2).

The duty produces labels. It does not produce a filterable corpus. A European marking obligation reaches European providers, not the back catalogue of the open web, not models served from outside the Union, and not the machine-written text already sitting in every crawl taken since 2023. The law now states what a careful publisher should do; nothing in it hands a crawler a switch for excluding what a careless one published three years ago.

None of this solves the problem yet. Provenance standards only work if adoption is close to universal, and a manifest or watermark that most of the web ignores provides no protection for a crawler that cannot distinguish marked from unmarked content at scale. The gap between "a provenance standard exists" and "training corpora are reliably filterable by provenance" is currently enormous, and closing it is a coordination problem across an adversarial ecosystem of publishers, platforms, and scrapers with no single actor able to compel compliance.

What Actually Follows

The honest reading of the evidence is that model collapse is a demonstrated, mechanistically understood phenomenon that occurs under a specific and avoidable condition: recursive training that replaces rather than accumulates real data, without provenance controls to distinguish the two. It is not an inevitability baked into the mathematics of scaling, and it is not a reason to treat synthetic data as uniformly hazardous — distillation is proof that synthetic data generated under a quality gradient and human oversight remains one of the more reliable ways to transfer capability.

What makes the risk consequential rather than academic is the collision of three separate trends that are each individually manageable but jointly harder to control: a finite and increasingly exhausted stock of human-generated training text, an open web whose newly produced content is substantially machine-authored and largely unlabeled as such, and a provenance infrastructure that exists in standards documents well ahead of where it exists in actual crawl-time enforcement.

Fixing any one of those trends would reduce the risk meaningfully. Fixing none of them, while continuing to scale training runs against a shrinking and increasingly contaminated commons, is the scenario in which the Nature paper's laboratory result stops being a controlled experiment and starts describing the default trajectory of the field.