Future of AI Models: A Computational perspective on Model collapse

AI summary

Overview

Research area: Natural Language Processing and data-centric AI — specifically the phenomenon of "model collapse," in which generative models trained on synthetic data lose diversity and semantic richness across generations.

Technical level: Intermediate. The paper is written for readers comfortable with basic linear algebra notation (vectors, means, covariance, cosine similarity), but the core argument and its empirical results are explained in accessible prose.

Scope: The paper measures how semantic similarity among English Wikipedia articles in the Common Crawl corpus changed year by year from 2013 to 2025, and extrapolates that trend to estimate a date for AI model collapse.

What This Paper Is About

As AI-generated text spreads across the web, the datasets used to train the next generation of models increasingly contain text produced by earlier models. The paper studies the risk that this recursive feedback loop erodes linguistic and semantic diversity — a process called model collapse — and asks when that erosion would become severe. The authors quantify this by measuring how similar documents became over time in a controlled, single-source text corpus and fitting a curve that projects forward to a predicted collapse date.

Key Contributions

1. A temporal similarity measurement over a real web corpus. The authors compute year-wise average cosine similarity of Transformer embeddings for English-language Wikipedia articles drawn from the Common Crawl corpus, covering 2013 through 2025.

2. A formal problem setup. The paper defines corpus-level parameters for measuring collapse: pairwise document similarity (q_{i,j,k}), average within-year similarity (q_i), the mean embedding vector (mu_i), and the covariance matrix (Sigma_i).

3. An exponential saturation model with a fitted timeline. The authors fit h(y) = 0.35 + 0.0935(1 - e^{-0.1029(y-2013)}) to the observed similarity trend and derive estimated years at which the metric reaches 90%, 95%, and 99% saturation.

4. A stated prediction for model collapse. Based on that fit, the paper estimates model collapse occurring after 2035, with an approximate collapse time of 2035.

Main Findings

• Similarity rises steadily over the whole period. The average cosine similarity values plotted run across a vertical axis from 0.36 to 0.42, with the year axis marked at 2014, 2016, 2018, 2020, 2022, and 2024, showing a consistent upward trend in textual homogeneity.

• An early rise predates public LLM adoption. The paper reports that similarity increased even before widespread public LLM use, which it attributes to earlier neural architectures — Long Short-Term Memory networks, Recurrent Neural Networks, and other language models — deployed for translation, summarization, and text normalization. Because these were not adopted at large scale, the paper says their overall impact on global linguistic homogeneity remained relatively limited.

• A significant acceleration around 2019 to 2021. The paper identifies a notable increase in that window, which it ties to the transformer architecture, transformer-based language models, and the release of GPT-2 and GPT-3 during 2019 and 2021.

• Fitted saturation timeline. Using the fitted exponential function, Table I reports 90% saturation in 2035, 95% in 2042, and 99% in 2057. The conclusion text states the same milestones as 2035, 2042, and 2058 — the table and the conclusion differ by one year on the 99% figure.

• Predicted collapse year. From the saturation analysis, the paper predicts AI model collapse to occur after 2035, defining collapse as a similarity rate of 90% or greater, with the approximate time of collapse being 2035.

• Connection to prior theory. The observed pattern is described as aligning with the "Forgetting the Tails of the Distribution" effect formalized in earlier work, in which iterative training on AI-influenced data erodes rare or unique features of a data distribution.

• Noise acknowledged in the estimate. The paper attributes minor fluctuations in the similarity trend to irreducible linguistic diversity, variation in corpus size across years, and finite sampling error, rather than to underlying semantic convergence.

• Context on synthetic content prevalence. The paper cites a 2025 Ahrefs study finding 74.2% of newly published webpages contained AI-generated material, estimates that 30–40% of the active web corpus is synthetic, reports a 2025 Elon University survey finding 52% of U.S. adults regularly use LLMs such as ChatGPT for writing, coding, or research, and cites estimates that 18% of financial consumer complaint records and 24% of corporate press releases contain LLM-assisted text.

• Context on AI in creative domains. The paper cites a systematic review analyzing 723 studies on AI in fine arts that retained 44 empirical works, with over half focused on visual arts and one quarter on AI in music generation; and a meta-analysis aggregating 28 studies with 8,214 participants finding that humans augmented with generative AI outperformed unaided humans on creativity tasks (Hedges' g = 0.27) while GenAI use substantially reduced idea diversity (g = -0.86).

Methodology in Plain English

The authors start from the Common Crawl database because it provides year-wise corpora, letting them track similarity over time. Because the raw crawl mixes many languages and many kinds of websites, they filter it in two stages. First, they isolate crawl paths containing "wikipedia.com" to keep a single source with a consistent author group. Second, they run language detection with the pycld2 library to exclude non-English material, limited to the first 2000 words of each page — if those words are English, the whole document is assumed to be English.

Each remaining document is converted into a fixed-length vector using a pretrained Transformer encoder, specifically the sentence-transformer model "BAAI/bge-large-en-v1.5," which projects text into a 1024-dimensional embedding space. These embeddings are meant to capture not just meaning but tone and writing style. The authors then compute pairwise cosine similarity between embeddings using NumPy and average it within each year to get q_i. Cosine similarity ranges from 0 (no relevance) to 1 (same contextual meaning), and the paper justifies this metric by pointing to its long use in information retrieval and its correspondence to how Transformer attention itself compares query and key vectors.

Finally, the year-wise averages are fitted to an exponential saturation curve h(y) = h_0 + a(1 - e^{-b(y - y_0)}), with the baseline h_0 and start year y_0 fixed to reduce model complexity given limited historical data. The parameters a and b are estimated by gradient descent using the SciPy library, minimizing a Euclidean loss summed over the observed similarity values. The resulting fit is h(y) = 0.35 + 0.0935(1 - e^{-0.1029(y-2013)}). Saturation years are then read off by solving for the year at which the exponential term equals the target percentage. Computation ran on a machine with an Intel Ultra 9 155H processor and a CUDA-enabled Nvidia RTX 4070 graphics processor with 32 GB of memory.

Why This Matters

Impact on research. The paper offers an empirical, data-driven complement to theoretical formalizations of model collapse from prior work. Rather than arguing from first principles that recursive self-training degrades generative systems, it measures an observable proxy — rising semantic similarity in a real web corpus — and attaches a timeline to it. It also raises the question of whether current training corpora already show measurable signs of collapse, which the authors assert the evidence supports.

Real-world applications:

• Training data curation pipelines. The similarity metrics described (q_i, q_{i,j,k}) could be used to monitor homogeneity in datasets before training, flagging corpora that have drifted toward synthetic uniformity.
• Legal and scientific writing. The paper notes that LLMs now automate legal contract analysis and assist scientific research; the erosion of stylistic and semantic diversity directly affects how varied such outputs can be.
• Financial and corporate communications. With an estimated 18% of financial consumer complaint records and 24% of corporate press releases containing LLM-assisted text, homogenization in these high-stakes registers has implications for how institutions detect and interpret authentic complaints or disclosures.
• Creative industries. Given the meta-analytic finding that generative AI use substantially reduced idea diversity (g = -0.86) while boosting individual creativity scores (g = 0.27), the paper's trend speaks directly to concerns about homogenization in graphic design, architecture, product prototyping, entertainment, and creative writing.

Industry relevance. Model builders depend on a stream of diverse, human-authored data to sustain generalization. If that stream narrows, the paper argues, models with ever-larger parameter counts — it contrasts GPT-2's roughly 1.5 billion data points in its largest model with GPT-3's roughly 175 billion parameters — risk overfitting to distributions whose effective information content resembles that of a much smaller but more diverse dataset. The extrapolated 2035 horizon gives data governance and dataset acquisition strategy a concrete planning reference.

Future Directions

1. Extend to multimodal datasets. The authors explicitly call for integrating multi-modal data, noting that the current analysis covers text only while image generation has already produced more than 15 billion AI-generated images, with estimates of over 30 million new images per day.

2. Account for accelerating generation rates. The projection assumes current rates of synthetic data production; the paper warns that more capable foundation models, multimodal architectures, or other disruptive technologies could dramatically alter that rate, shifting the timeline in either direction.

3. Evaluate mitigation techniques. The paper suggests analyzing the effects of measures such as adaptive feedback between human and synthetic content generation, and developing mechanisms to preserve data diversity in large-scale AI ecosystems.

4. Refine estimates as more data accumulates. Because large-scale public deployment of generative AI began only recently, the authors describe the projection as a preliminary approximation and call for subsequent research with richer datasets — including analysis of how corpus size variation and irreducible diversity affect the measured trend.

Target Audience

This paper is most useful to AI researchers and graduate students working on data-centric AI, dataset curation, and the sustainability of training corpora; to ML engineers and data governance teams at organizations that acquire or filter web-scale training data; and to policy and strategy readers who need a quantitative handle on how quickly synthetic content could erode the value of public text. Readers without a mathematical background can follow the argument and results, but the problem formulation in Section 2 and the fitting procedure in Section 5.3 assume comfort with vector notation and exponential curve fitting.

Authors’ abstract

Artificial Intelligence, especially Large Language Models (LLMs), has transformed domains such as software engineering, journalism, creative writing, academia, and media (Naveed et al. 2025; arXiv:2307.06435). Diffusion models like Stable Diffusion generate high-quality images and videos from text. Evidence shows rapid expansion: 74.2% of newly published webpages now contain AI-generated material (Ryan Law 2025), 30-40% of the active web corpus is synthetic (Spennemann 2025; arXiv:2504.08755), 52% of U.S. adults use LLMs for writing, coding, or research (Staff 2025), and audits find AI involvement in 18% of financial complaints and 24% of press releases (Liang et al. 2025). The underlying neural architectures, including Transformers (Vaswani et al. 2023; arXiv:1706.03762), RNNs, LSTMs, GANs, and diffusion networks, depend on large, diverse, human-authored datasets (Shi & Iyengar 2019). As synthetic content dominates, recursive training risks eroding linguistic and semantic diversity, producing Model Collapse (Shumailov et al. 2024; arXiv:2307.15043; Dohmatob et al. 2024; arXiv:2402.07712). This study quantifies and forecasts collapse onset by examining year-wise semantic similarity in English-language Wikipedia (filtered Common Crawl) from 2013 to 2025 using Transformer embeddings and cosine similarity metrics. Results reveal a steady rise in similarity before public LLM adoption, likely driven by early RNN/LSTM translation and text-normalization pipelines, though modest due to a smaller scale. Observed fluctuations reflect irreducible linguistic diversity, variable corpus size across years, finite sampling error, and an exponential rise in similarity after the public adoption of LLM models. These findings provide a data-driven estimate of when recursive AI contamination may significantly threaten data richness and model generalization.