Writing in Symbiosis: Mapping Human Creative Agency in the AI Era

AI summary

Overview

Research area: AI safety and ethics / computational social science — specifically how human creative writing is changing alongside large language models.

Technical level: Intermediate. The paper uses stylometry, perplexity-based measures, HDBSCAN clustering and change-point detection, but explains each step in accessible terms and reports results as behavioral archetypes rather than model internals.

Scope (one sentence): Using a 50,728-document longitudinal corpus of Discord messages and arXiv preprints spanning January 2021 to December 2024, the paper tests whether human writing responds to LLMs as a single homogenizing trend or as structured, author-level "Dual-Track Evolution" of thematic convergence plus stylistic differentiation.

What This Paper Is About

Most prior work on LLMs and writing argues that machine-assisted text drives stylistic homogenization — a recognizable "AI style" spreading across the internet and academia, with reduced creative diversity. This paper argues that framing is too unidirectional, because it treats humans as passive recipients rather than agents making adaptive choices. The goal is to quantify, at the level of individual authors, how writing style and topic engagement changed across the pre-LLM/post-LLM boundary, and whether those changes form distinct, recognizable patterns.

Key Contributions

1. A "Dual-Track Evolution" hypothesis, tested empirically. The paper proposes that the human response to LLMs splits into two tracks: thematic convergence on AI-related topics, coupled with structured stylistic differentiation rather than uniform convergence.

2. The Creative Archetype Map. An author-level framework derived from unsupervised clustering of stylistic change vectors, yielding three emergent archetypes — Adopters, Resistors, and Pragmatists — plus a Neutral/Noise group. The authors describe this as, to their knowledge, the first systematic quantification of dual-track human-AI coevolution at author resolution across both social and formal discourse genres.

3. A perplexity-gap methodology with pre-LLM judges. GPT-2 Medium (355M parameters) trained exclusively on pre-2022 corpus data (847M tokens) serves as a "pre-LLM judge," compared against a frozen Llama-3-8B-base "current" baseline, with the difference normalized in nats-per-character and z-scored within author.

4. Evidence on detection and creative diversity. The archetype structure is used to argue that binary human-vs-machine detection is insufficient, and that a majority of authors (Resistors and Pragmatists) appear to retain non-AI stylistic signatures.

Main Findings

• Three archetypes emerge from data, not from predefined categories. HDBSCAN clustering on standardized change vectors for 2,100 authors produced: Adopters (n=370, 18%), with the highest perplexity gaps and reduced lexical diversity; Resistors (n=442, 21%), with low or negative perplexity gaps indicating text that remains challenging for both pre-2022 and current models; and Pragmatists (n=866, 41%), the largest group, showing moderate stylistic adaptation alongside high engagement with AI-related themes. A Neutral/Noise group accounted for n=422 (20%).

• Cluster quality and robustness. Silhouette 0.426 (95% CI: 0.419–0.433), robustness ARI 0.891 (95% CI: 0.884–0.898), Davies-Bouldin 1.73. Bootstrap consistency was 89% stable membership, and archetypes predicted future author text with AUC = 0.813 (95% CI: 0.798–0.828).

• Universal thematic convergence. AI-related thematic content rose significantly in both genres after November 2022. Change-point analysis placed the thematic structural break in Q1 2023.

• A dynamic, two-phase stylistic arc. Mean perplexity gaps rose 23% in social discourse and 15% in formal writing by mid-2023, then reversed — declining 18% in social discourse and 12% in formal writing below peak values. The PELT algorithm placed the stylistic break in Q2 2023, suggesting cultural attention shifted first and linguistic adaptation followed.

• Cross-validation and external validation. Clustering cross-validation accuracy 89.3% (95% CI: 87.1–91.5%); held-out arXiv accuracy 89.1% (86.8–91.4%, n=483 authors). A null model performed worse (silhouette 0.31 vs 0.43, p<0.001).

• Style survives topic controls. AI-likeness remained predictive controlling for Flesch-Kincaid grade level, type-token ratio and sentence length (partial r=0.34, p<0.001). Post hoc residualization on 100-topic transformer topic mixtures confirmed stylistic patterns persist after controlling for thematic content.

• Temporal robustness of archetype assignment. 84% consistency in archetype assignment across alternative boundary definitions, rising to 91% for extreme changes.

• Detection benchmark reported in the appendix. AI detection with AUC = 0.852 ± 0.012, Precision = 0.832, Recall = 0.791, F1 = 0.811, Brier = 0.184.

• Effect sizes. Social AI-likeness d=0.706, formal AI-likeness d=0.312, archetype ANOVA η²=0.127.

Methodology in Plain English

Corpus. The researchers assembled over 50,000 documents from two genres to capture both edited and conversational writing: informal public Discord messages (from the Discord Unveiled dataset, CC BY 4.0) and formal computer science preprints from arXiv, spanning January 2021 to December 2024. The study treats November 30, 2022 (± 7 days) as the pre/post-LLM boundary. Authors needed at least 10 posts before and after the boundary to be included.

Separating style from topic. Because subject matter naturally shifts over time, the team used topic-controlled stratified sampling for Discord (balanced quotas: Gaming 23%, Tech 31%, Social 28%, Other 18%) and monthly sampling for arXiv. They later residualized on a 100-topic transformer topic model to confirm that style effects survived after removing topic effects. AI-topic share was tracked separately with a 47-term lexicon (AI, ML, neural, transformer, GPT, ChatGPT, and similar) using TF-IDF weighting at threshold 0.23 — keeping theme measurement separate from style measurement.

The perplexity-gap measure. The core stylistic signal comes from comparing how "surprising" a text is to two models. A GPT-2 Medium model was fine-tuned only on pre-2022 data (847M tokens, 355M parameters, 41.7 hours of compute on NVIDIA A100 GPUs) so it would have no exposure to AI-generated text. That model acts as the pre-LLM judge. Llama-3-8B-base serves as the frozen modern baseline. The perplexity gap is the difference between the two models' negative log probabilities, normalized per character so that different tokenizers can be compared, then z-scored within each author. Higher values mean text that is easy for a modern model but hard for a pre-2022 model — a temporal signature consistent with LLM-era patterns.

Change vectors and clustering. For each author, the team computed a change vector across seven z-scored features: perplexity gap, type-token ratio, Flesch-Kincaid grade level, passive voice ratio, first-person pronoun frequency, punctuation density, and mean sentence length. HDBSCAN (min_cluster_size=15, min_samples=5, euclidean metric) clustered these vectors, chosen because it can find clusters of varying shapes and density and can label outliers as noise rather than forcing every author into a category. Results are visualized in 2D using change in AI-likeness against change in AI-topic share.

Controls and validation. Fixed-effects models included author and server-category fixed effects with HC3 robust standard errors and Holm-Bonferroni correction. Validation used 5-fold stratified cross-validation (70/15/15 split), 1,000 bootstrap iterations, winsorization at 2.5/97.5 percentiles, multiple imputation with m=5 chains, and three random seeds (42 main, 1337 robustness, 2024 validation). External validation used independent AI detectors and held-out arXiv authors.

Why This Matters

Impact on research. The paper pushes back on the homogenization narrative by reframing human-AI interaction as coevolution with human agency at the center. It argues that binary human-vs-machine detection is structurally inadequate: an Adopter's text can be statistically closer to AI output than a Resistor's, so archetype-aware models are needed. It also offers a reusable measurement design — pre-LLM judge plus frozen modern baseline — for separating style drift from topic drift in longitudinal writing data.

Real-world applications:

• AI detection and academic integrity tools: detectors could be calibrated to distinguish stylistic adaptation from actual machine generation, reducing false accusations against writers who have absorbed AI-era conventions.
• Publishing and editorial policy: journals and platforms could use archetype distributions to monitor whether human stylistic diversity is being preserved in their venues over time.
• Creative tool design: writers and toolmakers could design assistants that support thematic exploration while leaving personal voice intact, mirroring the Pragmatist pattern.
• Linguistic and cultural preservation: the paper raises concerns about erosion of linguistic diversity and cultural marginalization, and the framework offers a way to monitor whether specific dialects or communities drift toward machine-like registers.

Industry relevance. The findings matter to any organization deploying LLMs for content, communications or knowledge work, since the observed adaptation patterns affect how human-authored material can be identified and valued. The paper also flags a fairness risk directly relevant to deployment: misuse of this kind of analysis for authorship policing could unfairly penalize non-native speakers or cultural dialects.

Future Directions

1. Extend beyond English. The authors explicitly note the analysis is limited to English-language communities, which may bias observed adoption patterns; multilingual replication is a clear open question.

2. Validate archetypes with human participants. The archetypes and the perplexity-gap methodology are described as statistical constructs that were not validated through direct human participant studies, leaving open whether writers recognize or endorse these categories.

3. Move from observation to causation. The study is explicitly observational and analytical rather than causal, so causal mechanisms behind adoption, resistance and pragmatic adaptation remain unidentified.

4. Build archetype-aware detection. Given that a simple human-vs-machine classifier struggles when an Adopter's text is statistically closer to AI output than a Resistor's, the paper points toward detection frameworks that account for adaptation diversity rather than assuming a single human style.

Target Audience

Researchers and practitioners in AI ethics and safety, computational linguistics, stylometry and authorship attribution, and science-and-technology studies who care about how human writing is changing alongside generative models. It is also useful for AI detection developers, journal editors and academic integrity officers, and for writers, educators and policymakers thinking about creative agency, attribution and the preservation of stylistic diversity. Readers need no deep machine-learning background, though familiarity with perplexity, clustering and change-point detection helps.

Authors’ abstract

The proliferation of Large Language Models (LLMs) raises a critical question about what it means to be human when we share an increasingly symbiotic relationship with persuasive and creative machines. This paper examines patterns of human-AI coevolution in creative writing, investigating how human craft and agency are adapting alongside machine capabilities. We challenge the prevailing notion of stylistic homogenization by examining diverse patterns in longitudinal writing data. Using a large-scale corpus spanning the pre- and post-LLM era, we observe patterns suggestive of a "Dual-Track Evolution": thematic convergence around AI-related topics, coupled with structured stylistic differentiation. Our analysis reveals three emergent adaptation patterns: authors showing increased similarity to AI style, those exhibiting decreased similarity, and those maintaining stylistic stability while engaging with AI-related themes. This Creative Archetype Map illuminates how authorship is coevolving with AI, contributing to discussions about human-AI collaboration, detection challenges, and the preservation of creative diversity.