A Comprehensive Dataset for Human vs. AI Generated Text Detection

AI summary

Overview

• Research area: Natural Language Processing — AI-generated text detection and model attribution
• Technical level: Intermediate (accessible to readers familiar with basic NLP and machine learning concepts)
• Scope: The paper releases a 73,000+ sample dataset of New York Times articles paired with versions rewritten by six different large language models, and reports baseline results for two classification tasks: human-vs-AI detection and identifying which model generated a given text.

What This Paper Is About

As large language models produce text increasingly indistinguishable from human writing, there is a growing need to detect machine-generated content and identify which model produced it. Progress in this area has been limited by a shortage of large, diverse datasets containing both authentic human writing and outputs from multiple modern LLMs under controlled conditions. This paper addresses that gap by building such a dataset from real journalism and establishing baseline benchmarks on it.

Key Contributions

1. A large-scale paired dataset: Over 73,000 text samples combining authentic New York Times articles (drawn from a base collection of more than 2.1 million articles spanning January 2000 to the present) with synthetic versions generated by six state-of-the-art LLMs.

2. Multi-model coverage with rich annotation: Each sample carries a binary label (label_A) indicating human versus AI origin, and a categorical label (label_B) identifying the specific generator — Gemma-2-9B, Mistral-7B, Qwen-2-72B, LLaMA-8B, Yi-Large, GPT-4o, or human-written — plus supporting metadata.

3. Two formalized benchmark tasks: Task A (human vs. AI-generated text classification) and Task B (model attribution for AI-generated text), with defined train/validation/test splits of 51,247 / 10,983 / 10,963 samples.

4. Baseline results using a rewriting-based method: A baseline inspired by the Raidar approach, reporting 53% accuracy on Task A and 5.04% accuracy on Task B, establishing reference points for future work.

Main Findings

• Task A accuracy of 53%: The baseline distinguishes human-written from AI-generated text only slightly better than random chance, indicating the task remains genuinely difficult with current methods.

• Task B accuracy of 5.04%: Attributing a text to its specific generating model is dramatically harder — with seven possible classes (six models plus human), this result is barely above chance and underscores how stylistically similar modern LLMs are to one another.

• LLMs edit their own output less: The core insight behind the baseline is that when an LLM rewrites text it (or another LLM) originally generated, it tends to make fewer changes than when rewriting human text — and even fewer changes when rewriting its own output.

• Word-level overlap between human and AI text: Word clouds of the two groups show key terms like "new york" and "united state" prominent in both, illustrating how closely AI-generated news prose mirrors the vocabulary and topic distribution of the source articles.

• Dataset balance: Each of the six LLMs contributes roughly equal numbers of articles (approximately 10,455–10,459 each), with the human-written class matching that scale, enabling fair cross-model comparison.

Methodology in Plain English

The researchers started with a large archive of New York Times articles. For each article, they took the abstract (the short summary at the top) and fed it to six different LLMs as a prompt, producing a synthetic article for each model. They also used each article's web URL to retrieve the full original human-written story.

This produced a one-to-many structure: one human story, plus six AI-generated counterparts, all derived from the same underlying topic. Each text was then labeled twice — once for binary origin (human or AI), and once for the specific generator.

For the baseline, they used a rewriting-based detection idea: they asked a fixed model (GPT-3.5-Turbo) to rewrite each input text with a standardized instruction like "Concise this for me and keep all the information." They then measured the Levenshtein edit distance (character-level insertions and deletions) between the rewrite and the original. Human-written text typically underwent many edits; AI-written text underwent far fewer. To make a prediction, they compared the rewrite distances across candidate models and picked the one with the smallest distance as the likely generator. If all distances exceeded a threshold — set to the median of the maximum edit distance observed across training samples — the text was classified as human-written.

Why This Matters

Impact on research: Existing AI-text-detection datasets often rely on synthetic prompts, student essays, or generic web content. This dataset grounds the problem in high-quality, long-form journalistic prose and systematically compares six leading LLMs under identical prompting conditions, filling a gap that previous benchmarks left open. The low baseline scores give the field honest, non-inflated reference points.

Real-world applications:

• Newsroom verification: Publishers and wire services can use detection and attribution tools to check whether submitted or syndicated content was machine-generated.
• Misinformation and fact-checking: Platforms combating AI-generated fake news — particularly around elections and public health — benefit from reliable origin detection.
• Academic integrity: Institutions assessing student work or peer-reviewed submissions gain a benchmark for building detection systems on realistic text.
• Content recommendation and annotation: Systems could flag or label content by origin, giving users transparency about what they are reading.

Industry relevance: The findings matter directly to AI labs (model provenance and watermarking), media companies (editorial trust), social platforms (content moderation at scale), and any organization deploying LLMs where provenance of text is a compliance or reputational concern. The fact that model attribution sits near chance at 5.04% is a meaningful signal for anyone assuming "which model wrote this" is currently a solvable problem.

Future Directions

1. Extending to other languages and modalities: The authors propose broadening the dataset beyond English text into multilingual and multimodal content, echoing efforts like the M4 and Factify series.

2. Harder adversarial conditions: Using in-context learning or fine-tuning, LLMs could be trained to generate more human-like text, making detection deliberately more challenging and the benchmark more rigorous.

3. Improving detection and attribution methods: The near-chance baselines leave substantial room for more sophisticated classifiers, stylometric analysis, and hybrid approaches.

4. Cross-model generalization: An open question is whether a detector trained on one model's outputs transfers to newer, unseen models — a critical concern given how quickly generative models evolve.

Target Audience

This paper is most valuable to NLP researchers working on AI-generated text detection, text provenance, and model attribution, as well as dataset builders seeking a realistic, multi-model benchmark grounded in journalistic content. It also serves fact-checkers, misinformation researchers, and media-integrity practitioners who need a realistic testbed for verification tools, and machine learning engineers building content-authenticity pipelines. Readers should have basic familiarity with classification tasks and evaluation metrics; the paper does not require deep technical background.

Authors’ abstract

The rapid advancement of large language models (LLMs) has led to increasingly human-like AI-generated text, raising concerns about content authenticity, misinformation, and trustworthiness. Addressing the challenge of reliably detecting AI-generated text and attributing it to specific models requires large-scale, diverse, and well-annotated datasets. In this work, we present a comprehensive dataset comprising over 73,193 text samples that combine authentic New York Times articles with synthetic versions generated by multiple state-of-the-art LLMs including Gemma-2-9b, Mistral-7B, Qwen-2-72B, LLaMA-8B, Yi-Large, and GPT-4-o. The dataset provides original article abstracts as prompts, full human-authored narratives. We establish baseline results for two key tasks: distinguishing human-written from AI-generated text, achieving an accuracy of 58.35\%, and attributing AI texts to their generating models with an accuracy of 8.92\%. By bridging real-world journalistic content with modern generative models, the dataset aims to catalyze the development of robust detection and attribution methods, fostering trust and transparency in the era of generative AI. Our dataset is available at: https://huggingface.co/datasets/Rajarshi-Roy-research/Defactify_Text_Dataset