Corpus Design: Domains, Documents, Rights, and Sampling

The unit should match the context required by the decision and the boundary used for splitting. The unit is also what a split can be grouped by. Choosing it decides, in advance, which kinds of leakage are even detectable later.

No single sample supports every claim about system behavior. A traffic sample describes who turned up, not who exists. The 250,000 responses a week that missed US first-dose vaccine uptake by 17 percentage points in May 2021 were a very large record of who was willing to answer on one platform. No additional week of them would have closed the gap — and the 1,000-response panel did not have that gap to close.

A corpus record should make two things inspectable: how the text was acquired, and what obligations travel with it.

Writing those answers down is a standing professional practice, not a courtesy. That was the argument of the data-statements paper, published in Transactions of the Association for Computational Linguistics in 2018. Bender and Friedman offered data statements as “a design solution and professional practice for natural language processing technologists”. Their case is half ethical and half arithmetic. A claim that a result generalises is a claim about a population. A dataset whose population was never described cannot support one. The practice, they argue, “will help alleviate issues related to exclusion and bias in language technology, lead to better precision in claims about how natural language processing research can generalize and thus better engineering results” — which is to say, better engineering.

Six years later a legislature wrote much the same checklist into law. The EU's AI Act, Regulation (EU) 2024/1689, was published on 12 July 2024. Its Article 10, “Data and data governance”, governs high-risk AI. Paragraph 3 reads: “Training, validation and testing data sets shall be relevant, sufficiently representative, and to the best extent possible, free of errors and complete in view of the intended purpose.” For systems in scope, the questions below stopped being professional courtesy and became compliance items.

A test document can appear in training through quotes, mirrored pages, paraphrases, templates, translations, or shared conversation fragments. Repeated authors and organizations can also leak distinctive style across random splits.

The size of the problem has been measured. Over 1% of the unprompted output of language models trained on standard datasets is copied verbatim from the training data. That is Lee and co-authors, in 2022. They also found a single English sentence of 61 words sitting in C4 over 60,000 times, until they removed it. Then the finding that reaches evaluation directly: “We can also reduce train-test overlap, which affects over 4% of the validation set of standard datasets, thus allowing for more accurate evaluation.” Models retrained on the deduplicated data emitted memorized text ten times less frequently.

Duplication does not raise the risk in proportion to itself. Kandpal and co-authors, working separately in 2022, found the relationship steeply superlinear on their own measurements: “a sequence that is present 10 times in the training data is on average generated ~1000x more often than a sequence that is present only once”. Ten copies of a test item are not ten times the exposure. They are roughly a thousand.

Use source-aware deduplication, group splits, time boundaries, and semantic overlap inspection. Keep protected challenge sets outside routine tuning and prompt development.

Write the plan before bulk collection, so that convenience does not silently become methodology.

What happens without a plan is on the record. In 2021 Dodge and co-authors set out to provide “some of the first documentation for the Colossal Clean Crawled Corpus (C4; Raffel et al., 2020), a dataset created by applying a set of filters to a single snapshot of Common Crawl”. That sentence already contains the problem. Some of the first — for a corpus whose English variant is 305 GB of compressed JSON holding 364,868,892 training documents and 156 billion tokens, released with only minimal documentation.

When they read it, they found “a significant amount of text from unexpected sources like patents and US military websites”. They found “machine-generated text (e.g., from machine translation systems) and evaluation examples from other benchmark NLP datasets”. And they found that “blocklist filtering disproportionately removes text from and about minority individuals”. Every one of those is a property of the sampling. None of them is visible from the size.

Whoever documents a corpus should name collection procedures, annotation processes, recommended uses, excluded uses, and known gaps. It should also report what cannot be measured because demographic or provenance information is unavailable.

That form has an origin and a name. Gebru and co-authors wrote it down in 2018 as one sentence: “By analogy, we propose that every dataset be accompanied with a datasheet that documents its motivation, composition, collection process, recommended uses, and so on.” The paper is called Datasheets for Datasets, and it reached Communications of the ACM in 2021.

It did not stay a paper. NIST's AI Risk Management Framework Playbook lists “Datasheets for Datasets” as an AI Transparency Resource under more than a dozen subcategories, and cites the Communications of the ACM article in its references, alongside the data statements guide. And Article 10(2)(h) of the AI Act requires of high-risk systems “the identification of relevant data gaps or shortcomings that prevent compliance with this Regulation” — absence, written down, as an obligation rather than a confession.

Silence is not neutrality. If a corpus lacks certain dialects, genres, or consent records, future teams need that limitation before interpreting model behavior.