Pretraining Data, Objectives, and the Behavior They Create

A training corpus is assembled from sources with different languages, rights, quality, duplication, and social context. The mixture weights determine which patterns receive repeated gradient updates.

Every line below has been measured. A single sentence counted 60,000 times in C4. A repetition boundary found across 400 training runs. An audit that forced a five-billion-pair image corpus off the internet. A numbered article of European law with a date on which it becomes enforceable.

Each stage changes the effective distribution seen by the optimizer. Sources are acquired with their origin, license, collection method, time and intended use recorded. Content is parsed and normalized while language, code and document boundaries are preserved. Quality, safety, privacy and policy filters are applied with measured error. Repetition and known evaluation overlap are reduced. Sampling weights, curriculum and sequence packing fix the mixture. The last stage is the one teams skip.

Retaining the evidence is now a legal obligation with a template and a deadline attached. Article 53(1)(d) of the EU AI Act, Regulation (EU) 2024/1689, requires providers of general-purpose AI models to “draw up and make publicly available a sufficiently detailed summary about the content used for training of the general-purpose AI model, according to a template provided by the AI Office”. The European Commission published that Explanatory Notice and Template on 24 July 2025. The AI Office can enforce the article from 2 August 2026. Versioned manifests, filter policies, corpus statistics and removal pathways are the artefacts that answer such a request. A team that did not keep them during training cannot reconstruct them afterwards.

Conflating them makes corpus claims impossible to audit. Filtering excludes records using rules, models, metadata or human policy. It targets quality, safety, language or rights, it can remove valuable minority content, and it needs precision and recall studies because it changes the represented world. Deduplication detects exact or near-repeated content at document or passage level. Decontamination removes overlap with evaluation or protected reference sets. The last two carry numbers.

Deduplication. One 61 word English sentence appeared in C4 more than 60,000 times. Lee and colleagues found it in 2022 and built the tools that took it out. Their abstract reports the rest: “We develop two tools that allow us to deduplicate training datasets—for example removing from C4 a single 61 word English sentence that is repeated over 60,000 times. Deduplication allows us to train models that emit memorized text ten times less frequently and require fewer training steps to achieve the same or better accuracy. We can also reduce train-test overlap, which affects over 4% of the validation set of standard datasets, thus allowing for more accurate evaluation.” They also found that over 1% of unprompted model output was copied verbatim from training data. So “reduces frequency distortion” means one sentence counted more than sixty thousand times. “Can lower memorization pressure” means a tenfold drop in how often memorized text is emitted. And the evaluation cost of skipping it lands on over 4% of the validation set of standard datasets.

Decontamination. The residual has been sized too. GSM1k is 1,205 grade-school math problems, written by people to mirror GSM8k; Zhang and colleagues commissioned it in 2024 to see what a fresh benchmark would show. It showed this: “When evaluating leading open- and closed-source LLMs on GSM1k, we observe accuracy drops of up to 8%, with several families of models showing evidence of systematic overfitting across almost all model sizes.” The gap tracked exposure. A Spearman's r² of 0.36 links a model's probability of generating a GSM8k example to its GSM8k-to-GSM1k performance gap. Decontamination cannot prove the absence of all contamination. A held-out twin benchmark is how you find out what the declared decontamination missed.

Causal next-token prediction rewards predicting future tokens from prior context. The other families are equally specific procedures, not categories.

A masked objective is a rule with numbers in it. BERT hides part of the input and trains the model to put it back: “In all of our experiments, we mask 15% of all WordPiece tokens in each sequence at random.” A chosen token is replaced with [MASK] 80% of the time, with a random token 10% of the time, and left as the original token 10% of the time. Devlin and colleagues published that rule in 2019. It produced a GLUE score of 80.5, a 7.7-point absolute improvement.

An image-text contrastive objective is likewise a stated task over a stated corpus. CLIP was given one job, matching the caption to the picture, over 400 million pairs. Its abstract: “We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet.” In 2021 the resulting model matched the original ResNet-50's ImageNet accuracy zero-shot, without using any of its 1.28 million labelled training examples.

Both results show the same thing from opposite directions: an objective can produce useful general representations without matching a downstream product. Later adaptation works best when the pretraining signal preserved the information that the target task needs — and 15% masking, or caption matching over web pairs, is a decision about exactly which information that is.

Use evidence that someone outside the team can inspect. Request a source taxonomy — major domains, languages, dates and collection pathways. Inspect the mixture weights against raw corpus size, since sampling frequency and availability are different quantities. Review filter errors on sensitive slices, measuring both exclusions and inclusions. Then two steps that teams treat as formalities. Each has a published result showing what they return.

Step 4, test memorization and contamination. This audit was run against GPT-2, and 1,800 candidate sequences were inspected by hand. Carlini and colleagues report the count: “In total across all strategies, we identify 604 unique memorized training examples from among the 1,800 possible candidates, for an aggregate true positive rate of 33.5% (our best variant has a true positive rate of 67%).” Some of the recovered sequences were present in only one training document. Larger models were more vulnerable than smaller models. A canary check is not a formality. A third of the candidates it surfaced were real.

Step 5, record removal capabilities. What that costs when it was not designed in has a date attached. In December 2023 an audit of LAION-5B counted the material every filter in that pipeline had passed. David Thiel, of the Stanford Internet Observatory, opens the report: “Through this process, we identified 3,226 dataset entries of suspected CSAM, much of which was confirmed as CSAM by third parties.” LAION withdrew LAION-5B from public circulation within days; its own account dates the removal 19 December 2023. Re-LAION-5B followed on 30 August 2024, eight months later, with 2,236 matched links removed — 0.000038% of the dataset, by LAION's figure. An outside audit, a same-week takedown of a corpus already used to train deployed models, and an eight-month rebuild to delete a fraction of a percent of a percent. That is what a removal pathway costs when it is improvised.

Data and objectives create broad capabilities, limitations, styles, and risks. They also determine which later behaviors are easy or expensive to shape. The numbers in this lesson all point the same way. The decisions are cheap while the corpus is being built and expensive afterwards. Deduplicating first costs a pass over the data; not deduplicating costs a tenfold increase in memorized emission and over 4% of the validation set of standard datasets. Decontaminating first costs a matching run; not decontaminating costs up to 8 points of measured accuracy that were never there. Recording provenance first costs a manifest; not recording it cost LAION eight months between the report and Re-LAION-5B.

Post-training can redirect the model's interface. It cannot guarantee removal of every learned association. 604 confirmed memorized examples came out of a model no one had asked to recite them. The next lessons examine scale, evaluation, and the adaptation pipeline built on this foundation.