Fine-Tuning Data Curation and Release Design

Fine-tuning data does more than teach a task. It teaches which requests receive detailed help, when the model asks a question, how it refuses, and which writing habits become the default.

So a dataset needs the same scrutiny as a product specification. Repetition and mixture weights can amplify an apparently minor example into a broad behavior change. That amplification has been measured. A 2022 deduplication study opens its abstract with the result rather than the method: “We develop two tools that allow us to deduplicate training datasets—for example removing from C4 a single 61 word English sentence that is repeated over 60,000 times.” Lee and colleagues wrote it.

No curator ever decided that one sentence deserved that much of the corpus. It arrived that way, and the optimizer read it as emphasis. The same paper reports that over 1% of the unprompted output of models trained on such data is copied verbatim from the training data. Deduplication makes models emit memorized text ten times less frequently. Train-test overlap affects over 4% of the validation set of standard datasets. Frequency was never neutral. It was a weight nobody had written down.

The dataset should be traceable from objective through release evaluation. A behavior contract fixes target actions, formats, boundaries and escalation. A coverage taxonomy spans tasks, domains, languages, difficulty, risk and ambiguity. Example provenance records source, author, policy version, generation method and review. Mixture and weighting decisions cover frequency, oversampling, curriculum and per-token loss. A protected evaluation holds deduplicated test cases, counter-metrics and broad regressions.

The provenance layer is not an invention of this lesson. Seven researchers proposed a standard document for it, borrowing the idea from the sheets that ship with electronic components: “By analogy, we propose that every dataset be accompanied with a datasheet that documents its motivation, composition, collection process, recommended uses, and so on.” The proposal is by Gebru, Morgenstern and five colleagues, and Communications of the ACM published it in 2021.

How far ordinary practice sits from that standard has also been measured. An audit of more than 1,800 text finetuning datasets, published in Nature Machine Intelligence in 2024, reports the state of the shelves those datasets are taken from: “We also observe frequent miscategorization of licenses on widely used dataset hosting sites, with license omission of 70%+ and error rates of 50%+.” Longpre and Mahari ran it with a larger team. A team that copies provenance from a hosting page is copying a field that is missing more often than not, and wrong more often than not when it is present.

For high-risk systems the top and bottom layers of this stack stopped being advice. Article 10 of the EU Artificial Intelligence Act, Regulation (EU) 2024/1689, makes them binding. Article 10(2) enumerates eight mandatory practices, among them data collection processes and the origin of data, data-preparation operations such as annotation, labelling, cleaning and aggregation, examination for bias, and identification of data gaps. Article 10(3) sets the bar for the sets themselves: “Training, validation and testing data sets shall be relevant, sufficiently representative, and to the best extent possible, free of errors and complete in view of the intended purpose.”

These defects can survive a low training loss. Each one below has been measured in a corpus or a benchmark that many teams used for years without noticing.

A small source can dominate if it is repeated or assigned greater loss weight. The 61 word sentence removed from C4 is the demonstration: one string, over 60,000 appearances, an effective weight no design document ever authorized.

A natural-frequency mixture samples examples according to observed volume. It reflects common traffic and preserves the existing product bias. It can ignore rare severe cases, and it needs explicit slice protection to stop the tail from disappearing.

A coverage-balanced mixture oversamples underrepresented tasks, languages, or risk cases. It improves exposure to important slices, but it changes effective prevalence and can distort default behavior. It requires calibrated evaluation against both the target slices and natural traffic.

A curriculum or staged mixture changes example composition across training phases. It can stabilize difficult adaptation, but it adds schedule and attribution complexity and may forget earlier behavior. It needs checkpoint comparisons to show what each phase actually did.

In all three cases the number worth recording is not how many examples a source contributed. It is how many times the optimizer saw them.

Preserve a clean chain from source examples to the approved checkpoint. Seven steps, and five of them now have a published measurement behind them.

1. Write the target taxonomy. Define behaviors, counter-behaviors, and important slices before acquisition. Coverage should be a specification, not an accident of what was easy to collect.

2. Acquire with provenance. Record expert, log, synthetic, and policy-derived sources. This is the datasheet Gebru and colleagues asked for. It is also the step the audit of more than 1,800 text finetuning datasets found omitted more than 70% of the time on widely used hosting sites, and wrong more than 50% of the time when it was present. Provenance copied from a download page is not provenance.

3. Normalize and deduplicate. Remove templates and cross-split near-duplicates. The payoff has been measured: memorized text emitted ten times less frequently, and the removal of pathologies on the scale of a single 61 word English sentence repeated over 60,000 times.

4. Review contradictions. Adjudicate policy versions, ambiguity, and acceptable alternatives. Budget for it using the rate found in the most scrutinized test sets in machine learning: at least 3.3% label errors on average across 10 datasets, at least 6% in the ImageNet validation set, with 51% of flagged candidates confirmed by human validation. A freshly assembled tuning set is unlikely to be cleaner than ImageNet.

5. Set mixture weights. Document why rare tasks or risks receive additional exposure. Record the effective frequency separately from the raw counts.

6. Lock protected tests. Prevent training, prompt, and selection leakage into release evaluation. Contamination inside C4.EN reached 24.88% of the target texts of one generation benchmark, and 53.6% input contamination for one GLUE field. Dodge and colleagues measured both, which is why this cannot be assumed. Article 10(2) of the EU AI Act turns the same discipline — origin of data, cleaning and aggregation, examination for bias, identification of data gaps — into a statutory checklist for high-risk systems.

7. Compare three systems. Base, prompt/context baseline, and tuned candidate under equal conditions. That comparison has been run head to head: “In this paper, we rigorously compare few-shot ICL and PEFT and demonstrate that the latter offers better accuracy as well as dramatically lower computational costs.” Liu and Tam reported it with five colleagues in 2022. Their (IA)^3 method and the T-Few recipe built on T0 outperformed the state of the art on the RAFT benchmark by 6% absolute, the first super-human result on it. The comparison is worth running because it can come out either way. Only a matched run tells you which.

A disciplined dataset release gives a visible reason for the tuned model’s behavior, and a path to correct it. An untracked mixture turns model weights into an opaque archive of policy decisions. The difference is auditable in a way the weights are not. A 61 word sentence repeated over 60,000 times. A benchmark 24.88% of whose targets are already in the corpus. A 3.3% floor on label errors. A licence field omitted more than 70% of the time. Each is a fact about a dataset that a datasheet would have recorded and a checkpoint alone never will.

For high-risk systems, Article 10(3) of the EU AI Act now requires those sets to be “relevant, sufficiently representative, and to the best extent possible, free of errors and complete in view of the intended purpose”. That is a claim someone has to be able to substantiate from documents.

The next lesson covers parameter-efficient adaptation — the family Liu and colleagues compared against in-context learning in 2022. Smaller trainable modules reduce update cost. They do not reduce the need for data governance and evaluation. (IA)^3 changed how many parameters move, not what the examples were teaching them.