Dataset Documentation, Governance, and Use Review

Different artifacts answer different questions about the same release, and the families they have to cover are not a matter of house style. The canonical form is the datasheet, proposed in 2018 by Gebru and colleagues and published in Communications of the ACM three years later: “By analogy, we propose that every dataset be accompanied with a datasheet that documents its motivation, composition, collection process, recommended uses, and so on.”

That proposal has an operational descendant inside a company. Microsoft's Aether Data Documentation Template, drafted in 2022, organises the same material under eight headings — Dataset Contents; Intended & Inappropriate Uses; Data Collection Procedures; Representativeness; Data Quality; Pre-processing, Cleaning, and Labeling; Privacy; and Additional Details on Distribution & Access. Its question 8 is literally “What are some tasks/purposes that this dataset is not appropriate for?” One is an academic proposal, the other a working internal template. They converge on the same five families of evidence, and both turn inappropriate use from a caveat somebody might add into a numbered question somebody has to answer.

Purpose and decision context — motivation, intended users, prediction or analysis task, and unacceptable uses. That is the datasheet's motivation and recommended uses, and the template's Intended & Inappropriate Uses with its question 8.

Composition and coverage — population, units, time range, geography, modalities, exclusions, and missing groups. Composition in the datasheet; Dataset Contents and Representativeness in the Aether headings.

Creation process — sources, consent, instrumentation, labeling, cleaning, transformation, and sampling. The datasheet's collection process; Data Collection Procedures plus Pre-processing, Cleaning, and Labeling in the template.

Quality and limitations — known errors, uncertainty, slice coverage, shift risks, and validation evidence. The Aether heading is Data Quality. It is also the heading whose contents a reviewer can most easily go and recompute.

Operations and governance — owner, access, retention, versioning, incident history, updates, and retirement. Privacy together with Additional Details on Distribution & Access, and the family that decays fastest once the release stops being maintained.

A speech dataset collected from call-center agents is proposed for a public accessibility product. Work this one as a constructed exercise rather than a reported case, and check each line against the evidence families above. Every fact a reviewer needs is a documentation field that either exists in the release or does not.

A strong governance process connects all three without confusing them.

Dataset documentation explains origin, composition, process, limitations, and maintenance. It supports discovery and makes assumptions visible. It needs an owner and a version. It does not grant permission to anybody for anything.

Use review determines whether a proposed use is allowed and appropriate. It weighs purpose and consequence, checks rights and population fit, can impose controls, and produces an accountable decision attached to a named body.

Ongoing monitoring checks whether the approved use stays within its conditions. It tracks drift and incidents, detects scope expansion, reviews retention and access, and can trigger reapproval or retirement.

The three fail in different ways, which is the practical reason to keep them apart. Documentation fails by going stale. Approval fails by being granted in the abstract. Monitoring fails by watching the dataset rather than the use.

Review the proposed use against the release evidence rather than approving the dataset in the abstract. A working example exists, and it is documented. UK Biobank holds a cohort of over 500,000 participants recruited between 2006 and 2010, opened to outside researchers through an online access process launched in April 2012. Sudlow and colleagues described how it runs in PLOS Medicine in 2015. Data-only applications are approved if the research is in the public interest. Applications that use depletable samples or require participant re-contact go through more rigorous scientific review. “Following initial assessment by the executive team, all applications are assessed and either approved or rejected (with right of appeal) by an independent Access Subcommittee.” Researchers receive only de-identified data and must sign a material transfer agreement not to re-identify participants and to use the data only for the approved research. In the first two years over 1,000 researchers registered and over 200 applications were submitted; roughly 85% were for data only. The HMA-EMA Catalogues of real-world data sources record the same arrangement from the regulator's side: biospecimen access requires a “Request to be made in the research application for approval”.

1. Name the proposed decision. Describe user, action, consequence, geography, and operating environment. UK Biobank's two-tier rule is keyed to exactly this: what the applicant intends to consume determines how hard the scrutiny is.

2. Compare populations. Assess coverage, exclusions, proxies, missing groups, and expected shift. A cohort recruited 2006-2010 answers some questions and silently fails others.

3. Check rights and purpose. Verify consent, license, contractual limits, retention, and deletion duties. The material transfer agreement is where those duties become enforceable rather than aspirational.

4. Review quality evidence. Inspect labels, slices, unresolved errors, provenance, and incident history. Recompute enough of it to know the narrative matches the artifact.

5. Set conditions. Define access, monitoring, human review, documentation, and expiry requirements. De-identification, a non-re-identification covenant, and approval scoped to one stated research question are three separate controls, not one.

6. Record the decision. Publish approver, rationale, dissent, review date, and triggers for reconsideration. A named Access Subcommittee with a right of appeal is what makes a decision reviewable later. An unnamed approver is accountable to nobody.

Datasets become obsolete when their population changes, rights expire, source systems are replaced, labels lose meaning, or a better release supersedes them. Leaving old versions discoverable without warnings invites accidental reuse.

Mark deprecated releases, identify active consumers, set migration dates, preserve required evidence, revoke access, and apply deletion policy to derived artifacts. Retirement should also update feature registries and training recipes.

Retirement rarely works by announcement alone, and the failure has been counted. Three Princeton researchers went through nearly 1,000 papers citing Labeled Faces in the Wild, MS-Celeb-1M and DukeMTMC. MS-Celeb-1M alone had 1,404 citing papers, 8 derivative datasets and 21 GitHub repositories of pre-trained models. The original licence permitted only non-commercial research use; 3 of those 21 repositories carried the same designation. MS-Celeb-1M and DukeMTMC were retracted in April 2019 and were still in use in 2020, largely through derivatives. Peng and colleagues state it flatly: “Our analysis shows that even after DukeMTMC and MS-Celeb-1M were retracted, their underlying data remained widely available and continued to be used in research papers.” The independent Exposing.ai investigation by Adam Harvey and Jules LaPlace documents the same pattern from the other side: “Despite the recent termination of the msceleb.org website, the dataset still exists in several repositories on GitHub, the hard drives of countless researchers, and will likely continue to be used in research projects around the world.” Revoking access is not revoking use. A step that says "identify active consumers" has 21 repositories and 8 derivative datasets to identify, for one dataset.

Deletion policy for derived artifacts is not a drafting preference either. It has been ordered. In 2021 the US Federal Trade Commission settled with the photo app developer Everalbum and required it to delete "(A) photos and videos of Ever app Users who requested deactivation of their accounts, (B) face recognition data that it created without obtaining Users' affirmative express consent, and (C) models and algorithms it developed in whole or in part using images from Users' photos." The third item is the one that mattered: an early and unusually explicit case of a regulator ordering the deletion of models built from improperly collected data, not just the data. Commissioner Rohit Chopra put the principle plainly: “First, the FTC's proposed order requires Everalbum to forfeit the fruits of its deception. Specifically, the company must delete the facial recognition technologies enhanced by any improperly obtained photos.” Everalbum's enterprise brand was Paravision. The Commission finalised the settlement on 7 May 2021 by a 4-0 vote, and the order runs for 20 years. A trained model is a derived artifact, and “models and algorithms it developed in whole or in part using images from Users' photos” is what a retirement plan has to be able to enumerate before anyone asks it to.

Where exact deletion is impossible, document residual copies, legal basis, access restrictions, and the limits of future reconstruction.

A consumer needs to know why a dataset exists, how it was collected, who is represented, which transformations were applied, and where it should not be used. Code and schema cannot answer those questions alone.

A datasheet or dataset card creates a structured conversation between creators, maintainers, reviewers, and downstream teams. Its value comes from evidence and ownership, not from the fact that somebody filled one in.

Language data got its own version of the artifact. Bender and Friedman proposed data statements in 2018: “In this paper, we propose data statements as a design solution and professional practice for natural language processing technologists, in both research and development.” The schema named curation rationale, language variety, speaker and annotator demographics, and speech situation and text characteristics.

What happened to that schema afterwards is the interesting part. It was revised after a 2020 workshop and republished in 2021 as Version 2, by the original authors joined by Angelina McMillan-Major. It now carries 15 numbered elements: Header, Executive Summary, Curation Rationale, Documentation for Source Datasets, Language Varieties, Speaker Demographic, Annotator Demographic, Speech Situation and Text Characteristics, Preprocessing and Data Formatting, Capture Quality, Limitations, Metadata, Disclosure and Ethical Review, Other, and Glossary. A documentation standard that was itself revised, versioned and republished three years after its proposal is the strongest available argument that the documents written against it are not finished objects either.

Documentation should therefore evolve with each release and link to executable artifacts such as contracts, quality reports, lineage, and approved access policies.