Building a Dataset That Represents the Job

A dataset is interpretable only when the team distinguishes who or what it wants to serve from what its collection process can actually observe. Target population, sampling frame and observed sample are three different things. Only the third one is in the file.

The gap between them does not stay hidden. It is legible afterwards, in the finished product. NIST's Face Recognition Vendor Test ran 189 mostly commercial algorithms from 99 developers over 18.27 million images of 8.49 million people, and reported in December 2019. False positive rates varied across demographic groups by factors of 10 to beyond 100. The direction of the error tracked where the system had been built: “A number of algorithms developed in China give low false positive rates on East Asian faces, and sometimes these are lower than those with Caucasian faces.”

None of those 189 algorithms shipped with a description of its sampling frame. At 8.49 million people, the frame described itself anyway — in which faces each system confused with which.

Sampling strategies are tools, not universal guarantees. A selection rule can do measurable damage while every person involved is being careful and publishing their method.

In 2019 the CIFAR-10 and ImageNet test sets were rebuilt from scratch, by re-running the original collection processes. Recht and colleagues report what happened next: “We evaluate a broad range of models and find accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet.” The obvious reading was that years of models had been quietly overfitted to the old test sets.

Then the replication itself was examined. Engstrom and colleagues showed in 2020 that the re-collection's own selection rule had introduced statistical bias. Once that was corrected, about 3.6% of the original 11.7% ImageNet drop remained unexplained. Roughly two thirds of a headline result about models turned out to be a property of how the second dataset had been assembled. The mechanism that decides which examples enter is not a neutral pipe. It is part of the measurement.

A coverage matrix makes broad claims testable. Its two hard steps — compare the sample against deployment, then price the empty cells — have both been carried out in public, with the numbers printed.

Step 3, done for real: in 2019 DeVries and colleagues scored six public object-recognition systems on the Dollar Street dataset, 135 classes photographed in 264 homes across 54 countries. “In particular, the accuracy of the systems is approximately 15% (absolute) higher on household items photographed in the United States than it is on household items photographed in Somalia or Burkina Faso.” Accuracy also ran about 10 points lower for households under US$50 a month than for those over US$3,500. Soap is soap. The systems disagreed by 15 points depending on whose bathroom it was in.

Step 4 has a regulator's version, with a minimum written into the cell. The FDA's 2013 guidance on pulse oximeters, issued 4 March 2013, set the calibration sample supporting clearance at 10 or more healthy subjects, and specified: “Your study should have subjects with a range of skin pigmentations, including at least 2 darkly pigmented subjects or 15% of your subject pool, whichever is larger.” Two out of ten counts as a filled cell.

Then the filled cell met patients. Wong and colleagues examined 87,971 patients in 215 hospitals and 382 ICUs, and published in JAMA Network Open on 3 November 2021. They found hidden hypoxemia — arterial oxygen low while the oximeter read acceptable — in 6.9% of Black patients against 4.9% of White patients, associated with mortality and later organ dysfunction. A cell can be occupied and still be empty of evidence.

Step 5 also has a worked instance. The response to the geographic hole was to go and collect the missing rows. A curated, fully labelled Dollar Street of 38,479 images was published in 2022, from homes with incomes as low as $26.99 per month, built precisely because existing datasets under-represent them.

Exact duplicates may arise from retries, copied records, mirrored files, or repeated imports. Near-duplicates include frames from the same video, templated messages, slightly cropped images, or multiple windows around one event.

Duplicates can dominate training. They can cross partition boundaries. Deduplication should use domain-aware identity and similarity checks, while preserving legitimate repeated events when recurrence itself matters.

Two of the most-used benchmarks in computer vision were partly testing models on pictures those models had already seen. Barz and Denzler published the count in the Journal of Imaging in 2020. Of CIFAR-10 and CIFAR-100 they write: “We find that 3.3% and 10% of the images from the test sets of these datasets have duplicates in the training set.” They then rebuilt the test sets. The fresh images came from the same domain. The training sets were left untouched, so pre-trained models stayed valid. They re-scored a range of standard architectures. The result was “a significant drop in classification accuracy of between 9% and 14% relative to the original performance on the duplicate-free test set”. Years of published comparisons had been decided partly on images the models had already been trained on.

People who cannot access a service, devices that fail before logging, and cases resolved outside the platform may never appear. Their absence can be a direct consequence of product design or institutional policy.

Sometimes the rule that removes them is written down, and someone measures what it removed. C4, the 750GB corpus behind T5, was cleaned by discarding “any page that contained any word” on a public list of bad words. Raffel and colleagues set the heuristic out in 2020, in the Journal of Machine Learning Research.

A year later somebody measured whose English that one line deleted. Dodge and colleagues, in 2021: “Using the most likely dialect of a document, we find that AAE and Hispanic-aligned English are removed at substantially higher rates (42% and 32%, respectively) than WAE and other English (6.2% and 7.2%, respectively).” Nearly half of the African American English, against roughly one document in sixteen of the White-aligned English. The filter was not even accurate on its own terms. Of a 100,000-document sample of what it excluded, only 31% was sexual in nature.

Nobody wrote a rule against a dialect. Somebody wrote a word list, and the word list did it. Dataset review should therefore ask who cannot generate a record, who declines participation, which failures stop telemetry, which filters ran before you saw the file, and which outcomes remain unobserved. These questions often require domain and user research. SQL alone is not enough.

A short dataset record can prevent hidden assumptions from spreading between teams.

The practice has a citable origin. Gebru and colleagues proposed “datasheets for datasets” in 2018, and the proposal was published in Communications of the ACM in 2021. The analogy is exact and deliberate. In the electronics industry every component ships with a datasheet describing “its operating characteristics, test results, recommended uses, and other information”. They propose that every dataset be accompanied by one recording “its motivation, composition, collection process, recommended uses, and so on”.

Almost every failure in this lesson would have been a line in such a record. The 79.6% in a composition field. The bad-words list in a processing field. The two darkly pigmented subjects out of ten in a collection field. The list below is that proposal at working length.