Vision Dataset Design, Annotation, and Provenance

An annotation is the final layer of several prior design choices, and a defect in a lower layer survives every improvement made above it. No annotation policy, however careful, can describe a population the sampling process never reached.

Two standard face benchmarks turned out to be lopsided at that bottom layer. Buolamwini and Gebru measured the composition and reported it in 2018: “We find that these datasets are overwhelmingly composed of lighter-skinned subjects (79.6% for IJB-A and 86.2% for Adience) and introduce a new facial analysis dataset which is balanced by gender and skin type.” On their balanced set, commercial gender classifiers erred on up to 34.7% of darker-skinned females against 0.8% for lighter-skinned males. The skew sits in the deployment-population layer and is visible there only as percentages of a benchmark's composition. It came out the other end as a forty-fold gap in error rate.

A national metrology institute reached the same place by a different route. NIST tested 189 algorithms from 99 developers on 18.27 million images of 8.49 million people. Its December 2019 report found false positive rates varying across demographic groups by factors of 10 to beyond 100. Two independent measurement efforts, one academic and one governmental, put numbers on the same layer of the stack.

Different label forms impose different cost, ambiguity, and model requirements. The choice is a contract. It fixes what the model can be asked to output, what the annotator has to decide, and which disagreements the guideline will have to resolve later.

Ambiguous instructions create systematic label noise that no optimizer can repair. The benchmarks the field trusts most are already carrying it. Ten widely used test sets were searched for mislabelled examples, and in 2021 Northcutt and colleagues reported what the search found: “Errors in test sets are numerous and widespread: we estimate an average of at least 3.3% errors across the 10 datasets, where for example label errors comprise at least 6% of the ImageNet validation set.” These are the test sets against which model rankings are declared.

The cause is frequently the ontology rather than the annotator. A re-annotation of the ImageNet validation set, published by Beyer and colleagues in 2020, found that roughly 29% of images either contain multiple objects or fall under a category matching several ImageNet synonym labels. For nearly a third of that set, a single forced label was never a description of the image. It was a choice among defensible answers, and the guideline had failed to make it in advance. Each question below is one of those choices, made once in the manual rather than silently and differently by every annotator.

Annotators may disagree because an image is low quality, the ontology is unclear, or the underlying concept is genuinely subjective. Majority vote can hide those causes.

How much disagreement survives review is itself a measurement. When the algorithmically flagged label-error candidates went to crowd workers, only 51% were confirmed erroneous. Roughly half of the suspicious labels were defended by human reviewers rather than overturned. On ImageNet validation, around 29% of images contain multiple objects or match several synonym labels. That is a structural reason for two careful annotators to differ and for both of them to be correct.

Preserve raw judgments, confidence, and adjudication outcomes where feasible. Some applications should model ambiguity or use soft targets rather than force a false single truth.

Quality control should diagnose policy, tooling, and evidence problems as well as individual mistakes. A review process that can only mark an annotator right or wrong will record ontology failures as personnel scores. Multi-object images, missing abstain options, undefined visibility rules — every one of them lands on somebody's record, and stays there.

Images from one video, patient, product unit, property, satellite pass, or camera site can be strongly correlated. Splitting individual frames at random may make recognizing the source look like task competence.

A chest radiograph turned out to be legible as a hospital. Zech and colleagues trained CNNs on 158,323 chest radiographs from three institutions, then asked the network to identify the source rather than the disease: “CNNs were able to directly detect hospital system of a radiograph for 99.95% NIH (22,050/22,062) and 99.98% MSH (8,386/8,388) radiographs.” The site was almost perfectly legible in the pixels. The consequence appeared when the boundary was moved. Pneumonia-detection AUC fell from 0.802 internally at MSH to 0.717 externally at NIH. That result was published in PLOS Medicine on 6 November 2018, and DeGrave and colleagues later found the same failure mode in COVID-19 chest radiograph models. It is not an idiosyncrasy of one hospital pair.

Choose group, temporal, geographic, device, or site boundaries that match the intended claim. Keep a final test set insulated from annotation-policy tuning.

Even a carefully rebuilt test set is contested evidence rather than settled ground. New CIFAR-10 and ImageNet test sets were constructed “by closely following the original dataset creation processes”, and the field was then measured on them: “We evaluate a broad range of models and find accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet.” Recht and colleagues read the gap as real: “the accuracy drops are not caused by adaptivity”. Not overfitting to a reused test set, then, but models failing on slightly harder images drawn the same way. An independent group tested that reading. Engstrom and colleagues showed part of the gap is an artefact of the replication procedure itself, leaving an estimated 3.6% ± 1.5% of the original 11.7% ± 1.0% ImageNet drop unaccounted for. How the boundary is drawn is not a preliminary to the measurement. It is a quantity that had to be measured in turn.

What the team writes down should let another team understand both the data and the limits of the claim — including the parts that later have to be withdrawn.

ImageNet's person subtree was audited category by category and the audit reported at FAccT 2020. Of its 2,832 categories, 1,593 were judged potentially offensive labels. Acting on that had a countable cost, and Yang and colleagues stated it in the paper: “The unsafe synsets are associated with 600,040 images in ImageNet. Removing them would leave 577,244 images in the safe synsets of the person subtree of ImageNet.” ImageNet's maintainers announced the removal of those synsets in a dated public update.

Enumerate the ontology, judge it category by category, count the affected images, publish the change with a date. That sequence is only available to a release that already recorded which categories exist, which images sit under each, and which version a published result was measured against. The artifacts below are what make a correction of that kind executable rather than merely desirable.