Privacy, Fairness, Security, and Media Provenance

A camera intended to count occupancy may also capture faces, health cues, screens, badges, behavior, and bystanders; stored embeddings can enable future linkage even when raw images are deleted.

Governance starts before training: purpose limitation, field of view, resolution, retention, access, inference scope, and acceptable secondary use.

European law now draws one of these lines in statute rather than in a design review. Article 5(1)(e) of the AI Act prohibits “the placing on the market, the putting into service for this specific purpose, or the use of AI systems that create or expand facial recognition databases through the untargeted scraping of facial images from the internet or CCTV footage”. The European Commission lists that practice among the ones it bans outright.

What the line is worth is easier to see once it has been priced. Clearview AI built a database of more than 30 billion photos scraped from the internet, and turned every face in it into a unique biometric code. The people depicted did not know, and did not consent. On 3 September 2024 the Dutch Data Protection Authority fined Clearview EUR 30.5 million, with further penalties of up to EUR 5.1 million if it fails to comply with the orders attached. “Facial recognition is a highly intrusive technology, that you cannot simply unleash on anyone in the world”, said Aleid Wolfsen, chairman of the Dutch Data Protection Authority. The underlying decision is dated 16 May 2024. Clearview did not object, so it cannot appeal.

Nothing in that database required a new sensor. Every one of the 30 billion faces was already a published image, and the inference — one biometric code per person — was added afterwards by someone the depicted person never met. That is the whole distance between what a system collects and what it can be made to know.

Controls should exist at every transition rather than only at storage. The last step is the one most often written as an intention rather than a procedure — and a court has already written it as a procedure.

Rite Aid ran face recognition in its stores. The FTC sued on 19 December 2023, and in February 2024 a federal judge signed the settlement. It bars Rite Aid for five years from deploying or using any facial recognition or analysis system in any retail store, retail pharmacy or online retail platform. It caps retention of biometric information at five years. And it puts a clock on what already exists: “Within forty-five (45) days after the effective date of this Order, delete or destroy all photos and videos of consumers used or collected in connection with the operation of a Facial Recognition or Analysis System prior to the effective date of this Order, and any data, models, or algorithms derived in whole or in part therefrom, and provide a written statement to the Commission, sworn under penalty of perjury, confirming that all such information has been deleted or destroyed;”

Read that against step 3 of the lifecycle. The order does not stop at the pictures. The data, models and algorithms derived from them fall under the same forty-five-day clock, and a named person has to swear to the result under penalty of perjury. That is what “verifiable deletion” costs when someone else defines it.

What produced the order is a lifecycle failure at the other end. The system ran from October 2012 to July 2020. The FTC alleged that it generated thousands of false-positive matches, and “was more likely to generate false positives in stores located in plurality-Black and Asian communities than in plurality-White communities”.

No single parity metric resolves all visual-product harms.

The measurements exist, and they are not close. In 2018 Buolamwini and Gebru tested three commercial gender classification systems on a benchmark balanced by gender and skin type. They reported “that darker-skinned females are the most misclassified group (with error rates of up to 34.7%)”. At the other end of the same benchmark: “The maximum error rate for lighter-skinned males is 0.8%”. One aggregate accuracy figure averages those two groups together and reports neither.

NIST ran a far larger study the following year. It put “a total of 18.27 million images of 8.49 million people through 189 mostly commercial algorithms from 99 developers”, and the December 2019 report states the result plainly: “Our main result is that false positive differentials are much larger than those related to false negatives and exist broadly, across many, but not all, algorithms tested. Across demographics, false positives rates often vary by factors of 10 to beyond 100 times. False negatives tend to be more algorithm-specific, and vary often by factors below 3.” Which error you choose to measure decides whether you see a disparity of 3 or one of 100. The same systems, the same images, two different findings.

Outcome equity is where the third column stops being a metric at all. Detroit police arrested Robert Williams in January 2020. Face recognition had flagged his driver's licence photo as a match to security video from a 2018 theft at a Shinola watch store. He was held for about thirty hours. On 28 June 2024 the parties settled Williams v. City of Detroit. Detroit agreed to pay $300,000. It also agreed to policies: no arrest on a face-recognition result alone, no photo lineup drawn solely from a face-recognition lead, training on the technology's higher misidentification rates for people of colour, and an audit of every case since 2017 in which face recognition produced an investigative lead followed by an arrest or arrest warrant. The court keeps jurisdiction to enforce the agreement for four years. Phil Mayor, senior staff attorney at the ACLU of Michigan, put it this way in the press release announcing the settlement: “Police reliance on shoddy technology merely creates shoddy investigations.”

Notice what the remedy is made of. Not a recalibrated threshold: sign-off rules, lineup rules, training, a retrospective audit back to 2017, and four years of court supervision. The error was in the model; the harm and the fix were both in the workflow around it.

Threats can alter inputs, training, software, or decisions. Two of the seven below have been measured in public, and the measurements are the reason to take the other five seriously.

Start with physical evasion. In 2018 Eykholt and colleagues stuck black and white stickers on a real stop sign, then filmed it from a moving vehicle. Their abstract reports both numbers: “With a perturbation in the form of only black and white stickers, we attack a real stop sign, causing targeted misclassification in 100% of the images obtained in lab settings, and in 84.8% of the captured video frames obtained on a moving vehicle (field test) for the target classifier.” They call the attack algorithm Robust Physical Perturbations (RP2). The two percentages are the same attack under two conditions, a lab number and a field number, and only one of them is the deployment number. NIST's adversarial machine learning taxonomy files the work under “physically realizable attacks”.

Then poisoning, which turns out to be a supply-chain problem wearing a data-quality costume. Dataset publishers hand out lists of URLs rather than images. NIST's taxonomy spells out what follows: “dataset publishers may provide a list of URLs to constitute a training dataset, and attackers may be able to purchase some of the domains that serve those URLs and replace the site content with their own malicious content”. Carlini and colleagues priced the attack in 2023: “By exploiting specific invalid trust assumptions, we show how we could have poisoned 0.01% of the LAION-400M or COYO-700M datasets for just $60 USD.” Ten popular datasets were vulnerable at the time. NIST's recommended defence is unglamorous — publish cryptographic hashes so downloaders can verify what they downloaded.

Bullets five and seven are therefore one attack seen from two ends: the poisoned example arrives because a dependency in the supply chain, an expired domain, changed hands for the cost of a dinner.

Security tests should reflect realistic capabilities and consequences.

Step 2 lists query budget beside equipment and physical proximity, which reads like padding until someone spends the budget. In 2016 Tramèr and colleagues attacked two live commercial services through nothing but their prediction APIs: “Given these practices, we show simple, efficient attacks that extract target ML models with near-perfect fidelity for popular model classes including logistic regression, neural networks, and decision trees. We demonstrate these attacks against the online services of BigML and Amazon Machine Learning.” Suppressing confidence values does not close the attack.

NIST records the same result in its section on model extraction: “The first model stealing attacks were shown by Tramer at al. [376] on several online ML services for different ML models, including logistic regression, decision trees, and neural networks”. It classes model extraction as a stepping stone to stronger white-box attacks. Steal a copy, then attack the copy at leisure with full knowledge of its weights.

That is what turns step 4's rate limits from a formality into a control. An adversary who can only ask questions is still an adversary. The number of answers you are willing to give is part of the attack surface, and it is one of the few parts a product team sets directly.

The product promises energy savings by estimating room use without identifying individuals. Every item below has a precedent earlier in this lesson: a scraped database nobody consented to, a forty-five-day deletion clock that reached the derived models, a false positive that cost thirty hours of detention, and a support interface that emulated user sessions.