Responsible AI
Privacy Threat Modeling for AI Systems
Map privacy assets, actors, data flows, attacks, unintended inference, and controls across AI development and deployment.
By the end you can
- Explain why AI privacy threat modeling covers raw and inferred data, actors, flows, purposes, attacks, harms, retention, and the full lifecycle
- Distinguish Data inventory, Privacy threat model, and Security threat model
- Identify evidence that connects data subjects and bystanders to harms and rights
- Design a review that moves from identify people and data to select and verify controls
Example
The counselling service that sent the intake answers to four ad platforms
People filling in an intake health questionnaire for an online counselling service were describing themselves to a counsellor. Their answers went to Facebook, Snapchat, Criteo and Pinterest, together with their email addresses and IP addresses. They had been promised the information would stay private.
On 2 March 2023 the Federal Trade Commission proposed an order against BetterHelp, Inc. It required $7.8 million and banned the company from sharing consumers' health data for advertising. The complaint runs to eight counts under section 5(a) of the FTC Act.
Nothing in that record is a break-in. The data went where the company sent it. A threat model that only looks for attackers finds nothing here.
- Direct data: Users gave BetterHelp their email addresses, their IP addresses and their answers to an intake health questionnaire.
- Third parties: Those email addresses, IP addresses and intake answers were disclosed to Facebook, Snapchat, Criteo and Pinterest — four advertising platforms, none of them the counsellor.
- Derived data: Beyond what a person types, an AI system infers health status, routines, risk and preferences. Those inferences are personal data that nobody ever entered in a field.
- Operational logs: Prompts, outputs, traces and tool calls enter observability systems, where retention is set by engineering practice rather than by the privacy notice.
- Expectation gap: The FTC priced the distance between what was promised and what was shared at $7.8 million plus a ban on disclosing health data for advertising. Deleting a visible history does not reach every copy, embedding, profile or backup.
Key idea
The promise was about the counselling
“We do not store the chat” is not a complete privacy claim. Training retention, safety logs, backups, embeddings, analytics, support access and derived profiles may preserve personal information elsewhere.
BetterHelp's promise was narrower still, and the FTC quoted where it was made: “At several points in the signup process, BetterHelp promised consumers that it would not use or disclose their personal health data except for limited purposes, such as to provide counseling services.” The sentence names one purpose. The disclosure the FTC charged was for another one — targeted advertising, at Facebook, Snapchat, Criteo and Pinterest, using the same intake answers.
Data-flow maps are imperfect when vendors, dynamic tools or user-created integrations change. Contracts, technical discovery, telemetry and periodic verification should challenge declared flows rather than transcribe them.
A promise that is literally true of one purpose can be false of the system: BetterHelp's held for the counselling relationship while the same intake answers went to four ad platforms.
Case
The deletion drill that never visits the model
Two attacks give the threat model its concrete shape. The first is membership inference. Give it a record and a trained model, and it recovers whether that particular record had been part of the training set. Shokri and colleagues demonstrated it against machine learning models in 2017.
A deletion drill runs across tables, logs, caches and backups. It never visits the model. The record is gone from every store the team can list, and the trained model can still answer the question its owner wanted closed.
Case
1,800 candidates from GPT-2, over 600 verbatim
The second attack pulled the training data itself back out of a released language model. Carlini and colleagues ran it against GPT-2 and reported the count in 2020: “we generate 1,800 candidate memorized samples … and find that over 600 of them are verbatim samples from the GPT-2 training data”.
At least a third of the queue was real training text. The record gives a floor rather than a rate: the remaining 1,200 candidates were left unconfirmed, not cleared. Some of the confirmed text was personally identifiable — names, phone numbers and email addresses.
Each of those sequences appeared in just one training document. That removes duplication as the explanation and leaves memorisation itself. The released weights are a copy of some of the data.
Figure
Threats against stages, and the cells nobody owns
Privacy threat modeling does its work before anyone chooses a control. It identifies personal data, inferred attributes, actors and purposes. Then it identifies data flows, attack surfaces, retention and possible harms. AI can create sensitive information even when users never provide a labeled field.
The threats to look for: unauthorized access, unintended reuse, overcollection, linkage, reidentification, memorization, inference, extraction, surveillance, manipulation and loss of control. The stages to look across: training, evaluation, deployment, telemetry, support, incident response and retirement.
Put the two lists on a grid. Each threat meets each stage, and the empty cells are the ones nobody owns.
BetterHelp's intake answers left at the advertising stage. Amazon's Alexa transcripts survived at the deletion stage. Neither is an attacker pathway. A threat list scoped to the counselling session, or to the voice recording, leaves both cells blank.
Each empty cell in that grid names a stage where no one has been asked to answer for a threat, and unassigned is the finding — not a blank.
Comparison
Data inventory, Privacy threat model, or Security threat model?
A data inventory lists fields. A privacy threat model explains how those fields turn into exposure. A security threat model is answering a different question about a different attacker.
An inventory of BetterHelp's stores would have been accurate. It would still not have raised the question the eight-count complaint raises. That question is about purpose: the same intake answers, sent to Facebook, Snapchat, Criteo and Pinterest for advertising. A security model finds no unresolved attacker pathway there either. There was no attacker.
Data inventory
Lists fields and repositories.
- Necessary for basic accountability
- May omit inferences and copies
- Often static and system-centered
- Example: tables and log stores
Privacy threat model
Explains how data can be exposed, inferred, linked, or misused.
- Includes actors, purpose, attacks, and harms
- Covers derived and transient data
- Supports control selection
- Needs updates after system changes
Security threat model
Focuses on confidentiality, integrity, and availability.
- Overlaps with privacy
- May miss lawful but unexpected use
- Strong for attacker pathways
- Should be combined with rights analysis
Analogy
A parcel that leaves copies at every checkpoint
A parcel is opened, photographed, summarized and logged at several checkpoints, and the box itself is destroyed at the end of the route. Destroying the box leaves every derived record exactly where it was. That is the Alexa finding in miniature: the voice recording deleted on request, the transcript of what the child said still sitting in other databases.
A photograph shows what the parcel held. An inference produces attributes the parcel never held. So the copies worth hunting for include ones nobody ever collected.
Privacy governance follows copies, transformations, inferences, and purposes—not only the primary database.
Visual
Bystanders belong in the first row
Bystanders belong in the first row, and most threat models leave them out. The rows below run through data and inferences, actors, flows and stores, and end at harms and rights.
The first row is also where the Alexa case sits. The people described were children. The person who had to ask for deletion was a parent. A map that starts from account holders reaches neither of them.
- 1
Data subjects and bystanders
People described directly or indirectly, including non-users.
- 2
Data and inferences
Raw inputs, labels, embeddings, profiles, outputs, and derived attributes.
- 3
Actors and processors
Controllers, vendors, operators, reviewers, attackers, and downstream recipients.
- 4
Flows and stores
Training corpora, prompts, logs, caches, vector stores, tools, backups, and reports.
- 5
Harms and rights
Exposure, discrimination, manipulation, chilling effect, identity risk, and loss of control.
Steps
The copies a deletion request misses
Follow the flows into logs, caches, vector stores and backups. Those are the copies a deletion request usually misses. There is a filed federal case that says so, with a price attached.
Amazon kept children's Alexa voice recordings indefinitely unless a parent asked for deletion. On 31 May 2023 the FTC and DOJ charged the company with violating the COPPA Rule. What happened when a parent did ask is in the FTC's press release: “And even when a parent sought to delete that information, the FTC said, Amazon failed to delete transcripts of what kids said from all its databases.”
The stipulated order carried a $25 million civil penalty. The deletion control worked on the store it named. The transcript is a derived copy in a different database. Step two of the review — trace every flow, including transformation, sharing, logging and caching — is what would have found it before the complaint did.
1. Identify people and data
Include bystanders, inferred attributes, and sensitive context.
2. Trace every flow
Follow collection, transformation, sharing, logging, caching, and deletion.
3. Model actors and abuse
Consider insiders, vendors, attackers, lawful misuse, and secondary use.
4. Connect to harm
Describe exposure, manipulation, discrimination, chilling, and loss of control.
5. Select and verify controls
Minimize, separate, encrypt, restrict, aggregate, delete, monitor, and test.
A sensitive attribute nobody ever collected
Inference is what keeps privacy work moving. A system can produce a sensitive attribute that was never collected from anybody. Neither an inventory of fields nor a search for attackers will list it.
So the review has to say when the owner must redesign, restrict, remedy or retire the model. The two FTC records show the register regulators actually use. For BetterHelp, $7.8 million and a ban on sharing consumers' health data for advertising. For Amazon, a stipulated order carrying a $25 million civil penalty. Both were reached without anyone breaking in.
Key takeaways
- AI systems can create sensitive inferences even when users never fill in a labeled attribute.
- Privacy analysis should cover training, inference, retrieval, tools, logs, support, backups and retirement. BetterHelp's failure sat at the advertising stage, Amazon's at the deletion stage.
- Deleting a primary record may leave embeddings, profiles, traces or vendor copies. Amazon deleted Alexa voice recordings when a parent asked and, the FTC charged, failed to delete the transcripts from all its databases.
- Security and privacy overlap, but lawful secondary use can be a privacy harm without a breach. BetterHelp disclosed email addresses, IP addresses and intake health-questionnaire answers to Facebook, Snapchat, Criteo and Pinterest.
- Bystanders and non-users may be represented in prompts, images, audio or derived records. The Alexa recordings were of children, and it was a parent who had to ask for deletion.
- Test privacy claims against actual flows and deletion behaviour: the FTC priced these two at $7.8 million and $25 million. The model itself is a store — membership inference and the 1,800-candidate GPT-2 extraction both read data back out of trained weights.