Skip to content
AI.info

ML data engineering

Privacy, Consent, Retention, and Deletion

Apply purpose limitation, minimization, consent, linkage risk, retention, deletion, and downstream propagation to ML datasets.

By the end you can

Example

Sensitive information can appear without an explicit sensitive column

Deriving features and linking tables can create new privacy and governance obligations. Each measurement below was made on a dataset that had already had the names taken out.

  • Location sequence: four points in time and space are enough to pick one person out of a million and a half. de Montjoye and colleagues took fifteen months of mobile-phone mobility data for one and a half million people across roughly 6,500 antennas, and reported the result in Scientific Reports in 2013: “In fact, in a dataset where the location of an individual is specified hourly, and with a spatial resolution equal to that given by the carrier’s antennas, four spatio-temporal points are enough to uniquely identify 95% of the individuals.” Uniqueness decays only as the 1/10 power of resolution, so blurring the trace buys almost no anonymity. The Article 29 Data Protection Working Party restated the finding in 2014: “They showed that 95% of the population could be singled-out with four location points, and that just two points were enough to single-out more than 50% of the data subjects”. Coarser data is not anonymous data.
  • Consumption history: Netflix released 100,480,507 ratings created by 480,189 subscribers between December 1999 and December 2005, with names replaced by numbers. Narayanan and Shmatikov took the release apart in 2008 and reported: “With 8 movie ratings (of which 2 may be completely wrong) and dates that may have a 14-day error, 99% of records can be uniquely identified in the dataset.” FTC staff opened an investigation on 13 October 2009. Netflix suspended its planned Prize 2, and the FTC closed the matter on 12 March 2010.
  • Text logs: support messages contain names, addresses, secrets, and medical details inside free-form content.
  • Graph linkage: pseudonymous accounts become identifiable through rare relationship patterns and public information.
  • Model feature: a set of weights is a data surface, not a summary that has forgotten its inputs. Carlini and eleven co-authors pulled 604 unique memorised training examples out of GPT-2, from 1,800 candidate generations — names, phone numbers, email addresses, IRC conversations and 128-bit UUIDs. Their 2021 abstract adds the part that matters: “Our attack is possible even though each of the above sequences are included in just one document in the training data.” One document was enough.

A technically useful dataset can still be inappropriate to build

ML data engineering decides what is collected, linked, retained, derived, shared, and made easy to query. Those choices create privacy, security, and social risk before any model is trained. Removing names is not enough. Rare combinations, location histories, device traces, free text, images, and linkage across tables can identify people or reveal sensitive attributes. Governance turns legal, ethical, contractual, and organizational requirements into technical controls. It covers purpose, minimization, access, retention, residency, deletion, audit, documentation, and approval.

The goal is not to collect everything and restrict it later. A safer design collects and derives only the evidence the decision justifies, then limits how long and how broadly that evidence can be used.

What follows is not a set of intuitions. It is what has actually been re-identified out of published data, what a regulator has actually ordered deleted, and what survived a deletion that was reported as complete.

Privacy and security are properties of the complete data lifecycle, not a final anonymization step.

Case

ZIP code, gender and birth date identified 87% of Americans

Three ordinary fields identify most of a country. Working from 1990 census data, Sweeney found that “87% (216 million of 248 million) of the population in the United States had reported characteristics that likely made them unique based only on {5-digit ZIP, gender, date of birth}”. Golle repeated the work on the 2000 census and reported “unique identification of fewer individuals (63% of the US population)”. Both numbers describe a table with no name column in it at all.

You do not need a released table of rows either. The US Census Bureau attacked its own published statistics. From roughly 6.2 billion Summary File 1 statistics, out of more than 150 billion released, it reconstructed record-level microdata for all 308,745,538 people counted in the 2010 Census. The reconstruction agreed exactly with the confidential Census Edited File on block, sex, age, race and ethnicity for 46.48% of the population. The Bureau then linked those records to Experian commercial data and reported: “Putative re-identifications were 138 million (45% of the 2010 Census resident population of the U.S.). Confirmed re-identifications were 52 million (17% of the same population).” Its Data Stewardship Executive Policy Committee concluded that the 2010 disclosure-avoidance method was no longer safe to use.

The re-identification figure is contested, and the disagreement is worth carrying rather than hiding. Muralidhar re-examined the attack in 2022, working through the same reported numbers — 238,175,305 putative identifications out of 308,745,538, and “at least 52 million” Americans, up to 179 million — while disputing that a single reconstruction proves confirmed re-identification. The dispute does not touch the first step. Published aggregates, with no microdata released at all, were enough to rebuild individual records.

Visual

Risk follows data from collection to disposal

Controls belong at every stage where the meaning or the exposure of the data changes. Every stage below has a documented failure attached to it: collection without express consent at Everalbum, publication of aggregates that could be inverted at the Census Bureau, and a deletion at Alexa that stopped at the recording and left the transcript in place.

FigureProcess · 5 steps

  1. 1. Collect

    Establish purpose, notice, consent or other basis, minimization, and safe instrumentation.

  2. 2. Store and classify

    Label sensitivity, encrypt data, separate identities, and enforce location or residency constraints.

  3. 3. Transform and link

    Review derived sensitivity, re-identification risk, joins, and feature appropriateness.

  4. 4. Access and use

    Apply least privilege, purpose-based approval, logging, environment isolation, and export controls.

  5. 5. Retain and delete

    Enforce schedules, legal holds, subject rights, backup handling, and downstream deletion propagation.

Derived features and linked datasets can be more sensitive than their individual sources.

Comparison

Minimization, de-identification, and access control address different privacy risks

No single control eliminates privacy risk across all uses. The second of the four below, de-identification, is the one most often mistaken for a finished job. It has a statutory version, so there is no need to guess what it involves. HIPAA's Safe Harbor method, at 45 CFR §164.514(b)(2), is a list of 18 identifier classes, (A) through (R), that must be stripped. ZIP codes may be kept only to their initial three digits, and only where that three-digit area holds more than 20,000 people; otherwise the field becomes 000. Clause (C) reads: “(C) All elements of dates (except year) for dates directly related to an individual, including birth date, admission date, discharge date, date of death; and all ages over 89 and all elements of dates (including year) indicative of such age, except that such ages and elements may be aggregated into a single category of age 90 or older;”

Set that against Sweeney's five-digit ZIP, gender and date of birth. Safe Harbor coarsens the ZIP, puts a population floor under it, removes the day and the month, and caps the age. Four rules, each one an admission that a rare value identifies. It is still not a guarantee. NIST Special Publication 800-188 describes Safe Harbor as working “by specifying that health information is considered to be de-identified through the removal of 18 kinds of identifiers”, and records that data de-identified this way “retains some risk of identification”: the risk “is very small, it is not zero”. One federal agency wrote the rule out in eighteen numbered clauses, and a standards body still calls the leftover risk real. That is why the other three columns exist.

FigureComparison · 4 columns

Data minimization

Avoid collecting or retaining data that is not justified.

  • Reduces breach and misuse surface directly
  • Requires feature-value and purpose review
  • Can limit future unplanned reuse
  • Example: store coarse region instead of precise location

De-identification or pseudonymization

Remove or transform direct identity fields.

  • Reduces casual identification and exposure
  • Linkage and rare combinations can remain identifying
  • Keys and mapping tables need separate protection
  • Example: replace customer ID with scoped token

Access control and isolation

Limit who and which systems can use the data.

  • Supports least privilege and purpose separation
  • Needs audit logs and timely revocation
  • Does not make inappropriate collection acceptable
  • Example: restricted environment for sensitive labels

Privacy-preserving computation

Reduce exposure through aggregation, noise, secure computation, or local processing.

  • Can provide formal or architectural protections
  • Introduces utility, complexity, and threat-model tradeoffs
  • Does not replace governance of inputs and outputs
  • Example: differentially private aggregate statistics

Key idea

Deletion must propagate through derived datasets and models

Removing one source row may not remove its copies, features, caches, training snapshots, exports, or its influence on trained models. What the response has to be depends on policy, law, contract, and technical feasibility.

Maintain lineage from subject-linked sources to derived assets. Design deletion or suppression workflows before the first request arrives, including backup and retry behavior.

Some artifacts may be retained under legal hold or documented exceptions. Those cases need restricted access, clear justification, and expiry review rather than silent noncompliance.

The GDPR writes this propagation duty into its own text. Article 17(2) covers a controller who has made the personal data public. That controller must “take reasonable steps, including technical measures, to inform controllers which are processing the personal data”. The obligation reaches “any links to, or copy or replication of, those personal data”. Available technology and cost of implementation bound the effort, not the obligation.

A court record shows the failure mode in its ordinary form. Amazon gave Alexa users a delete button and it worked on the recordings. The complaint in United States v. Amazon.com, Inc., filed on 31 May 2023, alleges what it did not reach: “Alexa deleted the voice recordings but retained written transcripts of those recordings in residual data stores.” The FTC and DOJ alleged that until mid-2019 those transcripts went on being used for product improvement, without users being told. The stipulated order of 19 July 2023 imposed a $25 million civil penalty and required deletion of inactive child profiles. It also barred Amazon from using deleted information to create or improve any “Data Product”, defined in the order as “any model, derived data, or other tool developed using Alexa App Geolocation Information, Voice Information, or a Child's Personal Information”. The delete button worked. The lineage behind it did not, and the remedy was written as an injunction rather than as a recommendation.

Whether the trained model itself still holds the data is no longer an open question either. Carlini and co-authors report: “we identify 604 unique memorized training examples from among the 1,800 possible candidates, for an aggregate true positive rate of 33.5%”. The European Data Protection Board cites that paper in Opinion 28/2024, adopted 17 December 2024, and reaches its position: “the EDPB considers that AI models trained on personal data cannot, in all cases, be considered anonymous”.

A deletion workflow is an end-to-end data-product capability, not a single SQL statement.

Example

Deletion reaches more than the raw table

A deletion request may touch several data surfaces that are easy to overlook.

  • Raw and curated rows keyed by direct or resolved identity.
  • Training snapshots and evaluation cohorts retained for reproducibility.
  • Feature-store values, caches, search indexes, and exported files.
  • Annotations, embeddings, aggregates, and synthetic records with residual linkage risk — the Alexa transcripts that outlived the deleted voice recordings in residual data stores were exactly this class of artifact.
  • Models and algorithms. Their removal is no longer only a matter of policy preference. On 11 January 2021 the FTC settled with Everalbum, Inc., developer of the Ever photo app, and the settlement reached the model: “Part III of the proposed order requires Respondent to delete (A) photos and videos of Ever app Users who requested deactivation of their accounts, (B) face recognition data that it created without obtaining Users' affirmative express consent, and (C) models and algorithms it developed in whole or in part using images from Users' photos.” The order runs for 20 years.

Steps

Map a deletion and retention obligation end to end

The map should connect legal or policy requirements to technical identities and evidence. Step 5 is the one that already carries an enforceable term. The stipulated order of 19 July 2023 in United States v. Amazon.com, Inc. forbids using deleted geolocation, voice or children's information to create or improve any “Data Product”. A backfill or a replay that puts deleted evidence back into a training set is a breach of an injunction, not a hygiene lapse.

FigureProcess · 5 steps

  1. 1. Resolve the subject

    Identify direct, aliased, household, device, and probabilistic identifiers.

  2. 2. Traverse lineage

    Find raw, derived, exported, cached, and released artifacts.

  3. 3. Apply the policy

    Delete, aggregate, restrict, retain under exception, or mark for expiry.

  4. 4. Reconcile completion

    Record successful actions, unreachable systems, and residual copies.

  5. 5. Update future builds

    Prevent deleted evidence from reappearing through replay or backfill.

Analogy

Borrowed data comes with conditions

Collection resembles borrowing a protected item for a stated purpose and a stated period. The borrower may copy notes only under the agreed rules, and must return or destroy them when the loan ends. A catalog tracks where the item and its authorized derivatives travelled. Access controls matter, but so do purpose, duration, and the rights of the owner. Data, unlike a book, can be copied perfectly and combined to reveal new information. Returning one copy does not prove that every derivative disappeared. The transcripts that stayed in residual data stores after the voice recordings were deleted are the notes kept after the item went back.

Privacy governance must track purpose and propagation, not only possession.

Key takeaways