Memorization, Membership Inference, Model Inversion, and Extraction

Models can leak information through memorized outputs, differential behavior, learned representations, or query access. A model is not automatically anonymous merely because raw training records are not stored or returned. The threats have names: membership inference, attribute inference, model inversion, training-data extraction, and model extraction. Whether any of them succeeds depends on the model, data rarity, overfitting, access, auxiliary knowledge, defenses, and attacker resources. Every item on that list is something a team can test. None of them is something a team can assume.

One item has been measured rather than asserted. Duplicated training data drives regurgitation, and not in proportion to the duplication. The rate at which a language model regenerates a training sequence rises superlinearly with the number of copies in the training set. Kandpal and colleagues put a size on that in 2022: “For instance, a sequence that is present 10 times in the training data is on average generated ∼1000× more often than a sequence that is present only once.” Ten copies instead of one, and the model hands the sequence back about a thousand times as often.

The same paper reports something worse. Existing methods for detecting memorized sequences have near-chance accuracy on non-duplicated training sequences. So the duplication a pipeline failed to remove drives the leak. And the records that slipped through as singletons are the ones a memorization detector is least able to flag in advance.

Five attacks, ordered by what the attacker wants. Whether a record was in the data: membership inference. What a sensitive attribute is: attribute inference. What the inputs looked like: model inversion, reconstructing representative or individual-level features from outputs. What the training text said: training-data extraction, eliciting memorized examples or rare sequences. What the model itself does: model extraction, approximating functionality through queries.

The last of the five is the one teams price wrongly, because it has been priced. In 2016 Tramèr and colleagues extracted 100%-equivalent copies of live ML-as-a-service models through the prediction API alone: “For both services, we show computationally fast attacks that use a small number of queries to extract models matching the targets on 100% of tested inputs.” An Amazon logistic regression on Digits took 650 queries and 70 seconds. Amazon on Adult took 1,485 queries. A BigML decision tree on German Credit took 1,150 queries and 631 seconds. A BigML decision tree on the Steak Survey took 4,013 queries and 2,088 seconds.

Training-data extraction and model extraction travel through the same query interface and want different things. A rate limit tuned to stop memorized text coming back is a result about memorized text. It says nothing about 650 queries in 70 seconds.

The closest published thing to the scenario every model owner fears is not an anonymous incident report. It is a production system with a name. In November 2023, over 10,000 unique verbatim-memorized training examples came out of ChatGPT (gpt-3.5-turbo) for $200 USD worth of API queries. A divergence attack raised the model's rate of emitting training data 150x above normal chatbot behaviour. The target was aligned, deployed and commercially operated. The attacker had a paid API key and nothing else. Read the five conditions below as the anatomy of that class of failure. Each one has been measured somewhere in the work above.

Testing starts with a named attacker and ends with a plan for the day a leak is confirmed. Define the attacker first — access, queries, auxiliary data, budget and objective — because every later result is scoped to that definition. Then prioritise the vulnerable data: rare, repeated, sensitive, identifying and high-consequence records. Then run the evaluations: memorization, membership, inversion and extraction. Then layer defenses: deduplicate, minimize, regularize, limit access, monitor, filter, and apply formal privacy where justified. Then prepare the response: deletion support, incident investigation, notification, retraining, and remedy for affected cases.

The third step is the one teams treat as vague, and it has a published shape. Plant a canary in the training data, then measure how hard it is to get back out. The 2019 paper The Secret Sharer inserted a single 9-digit canary once into the Penn Treebank corpus and recovered it with about 10^5 queries using shortest-path search. Brute force over the 10^9 candidates would have cost four orders of magnitude more. Carlini and colleagues turned that into an exposure metric a team can track across training runs. Pointed at real data rather than planted canaries, the same method found 10 pre-existing secrets heavily memorized in character-level models trained per Enron employee: “Three secrets are extractable in < 1 hour; all are heavily memorized.”

A canary inserted once and found is a memorization result you can report. A canary you never inserted is the reason a team has nothing to report.

De-identification, privacy testing, and a formal mechanism answer three different questions. Only the third produces a bound rather than a result.

Model privacy attacks are evaluated against an assumed attacker. So those assumptions belong in the record beside the results: the access, the queries, the auxiliary data, the budget and the objective. Write them down before the tests rather than reconstruct them after. Paragraph 55 of Opinion 28/2024 is the reason. Testing against widely known, state-of-the-art attacks is evidence for resistance to those attacks, and to nothing else. A record that omits the attacker records a conclusion without its scope.

Decide, in advance, which leakage finding would make the owner redesign, restrict, remedy, or retire the model. A named individual recovered verbatim, as 46 were from GPT-2? A membership test that separates members from non-members at the rates the 2017 work reported? A functional copy taken in 650 queries? Naming the trigger before the result arrives is what keeps the answer from being written by whoever is least inconvenienced by it.