Skip to content
AI.info

Responsible AI

Synthetic Data, Privacy-Preserving Evaluation, and Residual Risk

Evaluate synthetic data privacy, fidelity, disclosure risk, rare-group utility, private evaluation, and evidence needed before release.

By the end you can

Comparison

NIST calls the first one the release-and-forget model

Public release, monitored sandbox, and query service differ mainly in what can be taken back. Only the first is irreversible.

The tiers are not this lesson's invention. They are a governance decision, taken in advance rather than left as a residue of how the project happened to end. NIST's 2023 guidance on de-identifying government datasets puts it this way: “Agencies should decide upon a data-sharing model, such as publishing de-identified data, publishing synthetic data based on identified data, providing a query interface that incorporates de-identification, or sharing data in non-public protected enclaves.”

Four named models, chosen up front. The document's own name for the public one is the “release-and-forget model”. And there is a reason the three columns below are not interchangeable. Once published, an organisation may find the data difficult or impossible to recall.

FigureComparison · 3 columns

Public synthetic release

Distributes generated records broadly.

  • Supports access and reproducibility
  • Creates irreversible leakage risk
  • Needs strong privacy testing or formal guarantees
  • Hard to revoke after copying

Controlled synthetic sandbox

Limits access and monitors use.

  • Reduces distribution risk
  • Can support iterative evaluation
  • Still requires privacy and output controls
  • Useful for higher-sensitivity data

Protected query service

Returns approved statistics or analyses.

  • Avoids releasing row-level data
  • Can apply DP and query controls
  • May limit research flexibility
  • Requires governance of repeated queries

Singling out, linkability, inference — not the absence of exact copies

Synthetic data are generated records intended to resemble selected properties of source data. They may reduce direct exposure. Privacy still depends on the generator, the training data, the release, what the attacker knows, and the evaluation.

The test that matters was written down and numbered long before any of these generators existed. It names three risks, not one. The EU's Article 29 Working Party set them out in 2014, in the opinion known as WP216: “An effective anonymisation solution prevents all parties from singling out an individual in a dataset, from linking two records within a dataset (or between two separate datasets) and from inferring any information in such dataset.”

Singling out, linkability, inference. WP216 says directly that removing identifying elements is not by itself enough. A duplicate check against the source table answers none of the three.

A release review should examine fidelity for the intended analyses, disclosure and membership risks, and outlier reproduction. It should examine rare-group utility, downstream bias, provenance, and the difference between internal testing and public release. Differential privacy can provide a formal bound when it is incorporated correctly. Most releases test the first item and stop.

Outlier reproduction is where the risk usually sits. A generator that copies no record can still copy a combination only a handful of people have.

A release approved on fidelity alone has been checked for usefulness and not for disclosure, and the two are not the same review.

Case

Controls at PG ≈ 1, one outlier below PG 0.005

That claim has been tested directly, and the measurements are what this lesson stands on.

Start with the design, because it is what makes the numbers readable. Five manually chosen outlier records were run against five randomly chosen ones, on the Texas and Adult datasets. Each dataset, raw and synthetic, held n = m = 1,000 records. The adversary trained on a reference dataset of l = 10,000 records, using 10 shadow models. Stadler and colleagues reported the results at USENIX Security in 2022, in the paper they called Synthetic Data – Anonymisation Groundhog Day.

The randomly chosen controls came out close to perfectly protected: privacy gain PG ≈ 1. Under an attack using the naive feature set F_Naive, 4 of the 5 selected Texas targets averaged a gain smaller than 0.8 across all three generative models. And 1 of the 5 consistently received a privacy gain close to 0 — PG < 0.005 — from synthetic data produced by a CTGAN-trained model.

So the protection is real, and it is not evenly distributed: “Our experimental evaluation reveals that synthetic data does not provide uniform protection against strategic adversaries, and some outliers remain highly vulnerable.”

A dataset-level leakage figure averages the PG ≈ 1 controls together with the target sitting below 0.005. It then reports something no individual in that file experienced. That is why the leakage measures usually published understate what an attacker can actually recover. And it is why records at the edge of the distribution are the ones a generator reproduces.

That is the combination only a handful of people have.

Visual

Thirty-four published tables, 97 million records back

The generator determines what can be learned. The release design determines who learns it. The two testing stages determine whether either answer was right.

The second step is the one routinely assumed away, on the theory that publishing statistics instead of records settles the question. It does not. A statistics-only release has been measured. The U.S. Census Bureau ran a simulated reconstruction and reidentification attack on the 2010 Census and published the result in 2023.

From only 34 of the published table sets it recovered census block, sex, age, race and ethnicity for the confidential person records: “Ninety-seven million person records (every resident in 70% of all census blocks) are exactly reconstructed with provable certainty using only public information.”

And the people who came back out by name were the ones at the edge again. Race and ethnicity were reidentified with 95% accuracy for 3.4 million population-unique individuals — persons with race and ethnicity different from the modal person on their census block. Repeated versions and repeated queries are the same arithmetic spread over time.

FigureProcess · 5 steps
  1. 1

    Source data and generator

    Determine what patterns, individuals, and rare combinations can be learned.

  2. 2

    Release design

    Control records, statistics, queries, access, and repeated versions.

  3. 3

    Privacy testing

    Probe nearest neighbors, membership, attribute disclosure, linkage, and outliers.

  4. 4

    Utility testing

    Compare intended analyses, downstream tasks, uncertainty, and group coverage.

  5. 5

    Governance

    Define purpose, access tier, documentation, monitoring, and withdrawal.

Example

Fifteen attributes, 99.98% of Americans

A health organization releases a synthetic patient dataset after confirming that no row exactly matches a real record. Researchers later infer rare diagnoses, because the generator preserves distinctive combinations from small groups. The duplicate check that cleared the release tests identity of rows. Uniqueness of combinations is a different quantity, and it has been measured.

It can also be predicted, and predicted well. Rocher and colleagues built a model that scores individual uniqueness with AUC of 0.84 to 0.97 across 210 populations, published in Nature Communications in 2019. Its headline figure is the one to read next to any duplicate check: “Using our model, we find that 99.98% of Americans would be correctly re-identified in any dataset using 15 demographic attributes.”

A rare diagnosis alongside an age band and a region is fewer than 15 attributes doing the same work.

  • No exact copies: Simple duplicate checks pass, because they compare rows for identity and nothing else.
  • Distribution fidelity: The generator preserves useful correlations and rare patterns — the same uniqueness scored at AUC 0.84 to 0.97 across 210 populations.
  • Disclosure risk: Distinctive combinations may remain linkable or reveal membership; 15 demographic attributes already re-identify 99.98% of Americans.
  • Utility disparity: Rare groups are smoothed away in some analyses and exposed in others.
  • Release assumption: The word “synthetic” is treated as equivalent to anonymous and safe, when WP216 asks about singling out, linkability and inference.

Key idea

At ε = 10 the outliers are still there

Resembling real data is both the source of utility and a source of privacy risk. A synthetic dataset cannot maximize unrestricted fidelity and simultaneously guarantee that every sensitive pattern is harmless. Privacy and utility tests are attack- and task-dependent.

The cap has a number attached to it. Even differentially private synthetic data built with ε = 10 may leave outliers vulnerable to linkage attacks. That is NIST's 2025 guidance on evaluating differential privacy guarantees, citing the Groundhog Day work for it. The same guideline still asks for the mechanism: “To provide robust privacy protection, including against novel developments in privacy attacks, synthetic data should be generated using differentially private algorithms.” Formal machinery, a stated ε, and residual exposure at the edge — all in the same document.

The other side of the trade is booked in the same place. Synthetic data can reduce accuracy for sub-populations, leading to systemic bias. That applies whether or not the generator is differentially private. Public release should therefore use conservative assumptions, because future auxiliary data and attacks cannot be recalled.

The health organisation checked that no synthetic row matched a real one. The rare diagnoses came back out through the combinations instead.

Fidelity has to be capped deliberately — every notch added for the analyst is disclosure risk that no test can certify away.

Example

Thirty-eight days, and about 1,500 copies already gone

A synthetic release is tested against a recipient, not against an average. It is also tested against the clock, because withdrawal is slower than distribution.

On 1 August 2016 the Australian Department of Health published a 10% sample of MBS/PBS claims on data.gov.au: roughly 3 billion lines of data, covering about 2.5 million Australians. University of Melbourne researchers reversed the provider-number encryption within a month. The file came down on 8 September 2016, 38 days after it went up. By then it had already been downloaded approximately 1,500 times.

Australia's Information Commissioner found in 2018 that the Department had breached Australian Privacy Principles 1, 6 and 11. The regulator's own remedy is the tier language this lesson uses: “For example, they should consider limiting release of unit level data about individuals to trusted recipients, rather than to the world at large.”

  • Attacker profile: List auxiliary information a motivated recipient may possess now or later; Stadler's adversary held a reference dataset of l = 10,000 records and 10 shadow models.
  • Rare-group test: Examine whether small populations are exposed, distorted, or erased — on the Texas targets, 4 of the 5 selected outliers came in below an average gain of 0.8 across all three generative models.
  • Task comparison: Re-run three intended analyses on real and synthetic data with uncertainty.
  • Revocation reality: Document what can and cannot be withdrawn after distribution; the MBS/PBS file took 38 days to come down and about 1,500 copies were already out.

Steps

Not whether to release, but which tier

Nothing here decides yes or no. It decides which tier of release the evidence will support. And NIST puts that decision at step 1, before the results arrive to argue with.

FigureProcess · 5 steps

  1. 1. Define release purpose

    Specify intended analyses, users, access, and prohibited uses.

  2. 2. Choose protection model

    Select access control, DP, query service, or other formal and operational safeguards.

  3. 3. Test disclosure

    Run membership, linkage, nearest-neighbor, rare-pattern, and outlier attacks.

  4. 4. Test utility and bias

    Compare downstream analyses, uncertainty, and group coverage.

  5. 5. Decide release tier

    Publish, restrict, sandbox, provide queries, redesign, or decline release.

Carry this synthetic data and private evaluation boundary forward

Synthetic records carry residual risk that no label removes. Disclosure testing and utility testing both belong in the release record.

Three findings are worth keeping together. One Texas target sat at PG < 0.005 while the controls sat at PG ≈ 1. Ninety-seven million person records were reconstructed from 34 published table sets. Fifteen demographic attributes re-identify 99.98% of Americans. None of the three is defeated by a duplicate check, and WP216 was already asking about all three risks in 2014.

State which linkage, rare-group, or utility finding would force the owner to redesign, restrict, remedy, or retire the model. State it before publication — the MBS/PBS file was downloaded about 1,500 times in the 38 days it took to withdraw.

Key takeaways