Exploring The Empirical Privacy Landscape of Synthetic Data

AI summary

Overview

Research area: Privacy-preserving machine learning — specifically, the trade-off between model capability and training-data leakage when language models are finetuned on sensitive records (cs.CR, with ties to privacy and language modeling).

Technical level: Intermediate. The paper assumes familiarity with finetuning, LoRA adapters, differential privacy (DP-SGD), and perplexity-based evaluation, but the experimental logic is explained plainly.

Scope (one sentence): A controlled comparison of data-level defenses (PII scrubbing, synthetic rewriting) and training-level defenses (DP-SGD, Goldfish) on a synthetic medical-record corpus seeded with canaries, measured on both privacy leakage and downstream capability for two open-weight models.

What This Paper Is About

Adapting language models to high-stakes domains such as medicine requires training on sensitive records, and models can memorize and later reveal those records. Existing defenses are usually studied one at a time, and usually with attention to either privacy or utility rather than both. This paper builds a single controlled testbed — synthetic patient records from Synthea plus planted "canary" secrets — so that several defenses can be ranked on the same two axes, and asks which methods actually buy a better capability–privacy trade-off.

Key Contributions

1. A controlled comparison harness. All methods start from the same corpus, the same target models, and the same LoRA training recipe, and are scored with two privacy metrics (verbatim perplexity, semantic perplexity) and two capability metrics (held-out record perplexity, medication perplexity). Data are newly generated with Synthea, so leakage can be attributed to the experiment rather than to pretraining exposure.
2. A canary design with graded duplication. 100 synthetic canary records each carry a fabricated, newly coined medication name as the planted secret, inserted at multiplicities of ×1, ×4, ×16, ×32 and ×64, yielding 7,340 training instances (5,000 non-canary records plus 2,340 canaries).
3. Identification of privacy-aware rewriting as the best-balanced method. A uniqueness-aware rewriting prompt — instructing a rewriter to replace rare, identifying details with plausible fictional values while keeping generic clinical content — outperforms plain paraphrasing and, unlike DP-SGD, keeps capability high.
4. Stress tests of the rewriting recipe. Ablations over rewriter model family and size, reasoning (test-time) budget, summary-based rewriting, and one-rewrite-for-five-epochs versus five-independent-rewrites-for-one-epoch, plus an exposure analysis and a comparison of LLM-judge scores against measured downstream outcomes.

Main Findings

• Direct finetuning is the capability leader and the leakage leader. It achieves the highest capability improvements for both target models and, for gpt-oss-20b, receives the second-lowest privacy rank.
• DP-SGD buys privacy at a capability cost. It reaches a relatively high privacy rank but a lower capability rank than most other trained methods; at ε=8 it achieves the lowest mean exposure among trained conditions (1.74 bits for Qwen3.5-4B-Base and 1.56 bits for gpt-oss-20b, with no rank-1 canaries), close to the no-finetuning values of 1.49 and 1.39 bits.
• PII scrubbing fails against non-identifier secrets. In its zero-shot application, the coined medications remain unredacted; exposure is essentially unchanged from direct finetuning (12.16 vs 12.22 bits for Qwen, 10.76 vs 10.98 for gpt-oss), and it generally tracks direct finetuning across duplication levels.
• Goldfish keeps capability but not privacy. It retains a high capability rank but performs poorly on privacy; its mean exposure is 10.38 bits (53% rank-1) and 8.90 bits (37% rank-1), and it levels off at 30–40% generation probability for the coined medication. It is, however, less sensitive to duplication, consistent with identical copies receiving the same mask.
• Privacy-aware rewriting gives the most balanced trade-off. For both target models it improves both privacy and capability ranks relative to standard rewriting. Compared with direct finetuning it shows only small differences in capability metrics but substantially higher verbatim and semantic perplexity.
• Stronger rewriters generally leak less, but not monotonically on capability. Later GPT-5 variants consistently improve privacy protection, while capability rises generally from GPT-5.1 through GPT-5.4 but not monotonically. GPT-5.5 as rewriter achieves higher privacy ranks but lower capability ranks than Qwen3.5-4B as rewriter, for both target models.
• More reasoning during rewriting is not a reliable lever. Enabling reasoning generally beats no reasoning, but increasing the reasoning budget does not monotonically improve either rank, and the best setting depends on the target model.
• Summarization adds no consistent benefit. Adding a summarize-then-expand step to privacy-aware rewriting generally yields similar or lower capability, potentially from information loss; the privacy effect varies by model. The appendix notes it remains a practical alternative when finetuning compute is limited.
• Independent rewrites do not pay off. One rewrite trained for five epochs slightly outperforms five independently generated rewrites trained for one epoch each, despite the extra generation cost.
• Duplication drives leakage, with exceptions. Direct finetuning on Qwen3.5-4B-Base generates a once-seen coined medication 16% of the time and a ×64 medication 99% of the time; gpt-oss-20b needs duplication before it memorizes (about once in 37,000 samples at ×1, rising to 69% at ×64). DP-SGD stays below 10⁻¹¹ at every duplication level; GPT-5.5 privacy-aware rewriting typically sits at the floor, with the jumps at ×16 and ×64 each traceable to a single leaked canary.
• LLM judges track privacy but not capability. Judged privacy correlates strongly with verbatim perplexity (Spearman's ρ = 0.89), while judged quality correlates negatively with measured capability (ρ = −0.64) — so data-level judging cannot replace downstream finetuning evaluation.
• The two metric pairs are not redundant. Verbatim and semantic perplexity correlate at 0.84 (Qwen) and 0.85 (gpt-oss) overall, but only 0.39 and 0.05 below a verbatim perplexity of 100; capability metrics correlate at 0.60 and 0.81 overall.

Methodology in Plain English

The team generated fully synthetic patient records with Synthea — names, addresses, SSNs, date of birth, problem lists, medications — using 5,000 records for training and reserving a disjoint held-out set. Into this corpus they planted 100 canary records that look like ordinary records but carry a made-up drug name (for example, "Mabsartinib") as the secret to be protected. Canaries were split into five groups of 20 and repeated ×1, ×4, ×16, ×32 and ×64 times. Because each duplicate level holds a different group of canaries, the no-finetuning model is used as a per-level reference for how intrinsically guessable each group is.

They then trained LoRA adapters on the corpus under six experimental settings: no finetuning, direct finetuning, PII scrubbing with the OpenAI Privacy Filter (identifiers replaced by placeholders), DP-SGD on the adapters (per-example clipping at C = 1, noise calibrated to δ = 10⁻⁵, main run at ε = 8, with ε ∈ {16, 32, 64} also trained, Poisson sampling with expected lot size 512), Goldfish (masking roughly 1/k of eligible token targets with k = 4), and rewriting — with privacy-aware rewriting as a prompt variant of rewriting. Two target models were used, Qwen3.5-4B-Base and gpt-oss-20b, under an identical recipe: rank 16, α = 32, zero dropout, all supported linear layers, AdamW at 2×10⁻⁴, batch size 4, 1,024-token sequences, bfloat16, two epochs, seed 0.

Privacy is measured two ways, both on the planted medication: verbatim perplexity, from the record prefix cut immediately before the drug name, and semantic perplexity, from a question whose unique correct answer is that drug. Capability is measured as held-out record perplexity over unseen records and medication perplexity for the final drug in a test record. For all four, lower is "stronger" in the respective direction — lower perplexity means more leakage for the privacy pair and better capability for the capability pair. Results are shown as ranks averaged across the two privacy and two capability metrics. A separate exposure analysis ranks each planted name against 6,300 unplanted alternatives (6,301 candidates, exposure range 0 to 12.62 bits, roughly 1.44 bits expected under random ranking).

Why This Matters

Research impact: The paper supplies a reusable, contamination-controlled protocol for comparing preprocessing defenses against training-time defenses on equal footing, and shows that data-level metrics (including LLM judgments) cannot substitute for measuring the finetuned model. It also documents that a formal guarantee (DP-SGD) and an empirical rewriting heuristic land in different places on the trade-off curve, which reframes the choice as a design decision rather than a solved problem.

Real-world applications:

• Adapting clinical or other regulated-record assistants where the raw records cannot be used directly but capability loss from DP-SGD is unacceptable.
• Preparing shareable or releasable synthetic corpora from sensitive sources without a formal privacy mechanism.
• Choosing a rewriter model and whether to spend inference compute, given that scaling the rewriter helps privacy more reliably than scaling reasoning effort.
• Auditing existing de-identification pipelines, which this work shows can leave non-identifier facts (a rare drug, in this case) fully extractable.

Industry relevance: Teams picking between differential privacy and data preprocessing now have evidence that privacy-aware rewriting is competitive on the trade-off while PII scrubbing and token-masking objectives (Goldfish) are weak against secrets that are not conventional identifiers. The paper is explicit that rewriting carries preprocessing cost — every record must pass through a generative model — which must be weighed against its gains, and that rewriting offers no formal differential-privacy guarantee.

Future Directions

• Formal grounding for rewriting. The authors note that one-to-one rewriting has only empirical privacy effects; whether uniqueness-aware rewriting can be given a formal guarantee, or bounded in combination with DP-SGD, is open.
• Why judged quality misleads. The negative correlation (ρ = −0.64) between LLM-judged rewrite quality and downstream capability is unexplained and suggests a need for better data-level proxies for capability.
• Reasoning budgets and rewriter scaling. Because more test-time compute yields inconsistent gains while stronger rewriters reliably improve privacy, mapping when reasoning helps — and why the best setting depends on the target model — remains unresolved.
• Beyond the Synthea setting. All records are synthetic and drawn from one generator and one domain; canary results also mix duplication with intrinsic canary difficulty, so replication on other data types and with pretraining-overlap controls would test generality.

Target Audience

Practitioners and researchers choosing privacy defenses for finetuning on sensitive text — ML privacy engineers, clinical-NLP teams, and policy-adjacent researchers evaluating synthetic data — as well as readers who want a concrete, quantitative picture of how DP-SGD, PII scrubbing, token masking, and LLM rewriting compare on the same axes. It is accessible to readers with intermediate ML background; no differential-privacy theory is required beyond the role of ε and δ as described.

Authors’ abstract

Methods for training language models on sensitive data differ in how they balance capability gains against the risk of leaking private information. In this paper, we present a controlled comparison of data-level and training-level interventions for navigating the capability-privacy tradeoff on synthetic medical records with canaries to probe for leakage. Direct finetuning performs best on capability metrics but leads to the most privacy leakage, while DP-SGD reduces leakage at substantial cost to capabilities. Generating synthetic data from these records offers a more favorable balance, particularly when guided by a uniqueness-aware instruction to replace private details. Relative to naive paraphrasing, privacy-aware rewriting improves privacy protection across a variety of different rewriting models. We also find that more capable rewriting models generally reduce the risks of downstream privacy leakage, although held-out fit does not improve monotonically and increased test-time compute yields inconsistent gains. Overall, these results highlight privacy-aware rewriting as a promising direction for achieving a more favorable trade-off between capabilities and empirical privacy compared to other training-level methods, and our evaluation framework supports comparisons of future models and methods.