EPSVec: Efficient and Private Synthetic Data Generation via Dataset Vectors

AI summary

Overview

• Research area: Natural Language Processing, with intersections in differential privacy (DP), synthetic data generation, and LLM activation steering.
• Technical level: Intermediate — readers benefit from familiarity with LLMs, embedding/activation spaces, and basic differential privacy concepts.
• Scope: The paper introduces EPSVec, a lightweight inference-time method that steers LLM generation using differentially private "dataset vectors" to produce high-quality synthetic text without incurring per-sample privacy costs.

What This Paper Is About

Many valuable text corpora — medical records, internal documents, user reviews — cannot be shared directly because they contain sensitive information. Synthetic data generated by LLMs can substitute for these corpora, but existing private generation methods are slow, data-hungry, and burn privacy budget with every token they generate. EPSVec's goal is to produce synthetic data that closely matches a private dataset's distribution, under formal differential privacy guarantees, using minimal compute and minimal private data.

Key Contributions

1. Dataset vectors. A new abstraction that represents a private dataset as a single direction in an LLM's activation space, computed as the mean difference between private data embeddings and synthetic reference embeddings. This captures subtle dataset-level properties (style, lexical patterns, semantic flow) that natural-language prompts cannot express.
2. Decoupled privacy and generation. The dataset vector is extracted and privatized once with Gaussian noise; every subsequent synthetic sample, regardless of length or count, is free under the privacy budget (post-processing preserves DP). This inverts the cost structure of prior methods, where privacy cost scales with tokens generated.
3. DP fixed-shot prompting. A technique that uses a DP-histogram mechanism over a synthetic candidate pool to select a small set of reusable exemplar prompts. These stabilize pretrained (base) model generation without leaking private data, and also improve the quality of the dataset vectors themselves.
4. Systematic evaluation of pretrained vs. instruction-tuned models. The paper empirically demonstrates that instruction-tuned LLMs suffer from diversity collapse and produce overly sanitized, easily separable text for private synthesis, motivating their use of base models.

Main Findings

• Large fidelity gains: EPSVec achieves an average 150% MAUVE improvement across IMDb, Yelp, BioRxiv, and OpenReview compared to the next-best method (PP++), with the largest gains in low-data regimes.
• Extreme compute and data efficiency: EPSVec requires the fewest private samples and the least amortized runtime among the compared baselines for generating 2K synthetic samples. It generates samples at roughly the same cost as zero-shot inference.
• Pretrained beats instruction-tuned: Both EPSVec and PP++ using pretrained models substantially outperform their instruction-tuned counterparts. The paper shows instruction-tuned models produce homogenized, overly fluent text with reduced lexical diversity (fewer unique opening 3-grams) that BERT classifiers distinguish far too easily from real data.
• Downstream utility matches real data: Synthetic data from EPSVec yields BERT classification accuracy comparable to training on real data across all four datasets.
• Baselines hit scalability walls: PP and PP++ cannot generate the full 2K samples on IMDb, BioRxiv, and OpenReview; PP++ fails entirely at ε=3. EPSVec handles both budget levels cleanly.
• Sequence-length robustness: Other methods fail to generate long samples under strong privacy budgets, while EPSVec is unaffected because its privacy cost is independent of generation length.
• Small data sufficiency: MAUVE improves rapidly even with only a small number of private examples (e.g., 500 per class) used to construct dataset vectors.

Methodology in Plain English

EPSVec works in four logical steps:

1. Extract hidden features. Run a frozen LLM over both the private dataset (e.g., real BioRxiv abstracts) and a matched synthetic reference set generated by prompting the same LLM. Average the hidden states across tokens for every example to get one vector per text.
2. Compute the dataset vector. Subtract each synthetic reference embedding from its paired private embedding. Averaging these differences yields a direction that captures the gap between what prompting produces and what real data looks like — the "dataset shift." Because both sides share the same coarse topic, subtraction cancels common features and isolates subtle dataset-specific structure.
3. Privatize the vector. Clip each per-example difference to bound sensitivity, then add Gaussian noise calibrated to (ε, δ). A theorem (Theorem 4.1) formalizes the guarantee: with appropriate noise, the released vectors are (Lε, Lδ)-DP by basic composition. Further subsampling amplification is proven in Theorem 4.2.
4. Steer generation. At each decoding step, add the privatized vector (scaled by a coefficient β) to the LLM's hidden states at selected layers. The model then generates freely — no further privacy bookkeeping — producing as many synthetic samples as desired.

Two supporting components accompany this pipeline: fixed-shot exemplars (DP-selected synthetic prompts reused throughout both vector extraction and inference) and the use of pretrained base models rather than instruction-tuned ones for their greater diversity.

Why This Matters

• Research impact: The paper reframes private synthetic data generation as a one-time representation extraction problem rather than a per-token DP problem, offering a different cost structure than any prior inference-time private method. It also contributes evidence to the growing literature on the limitations of instruction-tuned models for faithful data simulation.
• Real-world applications:
  • Healthcare: Generating synthetic biomedical abstracts or clinical notes for model training without exposing patient records.
  • Enterprise analytics: Sharing realistic user feedback or internal documents with third-party tooling under privacy constraints.
  • Peer review and scientific publishing: Producing synthetic review corpora that preserve decision patterns without attributing opinions to identifiable reviewers.
  • Small-data domains: Supporting teams that have only a few hundred sensitive examples but need thousands of training samples.
• Industry relevance: The method drops compute requirements to near zero-shot inference levels and eliminates per-token privacy accounting, making private synthetic data generation practical for production pipelines where DP fine-tuning or large-batch private prediction were previously too expensive.

Future Directions

1. Extension beyond text. The dataset-vector abstraction is modality-agnostic in principle; applying it to image, audio, or multimodal data is a natural next step.
2. Better reference distributions. Dataset vectors encode the gap between prompting and real data. Improving how the reference set is created (through stronger DP scaffolding or model ensembles) could sharpen the captured signal further.
3. Adaptive layer and coefficient selection. The paper uses fixed layers (18–21) and β values; automatic per-dataset selection, perhaps via private validation, remains an open problem.
4. Interaction with DP fine-tuning. Combining dataset-vector steering with lightweight private adaptation — where the vector acts as a prior and DP-SGD refines it — could blend the efficiency of one with the fidelity of the other.

Target Audience

Researchers and practitioners working on privacy-preserving machine learning, synthetic data generation, and LLM-based data augmentation. It is especially relevant to applied scientists who need to produce usable synthetic corpora under tight DP budgets or scarce private data, and to those interested in interpretable control of LLM generation through activation-space interventions.

Authors’ abstract

High-quality data is essential for modern machine learning, yet many valuable corpora are sensitive and cannot be freely shared. Synthetic data offers a practical substitute for downstream development, and large language models (LLMs) have emerged as powerful engines for generating it. However, existing private text generation methods are severely inefficient: they are data-intensive, computationally slow, and often require large private corpora or batch sizes to achieve usable quality. We introduce EPSVec, a differentially-private lightweight alternative that steers LLM generation using *dataset vectors*--directions in activation space that capture the distributional gap between private data and public priors. EPSVec extracts and sanitizes steering vectors just once and then performs standard decoding. This decouples the privacy budget from generation, enabling arbitrarily many synthetic samples without additional privacy cost and yielding strong fidelity even in low-data regimes. Furthermore, we enhance our method by utilizing pretrained (base) models and introducing fixed-shot prompting to boost generation diversity and fidelity. Our experiments demonstrate that EPSVec outperforms existing baselines in distributional alignment and downstream utility, particularly in low-data regimes, while significantly reducing computational overhead.