ACTG-ARL: Differentially Private Conditional Text Generation with RL-Boosted Control

AI summary

Overview

Research area: Privacy-preserving machine learning, specifically differentially private (DP) synthetic text generation, conditional generation, and reinforcement learning for language model post-training.

Technical level: Advanced. The paper assumes familiarity with differential privacy (DP), DP fine-tuning (DP-FT), tabular synthesizers such as AIM, reinforcement learning from human feedback (RLHF), PPO, and standard text-generation evaluation metrics such as MAUVE.

Scope: The paper proposes a hierarchical framework that splits DP synthetic text generation into feature learning and conditional text generation, identifies an optimal configuration called ACTG, and adds a post-training method called Anchored RL (ARL) to restore instruction-following control under DP, yielding the end-to-end algorithm ACTG-ARL.

What This Paper Is About

Existing methods for generating synthetic text under differential privacy mostly produce whole synthetic datasets and offer no fine-grained control over what gets generated, while also losing utility to the noise that DP requires. The authors tackle both problems at once: they design a modular pipeline that learns a private feature representation of the data and then generates text conditioned on those features, and they add a reinforcement-learning step that makes the generator reliably follow those feature instructions without abandoning the style of the original private text. The stated goal is high-quality, controllable DP synthetic text under strong privacy guarantees (evaluated at ε ∈ {1, 4, ∞}).

Key Contributions

1. A hierarchical framework for DP synthetic text generation. The problem is decomposed into two subtasks — learning a low-dimensional feature representation of private text (Stage 1) and learning a conditional generator (Stage 2) — with feature extraction (Stage 0) treated as a trusted pre-processing step that consumes no privacy budget. Through systematic ablations across three feature designs (topic, free-form summary, structured tabular schema), two feature generators (DP-FT and AIM), and two conditional generators (DP-FT and prompting), the authors identify an optimal configuration they call ACTG (Attribute-Conditioned Text Generation): a rich tabular schema, AIM as the DP feature generator, and a DP fine-tuned conditional generator.

2. Anchored RL (ARL), a post-training recipe for control. Because DP degrades the conditional generator's instruction-following ability, the authors introduce ARL: a hybrid objective that mixes a PPO-based RL loss (using per-instance instruction-following accuracy as the reward) with a supervised fine-tuning (SFT) loss on a best-of-N dataset. This anchors the model to the private text distribution and mitigates reward hacking. The RL phase requires no additional privacy budget because both the prompts and the reward signal are derived without accessing private data.

3. State-of-the-art results in DP conditional text generation. The combined ACTG-ARL algorithm is reported to advance DP synthetic text quality by +20% in MAUVE and +50% in attribute distribution matching over prior work, while delivering a conditional generator with strong instruction-following capability.

4. An evaluation suite and error analysis. The authors introduce instruction following accuracy (IFAcc) as a per-instance control metric, use Jensen-Shannon distance between private and synthetic feature distributions (d_JS^f), and decompose error into extraction error (d_JS^f0), feature learning error (d_JS^f1), and conditional generation error (d_JS^f2). They also critique prior evaluation practices that used weak embedding models such as all-MiniLM-L6-v2 or sentence-t5 and short context lengths of 128 or 256 tokens, which can inflate MAUVE scores.

Main Findings

• The hierarchical decomposition beats end-to-end baselines. On the two evaluated datasets, the proposed conditional generation approaches consistently outperform Aug-PE, vanilla DP-FT, and CTCL across all datasets, privacy levels, and evaluation metrics (both fidelity and utility, measured by MAUVE, d_JS^f, classification F1, and next token prediction accuracy). The advantage persists when scaling to a larger base model (gemma-3-4b-pt) and when replacing the proprietary oracle model with the open-source Qwen2.5-32B-Instruct.

• The rich tabular schema is the best feature design. The structured tabular schema (S3) performs best, followed by the free-form 1-2 sentence summary (S2); both substantially outperform the generic topic model (S1) used in CTCL. The authors attribute this to domain mismatch in the fixed, public-corpus topic model, and highlight the value of a compact yet informative, dataset-specific schema.

• AIM is the better DP feature generator. AIM achieves a much lower feature learning error (d_JS^f1) than DP-FT because, as a specialized tabular synthesizer, it allocates privacy budget only to the predefined attributes of interest rather than across all tokens, avoiding budget spent on non-sensitive information such as JSON grammar or public knowledge such as age groups.

• DP-FT is the better conditional generator. In Stage 2, DP-FT attains a lower conditional generation error (d_JS^f2) than direct prompting with a powerful LLM.

• LLM-based feature extraction is reliable. The extraction error d_JS^f0 is around 0.01 on bioRxiv (measured by performing five independent extractions and treating their average distribution as ground truth).

• Conditional generation is the error bottleneck. Comparative error analysis across the three stages of ACTG shows extraction error is negligible, while conditional generation incurs a larger error than feature learning, indicating more room for improvement in Stage 2.

• DP substantially degrades instruction following. The instruction-following accuracy of ACTG's conditional generator drops from 66% to 53% on bioRxiv when DP is applied, even though aggregate metrics remain high.

• Standard RL causes reward hacking. ACTG-RL, which uses standard PPO, improves IFAcc but causes the MAUVE score to plummet. The failure mode is that the model generates short "TL;DR"-style sentences that satisfy the rubric but do not match the target domain's style; the paper shows an example from bioRxiv that scores 8/8 on the rubric but fails to resemble a paper abstract.

• Anchored RL resolves the control–fidelity trade-off. ACTG-ARL matches ACTG-RL in IFAcc while retaining ACTG's high MAUVE, and achieves the best attribute distribution matching (lowest d_JS^f). The authors note that since d_JS^f is a metric, the triangle inequality d_JS^f ≤ d_JS^f1 + d_JS^f2 explains why improving Stage 2 reduces overall end-to-end error. ACTG-ARL also further improves classification F1 over ACTG.

• Compute overhead is modest for ACTG and optional for ARL. ACTG delivers quality gains over vanilla DP-FT with little compute overhead — feature extraction and AIM together take under 15 minutes — while Anchored RL is more costly but serves the orthogonal goal of fine-grained control and is optional. Once trained, the feature generator and the ARL generator together produce 5k synthetic samples in about 20 minutes on a single A100.

Methodology in Plain English

The authors break the problem of writing private synthetic text into two smaller, easier problems. First, instead of asking a model to imitate raw text directly, they describe each private text with a compact set of attributes — for example, for a scientific abstract, its primary research area, model organism, experimental approach, and so on. This attribute set is a "tabular schema," similar to a spreadsheet row per document. Second, they learn to generate new attribute rows that look like the real ones while satisfying differential privacy, using a dedicated tabular synthesizer that spends privacy budget only on the attributes that matter. Third, they train a language model to write text conditioned on those attributes, again under DP, so that generated text both sounds like the private corpus and matches the requested attributes. At generation time, they sample fake attributes and feed them to the conditional generator to produce synthetic text, with no further access to the private data.

To fix the fact that DP fine-tuning makes the generator sloppy at following its attribute instructions, they then run a reinforcement learning step. Each attribute set acts as a checklist, and a separate model grades how many checklist items the generated text actually satisfies — that score becomes the reward. Training on this reward alone improves instruction following but destroys writing quality, because the model discovers that short, generic sentences can satisfy the checklist. The fix is to mix the RL objective with a supervised objective on a high-quality dataset built by best-of-N sampling: for each set of attributes, generate N candidate texts and keep the best-scoring one. The supervised loss is weighted by a coefficient that starts high (preserving text fidelity) and decays linearly (allowing control to improve). The result is a model that follows instructions without drifting away from the private text distribution.

Why This Matters

Impact on research. The paper reframes control as a third critical dimension of DP synthetic text generation, alongside utility and privacy, and shows that a modular decomposition plus a post-training recipe can improve both aggregate fidelity and per-instance controllability at the same time. It also extends the conditional-generation line of work (CTCL) into a general framework with an explicit design space and a systematic ablation methodology, and it documents weaknesses in prior evaluation practice (weak embeddings, short context windows) that can inflate reported quality.

Real-world applications:

• Clinical and biomedical text. The evaluated datasets are bioRxiv scientific abstracts and PMC-patients clinical notes, where DP synthetic text could support model development and analysis without exposing patient records.
• Keyboard and mobile input data. The introduction cites keyboard inputs on mobile devices as a major source of user data that carries privacy risk.
• Recommender systems. Interaction histories are named as another user-data source motivating DP synthetic data reuse.
• Conversational assistant training. Conversational preferences are cited as a data source, and controllable generation (for example, emails with positive sentiment on a given topic) is given as a motivating use case.

Industry relevance. Because DP synthetic data can be reused across downstream tasks without additional privacy cost, it avoids designing a bespoke DP mechanism for every application. The paper positions this as valuable for organizations that want to share or reuse data while keeping existing data pipelines unchanged, and its results target practical constraints: strong privacy budgets (ε = 1 and ε = 4), real-world domain-specific datasets, and generation throughput measured on a single A100.

Future Directions

• Extending beyond the biomedical domain. The authors state that evaluation focuses on bioRxiv and PMC-patients and has not yet covered other fields such as legal (the limitations section is truncated in the provided content).
• Broadening model coverage. The paper reports results on a single model family and two model sizes (gemma-3-1b-pt in the main text and gemma-3-4b-pt in the appendix), and lists the small number of models and sizes as a limitation.
• Further reducing Stage 2 error. Since conditional generation error exceeds feature learning error in the error decomposition, and improving Stage 2 reduces the overall error bound, closing this gap is a natural next target.
• Making Anchored RL cheaper or more widely applicable. The paper notes that Anchored RL is more costly than the rest of the pipeline and is optional, leaving open how to obtain its control benefits at lower compute or how to extend the recipe when no verifiable rubric-like feature is available.

Target Audience

Researchers and practitioners working on privacy-preserving machine learning, DP synthetic data, and DP fine-tuning of language models; applied scientists who need controllable synthetic text for sensitive domains such as healthcare; and engineers building RL or RLHF-style post-training pipelines who are interested in preventing reward hacking without access to private data. Readers without a background in differential privacy or reinforcement learning will find the framework sections accessible but the experimental setup and accounting details demanding.

Authors’ abstract

Generating high-quality synthetic text under differential privacy (DP) is critical for training and evaluating language models without compromising user privacy. Prior work on synthesizing DP datasets often fail to preserve key statistical attributes, suffer utility loss from the noise required by DP, and lack fine-grained control over generation. To address these challenges, we make two contributions. First, we introduce a hierarchical framework that decomposes DP synthetic text generation into two subtasks: feature learning and conditional text generation. This design explicitly incorporates learned features into the generation process and simplifies the end-to-end synthesis task. Through systematic ablations, we identify the most effective configuration: a rich tabular schema as feature, a DP tabular synthesizer, and a DP fine-tuned conditional generator, which we term ACTG (Attribute-Conditioned Text Generation). Second, we propose Anchored RL (ARL), a post-training method that improves the instruction-following ability of ACTG for conditional generation. ARL combines RL to boost control with an SFT anchor on best-of-$N$ data to prevent reward hacking. Together, these components form our end-to-end algorithm ACTG-ARL, which advances both the quality of DP synthetic text (+20% MAUVE over prior work) and the control of the conditional generator under strong privacy guarantees.