Beyond One-Size-Fits-All: Neural Networks for Differentially Private Tabular Data Synthesis

AI summary

Overview

• Research area: Differentially private (DP) tabular data synthesis, comparing statistical/graphical-model approaches against neural network (NN) generators.
• Technical level: Advanced. The paper assumes familiarity with differential privacy definitions, zero-Concentrated DP (zCDP), the Gaussian and exponential mechanisms, and probabilistic graphical models (PGMs). The plain-language sections below are written to be readable without that background.
• Scope: The paper proposes MargNet, a feature-DP neural synthesis algorithm that trains a generator against adaptively selected, privatized full marginals, and argues that the choice between statistical and NN-based synthesis should depend on how densely correlated a dataset is rather than being one-size-fits-all.

Authors: Kai Chen, Chen Gong, and Tianhao Wang (University of Virginia, Charlottesville, USA). Venue: Proceedings of the 2026 ACM SIGSAC Conference on Computer and Communications Security (CCS '26), November 15–19, 2026, The Hague, Netherlands. DOI 10.1145/3830454.3846700. Source code released at https://github.com/KaiChen9909/margnet.

What This Paper Is About

The prevailing consensus in DP tabular data synthesis is that statistical models—especially AIM—outperform neural network methods. The authors argue this conclusion was largely drawn from datasets with sparse correlations, and that it misses a fundamental difference: AIM explicitly parameterizes the joint distribution over small cliques in a graph, while neural networks estimate a distribution by generating samples directly. When correlations are densely distributed, graphical modeling becomes expensive or intractable, and NN-based synthesis can become competitive. The paper's goal is to build an NN-based method that better exploits the generative capacity of neural networks, and to evaluate both paradigms across sparse- and dense-correlation regimes.

Key Contributions

1. A new NN-based synthesis algorithm, MargNet. It unifies adaptive full-marginal selection and differentiable marginal fitting in a single training pipeline. Rather than relying on noisy gradient privatization (as in DP-SGD methods) or weaker feature supervision, MargNet uses privatized full marginals as adaptive training targets, so the generator receives stronger distributional supervision under the same privacy constraints.

2. A broader empirical evaluation across data characteristics. The authors systematically compare statistical and NN-based synthesizers on both sparsely correlated and densely correlated datasets, covering different dimensionalities and correlation structures, and explicitly examine how relative performance changes across correlation regimes.

3. A more nuanced perspective on algorithm selection. The results indicate that the relative strengths of statistical and NN-based methods depend strongly on dataset characteristics. Prior conclusions about the superiority of graphical-model methods are described as primarily derived from sparsely correlated benchmarks, while NN-based synthesis is especially competitive in the densely correlated regimes studied.

Main Findings

• AIM remains strong on sparsely correlated data: On sparsely correlated datasets, the prior state-of-the-art statistical baseline AIM continues to achieve high utility.
• MargNet is comparable with a speedup on sparse data: In that same regime, MargNet achieves performance comparable to AIM with a significant speedup. The paper's abstract states the speedup is significant but the provided content does not report its magnitude.
• MargNet wins in most dense-correlation settings: On densely correlated datasets, MargNet achieves the best synthesis utility in most settings.
• Graphical models degrade under dense correlation: The authors illustrate with an example of six attributes where seven marginals yield three cliques of at most three attributes, versus a denser case where PMG is forced into cliques of four to five attributes. Assuming a domain size of 100 per attribute, the state space of the largest clique grows from 10^6 to 10^10, which the paper calls computationally infeasible to estimate.
• Prior evidence was suggestive but incomplete: The authors cite Ganev et al. (2024), which showed statistical models including MST and PrivBayes fail to capture distributions in high-dimensional settings (e.g., d ≥ 128). The authors argue this evidence is not conclusive because it omits stronger graphical models like AIM and feature-based NN approaches like GEM and DP-MERF, focuses on dimensionality rather than correlation distribution, and proposes no algorithmic improvement.
• Diagnosed weaknesses of earlier NN methods: PATE-GAN provides only binary "yes"/"no" voting signals; DP-MERF uses random Fourier feature mean embeddings that require infinite dimensions to capture a distribution accurately but are truncated in practice; GEM allocates budget to each selected marginal query individually instead of measuring the full marginal once and reusing the statistics, and follows a predetermined number of feature-selection rounds rather than adapting to the remaining budget. DP-SGD-based methods face a trade-off between optimization quality and privacy, limiting iterations and injecting noisy gradients.
• Theoretical fitting error analysis: The paper includes a theoretical evaluation of MargNet's fitting error from two angles, selected marginal errors and unselected marginal errors. The provided content is truncated at the start of Section 5.1, so the results of that analysis are not reported here.
• Experimental specifics are not in the provided content: Dataset names and sizes, utility metrics, and the numeric results behind the sparse/dense comparisons are not included in the truncated text, so no figures are given here.

Methodology in Plain English

MargNet follows a feature-DP paradigm: instead of privatizing gradients during training, it first extracts and privatizes statistical summaries of the data (marginals, i.e., frequency counts of attribute combinations), then trains a neural generator to match those noisy summaries.

The training loop has four stages. First, the total privacy budget ρ is divided into a selection budget ρ_s and a measurement budget ρ_m, controlled by a hyperparameter c that caps the number of selection steps. In the version presented, ρ_s = 0.1ρ/c and ρ_m = 0.9ρ/c. Second, a warm-up step measures all one-way marginals with Gaussian noise of scale 1/sqrt(2ρ_m) and fits the model on them. Third, an iterative loop selects two-way marginals and retrains the model. Fourth, the fitted model generates synthetic data.

Marginal selection uses the exponential mechanism with a score that trades off expected estimation improvement against expected noise: q_i = r_i(||M_i(G^{k−1}) − M_i||_1 − n_i/sqrt(πρ_m)), where the first term is how far the current generator's marginal is from the real one and the second subtracts the expected ℓ1 norm of DP noise. The paper's Theorem 1 states that the sensitivity of this score is r_i; r_i is set to 1.0 in the experiments. The selection procedure then measures the chosen marginal with Gaussian noise and adds it to the training set.

Budget management is adaptive. If the improvement from a newly selected marginal falls below the expected noise error and that marginal is selected for the first time, MargNet doubles ρ_m and ρ_s. This first-time-only condition is the authors' refinement over prior work; it prevents repeated budget escalation for the same marginal and preserves budget for additional iterations. If the remaining budget cannot cover another iteration plus its measurement, the leftover is allocated to a final round that terminates the loop.

Training is differentiable. Samples from the generator represent categorical values as soft probability distributions rather than discrete one-hots; the model averages these soft probabilities across a batch to obtain empirical marginal frequencies, and scales them into a marginal estimate. The loss is a weighted squared error between the generated marginals and the privatized targets: L(θ,S) = Σ w_i ||M_i(D') − M̃_i||²_F. Weights are designed so more reliable (less noisy) measurements get higher weight, and newly selected marginals get an extra factor of d, the dimensionality, to counteract signal dilution as the number of training targets grows: w_i ∝ sqrt(ρ_{m,i}) for previously selected marginals and d·sqrt(ρ_{m,i}) for newly selected ones.

Privacy accounting uses zCDP, chosen for consistency with prior work. The Gaussian mechanism provides 1/(2σ²)-zCDP and the exponential mechanism provides ε²/8-zCDP; composition and post-processing properties let the pieces be combined. Theorem 2 states that MargNet satisfies ρ-zCDP for any ρ > 0, with the proof based on the privacy filter and given in Appendix B of the paper. The authors acknowledge that zCDP can give less tight privacy bounds and less flexibility than more refined accounting frameworks in certain settings.

Why This Matters

Impact on research. The paper pushes back on a settled-seeming consensus in the DP tabular synthesis literature. Rather than claiming NN methods are universally better, it argues that the correct comparison depends on correlation density, and it supplies an algorithm designed to make the NN case competitive. This reframes benchmark conclusions as regime-specific and gives future evaluations a reason to report correlation structure alongside dimensionality.

Real-world applications. The paper's own motivation centers on sharing useful synthetic datasets with quantifiable privacy guarantees. Specific applications are not enumerated in the provided content beyond the following; the list below reflects the settings the paper discusses or directly implies:

• Releasing synthetic versions of tabular records held by researchers and practitioners, as the paper describes the core use case for DP tabular generation.
• IoT and sensor-based data, which the paper cites (Reiss, 2012) as an example of datasets whose attributes are densely correlated with each other.
• Domains where statistical/graphical synthesizers become computationally intractable because the dependency graph turns dense—the paper's motivating failure mode for AIM-style PGM modeling.
• Workloads where synthesis must be fast as well as accurate, given that MargNet offers a significant speedup over AIM in the sparse-correlation regime.

Industry relevance. Organizations with tabular data often cannot share raw records. The paper's message that no single algorithm dominates across data regimes is directly useful for practitioners choosing a synthesizer: an AIM-style statistical method may remain the right pick for sparse-correlation workloads, while a NN-based method such as MargNet may be preferable when correlations are dense. The released source code lowers the barrier to reproducing and adopting the method.

Future Directions

• Characterizing the crossover point. The paper shows performance depends on correlation regime, but the provided content gives no formal criterion for deciding when a dataset is "dense enough" that NN synthesis becomes the better choice. A decision rule derived from measurable dataset statistics is a natural next step.
• Understanding the theoretical error terms. Section 5 sets up an analysis of selected and unselected marginal errors, but the provided content is truncated before any results. Completing and validating that analysis against empirical error would strengthen the paper's claims.
• Extending beyond two-way marginals. The adaptive selection loop as presented selects two-way marginals, with all one-way marginals measured up front. Whether the framework scales to higher-order marginals in very densely correlated data is an open question raised by the design.
• Tighter privacy accounting. The authors explicitly note that zCDP may give less tight bounds and less flexibility than refined accounting frameworks in some settings, leaving room for improved accounting within the same algorithmic structure.
• Broadening the data regimes studied. The empirical evaluation covers dimensionality and correlation structure; the paper does not report how other data characteristics (for example, domain sizes or mixed attribute types) shift the comparison, which the provided content leaves unaddressed.

Target Audience

Researchers and graduate students working on differential privacy, private data synthesis, and tabular generative modeling will get the most from this paper, particularly those who treat AIM as the default state of the art and want a counterpoint. Practitioners who must select a synthesizer for a real tabular dataset—especially ones with densely correlated attributes such as sensor or IoT data—will find the regime-dependent conclusion and the released implementation practically useful. Readers without a background in DP mechanisms, zCDP, and graphical models will need to work through Sections 2 and 3 before the algorithm's design choices make sense.

Authors’ abstract

In differentially private (DP) tabular data synthesis, the consensus is that statistical models are better than neural network (NN)-based methods. However, we argue that this conclusion is incomplete and overlooks the challenge of densely correlated datasets, where intricate dependencies can overwhelm statistical models. In such complex scenarios, neural networks are more suitable due to their capacity to fit complex distributions by learning directly from samples. Despite this potential, existing NN-based algorithms still suffer from significant limitations. We therefore propose MargNet, incorporating successful algorithmic designs of statistical models into neural networks. MargNet applies an adaptive marginal selection strategy and trains the neural networks to generate data that conforms to the selected marginals. On sparsely correlated datasets, our approach achieves utility close to the best statistical method while offering an average 7$\times$ speedup over it. More importantly, on densely correlated datasets, MargNet establishes a new state-of-the-art, reducing fidelity error by up to 26\% compared to the previous best. We release our code on GitHub.\footnote{https://github.com/KaiChen9909/margnet}