On Inherent Privacy of Posterior Sampling: A Unified Rényi-Divergence Framework

AI summary

Authors: Ruyu Zhou (Department of Computer Science and Statistics, University of Rhode Island), Fang Liu (Department of ACMS, University of Notre Dame) arXiv: 2610.19573v1 [stat.ML], 14 Oct 2026

Overview

Research area: Differential privacy (DP) theory and Bayesian posterior sampling, specifically the study of privacy guarantees that arise "for free" from the intrinsic randomness of sampling rather than from injected noise.

Technical level: Advanced. The paper assumes fluency in Rényi divergence, Rényi differential privacy (RDP), zero-concentrated DP, Gaussian DP, exponential families, and tail conditions such as sub-Gaussianity and sub-exponentiality.

Scope (one sentence): The paper develops a unified Rényi-divergence framework that exactly characterizes the privacy loss of One-Posterior-Sampling (OPS) in terms of two posterior exponential moments of the single-record log-likelihood ratio, and derives explicit DP guarantees under uniformly bounded, sub-Gaussian, and sub-exponential tail regimes for a broad class of Bayesian models.

What This Paper Is About

Most differentially private methods achieve their guarantees by deliberately adding calibrated noise to an algorithm's output. The authors study an alternative: when a procedure is already random — as Bayesian posterior sampling is — that built-in randomness can itself provide privacy protection "for free," without external perturbation, which better preserves model utility.

The core problem is that existing guarantees for One-Posterior-Sampling are limited: many require uniformly bounded likelihood ratios, bounded sample or parameter spaces, or modified pseudo-posterior mechanisms, and they often yield loose privacy-loss bounds. The goal of this paper is to characterize the inherent privacy of unmodified OPS exactly and sharply under weaker and more broadly applicable conditions.

Key Contributions

1. An exact characterization of the privacy loss of OPS. The authors prove that the Rényi divergence between the posteriors formed over two neighboring datasets is fully determined by two specific exponential moments — evaluated at parameters −1 and α−1 — of the single-record log-likelihood ratio under one of the two posteriors.

2. Explicit DP guarantees under three tail regimes. They derive concrete bounds under a uniformly bounded log-likelihood ratio, a sub-Gaussian log-likelihood ratio, and (as stated in the abstract and introduction) a sub-exponential log-likelihood ratio, and show the results are compatible with multiple DP notions including RDP, zCDP, GDP, and (ε, δ)-DP.

3. Application to a broader class of Bayesian models than previously studied. The framework is applied to categorical likelihoods with arbitrary priors, the Gaussian–Gaussian conjugate pair, generalized linear models from the exponential family with Gaussian priors, and linear regression with Gaussian likelihood and Gaussian priors on the coefficients.

4. Recovery and tightening of existing results. The bounds recover prior OPS privacy guarantees as special cases — frequently with tighter privacy loss bounds — and show how sample size, model structure, the neighboring relation (bounded or unbounded), and the prior jointly determine the inherent DP guarantee.

Main Findings

• The privacy loss decomposes exactly into two terms. Writing the pointwise privacy loss as c₁(θ) = log[ℓ(θ; Y) / ℓ(θ; Y′)] and the marginal log-likelihood ratio as c₂ = log[p(Y′)/p(Y)], the total loss is c(θ) = c₁(θ) + c₂. The Marginal–Pointwise identity (Lemma 2) states c₂ = log 𝔼_{θ∼π(θ|Y)}[exp(−c₁(θ))], and if c₁(θ) lies in [a, b] for all θ, then c₂ lies in [−b, −a].

• Exact Rényi divergence formula (Theorem 3). For order α > 1, with P = π(θ|Y) and Q = π(θ|Y′): D_α(P‖Q) = c₂ + (α−1)⁻¹ log 𝔼_{θ∼P}[e^{(α−1)c₁(θ)}], equivalently log[(𝔼_{θ∼P}[e^{−c₁(θ)}]) · (𝔼_{θ∼P}[e^{(α−1)c₁(θ)}])^{1/(α−1)}]. This is the foundational result: the divergence depends only on exponential moments of c₁ at t = −1 and t = α−1.

• RDP requires the maximum of both directions (Remark 4). Because d(Y, Y′) = 1 ⟺ d(Y′, Y) = 1, an (α, ε)-RDP guarantee requires sup over neighboring pairs of max{D_α(P‖Q), D_α(Q‖P)} ≤ ε. The reverse direction uses the negated log-likelihood ratio c̃₁(θ) = −c₁(θ) with expectations taken under Q.

• Uniformly bounded ratio: a sharp, shift-invariant, order-dependent bound (Corollary 5). With c₁(θ) ∈ [a, b], Δ = b − a, A = e^{−Δ} ∈ (0,1), and B = e^{(α−1)Δ} > 1, the bound is ε_ub(α, Δ) = log[(α−1)(B−A) / (α(B−1))] + (α−1)⁻¹ log[(B−A) / (α(1−A))]. Two properties follow: the bound is shift-invariant (depends only on the range Δ, not on the particular a and b) and monotonically increasing in Δ (a wider range means weaker privacy). The bound is sharp over all random variables supported on [a, b], though not necessarily attained by an actual neighboring posterior pair.

• Pure DP is the α → ∞ limit, and prior work is a special case (Corollary 6). max{D_α(P‖Q), D_α(Q‖P)} ≤ b − a = Δ = sup_{α>1} ε_ub(α, Δ), so OPS satisfies pure ε-DP with ε = Δ. This recovers results such as Theorem 2 of Dimitrakakis et al. (2014), Theorem 8 of Dimitrakakis et al. (2017), and Theorem 1 of Wang et al. (2015) in the symmetric case a = −b.

• An attainability gap is closed. The authors show Δ is generally unattainable at finite α unless a = b: attaining it would require simultaneously that (α−1)⁻¹ log 𝔼_{θ∼P}[e^{(α−1)c₁(θ)}] = b and c₂ = −a, but c₂ decreases monotonically with c₁(θ), so the first condition forces c₁(θ) = a almost surely while the second requires c₁(θ) = b almost surely. Corollary 5 accounts for this interdependence explicitly.

• Sharper composition accounting. For fixed Δ, expanding as α → ∞ gives ε_ub(α, Δ) = Δ − α⁻¹ log(α(1 − e^{−Δ})) + O(α⁻¹), so ε_ub converges to the pure-DP bound Δ at rate Θ(log α / α). Prior work provides only the static worst-case limit, whereas Corollary 5 gives the full spectrum over all α > 1.

• Categorical likelihood example (Example 7). For bounded neighboring (one record switches category from k to j), c₁(θ) = log(θ_j/θ_k); for unbounded neighboring (one record added or removed), c₁(θ) = log θ_j. Restricting θ_j ∈ [η, 1−η] for η ∈ (0, 1/2) gives range Δ_bounded = 2 log[(1−η)/η] and Δ_unbounded = log[(1−η)/η], so bounded neighboring dominates. At α = 2 the bound reduces to ε_ub(2, Δ) = log([1 + cosh(Δ)]/2). Numerically:

  • η = 0.01: (2, ∼7.804)-RDP and (∞, ∼9.190)-RDP
  • η = 0.1: (2, ∼3.033)-RDP and (∞, ∼4.394)-RDP
  • η = 0.3: (2, ∼0.646)-RDP and (∞, ∼1.695)-RDP

• Sub-Gaussian ratio: linear scaling in α, no pure DP (Corollary 8). If log 𝔼_{θ∼P}(e^{t(c₁(θ)−μ)}) ≤ t²σ²_{Y,Y′}/2 with variance proxy σ²_{Y,Y′} and σ̄² = sup over ordered neighboring pairs, then max{D_α(P‖Q), D_α(Q‖P)} ≤ ε_subGaus(α, σ̄²) = ασ̄²/2. Because this grows linearly in α and diverges as α → ∞, the sub-Gaussian regime yields no pure DP guarantee, but it maps directly onto relaxed notions: ρ-zCDP with ρ = σ̄²/2, μ-GDP with μ = σ̄, and (ασ̄²/2 + log(1/δ)/(α−1), δ)-DP.

• Interpretation as a Gaussian mechanism (Claim). Since the Gaussian mechanism with scale σ² satisfies (α, α/(2σ²))-RDP, the authors state that when c₁(θ) is sub-Gaussian, OPS's inherent guarantee is conceptually equivalent to applying the Gaussian mechanism with scale parameter σ̄^{−2}.

• Generalized linear models with Gaussian priors (Proposition 9). Under a bounded record-level score condition ‖∇_β log p(y_i|x_i, β)‖ ≤ L and prior π(β) = N(0, τ²I_d):

  • Bounded neighboring: c₁(β) is sub-Gaussian with variance proxy σ²_{Y,Y′} ≤ 4L²τ², giving ε_GLM,bounded(α) ≤ 2αL²τ², i.e. a 4-fold increase relative to the unbounded case. This matches the strongly log-concave Lipschitz-perturbation analysis of Gopi et al. (2022).
  • Unbounded neighboring: σ²_{Y,Y′} ≤ L²τ², giving ε_GLM,unbounded(α) ≤ αL²τ²/2.
  • Taken together, OPS satisfies (α, 2αL²τ²)-RDP.

• Dependence structure of the GLM bound. The bound scales quadratically in the score constant L (so records with large score norm substantially weaken the guarantee, consistent with the practice of clipping and normalization) and linearly in the prior variance τ² (a less informative prior lets the likelihood dominate, weakening the inherent guarantee).

• Neighboring relation can change the qualitative guarantee. The paper reports that the choice of bounded versus unbounded neighboring relation can fundamentally affect the tail behavior of c₁(θ), leading to qualitatively different inherent privacy guarantees; under unbounded neighboring, the two directions of the Rényi divergence can differ even in their range of finite orders.

Methodology in Plain English

The authors start from the standard Bayesian posterior formula and write down the privacy loss random variable for releasing a single posterior draw θ. Because the prior cancels when the posteriors over two neighboring datasets are divided, the loss splits neatly into two pieces: a pointwise part c₁(θ) that measures how the likelihood of that specific θ changes between the two datasets, and a marginal part c₂ that measures how the total evidence changes.

They then prove that c₂ is itself the logarithm of an exponential moment of c₁ under the observed posterior, which links the two pieces together. Substituting this into the definition of Rényi divergence produces a single exact expression in which the divergence of any order α depends only on two exponential moments of c₁.

The rest of the paper is a program of bounding those moments under successively weaker assumptions. Each assumption — uniform boundedness of c₁, then sub-Gaussianity, then sub-exponentiality — corresponds to a different family of likelihoods and yields a different style of DP guarantee. Because the exact expression holds for any ordered pair of neighboring datasets, they must bound it in both directions, so they also analyze the reverse divergence with the negated log-likelihood ratio. Finally, they instantiate the general bounds on concrete models by computing what c₁ looks like for categorical data, conjugate Gaussian models, GLMs, and Bayesian linear regression, and by inserting the resulting ranges or variance proxies into the general formulas. Conversion results from the DP literature let the RDP bounds be read off as zCDP, GDP, or (ε, δ)-DP guarantees.

Why This Matters

Impact on research. The work reframes inherent privacy of posterior sampling as a computable problem about exponential moments rather than a worst-case boundedness assumption. It provides sharper, order-dependent bounds that are more useful for composition-based privacy accounting, unifies several earlier OPS results as special cases (Dimitrakakis et al. 2014, 2017; Wang et al. 2015; Geumlek et al. 2017), and extends coverage to model classes such as GLMs with Gaussian priors and categorical likelihoods with arbitrary priors that were not previously handled under a single framework. It also clarifies the role of the neighboring relation: bounded versus unbounded neighboring can produce different worst-case behavior, and the 4-fold difference in the GLM case is a concrete quantification of that effect.

Real-world applications (as implied by the models studied):

• Categorical and count data such as survey responses, disease codes, or product categories modeled with Categorical or Dirichlet–Categorical likelihoods, where the paper gives explicit (α, ε)-RDP values for bounded and unbounded neighboring.
• Synthetic data generation and randomized prediction. A single posterior draw defines a complete randomized predictor or synthetic-data generator, and any number of predictions or synthetic records computed from that draw inherit its privacy guarantee by post-processing, with no composition cost.
• Generalized linear models applied to sensitive tabular data — for example logistic or Poisson regression with Gaussian priors on the coefficients — where the guarantee depends on a bound on the record-level score norm and on the prior variance.
• Bayesian linear regression with Gaussian likelihood and Gaussian priors on the coefficients, a standard setting for releasing regression summaries or predictions from private data.

Industry relevance. The framework gives practitioners a way to quantify how much privacy a Bayesian workflow provides without adding noise, which matters for utility-sensitive pipelines. The identified dependencies are directly actionable: because the guarantee worsens quadratically in the score constant L, clipping and normalizing records is justified by the theory, and because it worsens linearly in the prior variance τ², choosing a more informative prior strengthens the inherent guarantee. The sub-Gaussian result is especially convenient for practice because it maps onto ρ-zCDP and μ-GDP, which are already implemented in privacy accounting libraries.

Future Directions

1. Full characterization of the sub-exponential regime. The abstract and introduction list sub-exponential tail behavior as one of the three regimes covered, but the provided body text does not present the sub-exponential results, so their explicit bounds, the models to which they apply, and whether the bounds diverge as α grows are not reported in the available content.

2. Completing the Gaussian-likelihood analysis. Section 5.2 is announced as covering the Gaussian–Gaussian conjugate model and Bayesian linear regression with Gaussian priors, but the provided text is truncated before these results are given. How the bounds depend on sample size and the prior in those models is not reported here.

3. Tightening the gap between the sharp bound and what valid neighboring pairs attain. The authors state that ε_ub(α, Δ) is sharp over all random variables supported on [a, b] but is not necessarily realized by a valid neighboring posterior pair. Characterizing which posterior pairs actually attain the bound would sharpen the guarantees further.

4. Extending the framework beyond the models considered. The paper notes prior work on collapsed Gibbs sampling for latent Dirichlet allocation, stochastic gradient MCMC, β-divergence-based sensitivity control, per-instance DP for linear regression, and posterior sampling under contamination models. Applying the exponential-moment framework to those procedures is a natural extension, as is empirically comparing the predicted RDP profile against observed privacy loss in real Bayesian pipelines.

Target Audience

Researchers and graduate students in differential privacy, Bayesian statistics, and statistical machine learning who already know DP and Rényi divergence and want sharper, more general guarantees for posterior-sampling mechanisms. It will also be useful to applied practitioners working with Bayesian models on sensitive data who need to decide whether posterior sampling alone provides adequate privacy or whether additional noise injection is required, and to industry privacy engineers who need to translate RDP or zCDP numbers into auditable guarantees for GLM-based or count-data workflows.

Authors’ abstract

Differential Privacy (DP) provides a rigorous framework for quantifying privacy guarantees. While most methods achieve DP by injecting calibrated random noise, the intrinsic randomness of certain procedures such as sampling can yield DP guarantees ``for free''. In this work, we develop a unified Rényi divergence framework to characterize the inherent DP guarantees of sampling from a Bayesian posterior distribution. We show that the privacy loss incurred by One-Posterior-Sampling (OPS) is governed by posterior exponential moments of the single-record log-likelihood ratio, and establish explicit DP guarantees under uniformly bounded, sub-Gaussian, or sub-exponential tail regimes. We apply the framework to representative Bayesian models, including categorical likelihoods with arbitrary priors, Gaussian-Gaussian conjugate pair, generalized linear models from the exponential family and linear regression with Gaussian likelihood, both with Gaussian priors on the regression coefficients. Our results elucidate sample size, model structure, neighboring relations in DP (bounded or unbounded), and how prior if applicable, jointly determine the inherent DP guarantees of OPS. Furthermore, our results recover the existing DP guarantees for OPS as special cases -- often with tighter privacy loss bounds -- and substantially broaden the class of Bayesian models for which the inherent DP guarantees of OPS can be rigorously characterized.