DP-AdamW: Investigating Decoupled Weight Decay and Bias Correction in Private Deep Learning

AI summary

Overview

Research area: Differential privacy and privacy-preserving deep learning, specifically the design of differentially private optimizers for training neural networks.

Technical level: Intermediate. The algorithms and experiments are accessible; Sections 3.2 and 3.3 contain advanced convergence analysis with weighted-norm inequalities that will be most readable for readers comfortable with optimization theory.

One-sentence scope: The paper introduces two new differentially private optimizers, DP-AdamW and DP-AdamW-BC, proves privacy and convergence guarantees for them, and benchmarks them against DP-SGD, DP-Adam and DP-AdamBC on image, text and graph node classification.

What This Paper Is About

Differentially private training is usually built on DP-SGD, which clips per-sample gradients and adds Gaussian noise, but its empirical performance lags behind the Adam-family optimizers that dominate non-private deep learning. This paper asks whether AdamW's key design choice, decoupling weight decay from the adaptive gradient update, transfers to the private setting, and whether correcting for the DP noise bias in Adam's second moment estimator (as DP-AdamBC does) still helps once decoupled weight decay is used.

Key Contributions

1. DP-AdamW: A differentially private AdamW optimizer that applies DP-SGD-style clipping and Gaussian noise to the gradient and then feeds the noised gradient into the AdamW update, with weight decay applied directly to the parameters instead of through the gradient.

2. DP-AdamW-BC: A variant that additionally subtracts the DP bias term Φ = (σC/B)² from the second moment estimate, replacing the update denominator with the square root of max(v̂_t − (σC/B)², γ), where γ is a variance floor.

3. Theory: A privacy theorem (Theorem 3.1) showing both optimizers inherit the same (ε, δ)-DP guarantee and privacy analysis φ(T, θ_i) as DP-SGD and DP-AdamW-BC's predecessor, via the adaptive postprocessing property, and two convergence theorems (3.5 for β₁ = 0 and 3.6 for general 0 < β₁ < 1) under bounded-below objective, bounded gradients (‖∇f_t(θ)‖ ≤ C₁ ≤ C) and L-Lipschitz gradient assumptions.

4. Empirical benchmark: A head-to-head evaluation of DP-AdamW and DP-AdamW-BC against DP-SGD, DP-Adam and DP-AdamBC on CIFAR-10 image classification, QNLI text classification and obgn-arxiv graph node classification at multiple privacy budgets.

Main Findings

• DP-AdamW beats DP-Adam everywhere: It outperforms DP-Adam across all tested privacy budgets and tasks, with the paper highlighting the moderate privacy setting ϵ = 3 as a particular strength.

• Headline gains over prior state of the art: The abstract reports that DP-AdamW scores over 15% higher than existing DP optimizers on text classification, up to 5% higher on image classification, and consistently 1% higher on graph node classification.

• Text classification (QNLI, BERT fine-tuning, mean over five random seeds): DP-AdamW reaches 77.78 (0.13), 79.26 (0.18) and 80.01 (0.19) at ϵ ≈ 1, 3 and 7, against DP-AdamBC's 58.32 (1.90), 61.42 (0.99) and 62.83 (1.60), and DP-SGD's 57.10 (1.59), 58.85 (1.20) and 58.29 (0.92). The paper states these are the best published DP optimizer fine-tuning results on QNLI to the authors' knowledge.

• Image classification (CIFAR-10, CNN): DP-AdamW reaches 52.59 (0.44) at ϵ ≈ 1, 59.26 (0.30) at ϵ ≈ 3 and 63.25 (0.53) at ϵ ≈ 7, against DP-Adam's 51.89 (0.69), 54.08 (0.41) and 62.24 (0.10). DP-SGD scores 52.37 (0.50), 57.30 (0.76) and 65.30 (0.33). The largest gain is at ϵ ≈ 3, where DP-AdamW and DP-AdamW-BC beat DP-Adam and DP-AdamBC by up to 5%. DP-AdamW is state of the art among the evaluated DP-SGD and DP-Adam variants at ϵ = 1 and 3, but DP-SGD remains highest at ϵ ≈ 7 (65.30).

• Graph node classification (obgn-arxiv): DP-AdamW scores 50.20 (0.75), 53.41 (0.37) and 54.78 (0.24) at ϵ ≈ 3, 6 and 12, versus DP-Adam's 46.55 (0.54), 51.98 (0.48) and 54.02 (0.18). DP-AdamBC is best at ϵ ≈ 3 (50.51, 0.56), while DP-AdamW and DP-AdamW-BC both beat DP-AdamBC at ϵ ≈ 12 and DP-AdamW beats it at ϵ ≈ 6.

• Bias correction hurts under decoupled weight decay: DP-AdamW-BC trails DP-AdamW on every task and privacy budget, in contrast to DP-AdamBC's reported improvement over DP-Adam. On text it trails by 2–3 percentage points; on CIFAR-10 it reaches 51.43 (0.52), 58.16 (0.72) and 62.01 (0.29); on obgn-arxiv it reaches 48.99 (0.67), 52.89 (0.40) and 54.53 (0.43).

• Accuracy rises with looser privacy: For all optimizers tested, test performance improves as ϵ increases, which the authors attribute to less noise being required.

• Convergence rate: With learning rate η = T^(−a) and β₂ = 1 − T^(−b) for b < 1 and a = 2/3, the average squared gradient converges as O(T^(−1/3+ζ) ln T) → 0, where ζ is arbitrarily small; the same bound holds for DP-AdamW-BC.

• Proposed explanation for the bias-correction regression: The correction term can drive the denominator v̂_t − Φ very small, clamping to γ and effectively freezing the adaptive schedule; combined with decoupled decay this produces large parameter steps that are not offset inside Adam's update. On text, the authors hypothesize the added variance floor γ interacts poorly with the small effective batch size and the already low intrinsic gradient noise of BERT fine-tuning.

Methodology in Plain English

The researchers take the DP-Adam recipe — clip each per-sample gradient to norm C, average, and add Gaussian noise of scale σC/B — and use that noised gradient as the input to AdamW instead of Adam. The only algorithmic difference from DP-Adam is that weight decay is applied directly to the parameters (the update adds λθ_{t−1} outside the adaptive term) rather than being folded into the gradient. A second variant subtracts the known noise variance (σC/B)² from the second moment estimate before taking the square root, guarded by a floor γ to avoid division by near-zero values.

For theory, they reuse the assumptions and proof outline from prior work on DP-AdamBC (bounded objective, bounded gradients, Lipschitz gradient) and add a new step: bounding the expected magnitude of the parameters, E‖θ_i‖², to account for the weight decay term. Privacy is argued by observing that the optimizer's updates use no private information beyond the noised gradients, so the postprocessing property of differential privacy carries the DP-SGD guarantee over unchanged.

For experiments they replicate the earlier DP-Adam/DP-AdamBC setup on the same CNN architecture and CIFAR-10 task, check that their DP-Adam and DP-AdamBC reproductions match the reported numbers, and then swap in the new optimizers. They evaluate at ϵ = 1, 3 and 7 for image and text and at ϵ ≈ 3, 6 and 12 for graph, reporting means and standard deviations (five random seeds for text) and tuning hyperparameters per privacy budget. Full experimental details are stated to be in Appendix C, which is not reproduced in the content available here; dataset sizes and the exact hyperparameter values are therefore not reported in this summary.

Why This Matters

Impact on research. The paper is the first to test whether AdamW's decoupling of weight decay survives the addition of DP noise, and it produces a counterexample to the prior claim that DP bias correction is broadly beneficial: the same correction that helps DP-Adam consistently harms DP-AdamW. It also supplies convergence proofs for the weight-decayed private setting, and notes that the same proof method can be imitated to show convergence for certain cases of non-private AdamW, which it flags as potentially of independent interest.

Real-world applications.

• Healthcare, which the paper's impact statement names explicitly as a sector where sensitive patient data could be used with stronger safeguards.
• Finance, also named in the impact statement as a domain that stands to gain from private training.
• Fine-tuning large language models on sensitive text, since the text experiments use pretrained BERT on QNLI, the setting where the gains are largest.
• Image classification and graph node classification on sensitive data, the other two benchmark families studied.

Industry relevance. Because DP-AdamW changes only how weight decay is applied and inherits the same privacy accounting as DP-SGD, it is a drop-in replacement for existing private optimizers in a training pipeline, requiring no change to clipping, noise calibration or privacy accounting. The reported accuracy improvements at tight budgets (ϵ = 1 and 3) are the regimes where commercial and regulated deployments typically operate.

Future Directions

• Explain the bias-correction anomaly. The authors call for further investigation into the interaction between DP noise, adaptive moment estimation, bias correction and regularization, and specifically into whether the γ variance floor is responsible for the degradation.
• Reconcile bias correction with decoupled weight decay. Can the correction be redesigned so it does not push the update denominator toward γ, or should it be dropped whenever decoupled decay is the primary regularizer?
• Broaden the empirical base. The current study covers one dataset and architecture per modality (CIFAR-10 CNN, QNLI with BERT, obgn-arxiv). Larger models, more tasks and different architectures are open questions, and the paper states that further avenues of work are offered in its concluding sections.
• Extend the theory. The convergence bound reaches a specific asymptotic rate only for a narrow hyperparameter regime (a = 2/3, b → 1⁻). Tighter rates, and formal convergence results for non-private AdamW via the same parameter-bounding technique, remain open.

Target Audience

Researchers and practitioners working on differentially private machine learning who are choosing or designing optimizers for private training. It is most useful to readers who already know DP-SGD and the Adam/AdamW update rules, and who want a concrete, benchmarked answer about whether decoupled weight decay and DP bias correction compose well. Readers interested only in the empirical comparison can skip Sections 3.2 and 3.3; readers interested in optimization theory will find the convergence proofs in Appendix B the novel formal content.

Authors’ abstract

As deep learning methods increasingly utilize sensitive data on a widespread scale, differential privacy (DP) offers formal guarantees to protect against information leakage during model training. A significant challenge remains in implementing DP optimizers that retain strong performance while preserving privacy. Recent advances introduced ever more efficient optimizers, with AdamW being a popular choice for training deep learning models because of strong empirical performance. We study \emph{DP-AdamW} and introduce \emph{DP-AdamW-BC}, a differentially private variant of the AdamW optimizer with DP bias correction for the second moment estimator. We start by showing theoretical results for privacy and convergence guarantees of DP-AdamW and DP-AdamW-BC. Then, we empirically analyze the behavior of both optimizers across multiple privacy budgets ($ε= 1, 3, 7$). We find that DP-AdamW outperforms existing state-of-the-art differentially private optimizers like DP-SGD, DP-Adam, and DP-AdamBC, scoring over 15\% higher on text classification, up to 5\% higher on image classification, and consistently 1\% higher on graph node classification. Moreover, we empirically show that incorporating bias correction in DP-AdamW (DP-AdamW-BC) consistently decreases accuracy, in contrast to the improvement of DP-AdamBC improvement over DP-Adam.