Enhancing DPSGD via Per-Sample Momentum and Low-Pass Filtering

AI summary

Overview

Research area: Differentially private machine learning — specifically, training deep neural networks with Differentially Private Stochastic Gradient Descent (DPSGD).

Technical level: Advanced. The paper combines a new optimizer design with convergence proofs and privacy accounting, and assumes familiarity with DP, gradient clipping, and non-convex optimization analysis.

Scope in one sentence: It proposes DP-PMLF, a DPSGD variant that pairs per-sample momentum with a post-processing low-pass filter to reduce DP noise and clipping bias at the same time, and validates it on image and sentence classification benchmarks.

What This Paper Is About

DPSGD gives formal privacy guarantees when training neural networks, but it hurts accuracy through two mechanisms: Gaussian noise injected to satisfy differential privacy, and bias introduced when per-sample gradients are clipped to a threshold. These two problems pull against each other — shrinking the clipping threshold lowers the injected noise but raises the clipping bias, and enlarging it does the reverse. This paper argues that both can be attacked simultaneously, because clipping bias is proportional to sampling variance, which per-sample momentum can reduce, while a low-pass filter can strip high-frequency DP noise from the aggregated signal without spending extra privacy budget.

Key Contributions

1. A combined method (DP-PMLF) that integrates per-sample momentum with low-pass filtering in a single DPSGD variant. The authors state this is the first approach to consider reducing DP noise and clipping bias at the same time.
2. A theoretical convergence result claiming faster convergence than vanilla DPSGD under stated assumptions (L-smoothness, bounded variance, bounded gradient, gradient auto-correlation, and independent sampling noise), alongside a proven (ε, δ) privacy guarantee.
3. A variance-reduction analysis showing that per-sample momentum shrinks gradient estimation variance by a factor ρ² that grows with the momentum factor β and momentum length k.
4. Empirical evaluation across four image datasets (MNIST, Fashion-MNIST, CIFAR-10, CIFAR-100) and four GLUE sentence classification datasets (MNLI, QNLI, QQP, SST-2), using a 5-layer CNN, ResNet-18, ViT, and a fine-tuned RoBERTa-base model, compared against vanilla DPSGD, LP-DPSGD, and InnerOuter.

Main Findings

• LP-DPSGD can underperform vanilla DPSGD. On CIFAR-10 with a 5-Layer CNN over 100 epochs at ε = 8, the authors report that LP-DPSGD does worse than plain DPSGD because the additional bias introduced by the low-pass filter outweighs its noise-suppression benefit (Figure 1a).
• InnerOuter fails when DP noise dominates. On CIFAR-10 at ε = 1, InnerOuter's outer momentum accumulates DP noise instead of separating it from the true gradient signal, and the authors note it lacks a theoretical proof of convergence (Figure 1b).
• Best ViT results across datasets and privacy budgets. In Table 1 (ViT; epochs = 25 for MNIST and Fashion-MNIST, 50 for CIFAR-10 and CIFAR-100), DP-PMLF reaches 80.65% ± 1.17 at ε = 1 and 81.93% ± 0.83 at ε = 8 on Fashion-MNIST, and 40.96% ± 1.18 / 51.47% ± 0.33 on CIFAR-10, versus 35.74% ± 0.26 / 47.74% ± 1.20 for DPSGD and 35.84% ± 0.63 / 48.37% ± 0.36 for LP-DPSGD.
• Large margins on CIFAR-100. DP-PMLF records 11.40% ± 0.21 at ε = 1 and 23.15% ± 0.52 at ε = 8, which the authors describe as a 4–5% margin over the next-best baselines at both privacy budgets.
• MNIST and Fashion-MNIST gains over DPSGD. On MNIST, DP-PMLF achieves 92.16% ± 0.05 at ε = 1 and 92.39% ± 0.07 at ε = 8, compared with 89.00% ± 0.06 and 88.95% ± 0.01 for DPSGD and 92.15% ± 0.15 / 92.43% ± 0.06 for InnerOuter.
• Consistent wins across architectures. On CIFAR-10 at ε = 1 (Figure 2), DP-PMLF reaches approximately 47% with CNN-5 — around 9% above the best baseline — nearly 50% with ResNet-18 (1–2% higher than the strongest competitor), and about 31% with ViT versus 23% for the best baseline.
• Strong gains on GLUE. In Table 2, at ε = 1 DP-PMLF leads baselines by over 4% on MNLI (56.81% ± 0.74) and nearly 3% on QNLI (72.38% ± 0.62). At ε = 8 it remains competitive or better (MNLI 75.56% ± 0.42, QNLI 86.96% ± 0.69, QQP 83.42% ± 0.52, SST-2 90.39% ± 1.03).
• Both components matter. Ablation studies (Figure 3, MNIST and CIFAR-10 on CNN-5 and ResNet-18, ε from 1 to 8) show DP-PMLF beats the version without per-sample momentum at all tested ε values, and beats the version without the low-pass filter on MNIST.
• One regime where smoothing hurts. On CIFAR-10, when DP noise is relatively small (ε > 6), excessive smoothing loses true gradient information and DP-PMLF falls about 0.5–0.7% below the variant without the low-pass filter.
• Privacy is preserved. Theorem 2 shows DP-PMLF is (ε, δ)-DP when the noise scale satisfies σ_DP ≥ c₂ q √(T log(1/δ)) / ε, with the filter applied as a post-processing step that consumes no additional budget.

Methodology in Plain English

The method changes what gets clipped and what gets filtered.

1. Per-sample momentum before clipping. Instead of clipping the raw gradient of each sample, DP-PMLF first forms an exponentially weighted average of that sample's gradients over the previous k iterations, normalized so the weights sum to one. Averaging over history reduces the sampling variance, and since clipping bias scales with sampling variance, the bias shrinks too.
2. Clip, aggregate, add noise. Each sample's momentum is clipped to a threshold C, the clipped momenta in the mini-batch are averaged, and Gaussian noise calibrated to the privacy budget is added.
3. Low-pass filter as post-processing. A linear filter with coefficients {a_r} and {b_r} is applied to the noisy aggregated momentum, suppressing high-frequency components where DP noise is spread evenly while retaining the low-frequency component where the true gradient lives. Because filtering happens after the noisy release, it is a post-processing operation and does not consume extra privacy budget. The coefficients are constrained so the filter preserves the signal mean.
4. Bias correction. A normalization term c_{m,t} is computed recursively and used to divide the filter output, correcting for initialization effects of the filter.
5. Model update. Parameters are updated with the corrected, filtered momentum scaled by the learning rate.
6. Validation. The authors combine a standard descent lemma with their filter representation and variance-reduction lemma to bound the expected squared gradient norm, then run image and sentence classification experiments, each repeated five times, reporting mean and standard deviation against DPSGD, LP-DPSGD, and InnerOuter.

Dataset sizes, the full hyper-parameter settings, and additional results are stated in the paper as being in the appendix.

Why This Matters

For research, the paper reframes the DPSGD utility problem: rather than trading noise against clipping bias, it shows a route to attack both, and it supplies a convergence bound where the clipping-bias term is modulated by a variance-reduction factor ρ. It also revisits the low-pass filter idea from LP-DPSGD and clarifies when that idea backfires, which sharpens the community's understanding of the noise-bias interplay.

Real-world applications, following the domains the paper cites as places deep learning needs sensitive data:

• Medical diagnosis, where patient records are the training data.
• Recommendation systems, which model individual user behavior.
• Autonomous driving, which relies on large volumes of sensor and driving data.
• Sentence classification and other natural language tasks, where text is fine-tuned from a pre-trained model (the paper demonstrates this with RoBERTa-base on GLUE).

Industry relevance: the paper notes that many existing DPSGD improvements either lack rigorous theoretical guarantees, work only for specific model architectures, or require access to public data — all of which "hinder the feasibility of DPSGD in real-world applications." DP-PMLF is presented as architecture-agnostic and public-data-free, and it applies equally to image and text workloads, which matters for organizations that must publish or deploy models with formal privacy guarantees.

Future Directions

1. Relax the assumptions. The authors plan to analyze the method under non-convex Polyak-Łojasiewicz conditions and (L₀, L₁)-smoothness.
2. Automate hyper-parameter choice. They intend to develop an adaptive method for selecting the per-sample momentum and low-pass filter hyper-parameters, which the paper currently treats as fixed settings.
3. Handle the low-noise regime. The finding that heavy smoothing loses gradient information when ε > 6 raises the question of how to make the filter adapt to the noise level instead of degrading accuracy.
4. Broaden the domain. They intend to apply the method beyond image and sentence classification, naming natural language processing tasks and reinforcement learning.

Target Audience

Researchers and graduate students working on differential privacy, private optimization, or trustworthy machine learning; practitioners who deploy or fine-tune models on sensitive data and need strong accuracy under tight privacy budgets; and engineers evaluating DPSGD variants who want a method that does not require public data, a bespoke architecture, or a modified privacy accounting pipeline. Readers without a background in DP or non-convex convergence analysis will find the theoretical sections demanding.

Authors’ abstract

Differentially Private Stochastic Gradient Descent (DPSGD) is widely used to train deep neural networks with formal privacy guarantees. However, the addition of differential privacy (DP) often degrades model accuracy by introducing both noise and bias. Existing techniques typically address only one of these issues, as reducing DP noise can exacerbate clipping bias and vice-versa. In this paper, we propose a novel method, \emph{DP-PMLF}, which integrates per-sample momentum with a low-pass filtering strategy to simultaneously mitigate DP noise and clipping bias. Our approach uses per-sample momentum to smooth gradient estimates prior to clipping, thereby reducing sampling variance. It further employs a post-processing low-pass filter to attenuate high-frequency DP noise without consuming additional privacy budget. We provide a theoretical analysis demonstrating an improved convergence rate under rigorous DP guarantees, and our empirical evaluations reveal that DP-PMLF significantly enhances the privacy-utility trade-off compared to several state-of-the-art DPSGD variants.