On Optimal Hyperparameters for Differentially Private Deep Transfer Learning

AI summary

Authors: Aki Rehn, Linzh Zhao, Mikko A. Heikkilä, Antti Honkela (Department of Computer Science, University of Helsinki, Finland) arXiv: 2510.20616v2 [cs.LG] — License: CC BY 4.0

Overview

Research area: Privacy-preserving machine learning, specifically differentially private (DP) deep transfer learning — fine-tuning pretrained models on sensitive data with DP optimizers such as DP-SGD and DP-Adam.

Technical level: Advanced. The paper combines a systematic empirical study with new theoretical results (an optimal clipping theorem and a per-step optimization bound) and assumes familiarity with DP definitions, gradient clipping, and noise calibration via the PRV accountant.

Scope in one sentence: The paper studies how the optimal gradient clipping bound C and batch size B vary with learning-problem difficulty — privacy budget, pretrained backbone capability, dataset, and compute budget — and explains the observed behavior through gradient-distribution shifts, a new optimal-clipping analysis, and cumulative DP noise.

What This Paper Is About

DP transfer learning is the current state-of-the-art approach for training large models under privacy constraints, but practitioners commonly fix the clipping bound C and batch size B to a single value across privacy levels, backbones, and compute budgets, tuning only the learning rate. The paper shows this practice is systematically suboptimal, and that the optimal (C, B) depend on how difficult the private fine-tuning problem is. It explains why the theory-based expectation that stronger privacy requires a smaller C contradicts experiments, where larger C often performs better under strong privacy.

Key Contributions

1. A systematic study of how the optimal clipping bound C and batch size B vary in DP deep transfer learning — primarily with the privacy budget ε, but also with other factors affecting the difficulty of the private fine-tuning task, such as the capability of the pretrained backbone and the amount of compute.

2. Empirical evidence contradicting prior theory: increasing C can improve results with smaller ε (Section 5.1) and with less capable pretrained models (Section 5.2). To explain this, the authors derive a novel optimal clipping result (Theorem 5.2) and show how it affects optimization progress (Theorem 5.4 and Corollary 5.5), which they note may be of independent interest.

3. A batch size selection rule under bounded compute (fixed epochs) that combines a lower bound on the number of optimization steps with maximizing the number of steps without increasing the total cumulative DP noise (Section 6).

4. A diagnostic framework for why fixed (C, B) fails: clipping is characterized as a form of gradient re-weighting (retained weight per class), and failures are traced to shifts in gradient distributions and cumulative DP noise.

Main Findings

• Theory and practice disagree on C: The current theoretical understanding suggests stronger privacy requires a smaller C, but empirical outcomes show larger C performing better under strong privacy. The paper attributes this mismatch to changes in the gradient distributions.

• Increasing σ shifts gradient norms upward: Under tighter privacy, gradient norm distributions shift toward larger norms. In Fig. 2 (SUN397, 8 epochs, δ=10⁻⁵, ViT-Tiny), the optimal clipping constant is larger with tighter privacy, and harder tasks (orange) shift the optimal C upward while easier tasks (blue) tolerate or benefit from smaller bounds.

• Backbone capability acts like privacy level: Switching from ViT-Base (more capable, ~85M parameters) to ViT-Tiny (less capable, ~5M parameters) at ε=0.25 shifts the optimal C upward, similar to the effect of decreasing ε. The paper argues a more capable backbone reduces gradient norms, which leads to a smaller C* in Theorem 5.2.

• Clipping is gradient re-weighting: The paper defines a per-class retained weight w_y(C) = (1/n_y) Σ_{i: y_i = y} min(1, C/||g_i||₂). Small C down-weights hard classes disproportionately; larger C preserves their gradients more evenly, recovering hard-class performance at the cost of slightly reduced accuracy on easier classes. This asymmetry grows with learning-problem difficulty.

• Automatic clipping underperforms on hard problems: Comparing AUTO-S (Bu et al., 2023) against properly tuned flat clipping (ViT-Tiny; 8 epochs for SUN397 and CIFAR-100, 32 epochs for Cassava; δ=10⁻⁵; mean over 3 repeats), AUTO-S roughly matches tuned C in easy-enough settings but performs notably worse on harder datasets (SUN397, Cassava), especially under tight privacy.

• Existing batch size heuristics fail under fixed epochs: The per-step averaged-gradient noise standard deviation criterion suggested by Ponomareva et al. (2023) shows no sweet spot in the fixed-epochs setting; it would always suggest full batch, which is suboptimal. Neither does its cumulative version saturate. The plot covers ViT-Tiny on CIFAR-100, 8 epochs, ε = 0.5, 1, 2, 4, 8, δ=10⁻⁵.

• Cumulative noise explains batch size better: Under a fixed-epoch constraint, cumulative noise σ√T (with T = E·N/B, σ from the PRV accountant) recovers a weaker "plateau" — in low-ε regimes σ√T remains nearly constant across a wide range of batch sizes. Tighter privacy broadens the flat region and favors smaller batches.

• A minimum number of steps is required: The right panel of Fig. 7 (CIFAR-100, ViT-Tiny, FiLM, N=50000, δ=10⁻⁵, averaged over 3 seeds) shows a minimum number of steps is needed for good performance, marked at 20 steps for that case. The actual minimum can vary between datasets (Fig. A35 shows SUN397). Increasing epochs flattens the σ√T curves and reduces sensitivity to batch size.

• Tuning guidance differs by condition (Table 1): Lower ε (tighter privacy) → increase C, decrease B. Stronger backbone / easier dataset → use smaller C. Weaker backbone / harder dataset → try larger C. Fewer epochs (limited compute) → avoid large B (which means more steps). More epochs → larger B becomes viable. General guidance → tune (C, B, η) jointly for each task.

• Learning rate interacts with batch size: The best learning rate under DP-Adam often scaled with √B, echoing non-private scaling rules (You et al., 2020; Malladi et al., 2022).

• Batch size range and compute trade-off: Batch sizes were searched exponentially from 256 up to full-batch. Doubling the batch size halves the number of optimizer steps under fixed epochs. The paper states that under fixed-epoch settings moderate-sized batches can outperform large ones, especially under high learning-problem difficulty where more iterations are needed for convergence.

• Full fine-tuning is an exception: When fine-tuning all model parameters on larger models, the increased parameter count raises average gradient norms, preventing the use of smaller clipping bounds — both easy and hard tasks then prefer abnormally large C.

Methodology in Plain English

The authors fine-tune pretrained models on several datasets under DP-Adam (and, in supplementary experiments, DP-SGD), varying three hyperparameters together: learning rate, batch size, and clipping bound. For image tasks they use Vision Transformers (ViT-Base, ViT-Tiny) pretrained on ImageNet-21k with the "AugReg" setup, and apply feature-wise linear modulation (FiLM) parameterization — freezing everything except the scale and bias of normalization layers plus the classification head, giving roughly 0.5–1.5% trainable parameters, with the classification head initialized to zero. They also test LoRA and full fine-tuning for images, LoRA for text (DistilBERT, ~66M parameters), and training from scratch with WideResNet-16-4 (described in the paper as about 2.8M parameters in one place and approximately 2.7M in another) on CIFAR-10.

Evaluation spans four datasets plus a 10% CIFAR-100 subset: SUN397 (397 classes, ~87k training examples, imbalanced; Fig. 1 uses "Full" = 87002), Cassava (5 classes, ~6k training examples, imbalanced and out-of-distribution relative to ImageNet-21k), CIFAR-100 (100 classes, 50k training examples), and 20 Newsgroups (~18k posts from 20 online newsgroups). Images are resized to 224×224. Validation splits are merged into training, and final accuracy is measured on the test set.

They ran exhaustive structured grid searches (batch sizes from 256 up to full-batch; clipping bounds and learning rates from dataset-specific geometric ranges — details in Table A1 and Appendix C) because, as they report, the learning rate often dominates optimization and masks the effects of the other hyperparameters under less thorough tuning. The privacy level is calibrated with the PRV accountant (Gopi et al., 2021) to tight budgets, with δ = 10⁻⁵ in all experiments. Privacy uses the add-remove neighborhood relation and sample-level privacy.

On the theory side, they analyze the standard (unnormalized) DP-SGD variant where noise scales explicitly with C, and prove which C minimizes the per-step mean squared error between the clipped noisy DP gradient and the true gradient (Theorem 5.2). They then relate that MSE to the expected per-step loss improvement (Theorem 5.4) and note that minimizing MSE minimizes that upper bound (Corollary 5.5). They are explicit that Theorem 5.2 is not used to find C* in practice, since it depends on the actual gradients, but only to describe how clipping behaves.

Why This Matters

Impact on research: The paper challenges a widespread convention — fixing clipping bound and batch size across tasks while tuning only the learning rate — and shows that this suppresses performance, especially when moving between loose and tight privacy and between plentiful and limited compute. It also supplies a theory-consistent explanation for a result that contradicted earlier theoretical guidance, and defines the retained-weight diagnostic as a reusable lens. The authors note that other lines of work (e.g., tuning C on a non-private model, or automatic clipping) only work in the specific regimes their analysis predicts.

Real-world applications:

• Private medical or health-record models fine-tuned on small, sensitive datasets, where tight privacy budgets and out-of-distribution data are common (the Cassava setting is an example of an out-of-distribution private task).
• Private text classification over user data, such as the 20 Newsgroups-style document or message categorization tested here with LoRA on DistilBERT.
• On-device or federated personalization, where a pretrained backbone is adapted to a user's private data under a fixed compute budget.
• Deployment settings with limited compute, where the fixed-epochs analysis of batch size directly applies.

Industry relevance: The paper provides concrete tuning rules (increase C and decrease B under tighter privacy; avoid large B with few epochs; tune C, B, and η jointly) that map directly to hyperparameter budgets in production DP training pipelines. It also highlights fairness implications: because clipping down-weights harder examples and the classes they dominate, fixed small clipping bounds risk systematically harming difficult subgroups.

Future Directions

• Extending the analysis beyond the fixed-epochs setting and clarifying when the cumulative-noise criterion should be preferred over fixed-step tuning of batch size.
• Understanding the full fine-tuning regime, where the increased parameter count raises gradient norms and both easy and hard tasks prefer abnormally large clipping bounds — the paper identifies this as the "odd-one out" and leaves the precise mechanism to further work.
• Developing practical methods to select C without access to the true gradients, since Theorem 5.2 characterizes C* but cannot be applied directly during DP optimization.
• Testing whether the difficulty-based tuning rules transfer to settings the paper only touches on, such as other optimizers, other dataset modalities beyond the image and text tasks studied, and the adaptive clipping and error-feedback alternatives discussed in related work.

Target Audience

Researchers and practitioners working on differentially private machine learning, especially those fine-tuning pretrained models under DP-SGD or DP-Adam and setting up hyperparameter searches. It is also relevant to theoretically inclined readers interested in clipping bias, gradient re-weighting, and privacy-utility trade-offs, and to engineers deploying DP training under fixed compute budgets who need concrete tuning guidance. The paper is written at an advanced level: the experimental sections are accessible with a working knowledge of DP training, while Sections 5.1 and 5.2 require comfort with optimization and MSE-based analysis.

Note: The paper's appendix includes a statement on LLM usage (ChatGPT and GitHub Copilot used for phrasing, code assistance, and plotting drafts, with all content reviewed and finalized by the authors).

Authors’ abstract

Differentially private (DP) transfer learning, i.e., fine-tuning a pretrained model on private data, is the current state-of-the-art approach for training large models under privacy constraints. We focus on two key hyperparameters in this setting: the clipping bound $C$ and batch size $B$. We show a clear mismatch between the current theoretical understanding of how to choose an optimal $C$ (stronger privacy requires smaller $C$) and empirical outcomes (larger $C$ performs better under strong privacy), caused by changes in the gradient distributions. Assuming a limited compute budget (fixed epochs), we demonstrate that the existing heuristics for tuning $B$ do not work, while cumulative DP noise better explains whether smaller or larger batches perform better. We also highlight how the common practice of using a single $(C,B)$ setting across tasks can lead to suboptimal performance. We find that performance drops especially when moving between loose and tight privacy and between plentiful and limited compute, which we explain by analyzing clipping as a form of gradient re-weighting and examining cumulative DP noise.