Towards Efficient and Scalable Implementation of Differentially Private Deep Learning

Thanks for your time and your insightful review.

Figure 5:

Thanks for the comments! It is correct, the ratio of the throughput is TF32/FP32 and that should be clarified in the figure caption. It will be updated for the camera ready version.

One thing that was not clear to me in Masked DP-SGD is how the batches are sampled at each step.

The masked DP-SGD batch sampling can be composed in two main steps. First, we sample the batch size $B^{(t)}$ from a Binomial distribution $\text{Bin}(N, q)$, where N is the number of samples and $q$ is the subsampling rate. We round $B^{(t)}$ up to the next integer multiple of physical batch size $p$: $B_+^{(t)} = p\lceil B^{(t)} / p\rceil$. Second we permute the data set, and pick the first $B_+^{(t)}$ elements as the full logical batch that we process as chunks of $p$ samples. Finally when the gradients are aggregated, we throw away (by zeroing the gradients) for the padded $B_+^{(t)} - B^{(t)}$ samples.

How to do this efficiently is not clear in cases where the dataset is too large to fit in memory. Perhaps Masked DP-SGD can be applied in conjunction with the method of Scalable DP-SGD proposed in Chua et al. (2024b), which is supposed to work when the dataset size is very large?

This is a very interesting future direction that we are happy to discuss! Indeed the current sampling procedure of the Masked DP-SGD might struggle with very large data sets, as permuting the indices becomes very expensive in that case. We could indeed combine the Chua et al. (2024b) approach for the subsampling with the masked approach by simply replacing the Truncate/Pad step in their approach with our masking step, and processing the minibatches again as chunks of $p$.

Thanks for your time and your thorough review.

What is the privacy guarantee of your proposed algorithm? How can you prove it?

The DP guarantees of the Masked DP-SGD follow from the standard analysis of Poisson subsampled DP-SGD in the add/remove adjacency. Note that the only difference we have to standard Poisson subsampling, is that in cases where the sampled batch size $B$ is not an integer multiple of the physical batch size $p$, we need to pad the batch with additional samples. We compute the clipped per-example gradients for these $B_+ = p \lceil B / p \rceil$ samples. However, when aggregating the clipped per-example gradients in a sum, we weigh the padded $B_+ - B$ samples with $0$. Hence, the sampling procedure we use is equivalent to that of Poisson subsampling, allowing us to analyze the privacy guarantees as the Poisson subsampled Gaussian mechanism.

What is the computational overhead of these DPSGD methods on datasets beyond CIFAR100?

This is an interesting question. In this work we focussed on cases where the complete dataset can be still handled by one machine and in these cases the computational overhead is mostly influenced by the model (see Figures 1 and 2). Handling cases where distributed computing is needed might require additional methodology that is orthogonal to our advancements (see also discussion with reviewer rRoP).

Lower precision using TF32 increasing the throughput is interesting but may not ensure its good performance on other tasks. [...] Since the privacy exhibits a tradeoff with accuracy, it may be beneficial to discuss accuracy in the experiments.

Thanks for the suggestion, we added new experiments below complementing the Table A2 (in the Appendix), where we compare accuracy for the ViT base model on CIFAR100. The new experiments measure the accuracy and throughput for the SVHN and CIFAR10 datasets with different numbers of examples per class and hyperparameters. We are comparing the accuracy and throughput between precision modes for private training with Opacus.

The throughput difference is the same for both datasets. Using TF32 is twice as fast as FP32. We tested the variability in accuracies across three repeats of the DP training with both TF32 and FP32. We computed the pairwise differences between TF32 and FP32 on different seeds and data sets. The differences between the two precisions have a mean of $\approx -4.3 \times 10^{-5}$ and std $\approx 4.2 \times 10^{-4}$. Using a pairwise t-test on the differences we cannot reject the null hypothesis that the accuracies have the same mean (p-value $\approx 0.74$). Furthermore, compared to the variance arising from DP (see Table above), the differences from changing the precision are negligible. This is in line with the previous results of the article in Table A2.

Why is accuracy missing from all experimental results? How does the sampling affect accuracy results?

The paper focuses on computational efficiency and as can be seen from Table A2 the optimizations have no impact (apart from tiny impact due to seeding noise) on test accuracy. We use the accuracy to test our implementations and check that everything is implemented correctly as no accuracy difference is expected unless using different precision where we observe little impact (see Table A2 and experiments above).

The general question on which sampling is optimal is a separate question and there is some early work looking at this question [1] but for fair comparisons between sampling in terms of accuracy the accounting must match the sampling method and so far tight accounting has been only established for Poisson subsampling [2]. Tight accounting for other sampling methods remains an active area of research.

[1] Chua et al. Scalable DP-SGD: Shuffling vs. poisson subsampling. NeurIPS 2024.

[2] Annamalai et al.. (2024) To Shuffle or not to Shuffle: Auditing DP-SGD with Shuffling. arXiv:2411.10614.

Thanks for your time and your careful review.

We respectfully disagree with your statement that “While it's a very solid technical work and important for practitioners, the paper lacks scientific novelty that would be expected at ICML.” because the call for papers of ICML mentions as topic of interest: “Machine Learning Systems (improved implementation and scalability, hardware, libraries, distributed methods, etc.)”. We believe that our work fits in the call for papers.

ICML Call for Papers has a section for review criteria, which starts with: “Submissions should report original and rigorous research of significant interest to the machine learning community.” We believe that according to your review, our paper satisfies this completely, when taking into account that the machine learning community includes both the researchers and practitioners.