Leveraging Randomness in Model and Data Partitioning for Privacy Amplification

We thank the reviewer for their detailed review of our paper and their valuable feedback.

Our paper is indeed the first to point out and quantify the privacy gain from model parallelism techniques already employed in federated learning.

Following the reviewer's suggestions, we have done more experiments for both centralized and federated settings. In the centralized setting with sample-level $(8, 10^{-5})$-DP, training ResNet101 with 3 submodels under the standard DP-SGD analysis achieves 79.80% accuracy and using our analysis accuracy increases to 82.43% (this is because our analysis accounts for the amplification gain for model splitting, allowing us to add less noise while still achieving the same DP guarantee); 8 submodels under the standard DP-SGD analysis has 76.80% accuracy and using our analysis accuracy increases to 80.52%. In federated setting with user-level $(8, 10^{-5})$-DP, 3 submodels with the standard analysis has 78.47% accuracy and using our analysis accuracy increases to 80.28%; 8 submodels with the standard analysis has 76.96% accuracy, and using our analysis accuracy increases to 79.11%. The experiments are run with 3 random seeds and all have standard deviations of around 0.7%.

The design choice to not partition the first and last layers of ResNet-101 is guided by insights from the model splitting literature (e.g. Dun et. al.). They argue that since these layers pick up important features, partitioning them leads to degraded accuracy. Following the guidance from this literature, we experimented with partial and not fully disjoint model splitting.

As for the comparison between Balanced Iteration Subsampling and Poisson Subsampling, we would like to emphasize a point which we will make more clear in the revision. Balanced Iteration and Poisson Subsampling achieve similar privacy-accuracy trade-offs experimentally. This is because standard training includes a large number of iterations, and for large numbers of iterations, both the training dynamic and privacy guarantees of the two become comparable as pointed out in the paper. The main advantage of Balanced Iteration Subsampling is that it is more practical and deployment-friendly especially in the federated setting. Poisson Subsampling has implementation-related drawbacks as it gives a variable load to each client, which can overwhelm the resources of the client and undermine fair use policies. The DP community has been interested in studying other (more deployment-friendly) data subsampling techniques whose utility and privacy guarantees are comparable to Poisson Subsampling, for example random shuffling and random check-ins (Balle et al 2020). Our paper fills this gap by showing that this is indeed the case for Balanced Iteration Subsampling. As for the actual experiments, for CIFAR-10 with WideResNet-40-4, $(8, 10^{-5})$-DP, 2000 iterations, using each sample 655 times for Balanced Iteration Subsampling and with probability $\frac{655}{2000}$ in each iteration for Poisson Subsampling, Balanced Iteration Subsampling injects noise with $\sigma = 10.17$ and achieves validation accuracy of $70.21\%$ (with standard deviation $0.69\%$), while Poisson Subsampling injects noise with $\sigma = 10.20$ and achieves validation accuracy of $70.13\%$ (with standard deviation $0.78\%$). ResNet-101 gives similar results. We will include these experiments and show more graphs to compare the privacy guarantees of the two subsampling methods in the revision.

In Figure 2, the thin line at $\gamma \approx 0$ is due to discretization of $\gamma$ values and the code arbitrarily ruling in favor of Balanced Iteration Subsampling when $\gamma = 0$.

On the concern about Corollary 3.7, we would like to clarify that the privacy gain we utilize comes from the random (and independent) assignment of submodels to the clients. So, although in Algorithm 1, model splitting in done centrally, each client independently receives a submodel. Algorithm 2's parameter dropout can be seen as forming $2^m$ submodels (where $m$ is the number of parameters subject to dropout), and each client independently receives a submodel, so in that perspective, dropout is fundamentally no different, and thus it is a corollary that follows.

The dataset neighboring relationship is indeed add-/remove-one, which was briefly mentioned in Section 2.

We thank the reviewer for their review and their valuable feedback. We will incorporate their comments in our revised paper.

Following the reviewer's suggestions, we have done additional experiments for both centralized and federated settings. In the centralized setting, training ResNet101 using 3 submodels achieves 79.80% accuracy with sample-level $(8, 10^{-5})$-DP guarantee for the final model under the standard DP-SGD analysis, and accuracy increases to 82.43% for the same privacy guarantee using our analysis (this is because our analysis accounts for the amplification gain for model splitting, allowing us to add less noise while still achieving the same DP guarantee); training ResNet101 with 8 submodels has 76.80% accuracy under the standard DP-SGD analysis and using our analysis accuracy increases to 80.52%. In the federated setting, training ResNet101 using 3 submodels achieves 78.47% accuracy and user-level $(8, 10^{-5})$-DP under the standard analysis, using our analysis accuracy increases to 80.28% under the same privacy guarantee; 8 submodels with the standard analysis has 76.96% accuracy, and using our analysis increases the accuracy to 79.11%.

We would like to clarify the main concern regarding the comparison between Balanced Iteration Subsampling and Poisson Subsampling. Even though we show that Balanced Iteration Subsampling can have better privacy guarantees in some regimes, as the reviewer points out Balanced Iteration and Poisson Subsampling achieve similar privacy-accuracy trade-offs experimentally. This is because standard training includes a large number of iterations, so both the training dynamic and privacy guarantees of the two become comparable (see experimental results below). The main advantage of Balanced Iteration is that it is more practical and deployment-friendly especially in the federated setting. Poisson Subsampling has implementation-related drawbacks as it gives a variable load to each client, which can overwhelm the resources of the client and undermine fair use policies. The DP community has been interested in studying other (more deployment-friendly) data subsampling techniques whose utility and privacy guarantees are comparable to Poisson Subsampling, for example random shuffling and random check-ins (Balle et al 2020). Our paper fills this gap by showing that this is indeed the case for Balanced Iteration Subsampling. As for the actual experiments, for CIFAR-10 with WideResNet-40-4, $(8, 10^{-5})$-DP, 2000 iterations, using each sample 655 times for Balanced Iteration Subsampling and with probability $\frac{655}{2000}$ in each iteration for Poisson Subsampling, Balanced Iteration Subsampling injects noise with $\sigma = 10.17$ and achieves validation accuracy of $70.21\%$ (with standard deviation $0.69\%$), while Poisson Subsampling injects noise with $\sigma = 10.20$ and achieves validation accuracy of $70.13\%$ (with standard deviation $0.78\%$). ResNet-101 gives similar results. We will include these experiments in the revision.

On the second concern, Theorem 3.5 and Corollary 3.7 indeed utilize only one of the two sources of randomness for amplification because different training dynamics require different privacy analyses. While our analysis does not fully capture all sources of randomness in the setting of Thm 3.5 (this would require an extension of our current analysis), we hope the reviewer will appreciate that we demonstrate a significant improvement over ignoring randomness entirely and defaulting to the standard DP-SGD analysis. We hope our paper will inspire the DP community to further explore such inherent gains and develop stronger mathematical tools for their accounting. This remains a promising direction for future work.

Regarding Line 196, which proposes using two clipping norms—one for the parts of the model with model splitting and another for the parts without—it is, in our view, the only viable approach. If a single clipping norm were used, the worst-case $\epsilon$ would occur when the gradient vector concentrates all its power on the part of the model without model splitting. In that scenario, there would be no privacy amplification at all. Therefore, two separate clipping norms appear necessary to the authors.

On the third point, our paper is not merely about introducing a theoretical tool to quantify privacy in specific cases; rather, we view the broader contribution of our paper as highlighting an important but previously overlooked insight —that inherent randomness in the training dynamics, such as model parallelism techniques already used in centralized and federated learning, can be leveraged for privacy amplification, at no additional cost. To our knowledge, this is the first work to identify and quantify such an amplification gain. Beyond the specific analysis presented, we aim to bring this perspective to the DP community and encourage further exploration of how different training dynamics yield privacy gains.

We thank the reviewer for their review and valuable feedback.

We would like to clarify their main question about "whether the proposed stronger privacy comes at the cost of worse utility or convergence rate compared to the canonical DP-SGD method." The main contribution of our paper is to develop a mathematical analysis that is able to quantify the privacy gain that comes from various sources of randomness that share a common structure. Model splitting is already used in the literature both in the centralized and federated settings. We are not proposing a new technique here; we are pointing out that model splitting has a free privacy gain which has remained unnoticed in the prior literature (which requires nontrivial analysis to quantify). Since we are not changing the training, same utility and convergence rates can now be achieved with better privacy guarantees. Conversely, for the same privacy, we can achieve better utility and convergence rates than canonical DP-SGD since we need to inject less noise (see experimental results below).

Balanced Iteration Subsampling is indeed a new subsampling method we propose. Balanced Iteration Subsampling is more deployment-friendly that Poisson subsampling, which has implementation-related drawbacks as it gives a variable load to each client. This can overwhelm the resources of the client and undermine fair use policies. The DP community has been interested in studying other (more deployment-friendly) data subsampling techniques whose utility and privacy guarantees are comparable to Poisson Subsampling, for example random shuffling and random check-ins (Balle et al). Our paper fills this gap by showing that Balanced Iteration Subsampling has comparable performance to Poisson subsampling (theoretically we can prove slightly better privacy guarantees for Balanced Iteration Subsampling, however in experiments we observe that Balanced Iteration Subsampling and Poisson subsampling have similar performance, see below for details). We will revise our paper to make these points more clear.

Regarding the question about whether we use "worst-case privacy, or average privacy across all samples", we would like to point out that we always use the standard definitions of $(\epsilon, \delta)$-DP and $(\alpha, \epsilon)$-RDP in the literature. The definition of $(\epsilon, \delta)$-DP can be intuitively thought as the privacy leak in the worst $\delta$-fraction of samples, so in that sense it makes intuitive sense that Poisson subsampling can have worse $(\epsilon, \delta)$-DP than Balanced Iteration Subsampling.

The baseline for our model splitting results is the canonical DP-SGD analysis as pointed out by the reviewer. For one iteration, the canonical DP-SGD analysis would give $\epsilon = \frac{\alpha c^2}{2\sigma^2}$ RDP in contrast to the RDP bound in Theorem 3.1 (this expression for $\epsilon$ does not have a dependence on the number of submodels because the canonical analysis does not leverage the privacy amplification gain due to model splitting, which is the contribution of our paper). Figure 1 presents a visual comparison of our analysis and the standard analysis. We will make this point clear in the revision. To put this into perspective, for $(\epsilon, \delta)$-DP with $\delta = 10^{-5}$, 1200 training iterations, data subsampling rate of 0.1, a noise standard deviation of 2, our analysis can guarantee $\epsilon = [3.5, 4.0, 5.0, 7.4]$ for $[8, 6, 4, 2]$ submodels, while without using our analysis, the best guarantee is $\epsilon = 11.0$ for all numbers of submodels. To give another example, with $\sigma = 6$, our analysis guarantees $\epsilon = [1.0, 1.2, 1.5, 2.1]$ while standard analysis gives $\epsilon = 3.0$. We will add these as graphs to the revised version. Note that because training remains the same and we only change the privacy analysis, utility remains the same in both cases.

Conversely, we can fix the same $(\epsilon, \delta)$-DP budget and compare utility. We have done such experiments for both centralized and federated settings. In the centralized setting with sample-level $(8, 10^{-5})$-DP, training ResNet101 with 3 submodels under the standard DP-SGD analysis achieves 79.80% accuracy and using our analysis accuracy increases to 82.43% (this is because our analysis accounts for the amplification gain for model splitting, allowing us to add less noise while still achieving the same DP guarantee); 8 submodels under the standard DP-SGD analysis has 76.80% accuracy and using our analysis accuracy increases to 80.52%. In federated setting with user-level $(8, 10^{-5})$-DP, 3 submodels with the standard analysis has 78.47% accuracy and using our analysis accuracy increases to 80.28%; 8 submodels with the standard analysis has 76.96% accuracy has using our analysis increases it to 79.11%.

We will add these experiments to the revised version of the paper, and incorporate reviewer's suggestion about the restructuring of the results.

We thank the reviewer for their review and their valuable feedback.

We would like to clarify their main concern regarding "a straightforward comparison between the proposed analysis and the analysis in the literature" for the model splitting methods in Section 3.2. We note that our paper is the first one to point out that model splitting has an inherent privacy amplification gain and to quantify this privacy gain. The baseline to compare this is the standard RDP analysis for DP-SGD which does not take this amplification gain into account since it has remained unnoticed and unquantified in the literature. For one iteration, the standard analysis in the literature would give $\epsilon = \frac{\alpha c^2}{2\sigma^2}$ in contrast to the RDP bound in Theorem 3.1 (this expression for $\epsilon$ does not have a dependence on the number of submodels, because again it does not take into account the fact that training uses model splitting). Figure 1 presents a visual comparison of our analysis and the aforementioned standard analysis. We will make this point clear in the revision. To put this into perspective, for $(\epsilon, \delta)$-DP with $\delta = 10^{-5}$, 1200 training iterations, data subsampling rate of 0.1, a noise standard deviation of 2, our analysis can guarantee $\epsilon = [3.5, 4.0, 5.0, 7.4]$ for $[8, 6, 4, 2]$ submodels, while without using our analysis, the best guarantee is $\epsilon = 11.0$ for all numbers of submodels. To give another example, with $\sigma = 6$, our analysis guarantees $\epsilon = [1.0, 1.2, 1.5, 2.1]$ while standard analysis gives $\epsilon = 3.0$. We will add these as graphs to the revised version.

Conversely, we can fix the same $(\epsilon, \delta)$-DP for the final model and compare accuracy. We have done such experiments for both centralized and federated settings. In the centralized setting with sample-level $(8, 10^{-5})$-DP, training ResNet101 with 3 submodels under the standard DP-SGD analysis achieves 79.80% accuracy and using our analysis accuracy increases to 82.43% (this is because our analysis accounts for the amplification gain for model splitting, allowing us to add less noise while still achieving the same DP guarantee); 8 submodels under the standard DP-SGD analysis has 76.80% accuracy and using our analysis accuracy increases to 80.52%. In federated setting with user-level $(8, 10^{-5})$-DP, 3 submodels with the standard analysis has 78.47% accuracy and using our analysis accuracy increases to 80.28%; 8 submodels with the standard analysis has 76.96% accuracy has using our analysis increases it to 79.11%.

As for the comparison between Balanced Iteration Subsampling and Poisson Subsampling, we would like to emphasize a point which we will make more clear in the revision. Balanced Iteration and Poisson Subsampling achieve similar privacy-accuracy trade-offs experimentally. This is because standard training includes a large number of iterations, and for large numbers of iterations, both the training dynamic and privacy guarantees of the two become comparable as noted in the paper. The main advantage of Balanced Iteration Subsampling is that it is more practical and deployment-friendly especially in the federated setting. Poisson Subsampling has implementation-related drawbacks as it gives a variable load to each client, which can overwhelm the resources of the client and undermine fair use policies. The DP community has been interested in studying other (more deployment-friendly) data subsampling techniques whose utility and privacy guarantees are comparable to Poisson Subsampling, for example random shuffling and random check-ins (Balle et al 2020). Our paper fills this gap by showing that this is indeed the case for Balanced Iteration Subsampling. As for the actual experiments, for CIFAR-10 with WideResNet-40-4, $(8, 10^{-5})$-DP, 2000 iterations, using each sample 655 times for Balanced Iteration Subsampling and with probability $\frac{655}{2000}$ in each iteration for Poisson Subsampling, Balanced Iteration Subsampling injects noise with $\sigma = 10.17$ and achieves validation accuracy of $70.21\%$ (with standard deviation $0.69\%$), while Poisson Subsampling injects noise with $\sigma = 10.20$ and achieves validation accuracy of $70.13\%$ (with standard deviation $0.78\%$). ResNet-101 gives similar results. We will include these experiments in the revision.

While we are aware of the paper by (Mironov et. al.) which also considers Poisson subsampling with the Gaussian mechanism; we use the stronger analysis in (Zhu & Wang) for Poisson subsampling.

Lastly, the design choice to not partition the first and last layers of ResNet-101 is guided by insights from the model splitting literature (e.g. Dun et. al.). They argue that since these layers pick up important features, partitioning them leads to degraded accuracy. Following the guidance from this literature, we experimented with partial and not fully disjoint model splitting.