Forward Learning with Differential Privacy

[W1] Misleading DP-SGD baseline

[RW1]: We understand your concern about the DP-SGD baseline and appreciate the opportunity to clarify. For DP-SGD experiments, we used the open-source OPACUS library with default settings for MNIST. We tested learning rates of 0.1 and 0.01, reporting the best-performing results (0.1 for all cases).

We agree that larger batch sizes generally improve the utility-privacy tradeoff in DP-SGD, as noted in previous literature. Specifically, performance was improved with a larger batch size when fixing privacy cost $(\epsilon, \delta)$. This improvement arises from increasing suitable noise scale $\sigma$ with a larger batch size. In contrast, higher noise scales significantly impair training with small batch sizes. In our experiments, we can also observe this improvement. For example, the privacy cost of training with $B=64$ and $\sigma=2$ is similar to, actually higher than, that of training with $B=500$ and $\sigma=8$. The valid acc. of the former setting is $68.73\%$, while that of the latter setting is $88.16\%$, indicating a better performance on both accuracy and privacy cost.

Besides, we kindly remind the reviewer that in our experiments, the performance of DP-SGD increases or for large batch size in most of case of $\sigma$ (2,4,8), and evidently drops only when $\sigma = 0.5$, where training resembles a non-privatized setting. We attribute this specific drop to hyperparameter or parameter interaction effects rather than inherent limitations of DP-SGD.

To further ensure the fairness of the baseline results, we extended our DP-SGD experiments across a wider range of learning rates (0.1, 0.05, 0.01, 0.005, 0.001) with larger batch sizes ($B=200, 500$). The results, presented below, show that the learning rate of 0.1, which was used in our paper, consistently achieves the best performance across almost all settings:

[W2] Report average and std for multiple runs

[RW2]: Thank you for the suggestion. We have rerun our experiments five times and calculated the average and standard deviation. These results have been incorporated into the Table 1 of the revised manuscript.

[W3] Experiments with more popular models

[RW3]: We appreciate the suggestion to extend our study to more complex architectures. This work investigates the connection between forward learning and differential privacy. While previous research has focused on improving performance, particularly in terms of gradient estimation accuracy, little attention has been given to fully exploring the potential of forward learning’s inherent characteristics. Our study aims to address this gap from a theoretical perspective, specifically how forward learning can be leveraged to provide privacy to neural networks in a natural way.

In response to your question, applying forward learning to large-scale neural networks remains a significant challenge. While current studies [1,2] suggest that forward learning algorithms can be used to fine-tune large models, to the best of our knowledge, no work has yet extended the application of forward learning to train large networks from scratch and achieve comparable performance. While Deepzero [3] claims success in this area, they rely on pruning techniques, and there are reproducibility issues, as indicated in their repository's issue channel. This suggests that there is still much work to be done in this direction.

We hope that our work can provide further support for forward learning research, with the added confidence that it inherently possesses the beneficial properties of differential privacy. This is another key message we aim to convey through our work.

[1] T. Ren et al., "FLOPS: Forward Learning with OPtimal Sampling." arXiv preprint, 2024.

[2] S. Malladi et al., "Fine-Tuning Language Models with Just Forward Passes." arXiv preprint, 2024.

[3] A. Chen et al., "DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Training." arXiv preprint, 2024.

[R1]: We appreciate your insightful comment regarding learning rates. Initially, we experimented with different learning rates and observed that the performance of DP-SGD at larger learning rates could rise but then suffers from more severe degradation, particularly with larger noise scales or smaller batch sizes. In our following hyperparameter search, we found that a learning rate of 0.1 offered the best overall performance across various settings. Therefore, we exhibit the results of the current learning rate in the paper, which, although it may not be optimal for every setting, represents a carefully tuned choice that balances performance across the board.

To address your suggestion, we conducted additional experiments with larger learning rates (1 and 2). The results, shown below, indicate that while these larger rates do not consistently outperform a learning rate of 0.1, which achieves the best average performance in most cases, they do yield improvements in some cases (e.g., larger batch sizes and lower noise levels). Thank you again for your instructive suggestion. We will include these additional results in the revised manuscript to provide a more comprehensive comparison.

Besides, we kindly remind the reviewer that, in the paper, we also didn't do hyperparameter searching for our DP-ULR in every setting of batch size and noise scale and applied consistent hyperparameters. For further enhanced comparisons, we selected several cases (not all cases due to the time limit), particularly the specific cases where DP-SGD benefits from larger learning rates, to showcase the improvements of DP-ULR with similar hyperparameter tuning. For example, with a slightly larger learning rate of 0.02, DP-ULR achieves the following performance improvements, which are still comparable to DP-SGD:

• $93.32\%$ training acc. and $93.36\%$ validation acc. for $B=200$, $\sigma_0=0.5$.
• $94.60\%$ training acc. and $94.63\%$ validation acc. for $B=500$, $\sigma_0=0.5$.
• $92.65\%$ training acc. and $92.71\%$ validation acc. for $B=500$, $\sigma_0=1$.
• $91.35\%$ training acc. and $92.04\%$ validation acc. for $B=500$, $\sigma_0=2$.

[R2]: Thank you for your thoughtful question and interest in our work. DP-ULR primarily aims to enable differential privacy in non-backpropagation (non-BP) algorithms. Consequently, the training capacity of DP-ULR aligns more closely with its foundational method, ULR, rather than BP. Moreover, DP-ULR inherits ULR's advantages, such as suitability for training black-box or non-differentiable models.

One significant practical application of DP-ULR is fine-tuning scenarios where the model backbone remains frozen. While DP-SGD requires gradient computations through frozen components using BP's chain rule, DP-ULR avoids this constraint. This makes DP-ULR particularly valuable when dealing with black-box backbones (e.g., GPT). In such cases, DP-SGD becomes inapplicable, whereas DP-ULR can perform privacy-preserving fine-tuning, such as through LoRA. This is critical for leveraging foundation models like GPT, which carry rich pretraining knowledge and are often fine-tuned with sensitive, domain-specific user data. By enabling private fine-tuning, DP-ULR directly supports real-world deployment needs.

Although this paper focuses on establishing the theoretical and practical foundations of DP-ULR, we believe it opens the door to scaling forward-learning-based approaches for practical applications, such as secure fine-tuning large-scale foundation models. We will clarify these potential applications in the revised manuscript.

[R3]: Thank you for your kind reminder. It was a compiling issue of tex. We will revise this with several missing words. The complete sentence is, "This stabilization is likely due to the increased sample diversity with larger batches, which reduces the non-isotropy of the variance and minimizes its impact on training performance."

[W1 & Q1] Lack of experiments and analysis on non-differentiable or black-box settings

[RW1 & RQ1]: Thank you for your thoughtful feedback. In the abstract, we initially highlight the general advantages of ULR to show the rationale for using it as the basis for our study. However, these advantages are not intended to be the primary selling points of DP-ULR, which focuses on leveraging the inherent randomness of forward learning algorithms to achieve differential privacy. Our work aims to inspire further exploration into the application of DP forward learning, including non-differentiable and black-box settings.

Nonetheless, our experiments partly demonstrate DP-ULR's capability in black-box settings. Specifically, while all layers are trained simultaneously, the gradient estimations of different layers are independent. When noise is added to the output of a layer for gradient estimation, other layers effectively function as black boxes. In contrast, backpropagation relies on computation graphs and cannot obtain gradients without access to subsequent layers. For non-differentiable modules, such as spiking neural networks (SNNs), ULR has been shown to outperform SGD, which requires smoothing approximations for backpropagation.

In the initial submission, we acknowledged the absence of direct experiments in these settings in the limitations section. In the revised manuscript, we have further revised the abstract and introduction to clarify that DP-ULR's suitability for non-differentiable or black-box settings is a potential rather than a demonstrated advantage. Thank you again for your constructive suggestions.

[W1 & Q2] Complexity analysis of DP-ULR

[RW1 & RQ2]: Thank you for the suggestion. For an MLP with $L$ layers and $N$ neurons per layer, the complexity of DP-ULR varies based on parallelism:

• No Interlayer Parallelism: The forward stage requires $L$ independent forward passes, resulting in a complexity of $O(L^2N^2)$. The gradient computation (backward stage) has a complexity of $O(LN^2)$.
• Full Interlayer Parallelism: A favorable property inherited from ULR is that both forward passes and gradient computations can be parallelized, reducing forward-stage complexity to $O(LN^2)$ and gradient computation complexity to $O(N^2)$.

Due to the same request, please refer to the response to Reviewer ZJUY for a detailed explanation.

[Q3] Relationship between noise redundancy and model size

[RQ3]: Thank you for the constructive suggestion. We conducted additional experiments on three MLP configurations: small, medium (2.5x parameters), and large (5x parameters). Both DP-ULR and DP-SGD were tested with a noise scale $\sigma_0 = 8$ and batch sizes $B = 64, 128, 256$. Each experiment was repeated five times, and the mean and standard deviations are reported below:

When the batch size is small, DP-ULR's advantage over DP-SGD decreases as the model size increases. This is likely due to DP-ULR's reliance on the minimum spectrum of the Jacobian matrix, introducing noise redundancy from non-isotropic variance, which scales with model size, as hypothesized by the reviewer. However, with larger batch sizes (128 and 256), the performance gap between DP-ULR and DP-SGD stabilizes, likely because increased sample diversity reduces the extent of non-isotropy, mitigating its impact. Besides, we can see that both DP-ULR and DP-SGD show uncommonly decreased performance with increased model size. We attribute this degradation to untuned training hyperparameters related to model size and argue that this doesn‘t affect our previous analysis.

We have included these results and discussions in Appendix C.5 of the revised manuscript to indicate the constructive insight pointed out by the reviewer.

Dear Reviewer pQjz,

Thank you for your time and thoughtful review. With one day remaining in the discussion phase, we kindly request your feedback on our responses and look forward to your updated evaluation. If you have any additional questions or concerns, please feel free to let us know.

[W1 & Q2] Insufficient differential privacy protection

[RW1 & RQ2]: Thank you for your insightful feedback. We agree that perturbing both inputs and labels could enhance the differential privacy guarantee, and this is a valuable direction for future exploration. However, we argue that perturbing the input $x$ alone is sufficient to guarantee DP. In deep learning, differential privacy hinges on the randomness of the algorithm's output (i.e., learned parameters), which are computed iteratively through gradients derived from both inputs and labels. By perturbing the input, we effectively randomize the gradients, thus ensuring privacy. Our theoretical analysis focuses on the distribution of output gradients when only the input is perturbed, providing solid DP guarantees. We choose to perturb the input rather than the label due to the forward learning algorithm's reliance on input perturbations for gradient estimation and the inherent randomness this introduces. This approach aligns with our goal of translating inherent randomness into differential privacy guarantees.

[W2 & Q3] Non-privatized computation of the minimum spectrum

[RW2]: We appreciate your concern regarding the non-privatized computation of the minimum spectrum. However, we argue that this operation does not affect the DP guarantee. Instead of influencing the expectation of the output gradient, the purpose of this computation is to ensure a consistent lower bound on variance, which is utilized to derive the privacy guarantee. This is comparable to gradient norm computation for clipping in DP-SGD, which is also non-privatized but does not affect the DP analysis. In our case, the privacy guarantee is derived solely from the distribution of output gradients, independent of the intermediate non-privatized computations.

[W3 & Q1] (Efficiency) comparison to ZeroDP and similar works

[RW3 & RQ1]: Thank you for highlighting the need for a clear comparison with ZeroDP [1] and similar works [2,3]. While existing DP zeroth-order methods, including ZeroDP, achieve privacy through two forward passes with added noise, our DP-ULR distinguishes itself in several key aspects:

• To obtain differential privacy, these algorithms add additional noise to the zeroth-order gradients or losses. In contrast, we explore the potential for utilizing inherent randomness "free lunch" provided by the noise injection operation in the forward learning algorithms to achieve differential privacy.

• Existing methods, such as Simultaneous Perturbation Stochastic Approximation (SPSA), add noise to parameters, which often have dimensions significantly larger than logits. This leads to higher computational costs and estimation variance, especially when training from scratch. Partly because of this, those existing methods are designed for fine-tuning pre-trained models. In contrast, DP-ULR operates directly on logits, enabling the training of deep learning models from scratch and reducing the computational overhead.

• ZeroDP's privacy cost scales quadratically with the number of repetitions $P$ (Theorem 4.1 in [1]), making it less scalable for larger $P$. In contrast, DP-ULR's privacy cost is independent of $P$, ensuring robust privacy guarantees and better scalability.

These distinctions highlight DP-ULR's efficiency advantages, in terms of both privacy cost and computational cost, over ZeroDP and similar existing methods. We have included a detailed discussion in Sections 2.3 and Appendix A.5 of the revised manuscript.

[1] Z Liu, J Lou, W Bao, Y Hu, B Li, Z Qin, K Ren. “Differentially Private Zeroth-Order Methods for Scalable Large Language Model Finetuning.” arXiv preprint, 2024.

[2] Liang Zhang, Bingcong Li, Kiran Koshy Thekumparampil, Sewoong Oh, Niao He. “DPZero: Private Fine-Tuning of Language Models without Backpropagation.” arXiv preprint, 2023.

[3] Xinyu Tang, Ashwinee Panda, Milad Nasr, Saeed Mahloujifar, Prateek Mittal. “Private Fine-tuning of Large Language Models with Zeroth-order Optimization.” arXiv preprint, 2024.

Dear Reviewer RAbX,

Thank you for your time and thoughtful review. With one day remaining in the discussion phase, we kindly request your feedback on our responses and look forward to your updated evaluation. If you have any additional questions or concerns, please feel free to let us know.

[W1 & Q2] The runtime of DP-ULR in comparison with DP-SGD

[RW1 & RQ2]: Thank you for your careful feedback. The runtime of our DP-ULR and DP-SGD is 23 seconds/epoch and 18 seconds/epoch on an A6000 GPU for the MNIST dataset, respectively. We kindly remind the reviewer that these reports have been included in the paper (lines 530–531).

[W1] Complexity of DP-ULR with computing Jacobian matrices

[RW1] Regarding time complexity, although our mathematical equations involve Jacobian matrices, we do not need to explicitly compute these matrices to calculate gradients. Instead, as detailed in the supplementary materials, the gradient computation function for each type of layer (e.g., Linear, Convolution) can be defined directly. For example, in linear layers, gradient computation is implemented using linear transformation. In convolution layers, it is implemented using convolution operations. Consequently, similar to backpropagation algorithms, the complexity of gradient computation in our DP-ULR for each layer matches the order of the forward operation.

For example, in an MLP with $L$ layers and $N$ neurons per layer, the complexity of one forward pass is $O(LN^2)$. For backpropagation in SGD/DP-SGD, the backward pass complexity is $O(LN^2 + LN^2 + LN) = O(2LN^2 + LN) = O(LN^2)$. For DP-ULR, the gradient computation complexity is $O(LN + LN^2 + LN) = O(LN^2 + 2LN) = O(LN^2)$. A key distinction is that SGD/DP-SGD incurs an additional $LN^2$ term, while DP-ULR incurs an additional $LN$. This arises because backpropagation requires matrix multiplication for computing gradients of the input to each layer, while in DP-ULR, gradients are computed by multiplying the loss and the noise (as in the last two terms of Equation 4), which has a complexity of $O(N)$.

Additionally, we kindly remind the reviewer that our DP-ULR includes sample-level gradient clipping to constrain sensitivity. We clarify the difference between DP-ULR and DP-SGD on this point as follows. Due to the need to clip each individual gradient, resulting in an independent backward pass for each example, DP-SGD is slower than traditional SGD. In contrast, the calculation of a single estimated gradient in a DP-ULR is inherently independent, and a single clipping can be performed without increasing the computational cost compared to ULR.

[Q1] Complexity of one step in DP-ULR

[RQ1]: For the complexity of one step in DP-ULR, in addition to the previous response, we consider an MLP with $L$ layers and $N$ neurons. The complexity depends on the parallelism setting:

• No Interlayer Parallelism: The forward stage involves $L$ forward passes (independently perturbing $L$ layers), resulting in a total complexity of $O(L^2N^2)$. The "backward" stage (gradient computation) has a total complexity of $O(LN^2)$.
• Full Interlayer Parallelism: A favorable property inherited from ULR is that the $L$ forward passes and gradient computations for $L$ layers can be executed in parallel. This reduces the forward stage complexity to $O(LN^2)$ and the gradient computation complexity to $O(N^2)$.

Dear Reviewer ZJUY,

Thank you for your time and thoughtful review. With one day remaining in the discussion phase, we kindly request your feedback on our responses and look forward to your updated evaluation. If you have any additional questions or concerns, please feel free to let us know.

[W1] Relation between $K$ and privacy cost; Potential unsatisfied assumption

[RW1]: We understand your concern about the relation between repeat time $K$ and privacy cost. Based on our theoretical results, when the noise scale $\sigma$ satisfies Equation (8), the privacy cost is independent of repeat time $K$. Therefore, the privacy budget does not restrict the choice of $K$.

We appreciate your observation that violating the full-rank assumption of the covariance matrix could impair privacy guarantees. To address this, we proposed two remedies outlined in lines 297–301 and Appendix A.4, including adding extra noise and altering the location where noise is added. These two remedy approaches ensure that privacy guarantees remain unimpaired even if the assumption does not hold.

[W2] Parameter selections

[RW2]: Thank you for your thoughtful feedback on parameter sensitivity.

Selecting parameters in DP deep learning is inherently challenging due to the privacy-utility tradeoff. For DP-ULR, we provide the following insights.

• For noise scale, we first kindly note that the target std level $\sigma_0$ in DP-ULR is equivalent to the noise scale in DP-SGD. For DP-ULR, the injected noise standard deviation $\sigma$ is determined by privacy requirements, specifically $\sigma_0$, indicating no need to select it.
• As for batch size, our experiments demonstrate that DP-ULR performs better with larger batch sizes, implying a higher sample rate $q$. The results also exhibit greater robustness than DP-SGD with smaller batch sizes in terms of higher target std levels.
• The rejection threshold primarily affects privacy cost without significantly impacting model performance. For datasets with $N > 10^5$, this choice is trivial as long as it is smaller than $qN$.
• Additionally, increasing target std level $\sigma_0$ allows for reducing the repeat time $K$, increasing the inherent randomness leveragable for privacy.

[W3] Differences between DP-ULR and ULR.

[RW3]: Thank you for your suggestion to clarify our contributions. We appreciate your careful attention to detail.

While the original ULR method involves noise injection into logits and gradient estimation, our DP-ULR introduces several key differences to ensure differential privacy. First, we employ a novel batch formation strategy involving sampling with rejection. Second, DP-ULR incorporates a privacy-controlling mechanism, including an additional forward pass without noise to compute covariance matrices and required noise scales. Finally, DP-ULR introduces per-example gradient clipping, which is not present in ULR.

These enhancements distinguish DP-ULR as a privacy-guaranteed extension of ULR. In the initial submission, we included the explanations scattered in section 3.2. To better clarify our contribution, we have revised our introduction section to highlight the distinctions (and maintain detailed explanations in Section 3.2). Thank you again for your constructive feedback.

[Q1] Description revision suggestion

[RQ1]: Thank you for pointing out this point of clarification. In the paper, we denote $v^l$ as the $l$-th layer's output and $x^{l+1}$ as the $l+1$-th layer's input. In DP-ULR, noise is added to the $l$-th module’s output, and the perturbed output is used as the input for the next layer ($x^{l+1} = v^l + z$). This approach enables us to estimate the $l$-th layer's gradient. In Appendix A.1, we provide a list of symbols used in our paper to help readers better understand our notations and avoid confusion. We appreciate your attention to detail.

Dear Reviewer JZt3,

Thank you for your time and thoughtful review. With one day remaining in the discussion phase, we kindly request your feedback on our responses and look forward to your updated evaluation. If you have any additional questions or concerns, please feel free to let us know.