Adaptive Sigmoid Clipping for Balancing the Direction–Magnitude Mismatch Trade-off in Differentially Private Learning

We are grateful for the reviewer’s insights.

– Does the deep learning model studied in the experimental section (Section 8) satisfy the assumptions made in the theoretical analysis (Section 7)? If not, it is unclear what the role of the theoretical section is, given that the motivation and intuition behind the proposed clipping method are already well-articulated earlier in the paper. Clarifying the connection between the theory and the experiments would strengthen the overall contribution.

The purpose of our theoretical analysis in Section 7 is to show that the proposed adaptive clipping retains the best-known convergence rate in the non-convex loss setting. This analysis serves as a basis for understanding the convergence behavior of the algorithm. It is natural to impose some assumptions on the loss function in order to theoretically characterize convergence. While these assumptions may not hold exactly in practice, they still offer valuable insight into how different parameters can affect the convergence bound.

The assumptions we use—such as smoothness and bounded gradients—are standard in the literature [1-4] and broad enough to cover a wide range of models and loss functions. Although some of these may only partially hold for the deep learning models used in our experiments, the theoretical results remain informative and provide useful guidance for the design and behavior of our method in practical settings.

We thank the reviewer for their suggestion on clarifying the role of the theoretical analysis, and we will reflect this discussion in the final version of the paper.

[1] Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, and George Karypis. Automatic clipping: Differentially private deep learning made easier and stronger. In NeurIPS, 2023.

[2] Xiangyi Chen, Steven Z Wu, and Mingyi Hong. Understanding gradient clipping in private SGD: A geometric perspective. In NeurIPS, 2020.

[3] Di Wang, MinweiYe, andJinhui Xu. Differentially private empirical risk minimization revisited: Faster and more general. In NeurIPS, 2017.

[4] Xinwei Zhang, Zhiqi Bu, Steven Wu, and Mingyi Hong. Differentially private SGD without 440 441 clipping bias: An error-feedback approach. In ICLR, 2024.

We appreciate the reviewer’s feedback.

Q1. Is it necessary to adopt the sigmoid function in this context? What are the unique benefits of using it compared to other functions, such as $\tilde{\mathbf{g}}{i,t} = \frac{C \mathbf{g}{i,t}}{\Vert \mathbf{g}{i,t} \Vert + r}$ in [3] and $\tilde{\mathbf{g}}{i,t} = \frac{C \mathbf{g}_{i,t}}{\Vert \mathbf{g}{i,t} \Vert + \frac{r}{\Vert \mathbf{g}{i,t} \Vert + r}}$ in [32]. A theoretical comparison and experimental results would be helpful.

The sigmoid function is simple yet well-suited to our goal of adaptively balancing the trade-off between direction and magnitude deviations. In particular, it enables control over the range of the linear span in Fig. 1 through the parameter $\alpha$. The sigmoid function provides greater flexibility and control compared with the clipping functions proposed in [32] and [3], even if those functions are extended with learnable parameters.

The structure of the clipping function in [32] is such that the span of its linear region changes within a small range as $r$ varies. We are not allowed to include plots at this stage, but this effect is visible when the function is plotted for various values of $r$. We will incorporate these plots in the final version of the paper. This behavior can also be understood by analyzing the function in extreme cases. Due to the presence of the term $\frac{r}{x + r}$ in the denominator of PSAC, varying $r$ from $0$ to $\infty$ causes the function to transition between two limiting values: $1$ and $\frac{x}{x + 1}$. Hence, the span of its linear region is restricted between these two curves, and adaptively updating $r$ does not effectively balance the trade-off between direction and magnitude deviations. In contrast, the sigmoid function used in AdaSig can effectively span from values close to 1 down to curves near 0, thereby flexibly balancing the trade-off.

While the clipping function in [3] does not suffer from this limited range of the linear span, it is less flexible than the sigmoid function in balancing the trade-off, as its values are less responsive to changes in $r$. Therefore, learning the optimal parameter requires greater variation across different values, whereas the sigmoid function, with its exponential change in $\alpha$, provides more flexible control over the linear region and thus, the direction-magnitude mismatch trade-off.

Q2. Would introducing a learnable parameter $r$ in the equation $\tilde{\mathbf{g}}{i,t} = \frac{C \mathbf{g}{i,t}}{\Vert \mathbf{g}{i,t} \Vert + \frac{r}{\Vert \mathbf{g}{i,t} \Vert + r}}$ in [32] also achieve comparable performance to AdaSig? It would be valuable to include experimental results evaluating a learnable $r$ in [32] for a direct comparison.

We first would like to clarify that deriving a differentially private update for the parameter of the clipping function is not straightforward and involves several steps that depend on the specific form of the clipping function. Specifically, the loss function must first be expressed in terms of the learnable parameter to compute its derivative w.r.t. that parameter. Then, to construct the private update, it is necessary to identify a constant upper bound on this derivative to guarantee bounded $\ell_2$-sensitivity, which allows the update to be privatized by adding a suitable amount of Gaussian noise.

Following the reviewer’s suggestion, we have carried out the derivations and conducted experiments for the method in [32]; however, since the calculations exceed the allowed space, we are unable to include them here. Furthermore, due to the extensive tuning required to find the optimal initial value for $r$, we were only able to perform experiments on the CIFAR-10 dataset. The best accuracy obtained by this method was $61.96 \pm 0.15\%$, which is not significantly different from that of the original method. Considering the limited span of the linear region in [32], as discussed in our response to Q1, we believe these results are intuitive, since even with a learnable $r$, it has limited capacity to balance the direction-magnitude mismatch trade-off.

We will conduct experiments on other datasets and include the results in the final version of the paper if the reviewer considers them necessary.

Q3. What is the main technical difference between AdaSig and the method in [11]? AdaSig appears to be a simple application of the method in [11]. A more detailed explanation would be beneficial for better understanding the novelty of AdaSig.

First, we emphasize that AdaSig does not use the method in [11]. In [11], vanilla clipping is extended by optimizing the clipping threshold over iterations to balance the trade-off between bias and variance. Therefore, [11] does not introduce a new clipping methodology, but rather a new approach to applying vanilla clipping. Furthermore, it is unclear whether the method in [11] provides a theoretical convergence guarantee, as none is given in that paper.

Despite both [11] and our work optimizing a learnable parameter, there are three key differences between them: (i) AdaSig introduces a novel clipping methodology, whereas [11] builds on existing vanilla clipping as mentioned above. (ii) The optimized parameters differ, with AdaSig tuning the saturation slope and [11] tuning the clipping threshold. (iii) The purpose of the optimization is fundamentally different. AdaSig aims to balance the trade-off between the two components of bias, namely the direction and magnitude deviations, while keeping the variance fixed. In contrast, [11] seeks to balance the trade-off between bias and variance. More specifically, since AdaSig uses a sigmoid-based clipping function, optimizing the saturation slope $\alpha$ does not affect the output range. As a result, the $\ell_2$ sensitivity remains constant when changing $\alpha$ in AdaSig, leading to a fixed variance for the Gaussian noise used in the privacy mechanism. On the other hand, in [11], updating the clipping threshold changes the $\ell_2$ sensitivity, which in turn affects the variance of the added privacy noise.

Q4. How does $\alpha$ evolve throughout the training process? Providing a traceable trajectory of $\alpha$ could offer valuable insights into the adaptive behavior of the model.

Since we are not allowed to include figures in our response, we provide a verbal description of the $\alpha_t$ behavior here and will include the plots in the final version.

For example, on CIFAR-10, we observe that the average $\alpha_t$ increases from the initial value of 1.0 to just below 1.5 within the first 20 iterations. It then steadily decreases until around iteration 200, after which it begins to rise again, ultimately surpassing 1.1 by the end.

Observing the changes in $\alpha_t$ over iterations across various datasets and models, we find no consistent trend in its variation. We conjecture that this is due to $\alpha_t$'s dependence on the gradients of individual samples, which can vary significantly across different datasets and model architectures.

We thank the reviewer for pointing out the typos. We will carefully revise the manuscript to correct them.

We thank the reviewer for their comments and address their concerns below.

1- Can you explain why a dynamic adaptation to $\alpha$ is needed and why a fixed $\alpha$ does not suffice?

The value of $\alpha$ affects the span of the linear region, as illustrated in Fig. 1: a smaller $\alpha$ leads to a larger linear region (e.g., see the curves for $\alpha = 0.8$ and $\alpha = 10.0$). Thus, varying $\alpha$ allows us to adjust the span of the linear region and control the trade-off between direction and magnitude deviations throughout training. In contrast, a fixed value of $\alpha$ results in a static linear region span. Since the distribution of sample gradient norms $||\mathbf{g}_{i,t}||$ can vary significantly during training, using a fixed $\alpha$ across all iterations cannot effectively balance this trade-off at all times.

To make this more concrete, consider Fig.1 and assume we have selected a fixed $\alpha = 0.8$ across all iterations. After several iterations, the maximum norm of the sample gradients within a batch may decrease significantly. In this case, it becomes possible to shrink the span of the linear region to reduce the magnitude deviation without causing excessive direction deviation. Recall from Fig. 1 that smaller linear regions, which correspond to larger $\alpha$ values such as $\alpha = 10$, introduce less magnitude deviation and more direction deviation. Thus, when the gradient norms are smaller, a larger $\alpha$ (that is, a narrower linear region) becomes preferable, as the risk of large direction deviation is mitigated due to the smaller gradient magnitudes. It is worth mentioning that this is just an illustrative case; in practice, gradient statistics may vary arbitrarily over time. Furthermore, the optimal adjustment of $\alpha$ depends not only on the maximum norm but on the overall distribution of the sample gradients. Our algorithm leverages this richer information to dynamically update $\alpha$ during training.

We have experimentally validated the benefit of using an adaptive $\alpha$ in Appendix H.2 in the supplementary file. In Table 8, we show that the performance achieved with a fixed $\alpha$ remains consistently below that of our adaptive approach, highlighting the benefit of dynamically adjusting $\alpha$ during training.

2- Can you explain what your algorithm is to update $\alpha$?

Our algorithm updates $\alpha$ to reduce the learning loss while preserving privacy. In brief, the procedure for updating $\alpha$ in the $t$-th iteration is as follows:

1. Computing $\hat{s}_t$ which is defined in equation (9b). This value is already computed in the model updating step in equation (10).
2. Computing $\alpha_{t+1}$ from equation (15), using the value of $\hat{s_t}$ and $\hat{r}_{t-1}$, which was computed in the previous step.
3. Computing $\hat{r}_t$ using equation (13) and the definition of $r$ provided in Lemma 5.1. $\hat{r}_t$ will be used in the next iteration for $\alpha$ update in step 2.

For further clarity, we kindly refer the reviewer to Algorithm 1 in Appendix A. We will also include additional clarifications in the algorithm description in the final version of the paper to make this procedure more transparent.

3- Can you explain the novelty of the idea w.r.t. Auto-S, NSGD and PASC? These also seem to address the same issues with somewhat similar ideas. I read the discussion in related works, but did not quite catch the difference. Examples may help.

Auto-S clipping (also known as NSGD) was proposed to eliminate the need for tuning the clipping threshold $C$ in vanilla clipping, which affects the bias-variance trade-off. This clipping operation is not designed to mitigate the deviation effects introduced by clipping. As shown in Fig. 1, Auto-S has a small linear region, which leads to significant direction deviation. This observation has also been validated numerically in [1].

Later, PSAC clipping was proposed to address the direction mismatch of Auto-S by assigning smaller weights to sample gradients with smaller norms. As shown in Fig. 1, the PSAC curve has a wider linear region to serve this purpose. However, the PSAC clipping has a fixed linear region; that is, it cannot adaptively expand or shrink its linear region to balance the trade-off between direction and magnitude deviations (see also our response to the first question).

In contrast, our proposed clipping method is based on an adaptive sigmoid function that can dynamically balance the aforementioned trade-off by adjusting its linear span through varying $\alpha$.

Minor comments: The explanations in lines 68-75 and 152-166 could be made more clear. They are important for understanding the paper, but not easy to follow.

In lines 68–75, we discuss the issue of magnitude deviation and its trade-off with the direction deviation. The span of the linear region in Fig. 1 governs this trade-off. A larger linear region—corresponding to smaller values of $\alpha$—reduces direction deviation but increases magnitude deviation. This occurs because, for gradients with small norms (points close to 0 on the horizontal axis), the red curve (with a larger linear region) scales down the norm of the clipped gradients more severely than the green curve (with a smaller linear region), as shown in Fig. 1. This effect is further illustrated in Fig. 2(c) and 2(d). The dashed green arrow in Fig. 2(c) corresponds to the aggregation of clipped gradients for a large $\alpha$ value (i.e., smaller linear region), the dashed red arrow in Fig. 2(d) represents the aggregation of clipped gradients with a small $\alpha$ value (i.e., larger linear region), and the dashed blue arrow shows the summation of unclipped gradients. As shown, the dashed green arrow has a closer norm to the dashed blue arrow compared with the dashed red arrow. In contrast, the dashed red arrow exhibits reduced direction mismatch relative to the dashed blue arrow, but at the cost of increased magnitude deviation. In summary, adjusting the value of $\alpha$ controls the size of the linear region and thus allows for balancing the trade-off between direction deviation and magnitude deviation.

Lines 152-166 also refer to the same aforementioned trade-off, providing further elaboration. We will revise the indicated sections of the paper to enhance clarity.

[1] Tianyu Xia, Shuheng Shen, Su Yao, Xinyi Fu, Ke Xu, Xiaolong Xu, and Xing Fu. Differentially 426 private learning with per-sample adaptive clipping. In Proc. of the AAAI Conference on Artificial 427 Intelligence, 2023.

I have no further comments. I have to admit that I do not completely get some aspects, but there does not seem to be any major flaws.

We thank the reviewer for their valuable feedback.

Room for improvement 1.) RDP bound in Noisy-SGD in [1]:

We appreciate the reviewer’s reference to [1], which provides an important contribution by showing that the RDP privacy loss of Noisy-SGD can converge to a bounded value as the number of iterations $T \to \infty$, in contrast to standard analyses where the privacy loss grows unbounded.

However, it is important to note that the analysis in [1] assumes unclipped gradients, as also mentioned in the discussion section of their paper. In contrast, our work focuses on the clipped setting; specifically, our analysis focuses on how adaptive clipping influences both training convergence and privacy guarantees.

We believe extending the theoretical results for RDP analysis in [1] to the clipped regime remains an open and nontrivial direction. Once addressed, it could potentially open the door to analyzing more advanced clipping strategies, including our proposed AdaSig method.

Room for improvement 2.) How does the following clipping result for DP compare with your proposed result? [2]. Please cite and compare. Similarly, do so with [3]? which is for a narrower class of functions.

We thank the reviewer for pointing out these two recent works [2] and [3]. We will cite these two works to suggest potential directions for future research. Inspired by [2], one possible direction is to extend our AdaSig approach to the federated learning (FL) framework. Moreover, drawing on the methodology of [3], developing a high-probability convergence analysis would be a valuable theoretical extension of our work. Below, we highlight the key distinctions between our work and these papers:

Reference [2]:

The work in [2] considers a Federated Learning (FL) setting and proposes an algorithm that combines vanilla clipping, momentum, and error feedback. The authors show that this combination achieves an optimal convergence rate under client heterogeneity.

While this is an important contribution for the FL setting, we note that the paper does not introduce a new clipping strategy, i.e., it relies on standard (vanilla) clipping. In contrast, our work focuses on the design of a new adaptive clipping mechanism, AdaSig, which dynamically balances direction and magnitude deviations in the clipped gradients, while maintaining optimal convergence guarantees. We note that a comparison with vanilla clipping used in [2] is already included in our experiments.

Moreover, our work focuses on centralized (non-distributed) learning, whereas [2] addresses the FL setting, where client heterogeneity is a key challenge. This challenge does not arise in our scenario. That said, our proposed AdaSig clipping strategy is general and can also be applied in the FL setting. Like any clipping method, it can be combined with techniques such as error feedback and momentum to enhance performance under heterogeneous clients. We view this as a promising direction for future studies.

Reference [3]:

The work in [3] provides a high-probability convergence bound for vanilla clipped SGD under $(L_0, L_1)$-smoothness assumptions and heavy-tailed noise. In contrast, our work focuses on proposing a new adaptive clipping strategy, AdaSig, which aims to balance the distortion introduced by clipping in both gradient direction and magnitude, and establishes a convergence bound for this adaptive method.

From a theoretical perspective, we note that the high-probability convergence analysis in [3] for vanilla clipping could potentially be extended to our adaptive clipping method. However, this extension is non-trivial and represents a promising direction for future work.

From an experimental standpoint, we emphasize that our proposed method has already been compared against vanilla clipping, which is the clipping method in [3].

• How are the seemingly similar-looking performance numbers in Table 1 justified as an improvement? Please provide some additional context to help understand the suggested gains. For example, PSAC 86.35 ± 0.19 Vs. proposed method at 86.68 ± 0.04 in the 2nd row of Table 1 with Fashion MNIST.

We would like to clarify that we performed a statistical $t$-test to assess the significance of the results. Specifically, the bold entries in Table 1 indicate cases where the $p$-value is less than 0.05, demonstrating that the observed improvements are statistically significant [4].

In the specific example mentioned (second row), although the mean values are close (86.35 vs. 86.68), the $p$-value is below 0.05, confirming the statistical significance of the improvement. Furthermore, our method achieves a smaller confidence interval compared with PSAC in this case, indicating greater stability. We also note that in several other cases (e.g., rows 3, 4, and 7), the improvements over PSAC are more pronounced.

– Figure 3 caption: ImageNette should be ImageNet.

We clarify that ImageNette [5] is a publicly available subset of ImageNet. It consists of 10 easily distinguishable classes from the original ImageNet dataset and is commonly used when full ImageNet training is computationally prohibitive. We will include a citation to [5] in the final version of the paper for clarity.

[1] J. Altschuler and K. Talwar, “Privacy of noisy stochastic gradient descent: More iterations without more privacy loss,” Advances in Neural Information Processing Systems, vol. 35, pp. 3788–3800, 2022.

[2] R. Islamov, S. Horvath, A. Lucchi, P. Richtarik, and E. Gorbunov, “Double Momentum and Error Feedback for Clipping with Fast Rates and Differential Privacy,” arXiv preprint arXiv:2502.11682, 2025.

[3] S. Chezhegov, A. Beznosikov, S. Horváth, and E. Gorbunov, “Convergence of Clipped-SGD for Convex $(L_0, L_1)$-Smooth Optimization with Heavy-Tailed Noise,” arXiv preprint arXiv:2505.20817, 2025.

[4] Greenland, S., et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. In European journal of epidemiology, 2016.

[5] J. Howard, ImageNette: A subset of 10 easily classified classes from ImageNet, fast.ai, 2019.