We thank the reviewer for their thoughtful and encouraging feedback. We appreciate the recognition of our paper’s clarity, the theoretical depth of our analysis, and the novelty of integrating streaming PCA into adaptive clipping for differentially private optimization. We are especially grateful for the acknowledgment of our method’s strong empirical performance and the practical significance of addressing gradient geometries in a computationally efficient way.

We respond to the specific points raised below.

Q1) Resource Overhead: A primary limitation of the method is ...

We thank the reviewer for raising this important point regarding the computational and memory overhead of our method. We acknowledge that storing and updating a full gradient covariance matrix can be expensive in large-scale settings. While much of modern ML focuses on high-dimensional data and large models, tabular datasets have long been, and continue to be, a central part of the ML landscape. These datasets typically have a relatively small number of features, making small models sufficient in many cases. At the same time, tabular data—particularly in domains like healthcare—often involve highly sensitive information, necessitating strict privacy guarantees. High privacy requirements typically limit the number of training iterations and lead to reduced utility. In such settings, our Algorithm 1 (with a full covariance matrix) is well suited: we can compute the full covariance matrix efficiently due to the manageable model size, and GeoClip’s fast convergence enables improved utility in just a few training iterations. We refer the reviewer to Tables 1–3 for results on tabular datasets, Figure 1 (left) for evidence of fast convergence, and Figure 1 (right) for how this fast convergence helps maintain strong privacy.

For larger models, our implementation of Streaming Rank-$k$ PCA maintains only the top-$k$ eigenvectors (out of $d$ total model parameters) and updates them using an $\mathcal{O}(dk^2 + k^3)$ SVD at each step (see the “Low-Rank PCA” section above Section 4). Since $k \ll d$, the cost is dominated by the linear dependence on $d$, making this step efficient even for large models. The memory overhead is also modest. For example, in our USPS experiment (Section 4.4), using a logistic regression model with $d = 2570$ and $k = 100$, Streaming Rank-$k$ PCA only stores a $d \times k$ eigenvector matrix and a length-$k$ eigenvalue vector—far smaller than the full $d \times d$ covariance matrix.

Although GeoClip introduces some computational and memory overhead, we believe this is well justified by the practical benefits. GeoClip converges significantly faster than baseline methods (see Figures 1 and 2), requiring fewer training steps to reach the target accuracy. While each iteration may be slightly more expensive than standard DP-SGD, this is partially offset by the reduced number of steps. Combined with improved privacy—since fewer steps lead to lower total privacy loss—these benefits make the overhead worthwhile.

Regarding the convergence of Streaming Rank-$k$ PCA, we empirically find that GeoClip with Streaming Rank-$k$ PCA maintains strong convergence behavior even for small $k$. As shown in Figure 2, the method still significantly outperforms baseline methods. A direct comparison between the full (Algorithm 1) and low-rank (Algorithm 2) approaches is included in our response to Reviewer 6gkB (Question 6), and we kindly ask the reviewer to refer to it. The results demonstrate that even with small $k$, the performance of Streaming Rank-$k$ PCA remains comparable to that of the full covariance method.

---

Q2) Lack of Large-Scale Experiments: While the technique is promising...

We thank the reviewer for highlighting this important direction. We agree that evaluating GeoClip on larger models is a valuable next step to further demonstrate the scalability of Algorithm 2 with Streaming Rank-$k$ PCA. This work serves as a proof of concept, and the results are encouraging enough to motivate applying the method to larger-scale models.

---

Q3) Missing Utility Guarantees: The paper does not provide explicit utility bounds ...

To relate our convergence bound to DP parameters, we note that the bound in Theorem 1 depends on the noise scale $\sigma$. Abadi et al. [1] show the following relationship between $\sigma$ and the privacy parameters $(\varepsilon, \delta)$, for some constant $c$, number of iterations $T$, and dataset size $n$:

$$\sigma = c \frac{\sqrt{T \log(1/\delta)}}{n \varepsilon}.$$

Substituting this into our convergence bound allows us to express it as a function of $\varepsilon$, $\delta$, and $n$.

Regarding the dependency of our convergence bound on dimensionality, if we simplify the expression in Theorem 1 using the bound on $\beta(a_t)$ from Equation (9) and the optimal transformation derived in Theorem 2 (note that the optimal solution occurs when the constraint in Equation (13) is active, i.e., $\operatorname{Tr}(M_t^\top M_t \Sigma_t) = \gamma$), the convergence bound depends on the dimensionality $d$ only through the term $\sum_{i=1}^d \sqrt{\lambda_{i}}$, where $\lambda_{i}$ is the $i$-th eigenvalue of the gradient covariance matrix. This reflects the effective dimension rather than the ambient dimension. In particular, if $\lambda_{i} = 0$ for some $i$, then that coordinate contributes nothing to the convergence rate. Thus, our bound naturally adapts to the structure of the gradient covariance and degrades gracefully in high-dimensional settings where the spectrum is concentrated in a low-dimensional subspace.

If accepted, we will include this discussion in the final version.

[1] Abadi et al., Deep Learning with Differential Privacy, in Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (CCS).

---

Q4) Convergence Under Approximation: What is the final convergence rate of GeoClip ...

We would like to highlight that although the proof of Theorem 1 assumes that $M_t$ is a full-rank $d \times d$ matrix, the same analytical steps extend to the case where $M_t$ is a $k \times d$ matrix with $k < d$, corresponding to a projection onto a lower-dimensional subspace. This is precisely the setting used in streaming rank-$k$ PCA. In this case, Equation (5) becomes:

$$\tilde{g}_t = M_t^+ \tilde{\omega}_t + a_t,$$

where $M_t^+$ denotes the (left) Moore-Penrose pseudoinverse of $M_t$. Consequently, in the convergence bound of Theorem 1, the term $\operatorname{Tr} \left[(M_t^\top M_t)^{-1}\right]$ is replaced by $\operatorname{Tr} \left[(M_t^\top M_t)^+\right]$, accounting for the lower-rank structure.

We thank the reviewer for highlighting this relevant line of work. While our current analysis does not explicitly assume a low-rank Hessian, our method is naturally aligned with this perspective. Intuitively, the directions where the loss changes most rapidly—i.e., those with high curvature—are also the directions where the gradients tend to vary most across data points. As a result, if the Hessian is approximately low-rank, most of the gradient activity is concentrated in the corresponding low-dimensional subspace. This helps explain why the principal components identified by Streaming PCA tend to align with the top eigenspace of the Hessian. Although we do not yet provide a formal analysis under this assumption, the connection is intuitive, and we see this as an interesting direction for future exploration.

---

Q5) Utility of Chosen Preconditioner: In Theorem 2, what is the final utility guarantee ...

We appreciate the reviewer’s comment. The final convergence bound for the optimal transformation matrix can be obtained by substituting the bound on $\beta(a_t)$ from Equation (9) and the optimal transformation derived in Theorem 2 into the expression in Theorem 1. We note that the optimal solution in Theorem 2 occurs when the constraint in Equation (13) is active, i.e., $\operatorname{Tr}(M_t^\top M_t \Sigma_t) = \gamma$.

The transformation matrix presented in Theorem 2 is chosen to minimize the convergence rate, given a fixed noise scale $\sigma$ determined by the $(\varepsilon, \delta)$-DP budget. While convergence rate is not a direct measure of utility, faster convergence implies that a target utility (e.g., loss or accuracy) can be achieved in fewer iterations. Since privacy loss accumulates over iterations, achieving convergence more quickly results in lower overall privacy cost. Therefore, this choice of preconditioner indirectly improves the privacy-utility trade-off.

---

Q6) Faster Preconditioning: Is it possible to prove faster convergence to optimal utility ...

We thank the reviewer for raising this question. In fact, our analysis in Theorem 1 was motivated by the intuition that faster convergence should enable the algorithm to reach a desired utility level in fewer iterations. While we did not explicitly optimize for the number of iterations, we focused on minimizing the convergence rate, which serves a similar purpose in practice. By accelerating convergence through a well-chosen preconditioner, we reduce the number of iterations required. These two objectives can be seen as closely related approaches to achieving efficient learning.

We hope this rebuttal effectively addresses the reviewer's concerns.

We thank the reviewer for their positive feedback and helpful comments. We are pleased that the reviewer found our methodology to be novel and insightful, appreciated the strong empirical results, and considered the paper well-written.

We respond to the reviewer’s comments in detail below.

I could not find some minor experimental details in the main paper or supplementary. The following is a list of some details, and I suggest the authors consider adding a section in the appendix providing more details for reproducibility.

We thank the reviewer for pointing this out—we agree that including these details is important for reproducibility. If accepted, we will add a dedicated section in the supplementary material that consolidates all relevant experimental details discussed here and includes additional information to improve clarity and completeness.

---

Q1) How was the lr tuned for experiments in section 4.1?

We tuned the learning rate via grid search over a logarithmic scale ranging from 0.0001 to 10.

---

Q2) Could authors provide exact details on the synthetic datasets used through the experimental section: e.g., what’s the exact distribution (i.e., how are the features correlated), what was the function for $y$ given $x$?

We appreciate the reviewer's comment. The synthetic dataset used in Section 4.1 consists of $n = 20{,}000$ samples and 10 features. To obtain the 5 correlated features, we first generate an $n \times 5$ matrix $Z$ and a $5 \times 5$ matrix $A$, each with entries drawn independently from the standard normal distribution. The correlated features are obtained by multiplying $Z$ by $A$. The remaining 5 features are drawn independently from a standard multivariate normal distribution. The full feature matrix $X$ is constructed by concatenating these two blocks. The target $y$ is generated using a linear function with Gaussian noise: $y = Xw + b + \epsilon$, where $w \sim \mathcal{N}(0, I_{10})$ is a weight vector, $b \sim \mathcal{N}(0, 1)$ is a scalar bias term, and $\epsilon \sim \mathcal{N}(0, 0.01^2)$ is i.i.d. noise.

For the synthetic dataset used in Section 4.4, the feature matrix $X$ is constructed in the same way. The logits are computed using the same linear function as before. The probabilities $p$ are then obtained via the sigmoid function, and binary labels are generated as $y = \mathbf{1}(p > 0.5)$ where $\mathbf{1}$ is the indicator function.

If accepted, we will add these details in the final version.

---

Q3) Could authors add what the grid search for hyperparameters were over for the various experiments, and details on the validation set (if it's not just the training set)

We thank the reviewer for the suggestion. In what follows, we explain how we tuned our hyperparameters for all experiments. If accepted, we will also include these details in the paper for completeness.

Parameter $\gamma$: As mentioned in the experimental results section on page 6, we set the parameter $\gamma$ to $1$, as its effect can be absorbed by tuning $h_2$. Therefore, we only tune $h_2$. We have also provided an explanation of how tuning $\gamma$ would have a similar effect as tuning $h_2$ in our response to Question 4 from Reviewer 6gkB, and we kindly refer the reviewer to that discussion.

Parameter $h_1$: We initially ran experiments using several different $h_1$ values within the range $[10^{-15}, 10^{-6}]$, and observed that our algorithms are robust to changes in $h_1$. Therefore, as stated in Section 4, we set $h_1 = 10^{-15}$ across all experiments.

Parameter $h_2$: We first tried $h_2$ values over a logarithmic scale in the range $[1, 1000]$, and observed that $h_2 = 1$ and $10$ perform consistently well across all datasets. Therefore, we select the best of these two values for each dataset in the final experiments.

Parameters $\beta_1, \beta_2, \beta_3$: We again tried several different values of $\beta_1, \beta_2$, and $\beta_3$ over the range $[0.9, 0.999]$, and observed that our algorithms are robust to the choice of these parameters. As stated in Section 4, we used standard values commonly used in optimization, such as $\beta_1 = 0.99, \beta_2 = 0.999$, and $\beta_3 = 0.999$, in all experiments.

Validation set: We would like to clarify that we used a separate validation set for tuning hyperparameters and choosing the best model for each dataset. As stated in Section 4, we split each dataset into 80-10-10 train-validation-test sets. For all experiments, we selected the best hyperparameters based on the model's performance on the validation set, trained the best model using different random seeds, and reported the average performance on the test set.

To further explore the questions raised by the reviewer, we also conducted an ablation study on $\gamma$, $h_1$, and $h_2$ using the USPS dataset, under the same settings as in Section 4.4. As shown, varying $\gamma$ does not significantly affect performance. The results indicate no noticeable impact from $h_1$, and only minor variations with $h_2$.

Test Accuracy (%) ( ablation study on $\gamma$ with $h_2 = 10$ and $h_1 = 10^{-15}$)

Test Accuracy (%) (ablation study on $h_2$ with $h_1 = 10^{-15}$ and $\gamma = 1$)

Test Accuracy (%) (ablation study on $h_1$ with $h_2 = 10$ and $\gamma = 1$)

---

Q4) Related, how was the batch size chosen in section 4.2 (it seems to be fixed for all methods?)

We specified the batch size used for each dataset in Section 4.2, within the captions of the tables presenting experimental results: batch size 32 for Diabetes, 64 for Breast Cancer, and 512 for Malware. If accepted, we will make these details more clearly stated in the main text.

---

Q5) How many trials were done for the experiments in section 4.1 and section 4.4? For the experiments in Section 4.1 and Section 4.4, we selected the best hyperparameters based on the model's performance on the validation set, trained the best model using $20$ different seeds, and reported the average performance on the test set. If the reviewer meant something else by ``trials", we humbly ask them to clarify it.

We hope our responses adequately address the reviewer's concerns.

We thank the reviewer for their valuable feedback. We are glad that the paper’s focus on improving gradient clipping in DP-SGD was recognized as important, and that our proposed coordinate transformation was seen as clean and intuitive. We also appreciate the positive feedback on the clarity of our theory and writing.

Below, we address the reviewer’s remaining comments.

Q1) While the idea of coordinate ...

We appreciate the reviewer’s comment. While our method can be seen as a generalization of AdaClip—as discussed in the paper—we emphasize that this extension is non-trivial, both conceptually and theoretically. Our analysis handles the complexity introduced by the basis transformation, going beyond AdaClip's coordinate-wise scaling.

Whereas AdaClip constructs clipping thresholds using only the coordinate-wise mean and variance of the gradients, our method estimates a structured transformation based on the eigen-decomposition of the gradient covariance matrix. Estimating eigenvalues and eigenvectors is a more sophisticated approach which then assures significantly better results. To this end, we designed an efficient streaming algorithm to estimate the top eigenvalues and eigenvectors in a scalable manner. Moreover, to the best of our knowledge, this is the first work to combine two lines of research: adaptive clipping and projection onto gradient subspace representations. We hope that our contributions in these directions are recognized as distinct and meaningful.

---

Q2) A key concern is the limited scope of empirical validation...

We appreciate the reviewer’s comment. To address this, we first clarify the experimental setups used in AdaClip [1] and in our work, and then highlight a challenging setting where GeoClip shows a clear advantage over existing baselines.

The AdaClip paper evaluated a synthetic regression task and MNIST, using only logistic regression and a one-hidden-layer neural network. For logistic regression, the full 784-dimensional MNIST input was used. For the neural network, the input was first reduced to 60 dimensions using differentially private PCA, followed by training a fully connected layer with 1000 hidden units. The privacy budget was split between the PCA step and model training. As far as we can determine from the cited paper, no CNNs or other deep architectures such as ResNets were evaluated, and MNIST was the only real-world dataset considered.

GeoClip with the full covariance estimator (Algorithm 1) was evaluated on synthetic, tabular, and image datasets. For tabular data, we used logistic or linear regression on Diabetes (10 features), Breast Cancer (30 features), and Android Malware (241 features). We also transferred a convolutional network pretrained on MNIST to Fashion-MNIST and fine-tuned the final fully connected layer under DP. This setting involved 510 trainable parameters. For higher-dimensional cases, Algorithm 2 (with streaming rank-$k$ PCA) was evaluated on a synthetic dataset and on USPS (256 features) using multiclass logistic regression (2570 parameters).

Regarding the practicality of GeoClip in challenging settings, we offer the following clarification. While much of modern ML focuses on high-dimensional data and large models, tabular datasets have long been, and continue to be, a central part of the ML landscape. These datasets typically have a relatively small number of features, making small models sufficient in many cases. At the same time, tabular data—particularly in domains like healthcare—often involve highly sensitive information, necessitating strict privacy guarantees. High privacy requirements typically limit the number of training iterations and lead to reduced utility. In such settings, Algorithm 1 is well suited: it allows for full covariance computation and benefits from GeoClip’s fast convergence, enabling improved utility in just a few training steps. We refer the reviewer to Tables 1–3 (tabular datasets) and Figure 1 (fast convergence) for supporting evidence.

For larger models and datasets, Algorithm 2 addresses similar challenges under strong privacy constraints. Our proposed streaming rank-$k$ PCA enables faster convergence, allowing a given level of utility to be achieved in fewer iterations. Please see Figure 2 in the main paper, as well as the privacy curves in Figure 4 of the supplementary material, which illustrate how fast convergence supports stronger privacy guarantees. We hope these helps address the reviewer’s concerns about the practicality of GeoClip.

We acknowledge that more experiments on large-scale settings would further support the scalability claims of Algorithm 2. This work serves as a proof of concept, and the results are encouraging enough to motivate applying the method to larger models.

[1] Pichapati, Venkatadheeraj, et al. ``AdaClip: Adaptive Clipping for Private SGD.'' arXiv preprint arXiv:1908.07643 (2019).

---

Q3) Related works that project gradients into lower-dimensional subspaces for utility improvement (see [1, 2]) ...

We appreciate the references provided by the reviewer. In what follows, we discuss the key differences.

A key difference between our approach and DP-SGD-JL [Bu et al., 2021] lies in the objective. DP-SGD-JL aims to improve computational efficiency—specifically memory and speed—by approximating per-sample gradient norms using Johnson–Lindenstrauss projections, while maintaining similar utility to standard DP-SGD. In contrast, our method is designed to improve the privacy–utility tradeoff.

We also differ from the approach in [Zhou et al., 2021], which assumes access to a public dataset drawn from the same distribution as the private data in order to estimate a low-dimensional gradient subspace, thereby avoiding additional privacy cost. Obtaining such a representative public dataset may not always be practical, and their analysis shows that the quality of the estimated subspace—and thus the effectiveness of the projection—depends on the size of the public data. In contrast, our method does not require a public dataset and incurs no additional privacy cost for identifying the subspace. Instead, we estimate the gradient geometry by reusing previously released noisy gradients.

If accepted, we will incorporate these comparisons into the paper.

---

Q4) The choice of the parameter $\gamma$ remains unclear, even when combined with $h_2$

In our formulation, $\gamma$ bounds the probability that a gradient is clipped (see the discussion above Equation (13)). The parameter $h_2$ also affects the clipping probability by influencing the clamped eigenvalues and consequently, the transformation matrix $M$. To simplify hyperparameter tuning, we set $\gamma = 1$ and tuned only $h_2$, since both parameters have similar effects.

We conducted an ablation study on $\gamma$ in our response to Question 3 from reviewer u6Le, and we kindly refer the reviewer to that discussion due to space limitations. If accepted, we will include a more extensive ablation study varying both $\gamma$ and $h_2$ independently.

---

Q5) A more principled or empirical justification of $h_1$ and $h_2$ is necessary.

We thank the reviewer for this comment. As noted on page 5, we introduce $h_1$ and $h_2$ to stabilize the transformation by avoiding issues from near-zero eigenvalues and limiting the influence of large, noisy ones. We had such ablation studies performed but missed adding them to the paper. We included them in our response to Question 3 from reviewer u6Le, and we kindly refer the reviewer to that discussion due to space limitations. The results show no impact from varying $h_1$, and only minor changes with $h_2$. We also refer the reviewer to our earlier response regarding the effect of $h_2$ on clipping.

---

Q6) If many eigenvalues of the covariance matrix are near zero...

We emphasize that both of our algorithms adapt noise and clipping to the geometry of the gradient distribution. Algorithm 2 (streaming rank-$k$ PCA) projects gradients from the $d$-dimensional space onto a $k$-dimensional subspace aligned with the top-$k$ eigenvectors of the gradient covariance. In contrast, Algorithm 1 (with full covariance) transforms gradients into the full eigenbasis and retains all $d$ directions. Crucially, noise is scaled by the corresponding eigenvalue, so directions with near-zero eigenvalues receive little to no noise. This makes Algorithm 1 a soft generalization of Algorithm 2: it retains all directions while still emphasizing dominant ones and is at least as expressive.

To support this, we compare the accuracy of GeoClip using Algorithm 1 and Algorithm 2 with $k=50$ on the same synthetic dataset from Section 4.4. As shown in Figure 2 of our paper, Algorithm 2 already outperforms other baselines. In the table below, we show that even with $k=50$, Algorithm 2 achieves comparable accuracy to Algorithm 1, which computes the full covariance. All results are averaged over 20 random seeds.

If accepted, these comparisons along with the corresponding plots, will be provided in the final version to further illustrate their relationship.

We hope these responses are satisfactory and convincing to the reviewer.

We thank the reviewer for the constructive and detailed feedback. The recognition of the principled motivation behind the method and the acknowledgment that extending Adaclip to the full-rank setting is a non-trivial and meaningful contribution are appreciated. It is encouraging that the resulting improvements were found to be significant, and that the experiments on small-scale datasets were seen as very convincing, demonstrating clear gains over the baselines.

We address the reviewer’s concerns below.

Q1) Small dataset

We appreciate the reviewer’s comment. While much of modern ML focuses on high-dimensional data, tabular datasets have long been, and continue to be, a central part of ML landscape. These datasets typically have a relatively small number of features, making small models sufficient in many cases. At the same time, tabular data—particularly in healthcare—often involve highly sensitive information, necessitating strict privacy. High privacy requirements limit the number of training iterations and lead to reduced utility. In such settings, Algorithm 1 is well suited: it allows to compute the full covariance matrix efficiently due to the manageable model size, and its fast convergence enables improved utility in just a few iterations.

We kindly refer the reviewer to our response to Question 6 from reviewer 6gkB, where we compare our two algorithms and show that even with a small $k$, the performance of Streaming Rank-$k$ PCA (Algorithm 2) remains comparable to that of Algorithm 1, which uses the full covariance matrix. This indicates that Algorithm 2 is effective under resource constraints.

We acknowledge that more experiments on large-scale settings would strengthen the scalability claims of Algorithm 2. This work serves as a proof of concept, and the results are encouraging enough to motivate applying the method to larger models.

---

Q2) Streaming Rank-k PCA

Our implementation maintains only the top-$k$ eigenvectors (out of $d$ total model parameters) and updates them using an $\mathcal{O}(dk^2 + k^3)$ SVD at each step (see the “Low-Rank PCA” section above Section 4). Since $k \ll d$, the cost is dominated by the linear dependence on $d$, making this step efficient even for large models if $k$ is small.

The memory overhead is also modest. For example, in USPS experiment (Section 4.4), using a logistic regression model with $d = 2570$ and $k = 100$, Streaming Rank-$k$ PCA only stores a $d \times k$ eigenvector matrix and a length-$k$ eigenvalue vector—far smaller than the full $d \times d$ covariance matrix.

Although it introduces some overhead, we believe it is well-justified by the practical gains. It converges significantly faster than baseline methods (see Figure 2), requiring fewer training iterations to reach target accuracy. While each iteration may be slightly more expensive than DP-SGD, this is partially offset by the reduced number of steps. Combined with improved privacy—since fewer steps lead to lower total privacy loss—these benefits make the overhead worthwhile.

If accepted, we will conduct more extensive benchmarking to compare the computation time and memory usage of GeoClip to the other versions of DP-SGD.

---

Q3) Decorreleted basis

While there may be a more direct relationship between input feature correlations and gradient coordinate correlations in linear models, this connection does not generally hold in nonlinear architectures. Nonlinear activations, weight sharing, and depth introduce interactions that can create dependencies between gradient coordinates—even when the input features are uncorrelated.

Consider a simple MLP with two independent inputs $x_1$ and $x_2$, a hidden layer computing
$h = \mathrm{ReLU}(w_1 x_1 + w_2 x_2)$,
and output
$y = \nu \cdot h$,
where $\nu$ is the output-layer weight. With squared error loss, the gradients are:

$$\frac{\partial L}{\partial w_1} = \alpha(x_1, x_2) \cdot x_1, \quad \frac{\partial L}{\partial w_2} = \alpha(x_1, x_2) \cdot x_2,$$

where

$$\alpha(x_1, x_2) = \left( \nu \cdot \mathrm{ReLU}(w_1 x_1 + w_2 x_2) - y_{\text{true}} \right) \cdot \nu \cdot \mathbf{1}_{w_1 x_1 + w_2 x_2 > 0}.$$

Although $x_1$ and $x_2$ are uncorrelated, both gradient components are scaled by the same nonlinear factor $\alpha(x_1, x_2)$, introducing dependency. These effects are amplified in deeper networks. Prior work shows that gradients often concentrate in low-dimensional subspaces [1], resulting in geometry that GeoClip is designed to exploit.

[1] Gur-Ari et al. Gradient Descent Happens in a Tiny Subspace.

---

Q4) LLM

We agree that evaluating the method on small LLMs would be a valuable direction to explore. This work was intended as a proof of concept, and our focus was on demonstrating the benefits of the proposed method in controlled settings. We plan to extend our experiments to include such models, particularly in the context of fine-tuning LLMs on sensitive datasets.

---

Q5) Missing baselines and related work

We appreciate the wide-ranging references provided by the reviewer. Our primary goal in comparing against existing variations of DP-SGD was to focus specifically on methods related to clipping. We acknowledge that some of the references pointed out by the reviewer make improvements to DP-SGD, but in aspects other than the clipping. For example, the first four references [Lee & Kifer (2020), Li et al. (2022), Bu et al. (2022), Bu et al. (2023)] focus on practical aspects such as improving speed, reducing memory usage, and lowering computational cost—directions that are orthogonal to our focus on the privacy-utility trade-off. On a different note, Béthune et al. (2024) takes a different approach by avoiding clipping altogether through Lipschitz-constrained networks, which is also complementary to our method. In the interest of maintaining focus, we elected not to include comparisons to these methods.

---

Q6) Decorrelation sanity check

The experiment in Section 4.1 involves linear regression on a synthetic Gaussian dataset with 10 features (five correlated). As suggested, we repeated the experiment with the same setup but with all 10 features drawn independently. We have collated our results in the Tables below.

In linear regression with uncorrelated features, gradient coordinates tend to become uncorrelated as the model is learned. Let

$x \sim \mathcal{N}(0, I_d)$

and

$y = (w^*)^\top x + \epsilon$

where $\epsilon$ is zero-mean noise with variance $\sigma^2$, independent of $x$. The gradient is $\nabla_w L = (w^\top x - y) \cdot x = e \cdot x,$ where $e = w^\top x - y = (w - w^*)^\top x - \epsilon.$

As training progresses and $w \to w^*$, the term

$(w - w^*)^\top x$

vanishes, and the residual becomes $e \approx -\epsilon$, which is independent of $x$. In this regime, the gradient simplifies to $\nabla_w L \approx -\epsilon x$, and the gradient covariance becomes

$$\mathbb{E}[\nabla_w L (\nabla_w L)^\top] \approx \mathbb{E}[\epsilon^2 x x^\top] = \mathbb{E}[\epsilon^2] \cdot \mathbb{E}[x x^\top] = \sigma^2 I_d.$$

This reflects uncorrelated gradient coordinates. This structure becomes more accurate later in training; during early iterations, the residual $e$ may still depend on $x$, and the gradient coordinates can be correlated. Thus, GeoClip tends to outperform AdaClip in the early stages of training (where only a small portion of the privacy budget has been consumed and the model operates in a higher privacy regime), and behaves similarly as the model converges. GeoClip consistently outperforms the baseline, as shown in Table 1.

Table 1 (Test MSE over 10 seeds)

We also conducted an additional experiment using the synthetic dataset from Section 4.4 (400 Gaussian features with logistic regression), this time with all features drawn independently. This illustrates how the nonlinearity induced by a sigmoid can introduce gradient correlations even under feature independence—structure that GeoClip is able to exploit. The results, shown in Table 2, align with our response to the reviewer’s earlier comment on the decorrelated basis.

Table 2 (%) (Test Accuracy over 10 seeds)

---

Q7) Quantile-based clipping

Combining with quantile-based clipping is unlikely to help, as our method already derives an optimal transformation matrix that includes both a change of basis and clipping.

---

Q8) Full covariance

At this point, we are not aware of differentially private algorithms that explicitly use the full gradient covariance; however, there is a growing interest in enhancing Adam-like optimizers with methods that require full gradient covariance matrices. Such approaches, with or without privacy could directly benefit from our Streaming Rank-$k$ PCA.

We hope these responses convincingly address the reviewer's concerns.