Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness

We thank the reviewer for the positive feedback. The remaining questions are addressed below.

Although the numerical results, especially the result in Fig. 2, are interesting, more experimental results on large-scale model training (e.g., the GPT-2 model series) and the ablation study are lacking and would be valuable.

To address the reviewer's comment, we carry out the following additional experiments:

• training a NanoGPT with 1B parameters in addition to the 124M parameter model in the paper
• training a ViT on ImageNet
• a sensitivity study of the radius $\rho$

NanoGPT 1B:

For the above NanoGPT experiments we observe that:

• The optimal learning rate transfers for ClippedScion (this is not the case for Adam). We make sure that the learning rate is not suboptimal for any of the methods by sweeping over a dense grid (multiples of 2) and ensuring that the best learning rate is not on the boundary of this grid.
• Clipping leads to a 10% speedup over Scion also for the 1B model. The speedup is measuring how much faster ClippedScion can achieve the same final validation loss of Scion.

ViT on ImageNet: We also run additional experiments of DeiT-base model on ImageNet following the Scion codebase, which contains a tuned Scion baseline. Specifically, we train a DeiT-base model for 200 epochs with cosine learning rate scheduling and make a reasonable guess of $\rho$ for ClippedScion. We observe that Unconstrained ClippedScion achieves 0.2% improvement over Unconstrained Scion. Note that this improvement is without tuning $\rho$, since one run for this additional experiment already required 16xH200 GPUs for 12h. We will include this experiment in the final version of the paper.

Sensitivity to $\rho$: We conduct additional experiments to test the sensititve to $\rho$. The sweep range is set according to the gradient norm from Figure 2 (right). The final performance is very stable across different $\rho$.

Note that the two extreme cases of steepest descent ($\rho=\infty$) and unconstrained conditional gradient ($\rho=0$) both have significantly worse performance, which provides further evidence of the advantage of the hybrid clipping method. We will include a plot in the final version of the paper.

Can the bounded variance assumption used for the stochastic convergence results in Section 4.2 be relaxed?

Yes, we expect that it should eventually be possible to relax the standard bounded variance assumption we have used, although we have not investigated this in detail. We are aware of several works that have relaxed this assumption in other algorithms (e.g., SGD) and it seems possible that the techniques used there could be combined with our analysis of GGNC to produce some results. Further studies on this, as well as clipping the gradient before adding it to the momentum estimator (which is known to be helpful for SGD with momentum in the case of heavy-tailed noise), are certainly fruitful directions to continue this work in.

Why is the generalized $(L_0,L_1)$-smoothness condition a suitable condition to study for neural networks, as claimed in Conclusion?

The best empirical evidence we are aware of is in a concurrent work on non-Euclidean $(L_0,L_1)$-smoothness [1] in which the assumption is emperically validated (see Section 5, specifically Figure 2 and Figure 4 on NanoGPT and CNN respectively). It seems that the assumption very closely matches the estimated Lipschitz constant between consecutive iterates (this is ultimately what is needed for the proof).

[1] Artem Riabinin, Egor Shulgin, Kaja Gruntkowska, Peter Richtárik "Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)"

Can GGNC be compared with the Muon optimizer [Jordan et al., 2024b], for example, on NanoGPT?

The main mechanism used in Muon is the spectral LMO, computed using the Newton-Schulz iterations. We compare in our paper to Scion, which we believe is a strong representative of a spectral LMO based method, since it performs slightly better than Muon on NanoGPT training and since it provides a way to directly see the effect of clipping. Between Scion and Clipped Scion, the only difference is the clipped stepsize. In contrast, if we tried to compare clipped Scion to Muon, we would have to account for the fact that Muon uses AdamW on the vector parameters and a different momentum, which would hinder interpretation.

By the way, there is a typo on Line 290, where Figure 2 should be replaced with Figure 1.

Thanks for catching the typo, we will correct it.

We thank the reviewer for the feedback and address the remaining concerns and questions below.

The empirical performance of the proposed algorithm appears nearly identical to the unclipped version.

We politely disagree. The difference in validation loss translates into a 10% speedup for NanoGPT, which we consider substantial. We additionally carry out the following experiments in order to provide further evidence:

• training a NanoGPT with 1B parameters in addition to the 124M parameter model in the paper
• training a ViT on ImageNet
• sensitivity of $\rho$

NanoGPT 1B:

For the above NanoGPT experiments we observe that:

• The optimal learning rate transfers for ClippedScion (this is not the case for Adam). We make sure that the learning rate is not suboptimal for any of the methods by sweeping over a dense grid (multiples of 2) and ensuring that the best learning rate is not on the boundary of this grid.
• Clipping leads to a 10% speedup over Scion also for the 1B model. The speedup is measuring how much faster ClippedScion can achieve the same final validation loss of Scion.

ViT on ImageNet: We also run additional experiments of DeiT-base model on ImageNet following the Scion codebase, which contains a tuned Scion baseline. Specifically, we train a DeiT-base model for 200 epochs with cosine learning rate scheduling and make a reasonable guess of $\rho$ for ClippedScion. We observe that Unconstrained ClippedScion achieves 0.2% improvement over Unconstrained Scion. Note that this improvement is without tuning $\rho$, since one run for this additional experiment already required 16xH200 GPUs for 12h. We will include this experiment in the final version of the paper.

Sensitivity to $\rho$: We conduct additional experiments to test the sensitivity to $\rho$. The sweep range is set according to the gradient norm from Figure 2 (right). The final performance is very stable across different $\rho$.

Note that the two extreme cases of steepest descent ($\rho=\infty$) and unconstrained conditional gradient ($\rho=0$) both have significantly worse performance, which provides further evidence of the advantage of the hybrid clipping method. We will include a plot in the final version of the paper.

The motivation for using gradient norm clipping is inadequately explained. The approach does not appear to improve the convergence rate.

Indeed, the convergence rate we find is matching the one given in [1]. Ultimately, this is a worst-case convergence rate and, in practice, the behavior of the two algorithms is not always the same, even if there exists some pathological function for which both algorithms would have the same rate of convergence. This is evidenced by our accompanying numerical experiments, which confirm the benefits of clipping under constant learning rate in a practical setting like training a NanoGPT network.

[1] Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls and Volkan Cevher, "Training Deep Learning Models with Norm-Constrained LMOs"

Is the $(L_0, L_1)$-smoothness condition valid in non-Euclidean settings? Do you have empirical verification of this assumption?

Yes, there is evidence to support that. In a concurrent work on non-Euclidean $(L_0,L_1)$-smoothness [1], the assumption is empirically validated (see Section 5, specifically Figure 2 and Figure 4 on NanoGPT and CNN respectively). It seems that the assumption very closely matches the estimated Lipschitz constant between consecutive iterates (this is ultimately what is needed for the proof).

[1] Artem Riabinin, Egor Shulgin, Kaja Gruntkowska, Peter Richtárik "Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)"

Compared to Pethick et al. (2025), the optimal convergence rate remains the same in the stochastic case. Why is there no improvement despite additional smoothness assumptions?

The $(L_0,L_1)$-smoothness assumption is a strictly weaker assumption (i.e., more general, thus allowing for a broader class of functions).

Pethick et al. (2025) use LMOs to avoid computing the dual norm, but your paper requires dual norm computation for gradient norm clipping. Is this why your learning rate depends on the Lipschitz constant while Pethick et al. (2025) does not?

Yes ClippedScion needs to set the stepsize clipping factor $\rho$, but can then converge faster in practice as seen in the experiments.

When obtaining $\mathrm{lmo}(d)$, $x$ is already constrained within domain $\mathcal{D}$. Why not simply control the norm-ball $\mathcal{D}$ directly?

Decreasing the radius is in some sense what Scion does when the stepsize is taken diminishing ($\rho$ and $\gamma$ plays the same role in the unconstrained case as e.g. seen in Theorem 4.5).

Thank you for the positive review - we remain available if there are any further questions.

We thank the reviewer for the thorough feedback and address the remaining concerns and questions below.

I would consider increasing my score if robust large-scale experiments with adaptive stepsizes and stronger baselines show clear benefits, or if you provide a cheap approximation of the lmo/sharp operator that preserves convergence.

Large scale experiments Regarding large scale experiments, the 10% speedup in Figure 2 is observed on a 124M parameter model (we will add the missing parameter count to the paper). We ran additional experiments to check if the conclusion is robust to even larger model sizes, specifically a 1 billion parameter model:

For the above NanoGPT experiments we observe that:

• The optimal learning rate transfers for ClippedScion (this is not the case for Adam). We make sure that the learning rate is not suboptimal for any of the methods by sweeping over a dense grid (multiples of 2) and ensuring that the best learning rate is not on the boundary of this grid.
• Clipping leads to a 10% speedup over Scion also for the 1B model. The speedup is measuring how much faster ClippedScion can achieve the same final validation loss of Scion.

We also run additional experiments of DeiT-base model on ImageNet following the Scion codebase, which contains a tuned Scion baseline. Specifically, we train a DeiT-base model for 200 epochs with cosine learning rate scheduling and make a reasonable guess of $\rho$ for ClippedScion. We observe that Unconstrained ClippedScion achieves 0.2% improvement over Unconstrained Scion. Note that this improvement is without tuning $\rho$, since one run for this additional experiment already required 16xH200 GPUs for 12h. We will include this experiment in the final version of the paper.

Cost of LMO computation/wall-clock time Regarding the computation of the lmo/sharp operator, the only relevant norm-choice for which these objects cannot be computed in closed form is the spectral norm. Fortunately, we can efficiently approximate the spectral LMO through a fast implementation of the Newton-Schultz iterations (see the experimental section). In practice, the Newton-Schultz iterations add a ~0.7% overhead, e.g., for NanoGPT (see [1], which our experiments are also in agreement with). This already small overhead shrinks proportionally with increasing bathsize and depth of the model.

Compared with the speedup over Adam, the overhead is negligible. The wall-clock time speedup of Scion over Adam for NanoGPT in Figure 2 is 34%, while for ClippedScion it is 39%. We will include wall-clock time in the final version.

[1] Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse and Jeremy Bernstein, "Muon: An optimizer for hidden layers in neural networks"

Sensitivity to ρ: Figures use a fixed clipping threshold, but how robust is performance to mis-specifying ρ by ±2×? A sweep would strengthen claims.

We conduct additional experiments to test the sensititve to $\rho$. The sweep range is set according to the gradient norm from Figure 2 (right). The final performance is very stable across different $\rho$.

Note that the two extreme cases of steepest descent ($\rho=\infty$) and unconstrained conditional gradient ($\rho=0$) both have significantly worse performance, which provides further evidence of the advantage of the hybrid clipping method. We will include a plot in the final version of the paper.

Notation is heavy, and several definitions (e.g., "product norm" radii) are buried mid-paragraph, making reproduction difficult.

For reproduction we have summerized the full algorithm in the appendix (Algorithm 3 and 4), which includes the per layer radii explicitly. This should provide an overview of what steps are needed for the implementation. We didn't include it in the main text due to space limitations, but we will consider pulling it up.

The supplementary also includes a compact pytorch implementation, which should make it straightforward to reproduce.

Experimental section omits many details (network width, regularization, hardware, etc.); Appendix C only lists a subset.

• The network width is indeed missing from the hyperparameter table, thanks for catching that. The NanoGPT width is 768 which makes it a 124 million parameter model, which is the default in the NanoGPT repository that we build on.
• The cifar10 experiment uses the default data augmentation from the airbench repository that we build on. We will add a description to the paper.
• The hardware used is specified in beginning of Appendix C.

Theory largely recycles existing proof templates for Euclidean clipping; novelty is modest.

We respectfully disagree with the framing of building on prior work as recycling existing proof templates. The way we are controlling the magnitude of the update is completely different to Euclidean clipping alone and so is the analysis that follows; prior works did not use non-Euclidean LMOs (e.g., spectral LMO) in combination with a clipped step size before our work, nor was it common in practice.

Prior concurrent works (Gorbunov et al., Vankov et al.) already extend clipping under relaxed smoothness; the non-Euclidean lift is incremental. Core idea—clip the gradient in a dual norm—has been folklore; the paper formalizes but does not fundamentally change practice.

Our change to a non-Euclidean setting goes much further than simply changing the norm in the clipping of the stepsize. The algorithm we are considering fundamentally alters the direction (and not just the magnitude) used to update the iterates through the use of the non-Euclidean LMO (a kind of nonlinear preconditioner). This significantly changes the algorithm, both in practice (we do not only clip the stepsize, we must also apply an LMO, e.g., Newton-Schulz iterations) and in its theoretical analysis (e.g., the direction of steepest descent used in the algorithm may not be unique, no inner product identities to rely on, etc).

Crucial practical aspects—computational cost of the sharp operator and lmo for deep networks, choice of radius ρ, and stability when the dual norm is ill-conditioned—are not analyzed.

We have elaborated on the cost of the sharp operator and LMO for deep networks in our previous responses but, to be clear, there is very little overhead coming from the computation of the sharp operator and LMO. Regarding stability when the dual norm is ill-conditioned, it's not clear what is meant here. There is no requirement on the dual norm besides needing the LMO to be accessible (either in closed-form or approximately using a subroutine like Newton-Schulz).

Assumption tightness: Is (L₀, L₁) actually satisfied for deep nets in the modular norm? Please supply empirical Lipschitz estimates or counterexamples.

Yes, there is evidence to support that. In a concurrent work on non-Euclidean $(L_0,L_1)$-smoothness [2], the assumption is empirically validated (see Section 5, specifically Figure 2 and Figure 4 on NanoGPT and CNN respectively). It seems that the assumption very closely matches the estimated Lipschitz constant between consecutive iterates (this is ultimately what is needed for the proof).

[2] Artem Riabinin, Egor Shulgin, Kaja Gruntkowska, Peter Richtárik "Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)"

Baselines: Why are AdamW, Lion, and normalized SGD with momentum absent? Demonstrating advantages over these would affect my significance assessment.

AdamW has been tested in this setting and was found to be worse than Adam. Therefore we do compare against Adam but not AdamW, which would be redundant for the modded-nanoGPT repository. We focus on comparing against one strong, well-tuned baseline rather than having many baselines:

• The Adam optimizer is tuned (across 12 runs) and warmup is added to the learning rate schedule (in contrast to Scion/ClippedScion), which significantly boosts performance.
• In addition to comparing to Adam, we compare against Scion since it can achieve state-of-the-art on NanoGPT experiments and since it provides a direct comparison to see the effect of clipping (no other changes are made simultaneously to the optimizer).

Adaptive stepsizes: Your theory requires γ ≤ 1/L₀. In practice, L₀ is unknown. Can backtracking or AdaGrad be combined with GGNC while retaining descent? Empirical evidence would increase the quality score.

We believe this is an interesting direction that we also mention in the conclusion, but it does not seem completely obvious how to generalize beyond the Euclidean case (despite our preliminary efforts). Besides this, the results we present are nonetheless novel (there were no previous results in this setting, even assuming knowledge of the constants) which is why we have left it for future work.

The paper includes a "Limitations" checklist entry but does not discuss computational cost, hyperparameter sensitivity, or failure modes in depth. These should be added.

We elaborate on the limitations in the checklist, thanks for pointing this out.

I thank the authors for their rebuttal, which has satisfactorily addressed my concerns. I'm increasing my score.