Large Learning Rates without the Agonizing Pain: Dispelling the Curse of Singularities in Deep Neural Networks

W1. While shown empirically, the link between singularities and instability isn't addressed theoretically.

W2. PSS requires a threshold and smoothing method. The article demonstrates the robustness of smoothing methods but not the threshold. Isn't the fine-tuning of the threshold similar to the fine-tuning of LR?

W3. The article shows limited comparisons with other stabilization methods mentioned in the related work section.

W4. Some figures lack clarity (e.g., unclear indicators in Figures 3 and 6).

W5. Effectiveness in downstream tasks isn’t fully explored, raising questions about potential drawbacks during training.

W6. Besides preventing explosions, results don’t clearly show faster or improved convergence at large learning rates.

W7. The Protection section notations are hard to follow.

Overall, I am not convinced that the paper's contribution is of great significance. The article seems to solve a marginal problem. The other regularization methods provide comparable results in preventing explosions, and the current work is not convincing that explosions are a frequent problem with a scheduler.

• W1: The authors severely under-compare and under-refer an alternative approach of dealing with learning rate -- automatic learning rate setters. There are many papers that propose automatic learning rate computational schemes that run without manual interventions and promise the same fruit as this work -- large learning rates that however still lead to stable training dynamics (see for instance [1-6]). Some of these papers also mention that the resulting average learning rate is much higher than a stable learning rate using vanilla optimizers. Most of the papers don't come with too prohibitive computational overhead. I would expect the paper to compare against the most competitive methods from this area and discuss them in related work.

• W2: The work lacks some ablation studies. In particular, the paper should investigate, whether some similar, but less principled operation on singular values would still give the same improvements or not. Examples of simpler ideas -- adding random noise to the matirx as a mean of smoothening its spectrum, setting just the first singular value to the value of the second one, or using hard (thresholded) rank instead of the stable rank.

• W3: There is a rather weak link between the "theoretical" motivation and the method itself. In particular, the authors just smoothen the singular values that are already dominant, thus not fixing the imbalance between large and small singular values, nor the number of the large singular values. For instance, for the convolution smoothing, the sum of dominant singular values stays the same, not balancing the large and small values at all. In fact, the fact that the method can recover good training even after the loss jump has happened suggests that the reason why the method works has nothing to do with balancing the singular values. Note that since we change only the top singular values, the mismatch between large and small singular values remains. This suggests that it might be important to re-weight the dominant directions, rather than balance them with the weak directions. To elaborate on this even further, left plot in Figure 6 even suggests that the loss explosion might not be caused by singularities at all. The effective rank in the collapsed solution stays constant for the steps of collapse, suggesting the loss might be exploding for completely different reasons.

• W4: The introduction of the stable jacobian energy (SJE) seems pointless in this paper. It is not used at all in connection with the proposed method (in fact I expected it to be used as the indicator to launch the mechanism, instead of gradient norm. That would fit the paper's motivation much better). Moreover, it does not serve as a convincing motivation behind the emergence of singularities, because the paper doesn't provide enough empirical or theoretical evidence for it. The only piece of evidence is Fig. 2b, but that only shows some correlation that provides very little insight in what is happening at the time of the loss explosions. I would suggest the authors to fully drop this aspect, or to provide much more discussion on it.

• W5: I disagree with the authors that the method is robust w.r.t. the choice of the hyperparameter $\tau$. If you pick too small $\tau,$ you might evoke the mechanism too often, resulting in a significant slowdown (maybe even non-convergence). If the $\tau$ is too big, you might need manual intervention because the smoothing might not be evoked even after the loss jump. Moreover, the hyperparameter is also dependent on both the exponential averaging coefficient $\alpha,$ as well as the learning rate itself. Thus, it might be quite tricky to choose the correct $\tau,$ which undermines the original motivation of this method to avoid hyperparameter optimization.

• W6: In the paper, the authors claim that the gradient clipping method is more computationally heavy (line 357) than PSS, but this disagrees with the evidence and also doesn't make sense since gradient clipping is extremely cheap. Moreover, the gradient clipping has only been demonstrated to perform worse for one particular setup. It would be nice to see a bit more convincing evidence that PSS significantly outperforms gradient clipping.

• W7: I don't think the method really allows to avoid learning rate tuning. I think one still needs to perform some learning rate tuning. This method just allows one to use higher values of lr safely.

[1] Jin, Yuchen, et al. "Autolrs: Automatic learning-rate schedule by bayesian optimization on the fly." arXiv preprint arXiv:2105.10762 (2021).

[2] Merrouchi, Mohamed, et al. "AutoLrOpt: An Efficient Optimizer Using Automatic Setting of Learning Rate for Deep Neural Networks." IEEE Access 12 (2024): 83154-83168.

[3] Yedida, Rahul, and Snehanshu Saha. "A novel adaptive learning rate scheduler for deep neural networks." arXiv preprint arXiv:1902.07399 (2019).

[4] Li, Yong, et al. "The improved training algorithm of back propagation neural network with self-adaptive learning rate." 2009 international conference on computational intelligence and natural computing. Vol. 1. IEEE, 2009.

[5] Subramanian, Shreyas, and Vignesh Ganapathiraman. "Zeroth Order GreedyLR: An Adaptive Learning Rate Scheduler for Deep Neural Network Training." 2023 IEEE 4th International Conference on Pattern Recognition and Machine Learning (PRML). IEEE, 2023.

[6] Wang, Kang, et al. "An automatic learning rate decay strategy for stochastic gradient descent optimization methods in neural networks." International Journal of Intelligent Systems 37.10 (2022): 7334-7355.

• I think a main weakness of the paper is that Section 2 could be largely skipped in favor of moving directly to the method in Section 3. Figure 3 shows that several metrics, beyond SR, can indicate training breakdowns effectively.
• The protection policy laid out in Section 3 seems somewhat ad-hoc. Would you be comfortable describing how you arrived at it and perhaps failed attempts on the way?
• SR is integrated into the methodology, but SJE is not. Removing SJE may not impact the paper significantly, as its contribution is primarily limited to Figure 2b.

-- The paper seems excessively empirical. The theoretical aspect might be studied more. It might be very difficult perhaps, but here there is not even an attempt. -- There is only an intuitively explanation about why the procedure introduced in the article should allow to avoid the singularities that cause instability. One might expect a bit more than that.