Fast Fractional Natural Gradient Descent using Learnable Spectral Factorizations

1. The main motivation of the method, which is the flexibility of choosing p, is mentioned but not very well established. I think the authors should at least make an effort on the numerical side. They mention that "This shows that the potential of using other fractional roots." by the end of the section 4, which seems weak as this is the main reason for potential practitioners to use this method over its simpler counterpart. Some results showing that the best training result can only be obtained by some p other than 1 or 2 can be very persuasive in this case.

2. In terms of the computational cost, the comparison between this method and previous methods is not clear. The authors calibrated the running time for each method in section 4 by adjusting the number of iterations to update the preconditioner, but it is still vaguely defined and not quantified. One good comparison is to those methods which directly compute the matrix decomposition using high-precision arithmetic. I think this will establish the other important motivation of the method.

3. The method suffers from a per-iteration error of $O(\beta_2^2)$, which is non-negligible for constant $\beta_2$. Also, the inverse approximation when calculating the Cayley map introduces another source of inaccuracy.

4. There are some notation ambiguity in the paper. For example, Kronecker product and functions like Mat($\cdot$) are not defined clearly. Diag($\cdot$) and diag($\cdot$) are used interchangeably. Eq 3 seems to come from nowhere when it first appears in the paper (no introduction before and after, no definition of $\mathcal{H}$). Claim 2 eventually becomes Lemma 2 in the appendix.

1. No numerical performance in large-scale setting is presented.

W1: The writing of the paper is hard to follow.

W2: The are no explicit formulation of lemmas in the paper.

W3: The key points in the theoretical analysis is missing.

1. The style of presentation is confusing in some parts of the paper. Next, I list some examples: a) Figure 1 shouldn’t be presented as is. It has two methods inside and half of the paper description. It ruins the style and flow of the paper. b) The authors present theoretical results as “claims”, however, next they prove it as lemmas. So, it would be reasonable to formulate them as lemmas in the main text as well. c) In lines 151-161 and 242-248, there is an extra spacing which shouldn’t be there. d) Some colorings are confusing. For example, line 353, 745, 833, 932. It might be forgotten highlights of the changes.

2. The experimental section. As the paper doesn’t provide convergence proofs, we can say it is mostly an empirical paper backed by theoretical intuition. a) Hence, the extensive experimental results are expected. In the paper, only one figure presents the performance of the methods for 3 vision transformers on one dataset. It does not seem enough for an experimental paper. So, I would recommend adding experiments for common DL benchmarks to understand the methods' performance in comparison to state-of-the-art methods. Some examples can be found here https://mlcommons.org/benchmarks/algorithms/ b) Another issue is that the experiments’ description is lacking proper details and descriptions. It is limiting the reproducibility of the results. There are no methods parameters, learning rates, and so on. The attached code may help as well.

Some minor misprints: a) Please, unify the use of “RMSprop”, as currently there are multiple variants of it (RMSprop, RmsProp, RMRprop) b) Line 709: misprint in “Given”

To strengthen the argument for using this method over existing alternatives, an ablation study examining the effects of different spectral factorization techniques on convergence speed and generalization performance would be beneficial.

While the paper demonstrates scalability in terms of Kronecker-structured preconditioners, additional details on computational complexity and memory requirements—particularly in comparison to popular methods like Adam or Shampoo—would offer a clearer understanding of the associated trade-offs.

Clarity Issues: Notation and concepts are unclear, making the method difficult to follow. For example:

• What is $B$ matrix?
• The purpose of learning the Kronecker factorization is under-explained
• What is matrix H? (In line 101)
• Meaning of the “mat” notation (line 107)
• What are matrices $S^K$ and $S^C$?
• What is $\otimes$ product?
• Description of the “Gaussian problem”? For that concept or notations, the authors should give an appropriate introduction and explanation and references

Weak Motivation: The paper provides an insufficient explanation as to why this complex approach is necessary or how it significantly advances existing methods. It does not provide new insight or guidelines (or at least does not explain clearly) that can potentially be utilized to result in efficient optimization or training.

Ambiguity in Implementation: The handling of factorization ambiguities in high-dimensional settings is unclear, particularly in terms of computational impact and scalability.

1. It is nots clear what is their contribution and what has been don ein previous work.

2. Description of the use of the gaussian variational approach in line 286 is very vague, it is not even clear why this approach is well-motivated or what is $\ell$.

3. Authors keep taking about suing local coordinates but it's not exactly what coordinate system for the manifold of matrices they are referring to, do you mean the normal coordinate system using the xponential map?

4. The definition of function $Tril$ is unclear, and authors define what they mean by $Cayley$ in the middle of the paper.

5. The high level ideas are vague and include terms that is not clear what they exactly refer to.

6. It would be nice if authors can provide further motivation or example on which practical case the kronecker-structured spectral parameterization is useful.

7. It is not clear what is the manifold metric the authors consider in their Riemannian appraoch. they connection of the gaussian variational framework, which seems to go back to work of Amari, to obtaining a manifold metric seems unclear.

Other issues: H is not defined in equation 3. U and Tril are not defined in algorithm box on line 162 Line 107 the notation is super unclear, is Mat(g) the diagonal matrix with entries of g? If so what is the difference between $G^\top G$ and $G G^\top$, and $S_C, S_K$? Authors refer to “root-free” method but they don’t define what they mean Equation 10 is not clear at all Authors don't provide any intuition of their approach/contribution e.g. for equation 10 or for the algorithm box in line 162

Minor issue: Claim 1 is named Lemma 1 in appendix