Retraction-free optimization over the Stiefel manifold with application to the LoRA fine-tuning

• Section 2.2 and a large part of Sections 2.1 play no role in the main paper. Why should they be included?
• The update rule (3) of the method is similar to the penalty method at Line 206, with the only difference being that the proposed method uses Riemannian gradients rather than the gradients of the objective. How could the paper justify its advantage over the penalty method? In theory, I imagine a result similar to Theorem 1 might hold for the penalty method. In practice, the penalty method is easier to implement, and it should be compared in the experiments. This is a fundamental issue that the paper could have addressed better.
• The other important issue is that, while theory does not involve the oblique manifold, the experimental section suddenly begins considering it. Enforcing the constraints of the oblique manifold is so easy that the point in considering a penalty-type method for oblique manifolds is not justified. This is a fundamental contradiction of the paper. Furthermore, Figure 1 and the experiments show that optimization on Stiefel manifolds, as the paper advocates, does not seem to be a strong point because optimization on oblique manifolds is simpler and achieves similar results and the question for authors is: Why should one look for a more complicated method? A sequence of experiments should be designed with relevant baselines in order to justify and motivate the proposed approach. In my opinion, fine-tuning LLMs with LoRA is not a good application scenario for the proposed method; all methods have comparable accuracy.
• Eq. (13) is not rigorous. It suggests optimization over the union of the oblique manifold and the Stiefel manifold, which is the oblique manifold.
• Algorithm 1 (Lines 324 - 348) is out of the blue. The main paper has little introduction to ADAM, and suddenly, many quantities related to ADAM show up in Algorithm 1 without explanation (in particular, Lines 335, 338). There is, therefore, no chance for a novice reader to make sense of Algorithm 1.
• The comparison in Figure 2 is misleading as I would imagine the computational cost per iteration of the proposed method is higher.

I started reading this manuscript with much anticipation, but my enthusiasm was short lived.

1. The contribution of this paper is unclear. Specifically, the proposed update rule (Equation (3) in the manuscript) follows the existing work, e.g. [1,2,3]. Additionally, the landing algorithm on the Stiefel manifold has been proposed in [2], and both the deterministic and stochastic versions on the generalized Stiefel manifold have been studied in [3]. More importantly, the manuscript claims that accommodating a constant penalty parameter is novel. However, a more general result has been established in prior work, e.g. [3, Theorem 2.8], which stated that the landing method provably converges for any fixed penalty parameter.
2. The literature review is insufficient. The manuscript omits two relevant works [2] and [3], where a landing algorithm on the Stiefel manifold was proposed in [2], and both the deterministic and stochastic versions on the generalized Stiefel manifold have been studied in [3].
3. The paper is not carefully written, with numerous typos influencing the readability (see Questions).

1. Important related work is missed involving Riemannian Preconditioned LoRA (https://arxiv.org/abs/2402.02347) and DoRA (https://arxiv.org/abs/2402.09353). Also LoRA+ (https://arxiv.org/abs/2402.12354).

To my understanding, Riemannian Preconditioned LoRA studies preconditioners for accelerating LoRA tuning derived from new Riemannian metric, which is in spirit close to current work since both consider the manifold geometry of LoRA. Though one is motivated by a different Riemannian metric and another is motivated by different retraction step.

DoRA proposes to decompose LoRA weight into magnitude and direction, which is in some sense also close to current work which considers B to be basis of manifold and A to be coordinates.

LoRA+ proposes to use different learning rates for A and B respectively, which is close to discussion right before paragraph 5.

2. Despite empirically verified, it is still not clear why in LoRA tuning, imposing B to be the basis and A to be coordinate would accelerate convergence. Theoretically speaking, they should cover same $X$ space ($X=BA$) and it feels to me vanilla LoRA parameters have more freedom to change compared to constraining $B$ to be on $B^TB=I$

3. Though in theory, the authors show that constant step size is enough, it looks like step size tuning is still needed for the experiments.

My main concerns with this work is about the experimental section:

1. The authors should make the code for their method available in the supplementary materials during the review period. 2The authors should include comparisons with the existing low-rank fine-tuning methods that incorporate the Stiefel manifold constraint. Without these comparisons, it is unclear whether the improved training speed and evaluation metrics are due to the proposed method itself or the addition of the Stiefel manifold constraint. 3.While the authors compared the number of epochs required for training, but have authors conducted comparisons in terms of actual runtime?