AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections

We thank the reviewer for the insightful and constructive comments. Below, we provide our detailed responses to each point. We hope these clarifications help address your concerns.

---

W1: The discussion about AltLoRA+ and transformation invariance

We thank the reviewer for the comment. AltLoRA+ aims for accelerating AltLoRA instead of achieving transformation invariance. That said, since our primary goal of AltLoRA+ is not to develop a fully transformation-invariant optimizer, we believe this limitation does not significantly impact its main contributions. In AltLoRA, our core contribution lies in inducing optimally aligned gradients and momentum within low-rank subspaces, which already demonstrates better performance compared to baseline methods. Besides the algorithm design, our theoretical guarantees, including stable feature learning, transformation invariance, and convergence analysis, provide a deeper understanding of the theoretical significance of gradient alignment in optimizing LoRA. To further accelerate convergence, we propose AltLoRA+, which incorporates element-wise second-order momentum in a manner similar to AdamW (for the explanation of why element-wise second-order momentum will break transformation invariance, please refer to the answer of Q1). This enhancement leads to additional performance gains over the baselines. The improvement from AltLoRA to AltLoRA+ can also be regarded as an ablation study showcasing the benefits of momentum acceleration, which highlights the significance of AltLoRA+.

---

W2: Narrowing the gap

Our work aims to better understand the limitations of optimization within low-rank subspaces and to design more efficient algorithms that align low-rank updates closely with full-gradient directions. While we do not claim to completely close this gap, our empirical results demonstrate that our method brings low-rank adaptation closer to full fine-tuning in empirical performance. Following the reviewer's suggestion, we hope to revise our claim like "it narrows this gap toward full fine-tuning in two out of three natural language generation tasks and in the majority of natural language understanding tasks."

---

Q1: Analysis on transformation invariance and aligned second-order momentum

First, we will clarify why the second-order momentum in AltLoRA+ breaks transformation invariance. Here, we will prove AltLoRA+ breaks scalar scale invariance (See Def 2 in [1] for the formal definition), as a weaker version of transformation invariance. For two pairs of low-rank adaptation $B_1 A_1 = B_2 A_2$, if we assume $A_2 = s A_1$, $B_2 = \frac{1}{s} B_1$, then

$\delta A_1 = -\eta \frac{\widetilde{\nabla }_{A_1} L}{\left(\nabla\_{A_1}L \odot \nabla\_{A_1}L\right)^{1/2}} = -\eta \frac{(B_1^\top B_1)^{-1} B_1^\top \nabla_w L}{\left(B_1^\top \nabla_w L \odot B_1^\top \nabla_w L\right)^{1/2}} = -\eta \frac{1}{s^2} \cdot \frac{(B_2^\top B_2)^{-1} B_2^\top \nabla_w L}{\left(B_2^\top \nabla_w L \odot B_2^\top \nabla_w L\right)^{1/2}} = \frac{1}{s^2} \delta A_2,$

where $\widetilde{\nabla }_{A_1} L$ is the scaled gradient. Therefore, the update would violate scalar scale invariance as $B_1 \delta A_1 = s B_2 \cdot \frac{1}{s^2} \delta A_2 = \frac{1}{s} B_2 \delta A_2 \neq B_2 \delta A_2.$

Secondly, it is possible to align the second-order momentum with the full gradient in a principled way, but doing so does not necessarily improve performance. Let $V_t^B$ denote the second-order momentum of $B$ at time $t$. For example, analogous to aligning first-order momentum in the low-rank subspace, we can formulate a similar problem for second-order momentum as

$\min_{\widetilde{V}^B_t} \|\|(A_t)^T V_t^B A_t - (A_{t+1})^T \widetilde{V}^B_t A_{t+1} \|\|^2,$

where we assume $V_t^B$ is aligned to the low-rank space at time $t$. Here, the optimal solution is

$\widetilde{V}^B_t = (P_{A_{t+1}})^T V_t^B P_{A_{t+1}},$

where $P_{A_{t+1}} = A_t A_{t+1}^T \left( A_{t+1} A_{t+1}^T \right)^{-1}.$ Then the updated second-order momentum for $B$ at time $(t+1)$ is $V\_{t+1}^B = \widetilde{V}^B_t + (\widetilde{\nabla}\_{B_t}L)^T\widetilde{\nabla}_{B_t}L$. However, our numerical study shows that the accuracy of this approach does not lead to performance gains. With the same setup in Table 2, the accuracy on the GSM8K test set is only 56.12. Thus, while principled alignment is possible, it appears less effective in practice. In contrast, AltLoRA+, which focuses on aligning first-order momentum efficiently, demonstrates stronger empirical performance.

---

Thank you for your constructive feedback. We will incorporate the new results in the revised version, and we hope our responses satisfactorily address your concerns.

---

[1] Yen J N, Si S, Meng Z, et al. LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization. arXiv preprint arXiv:2410.20625, 2024.

We thank the reviewer for the insightful and constructive comments. Below, we provide our detailed responses to each point. We hope these clarifications help address your concerns.

---

W1: The convergence analysis and stochastic optimization

Our key contribution is to efficiently align both the gradient and momentum with the full gradient. The theoretical guarantees we establish provide insights into why the alignment intuition helps to improve performance in numerical studies. In contrast to the baseline work LoRA-Pro, which only proposed the theoretical motivation, our work provides formal guarantees on stable feature learning, transformation invariance, and convergence, thereby deepening the understanding of the importance of gradient alignment.

In particular, for the convergence analysis, [1][2] also analyze LoRA on an over-parameterized two-layer ReLU neural network tuning problem. Our convergence analysis demonstrates that the proposed alternating scaled gradient descent method for LoRA achieves the same convergence rate as joint optimization (as established in [1] using the same toy model). More importantly, we show that the convergence rate of alternating scaled gradient descent is independent of the condition number, and that alternating updates allow for a wider range of learning rates compared to joint updates. These results constitute the core theoretical contributions of our convergence analysis for alternating projection in LoRA training.

Regarding the stochastic gradient, thank you for raising this point.

First, for the algorithm design, we do implement stochastic batch gradients in our fine-tuning process, as it is standard practice in PEFT. In the revised version, we will add the corresponding description to clarify this in our Algorithms 1 and 2. For example, for the update of $A$ in L167, we will explicitly note: "Only backpropagate with respect to $A_t$ and obtain the stochastic batch gradient $\nabla_{A_t}L$."

Second, while prior gradient alignment works also adopt stochastic gradients in practice, they typically do not analyze how stochastic gradients impacts the gradient alignment effect. Extending our theoretical analysis to explicitly incorporate stochastic gradient behavior is a promising direction for future work, particularly to better bridge the gap between LoRA and full fine-tuning from a stochastic optimization perspective.

---

W2: The performance and efficiency trade-off

Thank you for the helpful suggestion regarding visualization. We have reported the performance in Table 2 and the memory cost in Table 3. To better illustrate the memory-performance trade-off of our algorithm, we would like to include a scatter plot in the revised version of the paper. For each task, the x-axis will represent memory cost, while the y-axis will correspond to the performance metric (e.g., accuracy). This figure will visually demonstrate how our method improves over baselines. Here, we provide a table to describe the results for the GSM8K task by fine-tuning the Llama3.1-8B model, which supports that our algorithms achieve better performance than LoRA with similar memory efficiency.

---

W3: Training Curves

We provide the training loss in the table below. As shown, our method not only converges to a lower training loss than full fine-tuning, but also does so more rapidly than most baselines.

The accuracy metrics reported in Table 2 are evaluated on the corresponding test sets, as described in Appendix E.1. Our method achieves higher scores, highlighting its superior generalization performance.

---

Q1: Epoch number

We have reported the number of training epochs in the experimental setup to avoid confusion. As reported in Line 897, the iteration number was set to 1, which corresponds to a single training epoch.

We agree that the number of epochs is an important training design, especially given its potential correlation with accuracy metrics. To further support the validity of our experiment, we add an ablation study for the training epoch with 5 training epochs under the setup of Table 2. With the increase in training epochs, our approach would outperform baselines as well.

---

Q2: Llama3-8B-Base

Using the Llama3-8B-Base model is a common practice in the literature. Among the most relevant baselines, LoRA-Pro and LoRA-GA adopt either Llama2-7B or Llama3-8B-Base as the pretrained model. Furthermore, several recently published works [4][5][6] on LoRA also use Llama3-8B-Base as the standard backbone.

For the Llama3-8B-Instruct model, we fine-tune it on the MetaMathQA dataset and evaluate it on the GSM8K test set, reporting accuracy as the primary metric. Our results demonstrate that even when using a stronger pretrained model, low-rank adaptation can still yield performance gains. More importantly, both AltLoRA and AltLoRA+ consistently outperform the standard LoRA baseline. We will add the full comparison in the revised revision.

---

Q3: Random projection and GaLore

We appreciate the clarification regarding the naming: it should be “GaLore“[5] instead of ”Ga-LoRA“. Additionally, we acknowledge that the variant using random projection is introduced in a separate work, GoLore. And we will include GoLore into our related work as well. We will revise the text accordingly to correct both the description and the naming to avoid confusion.

---

Thank you for your constructive feedback. We will incorporate the new results in the revised version, and we hope our responses satisfactorily address your concerns.

---

[1] Zhang, F. and Pilanci, M. Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models. In International Conference on Machine Learning (pp. 59641-59669). PMLR.

[2] He, Y., Li, P., Hu, Y., Chen, C., and Yuan, K. Subspace Optimization for Large Language Models with Convergence Guarantees. In Forty-second International Conference on Machine Learning. 2025.

[3] Zhang, J., Chiu, H.M., and Zhang, R.Y. Accelerating SGD for Highly Ill-Conditioned Huge-Scale Online Matrix Completion. Advances in Neural Information Processing Systems 35 (2022): 37549-37562.

[4] Liao, X., Li, S., Xu, Y., Li, Z., Liu, Y. and He, Y. GaLore+: Boosting Low-Rank Adaptation for LLMs with Cross-Head Projection. arXiv e-prints, pp.arXiv-2412.

[5] Zhang, C., Lianhai, R.E.N., Cheng, J. and Li, Q. From Weight-Based to State-Based Fine-Tuning: Further Memory Reduction on LoRA with Parallel Control. In Forty-second International Conference on Machine Learning. 2025.

[6] Huang, Q., Ko, T., Zhuang, Z., Tang, L., and Zhang, Y. HiRA: Parameter-Efficient Hadamard High-Rank Adaptation for Large Language Models. In The Thirteenth International Conference on Learning Representations. 2025.

[7] Recht, B., Maryam F., and Parrilo, P. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Review 52.3 (2010): 471-501.

The authors thank the reviewer for insightful and constructive comments. Below, we provide our detailed responses to each point. We hope these clarifications help address your concerns.

---

W1: Writing Quality

Thank you for bringing these presentation issues to our attention. We will relocate the caption of Table 2 to the correct position in the revised version. In L601, the reference should be “LoRA-GA[65]” rather than “LoRA-Pro~[66]”. LoRA-GA indeed introduces a novel initialization approach that more closely aligns LoRA with full fine-tuning at the first step.

---

W2: Missing Ablation on Momentum Alignment

We include ablation studies to evaluate the effect of momentum alignment. The AltLoRA w/o MA variant differs from AltLoRA only in that momentum is not projected onto the low-rank subspace (momentum alignment). Across the tasks of natural language generation, AltLoRA outperforms AltLoRA w/o MA, achieving higher accuracy and lower variance. This demonstrates the advantages of aligning momentum with low-rank spaces.

---

Thank you for your constructive feedback. We will incorporate the new results in the revised version, and we hope our responses satisfactorily address your concerns.

The authors thank the reviewer for insightful and constructive comments. Below, we provide our detailed responses to each point. We hope these clarifications help address your concerns.

W1: Computing $(B^TB)^{-1}$

Sorry for the confusion. Please note that we have clarified at L121 and L149 of the original submission that we didn't directly compute $(B^\top B)^{-1}$ but used a regularized inverse instead. We use $(B_t^\top B_t)^{-1}$ in Algorithms 1 and 2 just for notational simplicity.

To be more specific, the full-rank assumption for $B$ is typically not satisfied during training according to our analysis. Thus, following the reasoning in our gradient-alignment analysis (see Corollaries 1 and 2), we introduce weight decay during the gradient alignment and compute the well-defined regularized inverse: $(B^\top B + \lambda \mathbb{I})^{-1}.$ This formulation ensures numerical stability and avoids undefined behavior during updates. In all experiments of the original submission, we have used $\lambda=1e^{-5}$.

We will revise the algorithm descriptions to replace $(B_t^\top B_t)^{-1}$ with $(B_t^\top B_t + \lambda \mathbb{I})^{-1}$ to avoid any ambiguity.

---

W2: Increasing training epoch

Thank you for the helpful suggestion. For fairness and direct comparability, our original experiments followed the evaluation protocol used in LoRA-GA (and LoRA-Pro), which trains for one epoch, and we reported the corresponding results.

To provide a more comprehensive evaluation, as the reviewer suggested, we have re-run all experiments in Table 2 using five epochs. The full results are reported below. We observe that our method continues to consistently outperform all baselines under this extended training regime, demonstrating that the improvements are robust beyond early convergence and hold across longer fine-tuning schedules. We would be glad to include these results in the revised manuscript for completeness.

---

W3: Why AltLoRA outperforms standard LoRA

We respectfully disagree with some of the reviewer's opinions and aim to clarify the reasons behind the improved performance of our method.

First and foremost, unlike full fine-tuning, the gradients in standard LoRA are confined to low-rank subspaces, which limits its effectiveness. Prior work, such as LoRA-Pro, has identified this limitation and attempted to align LoRA gradients to approximate the full gradient. However, LoRA-Pro does not yield a unique update solution and demonstrates unstable performance. In contrast, our alternative optimization strategy addresses this issue by better aligning the low-rank gradient with the true full gradient. This alignment underlies the superior performance of AltLoRA compared to standard LoRA.

Regarding theoretical analysis, we emphasize that our work is not purely theoretical or solely focused on convergence proofs. Instead, our goal is to develop an effective LoRA algorithm supported by theoretical insights that justify our design choices. Specifically, we have demonstrated that our approach achieves stable feature learning and transformation invariance—property not present in standard LoRA but crucial for non-asymptotic training behavior. Furthermore, the convergence guarantee, albeit established on a simplified two-layer ReLU network, provides additional evidence of the method’s design soundness.

Finally, we note that the strategy of freezing layers, which may appear as an empirical heuristic, fundamentally differs from our alternating optimization approach. Alternating optimization is a well-established technique with decades of study. Recent works—[1] in LoRA and [2] in matrix factorization—also show that alternating optimization can be more stable and effective than joint optimization in certain settings.

---

W4: Performance as rank increases

In Table 2, we observe that performance degrades as the rank increases. A similar trend appears in the original LoRA paper (Tables 15 and 18 in [3]). We hypothesize that larger ranks introducing more trainable degrees of freedom can easily lead to overfitting when learning from a small sample size. Consistent with this view, in the five-epoch setting, the performance of AltLoRA and AltLoRA+ with higher rank improves and exceeds the default setting (rank = 8).

---

W5: The superiority of AltLoRA

By computing $(B^{\top}B + \lambda\mathbb{I})^{-1}$ instead of $(B^{\top}B)^{-1}$, we eliminate the theoretical concern raised in W1. As a result, the downstream argument in W5, which relies on W1, no longer applies. To examine whether the superior performance of zero initialization over Gaussian initialization for $B$ is related to W2, we revisit the same setup in Table~8 under a 5-epoch training schedule. We find that, even with more steps, zero initialization of $B$ yields a higher accuracy (74.52) compared to Gaussian initialization (74.31), which shows a consistent result with our manuscript.

---

Thank you for your constructive feedback. We will incorporate the new results in the revised version, and we hope our responses satisfactorily address your concerns.

---

[1] Chen S, Guo Y, Ju Y, et al. Robust federated finetuning of llms via alternating optimization of lora. arXiv preprint arXiv:2502.01755, 2025.

[2] Jia, Xixi, et al. Preconditioning matters: Fast global convergence of non-convex matrix factorization via scaled gradient descent. Advances in Neural Information Processing Systems 36 (2023): 76202-76213.

[3] Hu E J, Shen Y, Wallis P, et al. Lora: Low-rank adaptation of large language models. International Conference on Learning Representations, 2022.

Summary

We appreciate the reviewer’s comment. Our PEFT methods provide improved gradient alignment through alternating projection and introduce a novel momentum alignment strategy. These techniques are supported not only by theoretical guarantees, including stable feature learning and transformation invariance, but also by fair and consistent empirical comparisons following the protocols of prior work.

Importantly, we have made concrete efforts to better integrate theory and practice. Our theoretical analysis focuses on the unregularized update, which is consistent with existing practices in the LoRA literature [1,2], and we adopt the regularized inverse $(B^TB+\lambda\mathbb{I})^{-1}$ during implementation, which is commonly used in practice for numerical stability [2].

---

[1] Zhang F, Pilanci M. Riemannian preconditioned LoRA for fine-tuning foundation models. ICML, 2024.

[2] Zhang Y, Liu F, Chen Y. LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently. ICML, 2025.

[3] Cheng C, Zhao Z. Accelerating gradient descent for over-parameterized asymmetric low-rank matrix sensing via preconditioning. ICASSP, 2024.

[4] Xu X, Shen Y, Chi Y, Ma C. The power of preconditioning in overparameterized low-rank matrix sensing. ICML, 2023.

[5] Sun Y, Li Z, Li Y, Ding B. Improving LoRA in privacy-preserving federated learning. ICLR, 2024.

[6] Liu Z, Han Z, Tang Y, et al. Efficient over-parameterized matrix sensing from noisy measurements via alternating preconditioned gradient descent. arXiv:2502.00463.

[7] Ward R, Kolda T. Convergence of alternating gradient descent for matrix factorization. NeurIPS, 2023.

[8] Chen S, Guo Y, Ju Y, et al. Robust federated finetuning of llms via alternating optimization of LoRA. arXiv:2502.01755, 2025.

[9] Pan R, Liu X, Diao S, et al. LISA: Layerwise importance sampling for memory-efficient large language model fine-tuning. NeurIPS, 2024.

[10] Luo Q, Yu H, Li X. BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models. NeurIPS, 2024.

[11] Sheng G, Zhang C, Ye Z, et al. Hybridflow: A flexible and efficient RLHF framework. EuroSys, 2025.

$\bullet$ On the alternating updates

$\circ$ We appreciate the reviewer’s thoughtful follow-up and would like to clarify that we do not claim that alternating optimization (M2) is inherently better or universally superior to joint optimization (M1). Rather, our argument is contextual and specific to the gradient alignment setting, where joint optimization often fails to yield a unique solution and can be empirically unstable, as noted in LoRA-Pro (Appendix D.1). In contrast, alternating updates lead to unique closed-form solutions and stronger theoretical guarantees (stable feature learning and transformation invariance) over literature, which would help to explain the superity of our methods over baseline.

Regarding the concern that M2 operates in a restricted subspace of M1, we respectfully disagree with this characterization. We note that truly restricted subspace optimization occurs in methods like FFA-LoRA [5], where only one sub-module (e.g., $A$) is optimized while the other ($B$) is frozen throughout training. In contrast, our alternating strategy updates both $A$ and $B$, sequentially projecting the full gradient onto the low-rank subspaces defined by $A$ and $B$, without freezing or excluding any parameter block.

Regarding the comparison with the alignment in LoRA-Pro and our core insight, LoRA-Pro pioneered the idea of gradient alignment, and it does so via joint updates over two low-rank spaces $A$ and $B$, which can overlap, making it impossible to derive a unique optimal update, as acknowledged in their analysis. This overlapping leads to empirical instability (see Appendix D.1 in LoRA-Pro). In contrast, our work demonstrates that alternating updates, by projecting the full gradient sequentially, avoid this overlap and result in more stable and efficient updates. Specifically, in LoRA-Pro’s formulation, for different pairs of gradient $A$ and $B$, the full model update is the same: project to the low-rank spaces of $A$ and $B$ $\frac{1}{s}B_t(B_t^TB_t)^{-1}B_t^T(\nabla_{W_t}L) + (\nabla_{W_t} L)A_t^T(A_tA_t^T)^{-1}A_t,$ and compensate for the overlapping part $-B_t(B_t^TB_t)^{-1}B_t^T(\nabla_{W_t}L)A_t^T(A_tA_t^T)^{-1}A_t.$ In contrast, our alternating projection avoids such overlap by updating one submodule at a time. We further extend gradient alignment to momentum alignment, making the method more practical and optimizer-friendly within LoRA’s memory budget. In summary, our contribution is not a blanket preference for alternating updates, but rather a principled design that improves stability, performance, and theoretical tractability in the context of gradient alignment.

$\circ$ Thank you for the observation. Our convergence analysis in Appendix D.3.2 establishes theoretical guarantees for the alternating update (M2), but does not claim it universally outperforms the joint update (M1). Our analysis demonstrates linear convergence at a rate independent of the condition number, which supports the theoretical soundness of the alternating scheme. However, we do not claim that this implies M2 is superior to M1 in all settings. We acknowledge that the original phrasing may have caused some confusion, and we will revise the text to make it clear that the theoretical guarantees are specific to the alternating scheme and are not intended as a universal comparison.

$\circ$ Thank you for the helpful clarification. Our use of the term "well-established" refers to the broader optimization literature, where alternating updates are commonly used for problems involving parameterized low-rank matrix decompositions [6,7], due to their numerical stability and ease of implementation. While we acknowledge that joint updates are more prevalent in recent PEFT works, alternating strategies have also been adopted in some recent works, such as [8].

Beyond PEFT, alternating updates are not rare in full-model fine-tuning. For instance, several recent approaches alternate between freezing and unfreezing subsets of layers during fine-tuning to improve stability and performance [9,10]. Furthermore, standard reinforcement learning settings routinely employ alternating updates, most notably between the actor and critic networks [11].

Our manuscript explicitly discusses and compares methods that adopt joint updates, including those mentioned by the reviewer, and we provide fair empirical evaluations following the evaluation protocols of LoRA-Pro and LoRA-GA. We agree that a stronger theoretical justification for alternating updates in the PEFT context is valuable, and our current convergence analysis aims to be a step in that direction. We will be careful with the wording in the revised version to avoid overstating the prevalence of alternating updates in the PEFT literature.

$\circ$ Thank you for pointing this out! The ``joint update" in Table 6 refers to joint scaled gradient descent with regularization. We will clarify this explicitly in the final version to avoid confusion.

Thank you for your kind response and for acknowledging our clarifications regarding epochs and ranks. Below, we further address and clarify the remaining concerns.

---

$\bullet$ On computing $(B^TB + \lambda \mathbb{I})^{-1}$

$\circ$ We now understand the reviewer’s original concern is not about rank deficiency but rather gradient instability when $B_t^TB_t\approx 0$, which is a relevant issue in early training. We appreciate the clarification. In our earlier response, we mentioned rank deficiency because the reviewer referenced the “undefined behavior” of $(B^TB)^{-1}$, which is typically ill-posed when $B_t$ is not of full rank.

Regarding the reviewer's concern about gradient explosion, we observe empirically that the update is stable even when $B_t^TB_t$ is small in early training steps. This stability is primarily due to the following algorithm design: we follow LoRA-Pro and adopt a cosine learning rate scheduler with a warmup ratio of 0.03. During the warm-up period (when $B_t^TB_t$ may be small), the magnitude of the learning rate starts at 0 and gradually increases to the default value (i.e., grid search $\\{1e^{-4},4e^{-5},1e^{-5} \\}$ in AltLoRA+). As a result, even when $B_t^TB_t$ is small, the effective step size for updating $A$ is controlled and does not lead to exploding gradient updates. Specifically, the magnitude of the update during early steps is approximately $\frac{\eta}{s^2\lambda}\nabla_{A_t}L$, which remains well-behaved due to small $\eta$ during warm-up. We support this with empirical evidence below, showing the norms of $A$ and $B$ updates as well as the loss remain stable during the early steps.

For AltLoRA:

For AltLoRA+:

$\circ$ Regarding the difference between Eq. (8) and Eq. (18), such a gap between the theoretical formulation and the practical implementation is consistent with existing practices in the LoRA literature. Prior works such as scaled AdamW [1] and LoRA-One [2] also precondition their gradients using $(B^\top B)^{-1}$ in theory, but apply $(B^\top B + \lambda \mathbb{I})^{-1}$ in practice for numerical stability. Notably, neither [1] nor [2] analyzes the regularized scaled gradient in their convergence guarantees. Our work closely follows this convention. Besides, LoRA-Pro assumes that the low-rank matrix is full-rank, yet also acknowledges that this assumption can be violated in early training (shown in Table 2 of LoRA-Pro). For convergence analysis, LoRA-Pro didn't establish convergence analysis.

Although the exact transformation invariance property of Eq. (8) may not strictly hold under Eq. (18), we find that in practice, the behavior of the update is very close due to the small value of $\lambda$. As we use a very small $\lambda$ (1e-5) in the implementation, a first-order Taylor expansion shows that $\frac{1}{s^2}(B_t^TB_t + \lambda\mathbb{I})^{-1}\nabla_{A_t}L \approx \frac{1}{s^2}(B_t^TB_t)^{-1}\nabla_{A_t}L$ up to the linear and higher order terms in $\lambda$, and in the special case where $\lambda = 0$, the two formulations are exactly equivalent. As such, the regularized update closely approximates the unregularized solution. For this reason, we conduct our theoretical analysis using the unregularized update in Eq. (8), which enables cleaner theoretical guarantees. Moreover, the stability and convergence observed in empirical results suggest that the effect of this perturbation is negligible in practice.

Analyzing convergence for the scaled gradient with adding term $\lambda$ is significantly more challenging, since the hyperparameter $\lambda$ affects both the effective $L$-smoothness and the PL inequality in [3], and may require a customized spectral initialization as discussed in [4]. Understanding how $\lambda$ influences training dynamics and generalization in LoRA is an open question and a promising direction for future work.

$\circ$ On "weight decay", we agree that our earlier phrasing may have been imprecise. To avoid this confusion, we will revise the sentence in L121 as follows:

"To mitigate this issue, we apply a Frobenius norm regularization to the two approximated gradients, which can stabilize the updates and remove the assumption of full rank."