ALLoRA: Adaptive Learning Rate Mitigates LoRA Fatal Flaws

1. In general, the theoretical justifications in this paper do not fully support the three flaws of standard LoRA as mentioned. Consequently, the motivation of this paper may not be convincing, at least to me.

   • For the first flaw, the connection between expectation and training steps may be incorrect. The expectation is over the randomness of V, whereas infinite training steps imply that W approaches the closed-form solution W*. To demonstrate that dropout is not suitable for fine-tuning with limited training steps, the authors need to prove that infinite training steps can lead to the expectation of the results. More importantly, how is "limited training steps" defined? Dropout with large rates outperforms that with small rates even for 8-epoch training, which is also considered limited compared to pre-training.
   • For the second flaw, it is unclear how the degenerate optimization landscape is related to the zero-initialization of LoRA. More importantly, why does the landscape become degenerate as the training set size (n) decreases? As far as I can see, this is related to X^TX, which may remain the same even after n decreases.
   • For the third flaw, it is unclear how the output's potential exponential growth with respect to the number of layers is connected to the performance of LoRA. If this connection is not made clear, we cannot conclude that it is harmful for the final fine-tuning performance.

2. Supposing the "flaws" are true, the paper fails to explain why ALLoRA is a good or necessary way to overcome them. More specifically, one could also directly remove dropout, the scaling factor, and so on to address these issues.

3. This paper primarily compares with DoRA and lacks comparisons with other more recent LoRA techniques, such as those in (Hayou et al., 2024a, 2024b), rsLoRA, Flexora, and others. Importantly, the improvement is quite marginal.

4. This paper lacks an ablation study to show how ALLoRA can help overcome each flaw independently. This is crucial for understanding the efficacy of ALLoRA.

5. The citation format needs to be refined. There are many places where the authors should use \citep to cite a paper, but they have applied \citet to cite the authors.

1. The novelties in this work are quite marginal. Although the authors have proposed the novel LoRA variant, it lacks of in-depth theoretical analysis on ALLoRA. The authors should try to have more comprehensive technical analysis why ALLoRA allows for improvement compared to vanilla LoRA and CoRA.

2. In Figure 1, the authors use two datasets MNIST and CIFAR10 to demonstrate the flaw of dropout. They plot the distribution of standard deviation of gradients. However, during the training, they consider a single mini-batch, instead of the regular one pass over the whole dataset in practice. This may cause the worse variance of the gradient during the training. That way, the claim that this is attributed to Dropout does not make much sense in this context. Also, in the caption, they have the conclusion "while the average variance of the gradient decreases slightly during training, the worst case increases, hence leading to unstable training in finetuning regimes". But this conclusion is not well supported by the two plots.

3. In Figure 2, I am confused how the authors use these two plot to demonstrate the dropout can hurt the fine-tuning of large models. Particularly, what is the purpose to show the right plot? Shouldn't it be a direct comparison between LoRA and ALLoRA in terms of accuracy, indicating that without dropout, ALLoRA can perform better? Also, the authors claim that 10 epochs is already a large number of finetuning iterations for practical scenarios. How to justify this? Is there any a standard to tell what number of iterations is large in finetuning?

4. In Section 3.2, the authors would like to show the second flaw of finetuning LLMs when the zero-initialization is used, which can cause poor optimization landscape. But how can we see this issue from this section? The authors claim that as the training set size decreases, the optimization landscape for $V$ will become degenerate. I am confused about this. Just from the Eqs. (1) and (2), how did the authors make this claim? They should at least show an example for the poor optimization landscape.

5. In Section 3.3, from Line 275 - 280, the derivation here is confusing. I understand that the authors would like have $(1+\eta)^L$ in the final term. However, when they define $C$ , how can the second inequality follow from? I believe $C$ is a consecutive product among all matrix norms of $W^*_1$,...,$W^*_L$. Also, can the authors tell in the paper why they need $\bar{m}$ in the paper? Please derive step by step clearly in the paper.

6. Notations are confusing. From the beginning, the authors didn't really define $W^*$. Then in line 306, they say $l$ is a learning rate, but in previous context, $l$ indicates the layer. Also, what does it mean by $f_o=\kappa:x\mapsto\eta x$? Please check thoroughly all notations in the paper to make sure they are clearly and properly defined.

7. In line 334, the authors mention that "We think a function that is inversely proportional to $||(BA)_{i,:}||$ is a good candidate to realize our idea". Why? You need justification here, not just saying that.

8. The Adaptive Learning in this work seems to have scaling factor in the functional, instead of directly applying to model parameters. However, to me, their effects are similar. That way, what is the exact difference and novelty in this study ALLoRA brings, compared to LoRA. Also, I didn't see how the second issue, poor optimization landscape has been addressed through ALLoRA.

9. The experimental results are not that promising. Though the authors present many results, from Section 4.4, the average improvement over all cases is 0.3%, which is even within the standard deviation, in my opinion. The same to Section 4.5. With such marginal performance improvement, what values does ALLoRA bring about?

10. Overall, though the topic in this work looks interesting, it requires a substantial amount of efforts from the authors to make it technically solid and sound, including more in-depth theoretical analysis, clarification on technical discussion, and more convincing experimental results.

W1. Figure 1 illustrates the distribution of the gradient's standard deviation as training progresses. While it is clear that the expectation in an OLS setting is affected by the standard deviation, the relationship between this formula and dropout remains unclear. For instance, the spiking gradients in Figure 1 could be due to out-of-distribution inputs. However, this does not seem to be directly related to dropout. In my view, to understand any potential connection between Figure 1 and dropout, it would be necessary to show that with dropout set to 0, the gradients remain stable, exhibiting low standard deviation without any spikes.

W2. The authors highlighted instability in LoRA-based fine-tuning methods with fewer epochs. However, through empirical experiments, it was not observed that training had actually stabilized or that the convergence speed had improved.

W3. The experiments lack baseline comparisons. For Table 1, experiments should include basic baselines, such as LoRA, and reference studies [1,2].

W4. (Minor) The ablation results in Figure 4 are not entirely clear. Personally, I believe that instead of using four separate plots to show relative improvement rates for the same data and parameters, it would be more effective to present this information in a single table.

[1] Zhang, Qingru, et al. "AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning." arXiv preprint arXiv:2303.10512 (2023).

[2] Jiang, Ting, et al. "MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning." arXiv preprint arXiv:2405.12130 (2024).

• Their analysis relies on the fact that we only fine-tune for a small number of epochs. This is highly subjective as the number of epochs required for a good fine tuning depends on the model and the dataset.
• Limited theoretical analysis of the adaptive learning rate’s convergence properties.
• The ripple effect argument lacks mathematical rigor, only establishing upper bounds without lower bounds or empirical validation.
• Use of simplified models may not accurately represent LLM fine-tuning dynamics.
• Absence of standard deviations in results makes it difficult to assess statistical significance.
• Lack of comparison with well-established benchmarks (e.g., GLUE).
• No comparison with similar approaches like LoRA+.
• Arbitrary choice of epoch numbers without clear justification.
• No discussion of computational overhead compared to standard LoRA.
• Unclear visualization of dropout’s impact in initial epochs.
• Limited justification for why/how ALLoRA ensures faster or more reliable convergence.

1. ALLoRA relies on a heuristic approach that seems more like a refined hyper-parameter adjustments rather than a fundamental improvement in LoRA architecture. The core changes such as adapting learning rates based on parameter norms, removing Dropout, and eliminating the scaling factor can be viewed as incremental modifications to existing techniques, not a novel structural design.

2. Limited experiments in terms of datasets, baseline models, comparison with other recent LoRA variants.

• Their experimental setup doesn't align with the standard setting. No GLUE benchmark experiments and the downstream task is limited to the commonsense reasoning task on LLaMA variants.
• Missing comparison to the various SOTA LoRA variants, only DoRA is used to compare the performance of ALLoRA on commonsense reasoning datasets.

3. Ablation study done in non-ideal settings.

• Ablation study not done on the same settings used for performance report, only done on midsized LLMs Qwen2-0.5B, Snowflake-Artic-L, and OpenELM-450M.

The primary issue with this paper lies in its clarity and coherence, particularly in Sections 3.1 and 3.2. The mathematical reasoning presented is often inconsistent with the conclusions drawn.

For instance, in Section 3.2, the claim that “as the training set size (n) decreases, the optimization landscape for V becomes degenerate” is not convincingly supported by the preceding discussion on the Hessian product.

In page 4, it is unclear to me how the formula (the "how far off" expectation bound) helps to argue the main conclusion of this section. The paragraph after the formula just does not make sense to me.

Additionally, there is a notable lack of comparison with relevant prior work that addresses similar challenges outlined in Sections 2 and 3. Specifically, regarding initialization, existing studies such as PiSSA[1], MiLoRA[2], and LoRA-GA[3] have already highlighted the benefits of non-zero initialization for LoRA. The discussion on scaling factor selection has also been explored by RSLoRA[4].

Furthermore, while the authors cite LoRA+[5] in relation to learning rate adjustments, there is no direct comparison between the fixed learning rate strategy of LoRA+ and the adaptive approach of ALLoRA. A thorough comparison with these established methods is necessary to substantiate ALLoRA’s claims and enhance the paper’s credibility. I agree that some of the mentioned works are very recent and may be considered concurrent work considering ICLR policy. However, given the pace of the development of this direction, and many of these recent papers dealing with very similar issues and adopting similar ideas (some from arguably more principled way), it is difficult to assess the novelty and contribution of the present submission without comparison to any of them. Even if I ignore all these recent works, I am not super exicited with the theoretical reasoning and empirical results of the paper. Hence, I think the submission may not pass the bar of ICLR acceptance.

[1] Meng, Fanxu, Zhaohui Wang, and Muhan Zhang. "Pissa: Principal singular values and singular vectors adaptation of large language models." arXiv preprint arXiv:2404.02948 (2024). [2] Wang, Hanqing, et al. "MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning." arXiv preprint arXiv:2406.09044 (2024). [3] Wang, Shaowen, Linxi Yu, and Jian Li. "LoRA-GA: Low-Rank Adaptation with Gradient Approximation." arXiv preprint arXiv:2407.05000 (2024). [4] Kalajdzievski, Damjan. "A rank stabilization scaling factor for fine-tuning with lora." arXiv preprint arXiv:2312.03732 (2023). [5] Hayou, Soufiane, Nikhil Ghosh, and Bin Yu. "Lora+: Efficient low rank adaptation of large models." arXiv preprint arXiv:2402.12354 (2024).