The Expressive Power of Low-Rank Adaptation

We sincerely thank the reviewer for acknowledging the theoretical significance and appreciating the empirical results of our paper. We are pleased to inform you that we have carefully addressed each of your concerns, as elaborated below.

Q1: A notation table would help understand all the notations since the paper is mostly about theoretical proof.

Thanks for the suggestion. Due to the page limit, We have summarized a list of the notations in Appendix A.

Q2: In section 2, why a L-layer (instead of one-layer) linear model is considered, which is still a linear model?

Correct. In our paper, we employ deep linear models (L-layer linear model) as a preliminary model, which serves as a foundation for extending our results to nonlinear models (i.e., FNN and TFN). Studying this toy model is a common technique in theoretical deep learning research, which offers valuable insights into deep nonlinear models, and has been employed in many notable studies, including those by Saxe et al., 2014, Kawaguchi, 2016, Lu & Kawaguchi et al., 2017, Hardt & Ma, and Laurent & Brecht, 2018.

Final Note: We want to thank you again for all the questions and comments you have provided. If there are any remaining questions, please do not hesitate to let us know. If our responses have resolved your concerns, we kindly request you to consider increasing your score and championing our paper.

References

• Saxe, Andrew M., James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. ICLR, 2014.
• Kawaguchi, Kenji. Deep learning without poor local minima. NeurIPS, 2016.
• Lu, Haihao, and Kenji Kawaguchi. Depth creates no bad local minima. arXiv, 2017.
• Hardt, Moritz, and Tengyu Ma. Identity matters in deep learning. ICLR, 2017.
• Laurent, Thomas, and James Brecht. Deep linear networks with arbitrary loss: All local minima are global. ICML, 2018.

We thank the reviewer for the encouraging feedback, especially for recognizing the theoretical contribution, timeliness, and importance of our paper in illuminating the empirical success of LoRA. Our response is detailed below.

Q1: While it is okay to not have them in this paper, I think it would be interesting to study other effects of LoRA theoretically. For example, how does LoRA affect generalization? What can we say about how fast LoRA can converge even if the target model can eventually be found by LoRA exactly.

We agree with the reviewer that it is an interesting research topic to explore the generalization and optimization problems of LoRA theoretically. While an in-depth analysis of these properties is beyond the scope of this paper, as the reviewer acknowledges, we have included additional simulation results and discussion based on your comments that provide initial insight into these issues. Please find our responses in the general comment to all reviewers and AC, specifically under "(Major Update #2) Illuminating the Optimization and Generalization Aspects of LoRA".

Final Note: We are excited that you consider our work both timely and theoretically strong. We plan to incorporate these responses into our updated version. Thank you once again for your insightful comments and for your encouraging feedback!

We wish to clarify that our primary objective is to understand the effectiveness of LoRA via theory, rather than experiments. We thank the reviewer for recognizing the thoroughness of our theoretical analysis and their contribution to the empirical understanding of LoRA. We believe this recognition affirms the significant theoretical contribution of our paper. While we acknowledge that not all of our findings may be directly applicable to practical settings, the theoretical insights provided by our work form a crucial foundation for future practical applications.

In light of the feedback, and although it was beyond the original scope of our work, we have conducted additional empirical studies and are excited to share these findings with you! If there are any remaining questions, please do not hesitate to let us know. If all of your concerns have been fully resolved, we kindly request you to consider increasing your score.

Q1: Given the widespread application of LoRA to various large language models (LLMs), there's an opportunity for the authors to substantiate their findings using models tasked with different challenges.

As per your question, we have conducted additional experiments using LLMs such as RoBERTa-base and RoBERTa-large on GLUE benchmark to substantiate our theoretical results. Please find our response in general comments to all reviewers and AC, specifically the first two findings under major update #1.

Q2: A multifaceted evaluation, including different performance metrics, would offer a more robust validation of the claims made in this work.

Our real-world experiments conducted on the GLUE benchmark include various metrics such as overall (matched and mismatched) accuracy for MNLI, Matthew's correlation for CoLA, Pearson correlation for STS-B, and accuracy for other tasks.

For our synthetic simulations, we have expanded the scope to include multi-class classifications. We quantize the results and optimize cross-entropy rather than MSE, and subsequently report the test accuracy.

As shown in Figure 5b (https://hackmd.io/_uploads/BkedJmrNT.png), consistent with our theoretical results, our construction achieves 100% accuracy when $R \geq 8$. The performance of gradient update is also similar to our observation when MSE is employed as the metric, particularly when MSE is plotted on a logarithmic scale (Figure 4: https://hackmd.io/_uploads/HksV_SBNp.png). This observation echoes the findings of Hui & Belkin et al. (2021), which indicate that optimizing MSE is fundamentally equivalent to optimizing cross-entropy.

The suboptimal performance of gradient update method in this simulation suggests that, despite LoRA's current impressive performance in practical applications, there is potential for further refinement.

Q3: Additionally, it is acknowledged within the community that even very low ranks (such as 4, 2, or even 1) can yield satisfactory fine-tuning results. Readers would benefit from an exploration into how low-rank adjustments are able to achieve effective fine-tuning.

We believe that our current results indeed explain when and why low-rank adjustments can yield satisfactory performance. We further elucidate this with additional empirical insights.

1. Theoretical Understanding: Our theoretical findings demonstrate that larger models requires lower ranks for effective fine-tuning. This insight is particularly relevant when employing large language models or stable diffusion models.

2. New Real-World Experiments: We have also added new experiments comparing RoBERTa-base and RoBERTa-large, as detailed in the first two findings of our summary in major update #1 in general comments to AC and all reviewers. These experiments further validate our theory, which suggests that larger models require smaller ranks for effective fine-tuning.

Q4: The authors are urged to establish a clearer connection between their theoretical discoveries and the empirical results previously reported for LoRA. Doing so could significantly streamline the hyper-parameter selection process for LoRA, reducing the effort required to fine-tune models effectively.

In response to your insightful suggestion to better integrate our theoretical findings with empirical observations, we have summarized both our theoretical and corresponding (additional) empirical results in our summary of major update #1 in general comments to all reviewers and AC. This enhanced and integrated discussion has been thoughtfully incorporated into the final version of our paper (Sec. H).

Final Note: Thank you for your detailed comments. We are excited that you found the theoretical study of our paper to be thorough. If there are any remaining questions, please do not hesitate to let us know. If our responses have resolved your concerns, we kindly request you to consider increasing your score and support the acceptance of our paper.

We sincerely appreciate your recognition of the novelty of our paper, its consistency with recent advances of LoRA, and the comprehensiveness of our theoretical and empirical results. Your concerns are addressed below. If our responses have resolved your concerns, we kindly request you to consider increasing your score and championing our paper.

Q1: From Figure 1, I can see that LoRA of FNN performs on par with gradient update, whereas LoRA of TFNs significantly outperform gradient updates. Could the author explain this performance difference?

In Fig. 1, we plot MSE, while in Fig. 5, we plot log(MSE). Sorry for the confusion, and we will make this clear in the revision. We have added Fig. 4 (https://hackmd.io/_uploads/Skgt22SV6.png) to report performances of FNN in log(MSE), and we observe that gradient update underperforms our construction on both FNN and TFN.

Q2: It is impressive that LoRA with rank=1 can match the performance of gradient update in Figure 3 (which is Figure 5 in the updated version). Does this mean the gradient update does not actually learn well?

The reviewer is correct that gradient update might fail to find the optimal LoRA parameters in the TFN cases. This underperformance of gradient update method, especially compared to our LoRA adapter construction, suggests that LoRA's current effectiveness might still have room for further improvement. A more sophisticated implementation could potentially yield even better results, indicating a promising direction for future exploration and development in this area.

Final Note: We are excited that you find our work novel and appreciate our theoreical study and the corresponding empirical validation. If you have any further questions, please do not hesitate to let us know. If our responses have resolved your concerns, we kindly request you to consider increasing your score and championing our paper.

2. When the frozen model is closer to the target model, a lower LoRA-rank is sufficient to attain the desired performance.

   • Our Theoretical Results: supported by Lemma 1, 2, and Theorem 5, 6

   • Empirical Observation: We have conducted new experiments to further support this claim. The following discussion has been incorporated into Sec. G.8 of our revised version.

     In these experiments, we compared the performance of pretrained RoBERTa-base with that of randomly initialized RoBERTa-base. The results clearly show that the pretrained model requires lower LoRA-ranks to achieve the desired performance, which aligns with our theoretical findings.

     Dataset	MNLI	MNLI	SST-2	SST-2	MRPC	MRPC	CoLA	CoLA	QNLI	QNLI	QQP	QQP	RTE	RTE	STS-B	STS-B
     Model	random	pretrained	random	pretrained	random	pretrained	random	pretrained	random	pretrained	random	pretrained	random	pretrained	random	pretrained
     R=2	.523	.861	.775	.950	.691	.892	.154	.632	.627	.928	.761	.891	.542	.780	.213	.907
     R=4	.535	.868	.788	.950	.696	.890	.145	.634	.625	.929	.768	.898	.542	.805	.224	.910
     R=6	.544	.868	.799	.948	.696	.892	.154	.629	.632	.931	.768	.900	.542	.773	.210	.909

3. LoRA outperforms tuning of the final layers if the quality of the shared representation is suboptimal.

   • Our Theoretical Results: This is supported by Corollary 4 and our newly added Lemma 4 in Sec. 3.3.

     Lemma 4: Let $D \geq 2$ and $\overline{f}$ be a one-layer target FNN. Assume that the elements of weight matrices $(W_l)_{l=1}^L$ are independently drawn from arbitrary continuous distributions. With probability 1, for any tuning of the last $L-1$ layers, the frozen model cannot be fine-tuned to match the target model, i.e., $f\neq\overline{f}$.

     This lemma highlights the limitations of final-layer tuning in a randomly initialized model for simple tasks achievable by a one-layer FNN. Conversely, Corollary 4 shows that LoRA can fine-tune a randomly initialized frozen model to align with any smaller target model, given a small rank requirement.

   • Empirical Observation: This finding aligns with Kornblith et al., (2019), emphasizing that the success of final layer tuning depends on the quality of the model's initial layers.

     We also added a simulation result comparing LoRA and final layer tuning with varying tunable parameters, as shown in Figure 6a. This validates our theory that, with equal tunable parameters, LoRA surpasses final layer tuning when the frozen model is randomly generated and the shared representation is poor.

4. In addition to applying low-rank updates to weight matrices, updating the bias is also crucial.

   • Our Theoretical Results: The update of biases plays a significant role in our proofs (see Sec. 3.2 and E.1).
   • Empirical Observation: Hu et al. (2022) recommend bias updates during LoRA training (see their 6th footnote). Our added toy experiments (Figure 6b in Section G.4) further examine this claim, underscoring the role of tunable bias vectors.

5. Tuning only the attention weights is sufficient to achieve good performance on TFNs.

   • Our Theoretical Results: This finding is supported by Theorem 7.
   • Empirical Observation: Hu et al. (2020) recommend applying LoRA only to the attention weights in Sec. 4.2.

6. Current optimization algorithms for LoRA training might be suboptimal.

   • Empirical Observation: Our current and newly added simulations, illustrated in Figures 4, 5, and 9, all support this claim. Thus, a more advanced algorithm could potentially lead to improved results, suggesting a promising avenue for future exploration and development in this field.