RaSA: Rank-Sharing Low-Rank Adaptation

Thank you for recognizing the significance, presentation, and methodology of our work!

W1 ..., in RaSA there is an underlying assumption that all shared layers have the same dimension. If so, it is worth mentioning it explicitly in the paper.

Thanks for pointing this out and we will add this assumption explicitly in the final version to make it clearly. In addition, we would like to emphasize that this assumption holds for the vast majority of widely used models (Llama, Mistral, Gemma, Qwen, ...).

W2 There are LoRA extensions that allow allocating a different rank for each layer, according to its importance (for example PRILoRA and others). It seems that in RaSA the rank should be similar across layers. If so, it is worth mentioning this fact and citing the relevant papers that do allow different rank allocation.

Thank you for pointing this out. RaSA does indeed assign a consistent rank across layers, which we will explicitly clarify in the final version of the paper. We will also include a more comprehensive discussion of rank-allocating methods such as PRILoRA and other relevant extensions, which will provide valuable context for readers.

W3 There are no comparisons in Table 1 to other LoRA extensions (AdaLoRA, PRILoRA), which may be of importance to the reader. Furthermore, it is not clear why the VeRA method is compared against, when its number of parameters is x13 less.

• We agree that including comparisons with methods like AdaLoRA and PRILoRA would provide additional context and be of interest to the reader. Please see General Response for details.
• VeRA, to out knowledge, is the first work to explore the concept of sharing rank across layers. We believe this makes it an interesting and relevant point of comparison for readers.

Thanks for the high assessment of this work!

W1 A great paper with almost no weaknesses. A small suggestion, the distinction between the different methods in Figures 4, 5, and 7 is only based on different color (red and blue), as printing on paper cannot distinguish them.

Thank you for this helpful suggestion! We will revise the figures in the final version to ensure better clarity.

Q1 I am curious why the performance is best when $k$ is around $r/8$ (line 249-250), and if possible, is there any theoretical explanation?

We have tried to explain this theoretically, but it is a challenging non-convex optimization problem. Therefore, so far, this remains empirical. It is worth noting that the best $k$ may also be related to the number of layers and the dimension of the model. We will clarify this in the final version. Thanks for bringing this up.

Thank you for acknowledging the contributions of our work!

W1 ... the authors only compare RaSA with existing peft methods, not including full fine-tuning.

Thanks for the suggestion. We agree that including full fine-tuning (FFT) would enhance the completeness of our work. Please refer to General Response for details.

Q1 In line 103, the authors claim that introducing a trainable diagonal matrix enables layer-specific weighting, actually, I am confused about this, maybe it could be explained specifically.

Thanks for pointing this out!

• Let's first explain why we call it weighting. Consider the matrices $\boldsymbol{B}=\begin{bmatrix}\boldsymbol{b}_1, \boldsymbol{b}_2, \cdots, \boldsymbol{b}_r\end{bmatrix}$, $\boldsymbol{A}=\begin{bmatrix}\boldsymbol{a}_1^T, \boldsymbol{a}_2^T, \cdots, \boldsymbol{a}_r^T\end{bmatrix}^T$, and a diagonal matrix $\boldsymbol{D}=\mathrm{diag}(d_1, d_2, \cdots, d_r)$, the product of these matrices can be expressed as:

  $\boldsymbol{B}\boldsymbol{D}\boldsymbol{A}=\sum_{i=1}^{r}d_i\boldsymbol{b}_i\boldsymbol{a}_i^T$.

  Here, each scalar $d_i$ acts as the weight of the component $\boldsymbol{b}_i\boldsymbol{a}_i^T$. This is why we call such a diagonal matrix $\boldsymbol{D}$ as a weighting.

• By "trainable," we mean it is not fixed or constant.

• By "layer-specific," we mean it it unique to each layer (not shared across layers).

Thank you for the positive feedback and helpful suggestions!

W1 Lack of baseline comparisions

Please see General Response.

Additionally, we note that EM-LoRA is still an anonymous submission to acl rolling review without open-sourced code. Therefore, we have not added EM-LoRA as a baseline and would be willing to add a disscusion about it in our final version.

W2 & Q1 Efficiency Concerns

• Training Overhead: We consider the extra computational time during training to be acceptable. The following table shows the time increments relative to LoRA, with RaSA incurring ~15% additional time. Notably, when compared to another high-rank method, MoRA, this 15% overhead is modest. Reviewer SwqF also agrees with this in Strengths (... with only a 10% increase in time, highlighting RaSA's promising advantages).

  r	Method	Llama-3.1-8B	Mistral-0.3-7B
  8	MoRA	25.0%	25.2%
  	RaSA	16.7%	13.1%
  16	MoRA	29.6%	30.8%
  	RaSA	14.3%	13.1%
  32	MoRA	24.0%	29.6%
  	RaSA	15.0%	15.7%

  Furthermore, we argue that parameter efficiency and inference efficiency are more critical metrics than training time when evaluating the efficiency of a PEFT method, as training is typically a one-time cost. RaSA fully inherits the advantages of LoRA in these two aspects (line 47-48: RaSA retains the core advantages of LoRA – keeping the same parameter overhead and allowing for easy merging back into the model).

• Performance Improvement: In Table 1, RaSA outperforms LoRA in PASS@1 by 5.4% in average, not 2%.

Q2 The performance improvement appears to be notably greater on Mistral-0.3-7B compared to Llama-3.1-8B. Do you have any insights into the reasons behind this difference?

That's a good question. We speculate that RaSA is more effective on tasks that are challenging relative to the model’s capacity. For the math and code tasks in our evaluation, these challenges are more pronounced for Mistral-0.3-7B than for Llama-3.1-8B (confirmed in Meta's blog of llama-3.1), which explains the larger performance gains.

Thank you. Can you kindly update the PDF so we can see it in context?

Dear Reviewers,

We have updated the PDF to reflect the corresponding revisions (highlighted in light blue). The additional baselines are placed in Table 3 of Appendix D (as they are still being completed).

Thank you!