RepLoRA: Reparameterizing Low-rank Adaptation via the Perspective of Mixture of Experts

We thank the reviewer for the feedback and would like to address the concerns as follows:

• Regarding the comparison with other variants of LoRA: Following the reviewer’s suggestion, we conducted an additional experiment on the image classification task using the FGVC dataset to compare RepLoRA with VeRA [1] and DoRA [2]. The results are presented below:

Furthermore, we would like to highlight that RepLoRA has already been compared to other methods that utilize hypernetworks to generate adapters in the main paper. Specifically, in the main text, we included a comparison with prefix tuning [3], which leverages an MLP to generate the adapters, specifically the prepended prompts. Building on the reviewer’s helpful suggestion, we will also incorporate comparisons with VeRA and DoRA in the revised version to provide a more comprehensive evaluation.

• Regarding the comparisons with comparable parameter sizes: In all experiments, we reparameterized the low-rank adapters using MLPs with a hidden dimension of $h=64$, which we identified as the optimal setting through extensive tuning in terms of both PPT and accuracy. To address this concern, we also conducted experiments with a reduced hidden dimension of $h=8$, resulting in a comparable parameter size. The results, presented below, show a slight overall drop; however, RepLoRA still significantly outperforms vanilla LoRA, demonstrating its robustness and practical benefits under tighter parameter constraints:

Following the reviewer's suggestion, we will include this finding in the appendix of the final manuscript.

References

[1] VeRA: Vector-based Random Matrix Adaptation. ICLR. 2024

[2] DoRA: Weight-Decomposed Low-Rank Adaptation. ICML. 2024

[3] Prefix-Tuning: Optimizing Continuous Prompts for Generation. ACL. 2021

We thank the reviewer for the valuable feedback and would like to address the concerns raised as follows:

• Regarding the discrepancy between the theoretical results and the experiment setup: Thanks for this feedback. We would like to clarify that the assumption in Section 4.2 that $A_Q=A_V$ is for simplicity and can totally be generalized to the setting where they share only the learnable matrix $A$. In particular, we can reformulate those matrices as $A_Q=W_{Q,1}A, A_V=W_{V,1}A$ for the simple linear reparametrization and as $A_Q=\sigma_1(W_{Q,1}A), A_V=\sigma_1(W_{V,1}A)$ for the non-linear reparametrization. Tailored to that setting, we need to add several additional terms involving parameters $W_{Q,1}$ and $W_{V,1}$ rather than merely parameters $W_{1}$ as in the current setting, making the convergence analysis unnecessarily complicated. Therefore, we assume without loss of generalization that $A_{Q}=A_{V}=W_{1}A$ or $A_{Q}=A_{V}=\sigma_1(W_{1}A)$ in Section 4.2 to simplify the analysis, making it more accessible.

• Regarding the experiment tying $A_Q$ and $A_V$: To further support our theoretical findings, we conduct experiments on FGVC datasets by tying $A_Q$ with $A_V$ and $B_Q$ with $B_V$ in RepLoRA. The results, summarized in the table below, show that tying $A_Q$ and $A_V$ slightly reduces the model's expressiveness, resulting in a modest drop in performance compared to the original RepLoRA with untied matrices. However, tied RepLoRA still significantly outperforms vanilla LoRA, which reinforces the practical value of our approach, even under constrained parameterization.

• Regarding the implications of Theorem 4.1: Thanks for your comment. We would like to emphasize that the results in lines 187-196 (right column) are consequences of Theorem 4.1 in our paper. In particular, by combining the result of Theorem 4.1 and the formulation of the Voronoi loss $\mathcal{D}_{1,r}$ in lines 167-173 (right column), we deduce the convergence rates of low-rank matrices estimation are slower than polynomial rates $O(n^{1-/2r})$ for all $r\geq 1$, where $n$ is the sample size. Due to the inequality $\log(n)<n$, these rates are even slower than the order $O(1/\log^{\tau}(n))$ for some positive constant $\tau$. As a result, to achieve a given approximation error $\epsilon=O(1/\log^{\tau}(n))$ of estimating the low-rank matrices, we need exponentially many data points $O(\exp(\epsilon^{-1/\tau}))$. We will consider formulating those implications into a corollary following Theorem 4.1 in the revision of our manuscript.

• Regarding the activation function: The nonlinearity was implemented using the sigmoid function. We appreciate the reviewer’s suggestion and will include this detail in the final revision.

We sincerely thank the reviewer for the constructive feedback and would like to address your concerns as follows:

Regarding the analysis of the theoretical results: Our theoretical analysis demonstrates that LoRA with reparameterization offers superior sample efficiency compared to LoRA without reparameterization. To empirically analyze this theoretical claim, we dedicated the final experiment in the experimental section to validate these theoretical findings. The results show that reparameterization in LoRA significantly improves sample efficiency, as illustrated in Figure 2.

Regarding the comparison with related works: Following the reviewer’s suggestion, we have included additional comparisons with VeRA [1] and DoRA [2] on the image classification task using the FGVC dataset, as detailed below:

The results demonstrate that RepLoRA outperforms both LoRA variants by large margins, emphasizing its practical advantages. In response to the reviewer’s suggestion, we will include this finding in the final revision.

References

[1] VeRA: Vector-based Random Matrix Adaptation. ICLR. 2024

[2] DoRA: Weight-Decomposed Low-Rank Adaptation. ICML. 2024

We appreciate the reviewer’s insightful feedback. In response to their concern, we’ve expanded our analysis to include additional comparisons with three LoRA adapters: VeRA [1], DoRA [2], and MoR [3], as suggested by the reviewer. These experiments were carried out on the image classification task using the FGVC datasets. For MoR, we specifically report results with 8 experts. The detailed results are presented below:

Following the reviewer’s suggestion, we will incorporate these results into the final revision.

When it comes to training, RepLoRA introduces only a minimal increase in parameters compared to LoRA, as seen in the number of parameters and PPT. As a result, RepLoRA does not introduce any significant additional training time, FLOPs, or memory usage when compared to LoRA. Additionally, as highlighted in the main text, the reparameterization matrices can be discarded during inference, making the inference process identical to that of LoRA.

References

[1] VeRA: Vector-based Random Matrix Adaptation. ICLR. 2024

[2] DoRA: Weight-Decomposed Low-Rank Adaptation. ICML. 2024

[3] MoR: Mixture of Ranks for Low-Rank Adaptation Tuning. Arxiv. 2024