BiLoRA: A Bi-level Optimization Framework for Low-rank Adapters

Weakness 2 addition: The influence of $T_1/T_2$.

Number of iterations: In the paper, we didn’t change the iteration numbers throughout our experiments and kept T1=T2=1. Actually there can be a set of hyperparameters that can all achieve good results because they can all keep good balance of inner/outer level and achieve 3 perspectives. 1) performing well on inner dataset; 2) performing well on outer dataset; 3) not overfitting on either subset and generalizing well on test dataset. We further experiment on different iteration numbers as follows. We experiment on DeBERTa-v3-base on CoLA dataset. For larger T1 and T2, one global step contains more training steps so we reduce the total global steps accordingly. All other hyperparameters are kept the same.

The total optimization steps for inner level and outer level are close for different T1/T2. Typically, a single inner optimization step is faster than a single outer optimization step due to the calculation of hypergradients of outer level. So using a larger T1 is also an efficient choice.

We don’t exactly tune the iteration numbers for 2 reasons.

• T1=T2=1 is an empirical choice in existing bi-level optimization tasks. The main effect of iteration numbers can be the balance of inner and outer level. Intuitively and practically, T1=T2=1 means BiLoRA frequently optimizes between the 2 levels, preventing the model from overfitting on either subset and effectively addresses overfitting.

• Simplicity. We expect BiLoRA to be powerful and effective, yet easy to use. The number of hyperparameters for BiLoRA is kept nearly the same as LoRA, which is much less than AdaLoRA.

Question 1: The orthogonality of optimal solutions.

We suspect the reason for “the performance is largely unaffected by the coefficient of this regularization” can be from these 3 aspects.

• The value distribution of $P^{T}P$ and $QQ^{T}$ in optimal solutions with regularization coefficient $\lambda=0.0$ is close to Identity Matrix. Despite magnitudes, singular vectors are largely orthogonal. When regularization coefficient $\lambda >= 0.1$, singular matrices are almost orthogonal.
• This “natural orthogonality" without regularization can be kind of due to the Normal Initialization we use for initializing all the singular vector matrices.
• Optimal solutions with different regularization coefficients including $\lambda=0.0$ can all contribute greatly to prevent overfitting.

First, we sample one of the singular vector matrices from the optimal solutions with reg coefficient=0.0 and 0.1. Since we use rank=8, the dimension of the resulting matrix is 8×8. We plot $P^{T}P$ in both AdaLoRA and BiLoRA. We sincerely refer you to the plots in Figure 6 (a)(b). The plots show that the value distribution of $P^{T}P$ from optimal solutions with reg coefficient=0.0 is very close to the Identity Matrix. When the reg coefficient > 0.0, the pseudo-singular vectors turn out to be almost orthogonal. These results can explain the robustness of BiLoRA across different regularization coefficients.

Secondly, this “natural orthogonality" can be kind of due to the Normal Initialization. We further plot the sampled matrix just after initialization. We sincerely refer you to the plots in Figure 6 (c). We use “nn.init.normal_()” for initializing all the singular vector matrices.

Third, optimal solutions with different reg coefficients can all contribute greatly to prevent overfitting. We plot the loss curves with reg coefficient=0.0 and 0.1. We sincerely refer you to the curve in Figure 7 and Figure 2.

Results demonstrate that with different regularization coefficients, BiLoRA can effectively prevent overfitting.

We appreciate your constructive feedback very much. We provide all the figures from Figure 2-7 in the Supplementary Section C-F of our paper. All the indexes of figures are referred to those of the updated paper. We sincerely refer you to the figures in our paper. We provide our response to your questions as follows.

Weakness 1: The basis of this paper.

The basis of BiLoRA can be concluded in 4 aspects.

• Limitations of existing fine-tuning methods.
• Inspirations and advantages of bi-level optimization.
• Accelerating the fine-tuning process.
• Opportunities for deeper understanding of PEFT process.

First, existing fine-tuning methods can suffer from overfitting. The large number of parameters in pre-trained models may make the full fine-tuning process more prone to overfitting[1]. Existing PEFT methods do not well solve this problem. The generalization capability of a fine-tuned model on the evaluation dataset can reflect the range of overfitting. We plot the training loss and evaluation loss of LoRA, AdaLoRA and BiLoRA to illustrate the generalization capability. We conduct experiments on RoBERTa-base on CoLA dataset and clip the first 2k steps. Figure 2(a) and Figure 2(b) show that for both LoRA and AdaLoRA, the evaluation loss starts to increase at or before 1k step while the training loss still decreases. The gap between the 2 losses is getting huge. This demonstrates the limitations of existing methods that they still suffer from overfitting issues. Figure 2(c) shows that both inner training loss and outer training loss are close to the evaluation loss. The evaluation loss is still decreasing after 2k steps and is obviously lower than that of LoRA and AdaLoRA. These results convincingly demonstrate the limitations of existing methods and the effectiveness of BiLoRA in addressing overfitting.

Second, we gain inspiration from DARTS (Differentiable Architecture Search) which utilizes a bi-level optimization for searching network architectures. DARTS uses training dataset and validation dataset separately for training operations and their corresponding proportions in 2 levels. DARTS conducts experiments on comparison between bi-level optimization and the normal training method(single-level). The latter performs much worse and they concludes that the latter “would cause α (analogous to hyperparameters) to overfit the training data, leading to poor generalization”[2]. We gain this inspiration of preventing overfitting and innovatively explore the possibilities of applying bi-level optimization to PEFT domain for addressing the existing limitations of overfitting.

Third, one important goal of PEFT is to accelerate the fine-tuning process. The bi-level optimization framework effectively accommodates larger learning rates by iteratively optimizing between the two levels without affecting the training stability. Through BiLoRA, we significantly reduce the training steps and training time needed for convergence.

Fourth, we expect BiLoRA to encourage deeper understanding of low-rank fine-tuning processes and more design potentials. We offer more flexible and direct interaction with singular values and singular matrices at separate levels and design different parameterization of singular values. More efficient methods and deeper insights can be inspired by this bi-level optimization framework.

Weakness 2: The influence of the dataset partition.

Data Partition: The Dataset Partition, together with learning rate can help keep the balance of inner/outer optimization, which can contribute greatly to preventing the model from overfitting. Lower level has more trainable parameters, so it’s natural to use more data for training singular vectors, while using the left for training singular values. We further experiment on DeBERTa-v3-base on CoLA and SST2 datasets to show the influence of different dataset partitions. We change the inner level dataset partition from 0.6 to 1.0 with 0.1 interval.

Results show that too small partitions(<=0.6) or too large partitions(1.0, only train singular matrix) can harm the overall performance.When the inner partition is too small, singular vectors are not well trained and when the inner partition is 1.0, singular vectors are not trained, which can cause a great performance drop. These results can also show that the two levels are both necessary in preventing overfitting and enhancing performances. In the paper, we don’t ever change the partition of the data and keep it 8:2.

[1] Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers.

[2] Liu H, Simonyan K, Yang Y. Darts: Differentiable architecture search.

Thank you for all the constructive feedback. We really appreciate your insightful suggestions.

We highly value your concerns regarding the motivation and the reason why techniques work well, and we genuinely provide the following response in the hope of alleviating your concerns.

Extensive experiment results demonstrate the better generalization capability of BiLoRA and loss curves convincingly illustrate that BiLoRA largely addresses the overfitting issue. There may be 3 main useful techniques for the improvement. 1) Bi-level optimization framework. 2) Parameterization of Singular Values. 3) Orthogonality Initialization and Regularization.

Bi-level Optimization Framework.

As far as we are concerned, the reason for utilizing bi-level optimization for low-rank adaptation and the powerful effect is mainly from 3 aspects:

• The overfitting issue has long been a hindrance to the further improvement of existing fine-tuning methods.
• Convincing results, conclusions and inspirations from DARTS. DARTS conducts experiments on comparison between bi-level optimization and the normal training method(single-level). Results demonstrate that the latter performs much worse and DARTS concludes that the latter “would cause α (analogous to hyperparameters) to overfit the training data, leading to poor generalization”. Bi-level optimization has been shown to improve the generalization capability.
• The nature of bi-level optimization. Iteratively optimizing between the 2 levels can intuitively prevent overfitting on either of the 2 subsets.

Parameterization of Singular Values.

The parameterization of singular values can help improve the performance because:

• Softmax and Approximately Binary Parameterization outperform Real Value (the parameterization in LoRA and AdaLoRA) because they yield positive values which meet the constraint that singular values need to be non-negative while Real Value does not.

Orthogonality Initialization and Regularization.

The Orthogonality Initialization and Regularization can help improve the performance due to:

• With $P$ and $Q$ to be almost orthogonal, $\Lambda$ can be exactly the singular value matrix of the low-rank adapters according to SVD. Thus the outer level is directly optimizing the singular values which can be viewed as importance of diifferent ranks.
• The Orthogonality Regularization can contribute to preventing overfitting since it helps optimizing the training loss while still keeping the evaluation loss decreasing rather than overfitting. This has been convincingly illustrated by the loss curves.

BiLoRA innovatively introduces bi-level optimization to the Parameter-Efficient-Fine-Tuning domain. It explores and demonstrates the huge potentials of utilizing bi-level optimization for improving the generalization capability of low-rank adapters and speeding up the fine-tuning process. We trust BiLoRA will help enhance the efficiency and performance of many models on many downstream tasks.

Thank you again for engaging with us and your valuable feedback.

Weakness 3 addition: The influence of pruning rates and weight decay in AdaLoRA.

The hyperparameters for AdaLoRA have been well tuned by the authors of AdaLoRA, so the results are already the best that can be achieved by AdaLoRA. We conduct several experiments trying to offer some insights about the influence of pruning rates and weight decay in AdaLoRA. We experiment on different pruning rates and larger weight decay on AdaLoRA. We experiment on DeBERTa-v3-base on the COLA dataset. Results can be seen as follows.

When applying larger weight decay in AdaLoRA, the performance drops. We show the training/evaluation loss curves of different weight decays. We sincerely refer you to the curves in Figure 3. There is no significant difference among the curves.

Pruning rates can be tuned by changing the target rank count. The target rank under current experimental settings is already 1 so we reduce the layer kinds from 6 to 4. Results can be seen as follows.

We also plot the training/evaluation loss curves of different pruning rates. We sincerely refer you to the curves in Figure 4.

Both pruning and weight decay cannot largely solve the overfitting problem. So we still consider this overfitting phenomenon to be an important issue for existing methods.

Weakness 4: Comparison between the distribution of the singular values learned in BiLoRA and AdaLoRA.

We plot the distribution of singular values learned both in BiLoRA and AdaLoRA. We use the singular values of RoBERTa-base on the CoLA dataset. AdaLoRA implements low-rank adapters to all linear layers including query, key, value, intermediate, layer.output and attention.output while LoRA and BiLoRA only implements low-rank adapters to query and key. We use abs for all the singular values of AdaLoRA. We sincerely refer you to the plots in Figure 5.

There are some similar features. 1) A lot of singular values are nearly zero. AdaLoRA gradually prunes ranks to make some of the singular values zero. BiLoRA utilizes bi-level optimization to automatically learn this distribution. 2) There are some important ranks with large singular values.

More results can be analyzed in future work in the distribution of singular values of optimal solutions. We expect BiLoRA can provide flexible interaction with singular values and help us better understand the process of low-rank fine-tuning.

We appreciate your constructive feedback very much. Since we are not able to submit a PDF, we provide all the figures from Figure 2-7 in the Supplementary Section C-F of our paper. All the indexes of figures are referred to those of the updated paper. We sincerely refer you to the figures in our paper. We provide our response to your questions as follows.

Weakness 1 and 3: The limitation of minimizing a single dataset’s training loss and the motivation of BiLoRA.

The motivation of BiLoRA can be concluded in 4 aspects.

• Limitations of existing fine-tuning methods.
• Inspirations and advantages of bi-level optimization.
• Accelerating the fine-tuning process.
• Opportunities for deeper understanding of PEFT process.

First, existing fine-tuning methods can suffer from overfitting. The large number of parameters in pre-trained models may make the full fine-tuning process more prone to overfitting[1]. Existing PEFT methods do not well solve this problem. The generalization capability of a fine-tuned model on the evaluation dataset can reflect the range of overfitting. We plot the training loss and evaluation loss of LoRA, AdaLoRA and BiLoRA to illustrate the generalization capability. We conduct experiments on RoBERTa-base on CoLA dataset and clip the first 2k steps. Figure 2(a) and Figure 2(b) show that for both LoRA and AdaLoRA, the evaluation loss starts to increase at or before 1k step while the training loss still decreases. The gap between the 2 losses is getting huge. This demonstrates the limitations of existing methods that they still suffer from overfitting issues. Figure 2(c) shows that both inner training loss and outer training loss are close to the evaluation loss. The evaluation loss is still decreasing after 2k steps and is obviously lower than that of LoRA and AdaLoRA. These results convincingly demonstrate the limitations of existing methods and the effectiveness of BiLoRA in addressing overfitting.

Second, we gain inspiration from DARTS (Differentiable Architecture Search) which utilizes a bi-level optimization for searching network architectures. DARTS uses training dataset and validation dataset separately for training operations and their corresponding proportions in 2 levels. DARTS conducts experiments on comparison between bi-level optimization and the normal training method(single-level). The latter performs much worse and they concludes that the latter “would cause α (analogous to hyperparameters) to overfit the training data, leading to poor generalization”[2]. We gain this inspiration of preventing overfitting and innovatively explore the possibilities of applying bi-level optimization to PEFT domain for addressing the existing limitations of overfitting. Extensive experiments demonstrate the effectiveness of BiLoRA.

Third, one important goal of PEFT is to accelerate the fine-tuning process. The bi-level optimization framework effectively accommodates larger learning rates by iteratively optimizing between the two levels without affecting the training stability. Through BiLoRA, we significantly reduce the training steps and training time needed for convergence, thus making PEFT more efficient.

Fourth, we expect BiLoRA to encourage deeper understanding of low-rank fine-tuning processes and more design potentials. We offer more flexible and direct interaction with singular values and singular matrices at separate levels and design different parameterization of singular values. More efficient methods and deeper insights can be inspired by this bi-level optimization framework.

Weakness 2: The clarification of "single training dataset".

When we mention that existing methods train "on a single training dataset", we mean training on a single training dataset in a single training process. Compared with this common practice in machine learning, we innovatively leverage the characteristics of bi-level optimization, optimizing low-rank adapters iteratively between upper and lower levels, to address the overfitting issues that arise in existing fine-tuning methods on a single dataset during a single training process. BiLoRA aligns with intuition: by iteratively optimizing between two levels, the model is less prone to overfitting on each of the two sub-datasets and the whole “single dataset”, resulting in enhanced generalization capability and better performance. Extensive experiments on multiple models and datasets strongly support this observation.

[1] Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers.

[2] Liu H, Simonyan K, Yang Y. Darts: Differentiable architecture search.

Dear Reviewer,

Thank you for your valuable insights and suggestions. We have tried our best to answer your questions in our author response, and we are also working to revise the paper following your suggestions. Given that the discussion period is ending soon, we were wondering if you could let us know if you have further questions or whether the author response addressed your concerns. We would be delighted to answer any further questions you might have.

Sincerely,

Authors

Weakness 3: It doesn't seem convincing to replace LoRA with BiLoRA due to the lack of theoretical insights.

We’d like to offer some empirical insights and notable advantages of BiLoRA in 2 aspects.

• Limitations of existing fine-tuning methods.
• Inspirations and advantages of bi-level optimization.

First, existing fine-tuning methods can suffer from overfitting. The large number of parameters in pre-trained models may make the full fine-tuning process more prone to overfitting[1]. Existing PEFT methods do not well solve this problem. The generalization capability of a fine-tuned model on the evaluation dataset can reflect the range of overfitting. We plot the training loss and evaluation loss of LoRA, AdaLoRA and BiLoRA to illustrate the generalization capability. We conduct experiments on RoBERTa-base on CoLA dataset and clip the first 2k steps. Figure 2(a) and Figure 2(b) show that for both LoRA and AdaLoRA, the evaluation loss starts to increase at or before 1k step while the training loss still decreases. The gap between the 2 losses is getting huge. This demonstrates the limitations of existing methods that they still suffer from overfitting issues. Figure 2(c) shows that both inner training loss and outer training loss are close to the evaluation loss. The evaluation loss is still decreasing after 2k steps and is obviously lower than that of LoRA and AdaLoRA. These results convincingly demonstrate the limitations of existing methods and the effectiveness of BiLoRA in addressing overfitting.

Second, we gain inspiration from DARTS (Differentiable Architecture Search) which utilizes a bi-level optimization for searching network architectures. DARTS uses training dataset and validation dataset separately for training operations and their corresponding proportions in 2 levels. DARTS conducts experiments on comparison between bi-level optimization and the normal training method(single-level). The latter performs much worse and they concludes that the latter “would cause α (analogous to hyperparameters) to overfit the training data, leading to poor generalization”[2]. We gain this inspiration of preventing overfitting and innovatively explore the possibilities of applying bi-level optimization to PEFT domain for addressing the existing limitations of overfitting. Extensive experiments demonstrate the effectiveness of BiLoRA.

Question 1: How does it achieve parameter-efficient finetuning?

In LoRA, for $h=W_{0}x$, the modified forward pass yields: $h = W_{0}x + \Delta Wx = W_{0}x + BAx$, where $\Delta W \in R^{d \times k}$, $A \in R^{r \times d}$, $B \in R^{k \times r}$ and $r \ll min\{d,k\}$. So the total number of trainable parameters is much smaller than full fine-tuning. In BiLoRA, P is of the same size as B, Q is of the same size as A and the singular value matrix $\Lambda \in R^{r \times r}$, which is much smaller than A and B. So the total count of the trainable parameters is almost the same as LoRA and much less than full fine-tuning. So this can effectively achieve parameter-efficient finetuning. In addition, BiLoRA accelerates the training process, thus making PEFT more efficient.

Question 2: How do you decide two datasets D1 and D2?

The Dataset Partition, together with learning rate can help keep the balance of inner/outer optimization, which can contribute greatly to preventing the model from overfitting. Lower level has more trainable parameters, so it’s natural to use more data for training singular vectors, while using the left for training singular values. We further experiment on DeBERTa-v3-base on CoLA and SST2 datasets to show the influence of different dataset partitions. We change the inner level dataset partition from 0.6 to 1.0 with 0.1 interval.

Results show that too small partitions(<=0.6) or too large partitions(1.0, only train singular matrix) can harm the overall performance.When the inner partition is too small, singular vectors are not well trained and when the inner partition is 1.0, singular vectors are not trained at all, which can cause a great performance drop. These results can also show that the bi-level optimization is efficient and the two levels are both necessary in preventing overfitting and enhancing performances. In the paper, we don’t ever change the partition of the data and keep it 8:2. Tuning the partition may further improve the overall performance.

[1] Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers.

[2] Liu H, Simonyan K, Yang Y. Darts: Differentiable architecture search.

We appreciate your constructive feedback very much. Since we are not able to submit a PDF, we provide all the figures from Figure 2-7 in the Supplementary Section C-F of our paper. All the indexes of figures are referred to those of the updated paper. We sincerely refer you to the figures in our paper. We provide our response to your questions as follows.

Weakness 1: It seems that the empirical model performance and training speed are mostly similar as LoRA.

For performance:

For NLU tasks, our method, BiLoRA, has demonstrated significant performance improvements over baselines on multiple datasets and models. As shown in Table 1 in the paper, using BiLoRA to fine-tune the RoBERTa model with different-sizes on the eight datasets of GLUE surpassed other baselines on almost all datasets. The average performance is significantly higher than other baselines, with 0.7, 0.8, 0.9 margin respectively.. On the DeBERTa-v3 model, AdaLoRA outperformed LoRA by 0.7 on average across the eight datasets. Our method, surpassing LoRA by 1.3, achieved a remarkably superior performance over LoRA and other baselines. To further demonstrate the performance improvement of BiLoRA, we further conduct experiments on GPT-2 on 2 more NLG datasets, DART and WebNLG.

To the best of our knowledge, the results of fine-tuning RoBERTa, DeBERTa and GPT-2 on these NLU and NLG tasks of BiLoRA are the best among all the existing fine-tuning methods. Results on different models and on both NLU and NLG datasets show that our method, BiLoRA outperforms other baselines by a large margin, demonstrating the effectiveness of our method.

For speed:

Firstly, we provide the total training steps needed for convergence for LoRA and BiLoRA in the following tablet. We use the results of RoBERTa-base on MNLI and SST2 datasets.

Secondly, we provide the total training time training steps needed for convergence for AdaLoRA and BiLoRA in the following tablet. We use the results of DeBERTa-v3-base on MNLI and SST2 datasets. The time is measured in min.

Results show that BiLoRA uses 6 times/11 times less steps than LoRA for convergence and that BiLoRA uses 3 times/8 times less steps than AdaLoRA for convergence. Results demonstrate that BiLoRA can converge much faster than LoRA, AdaLoRA and take much less time for training than baselines. BiLoRA is especially time-efficient on larger datasets such as MNLI, which can reduce the total time of LoRA from 3191 min to 1407 min.

Weakness 2: It'd be better to see results on LLMs (>10B) in NLG tasks.

Due to constraints of time and GPU resources, we are not able to conduct experiments on very large-scale LLMs. To demonstrate the effectiveness of BiLoRA when scaling up, we further conduct experiments on DeBERTa-xxlarge(1.5B) on 2 more datasets (SST2, QNLI) to demonstrate the effectiveness of BiLoRA.

Results demonstrate the effectiveness of BiLoRA when scaling up to large language models.

Dear Reviewer,

Thank you for your valuable insights and suggestions. We have tried our best to answer your questions in our author response, and we are also working to revise the paper following your suggestions. Given that the discussion period is ending soon, we were wondering if you could let us know if you have further questions or whether the author response addressed your concerns. We would be delighted to answer any further questions you might have.

Sincerely,

Authors

Question 1:Does BiLoRA also apply iterative pruning of singular values (similar to AdaLoRA)?

In the paper, BiLoRA doesn’t prune singular values. In fact, the pruning process can be applied easily to BiLoRA with only several code lines since singular values are separately in the upper level and can be interacted with directly. We expect BiLoRA to be powerful and effective, yet easy to use. The number of hyperparameters for BiLoRA is kept nearly the same as LoRA, which is much less than AdaLoRA. For such simplicity, we don’t apply pruning which will introduce some extra complexities. After all, BiLoRA is highly compatible with existing low-rank adaptation methods and we are looking forward to combining other LoRA methods with BiLoRA to achieve higher efficiency and better performance.

Question 2: Total training steps and per-update cost for LoRA, BiLoRA and AdaLoRA.

Firstly, we provide the total training steps needed for convergence for LoRA and BiLoRA and the per-update cost for the two methods in the following tablet. We use the results of RoBERTa-base on MNLI and SST2 datasets. The per-update cost is measured in min/k.

Results show that BiLoRA uses 6 times/11 times less steps than LoRA for convergence. The per-step cost for BiLoRA is roughly 2.5 times as LoRA, since BiLoRA needs to iteratively optimize between the two levels and the calculation of outer hypergradients can cost more than simple gradient calculation. Results demonstrate that BiLoRA can converge much faster than LoRA and takes much less time for training than LoRA.

Secondly, we provide the total training steps needed for convergence for AdaLoRA and BiLoRA and the per-update cost for the two methods in the following tablet. We use the results of DeBERTa-v3-base on MNLI and SST2 datasets. According to AdaLoRA paper, it can be seen that each training epoch of AdaLoRA is longer than LoRA, thus intuitively BiLoRA is also faster than AdaLoRA. The per-update cost is measured in min/k and time is measured in min.

Results show that BiLoRA uses 3 times/8 times less steps than AdaLoRA for convergence. The per-step cost for BiLoRA is roughly 1.7/2.0 times as AdaLoRA. Results demonstrate that BiLoRA can converge much faster than LoRA, AdaLoRA and take much less time for training than baselines.

Question 3: Does removing the regularization, such as just sigmoiding over the real values help?

It’s a really good suggestion of just using sigmoid over the real values without regularization. We further conduct experiments on RoBERTa-base on MNLI, SST2 and MRPC datasets.

Results show that using sigmoid over the real values without regularization can achieve equal or better performance than with regularization and achieve better performance than Softmax on MRPC dataset. The design purpose of Binary with regularization is to automatically assign either a high or low importance to each singular vector pair with the corresponding singular value as zero or one. Since singular values can be seen as importance scores of different ranks, the ranks with zero singular values can be pruned. The whole method can serve as an automatic rank selection mechanism. In the future, we will further explore the potentials of this design and improve the performance.

We appreciate your constructive feedback very much. Since we are not able to submit a PDF, we provide all the figures from Figure 2-7 in the Supplementary Section C-F of our paper. All the indexes of figures are referred to those of the updated paper. We sincerely refer you to the figures in our paper. We provide our response to your questions as follows.

Weakness 1: Motivation is not well supported.

The motivation of BiLoRA can be concluded in 4 aspects.

• Limitations of existing fine-tuning methods.
• Inspirations and advantages of bi-level optimization.
• Accelerating the fine-tuning process.
• Opportunities for deeper understanding of PEFT process.

First, existing fine-tuning methods can suffer from overfitting. The large number of parameters in pre-trained models may make the full fine-tuning process more prone to overfitting[1]. Existing PEFT methods do not well solve this problem. The generalization capability of a fine-tuned model on the evaluation dataset can reflect the range of overfitting. We plot the training loss and evaluation loss of LoRA, AdaLoRA and BiLoRA to illustrate the generalization capability. We conduct experiments on RoBERTa-base on CoLA dataset and clip the first 2k steps. Figure 2(a) and Figure 2(b) show that for both LoRA and AdaLoRA, the evaluation loss starts to increase at or before 1k step while the training loss still decreases. The gap between the 2 losses is getting huge. This demonstrates the limitations of existing methods that they still suffer from overfitting issues. Figure 2(c) shows that both inner training loss and outer training loss are close to the evaluation loss. The evaluation loss is still decreasing after 2k steps and is obviously lower than that of LoRA and AdaLoRA. These results convincingly demonstrate the limitations of existing methods and the effectiveness of BiLoRA in addressing overfitting.

Second, we gain inspiration from DARTS (Differentiable Architecture Search) which utilizes a bi-level optimization for searching network architectures. DARTS uses training dataset and validation dataset separately for training operations and their corresponding proportions in 2 levels. DARTS conducts experiments on comparison between bi-level optimization and the normal training method(single-level). The latter performs much worse and they concludes that the latter “would cause α (analogous to hyperparameters) to overfit the training data, leading to poor generalization”[2]. We gain this inspiration of preventing overfitting and innovatively explore the possibilities of applying bi-level optimization to PEFT domain for addressing the existing limitations of overfitting. Extensive experiments demonstrate the effectiveness of BiLoRA.

Third, one important goal of PEFT is to accelerate the fine-tuning process. The bi-level optimization framework effectively accommodates larger learning rates by iteratively optimizing between the two levels without affecting the training stability. Through BiLoRA, we significantly reduce the training steps and training time needed for convergence, thus making PEFT more efficient.

Fourth, we expect BiLoRA to encourage deeper understanding of low-rank fine-tuning processes and more design potentials. We offer more flexible and direct interaction with singular values and singular matrices at separate levels and design different parameterization of singular values. More efficient methods and deeper insights can be inspired by this bi-level optimization framework.

Weakness 2: A more thorough comparison against AdaLoRA.

In the paper, we compare BiLoRA with AdaLoRA on DeBERTa-v3. To have a more thorough comparison with AdaLoRA, we further conduct experiments on RoBERTa-base on 4 NLU tasks and GPT-2 on 2 NLG tasks. The experiment results can be seen as below, higher is better for all the scores. First, we compare AdaLoRA with BiLoRA on 4 NLU datasets.

Second, we compare AdaLoRA on 2 NLG datasets, WebNLG and DART. Results of other methods are from LoRA as a score reference.

Results on different models and on both NLU and NLG datasets show that our method, BiLoRA outperforms LoRA, AdaLoRA and other baselines by a large margin, demonstrating the effectiveness of our method.

[1] Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers.

[2] Liu H, Simonyan K, Yang Y. Darts: Differentiable architecture search.

Thank you for all the constructive feedback. We really appreciate your insightful suggestions.

We highly value your concerns regarding the motivation, and we genuinely provide the following response in the hope of alleviating your concerns.

As far as we are concerned, the reason for utilizing bi-level optimization for low-rank adaptation is mainly from 3 aspects.

• Convincing results and inspirations from DARTS. DARTS conducts experiments on comparison between bi-level optimization and the normal training method(single-level). Results demonstrate that the latter performs much worse and DARTS concludes that the latter “would cause α (analogous to hyperparameters) to overfit the training data, leading to poor generalization”.
• The nature of bi-level optimization. Iteratively optimizing between the 2 levels can intuitively prevent overfitting on either of the 2 subsets.
• The overfitting issue has long been a hindrance to the further improvement of existing fine-tuning methods.

Extensive experiment results demonstrate the better generalization capability of BiLoRA and loss curves convincingly illustrate that BiLoRA largely addresses the overfitting issue.

BiLoRA innovatively introduces bi-level optimization to the Parameter-Efficient-Fine-Tuning domain. It explores and demonstrates the huge potentials of utilizing bi-level optimization for improving the generalization capability of low-rank adapters and speeding up the fine-tuning process. We trust BiLoRA will help enhance the efficiency and performance of many models on many downstream tasks.

Thank you again for engaging with us and your valuable feedback.