Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matrices

We are grateful for your constructive comments. We have revised our paper according to your constructive suggestions.

Strengths: The writing of this paper is good, making the paper easy to follow. The method is reasonable and easy to reproduce, and the experiments demonstrate the efficiency of the method. The challenge with directly fine-tuning a large language model is the memory cost. This method provides a way to update LoRA to fine-tune the backbone. In this case, the model can train the backbone directly without incurring a significant GPU memory cost.

Question 1: Since LoRA is a framework, it would be beneficial if the author could apply Delta-LoRA to more large language models. The paper doesn't clearly explain why Delta-LoRA works. In terms of novelty and significance, I believe it is borderline.

Our Reply:

We have used a variety of language models(e.g. GPT-2 and Bart-Large as well as LLaMA-7B with 7 Billions parameters) to conduct experiments with our Delta-LoRA. Based on the experimental results, we believe that our Delta-LoRA can generalize well on other LLMs.

We would like to clarify the following three strengths of Delta-LoRA.

• Mathmaticlly, $g_W = g_{AB}$. Therefore, it is reasonable to update the pretrained $W$ by $\Delta AB$.
• Delta-LoRA showcases the lowest cosine similarity with the pretrained model. According to our understanding experiment in Section 5.4, we found that Delta-LoRA has the lowest similarity with the pretrained model among all baselines, underscoring that our approach induces the most significant modifications to the final matrix $\boldsymbol{W}^{∗}$.
• Delta-LoRA shows better performance compared with LoRA on LLaMA-7B. According to the evaluation experiment in our paper, we found that Delta-LoRA can yield higher evaluation result than LoRA with Alpaca dataset.

Question 2: Can the author apply Delta-LoRA to llama2 and provide the experimental results?

Our Reply: Thank you very much for your suggestion. We have tested the LLaMA-7B, following the setting in most papers. The results can be found in Section 5.1 in the revised manuscript. We can see that our Delta-LoRA has around 10% improvement compared with LoRA evaluated by LLMs.

We will really appreciate if you can reconsider your score based on our revisions.

We hope our response addresses your concerns; if so, we would really appreciate it if you would reconsider your score accordingly. Please let us know if you have additional questions.

Question 2.2: Why does the author only try the experiments with cases with rank = 8? How about other cases?

Our Reply:

Thanks for your constructive comments. We would like to emphasize that we use the same training setups with LoRA. We use the rank=8 on 8 datasets in GLUE benchmark, while rank=4/8 for NLG tasks. We explore more ranks in the following works. The table in the following is the experiment that we use different ranks on CoLA dataset.

We can see a better choice of rank can further improve Delta-LoRA.

Question 2.3: Why is the learning rate fixed? In my intuition, the only difference between the Delta-LoRA seems to be the resulting learning rate as I explained in the first bullet point. Therefore, it is unclear whether the reported performance of Delta-LoRA happen to outperform LoRA in this learning rate.

Our Reply:

To make fair comparison, we use the same learning rate as reported in the original paper and code(e.g. 2e-4 on E2E Challenge dataset). Besides, we apply linear learning rate scheduler at the training stage, which is same as the setting of LoRA. We could use better hyper-parameters to further improve the results of our Delta-LoRA with standard errors.

Question 2.4: Why are the trainable parameters of Delta-LoRA set to be higher than LoRA in multiple experiments? (Table 1, 4). It is very unclear whether the improvement comes from the additional trainable parameters or extra updatable parameters (or dropouts).

Our Reply:

The Extra Updatable Trainable Parameters means the adjustable pretrained parameters, which we update by using the delta of low rank matrices, without extra cost of GPU resources. For example, since LoRA couldn't update the pretrained $\boldsymbol{W}$, so that its Extra Updatable Trainable Parameters are 0. The Table 6 shows the Delta-LoRA+LoRA module, which means that we add the Dropout layer in Delta-LoRA module. As we reported, only using the Delta-LoRA update strategy indeed prompts the model to have better performance, while removing the Dropout layer can have more performance gain.

Question 3.1: [Medium] The improvements look not significant.

Our Reply:

We have decent improvement on most NLG datasets (e.g. 1.24 on E2E, 0.91 on WebNLG, 10.4% on LLaMA-7B with Alpaca dataset), and some NLU datasets (e.g. 0.65 on CoLA). More importantly, we use the same learning rate and same model architecture with LoRA, we do not search the best hyper-parameter and show the real performance of our Delta-LoRA in our paper. The Delta-LoRA has consistent performance improvement over 4 NLG datasets and 8 NLG datasets, which is a general method and can be applied to multiple domains.

Question 3.2: [Medium] The code is not publically available thus I cannot verify the reproducibility of the experiment results.

Our Reply:

We will release our models and all codes after the decision of this conference.

Question 3.3: [Minor] Listing extra updatable parameters as a column in the tables looks very confusing. For example, Table 4 made me think that Delta-LoRA uses 72M updatable parameters but achieves worse results than Fine-Tuning in the third row.

Our Reply:

We would like to modify our paper to clarify the main ideas and contributions as well. Generally, The Extra Updatable Trainable Parameters claimed in our paper means the adjustable pretrained parameters but does not consume more memory, which we update by using the delta of low rank matrices.

We will really appreciate if you can reconsider your score based on our revisions.

Thank you very much for your valuable and constructive comments.

Question: [Major] Delta-LoRA seems to be fundamentally equivalent to LoRA. I am very unclear about the fundamental difference between updating the pretrained weights using Delta-LoRA and LoRA. Basically, both of them are performing low-rank updates. Updating the pretrained weights does not seem to have any benefits, and if we treat W+AB as a whole, then, what you are doing can be viewed as LoRA but applied with a different learning rate. Please correct me if I am wrong.

Our Reply: We would like to clarify some points and help you understand our Delta-LoRA better. We want to point out that Delta-LoRA is fundamentally different from LoRA in the following three aspects:

• Given $\boldsymbol{W + AB}$, $\boldsymbol{W}$ is fixed in LoRA, while $\boldsymbol{W}$ will be updated in our Delta-LoRA. This is the largest difference between LoRA and Delta-LoRA. This modification can enhance the representative ability of the model.
• $\text{Rank}(\Delta \boldsymbol{W_{Delta-LoRA}}) = \text{Rank}(\boldsymbol{W}^{(T)} - \boldsymbol{W}^{(0)} + \boldsymbol{AB})>\text{Rank}(\Delta \boldsymbol{W}_{LoRA}) = \text{Rank}(\boldsymbol{AB})$. The rank of the learned incremental weight matrix in our Delta-LoRA is larger than that in the original LoRA.
• The gradient flow is different between LoRA and Delta-LoRA. We can know that $\frac{\partial \mathcal{L}}{\partial A} = B^{\top}\cdot\frac{\partial \mathcal{L}}{\partial W}$ and $\frac{\partial \mathcal{L}}{\partial B} = \frac{\partial \mathcal{L}}{\partial W}\cdot A^{\top}$ according to the chain rule.

Here is the back-propagation process of LoRA: $\frac{\partial \mathcal{L}}{\partial A^{(t+1)}}= \frac{\partial \mathcal{L}}{\partial W^{(t+1)}}\cdot (B^{(t+1)})^{\top}=\frac{\partial \mathcal{L}}{\partial W^{(0)}+(A^{(t)}+\Delta A^{(t)})(B^{(t)} + \Delta B^{(t)})}\cdot (B^{(t)}+\Delta (B^{(t)}))^{\top}$ $\frac{\partial \mathcal{L}}{\partial B^{(t+1)}} = (A^{(t+1)})^{\top} \cdot \frac{\partial \mathcal{L}}{\partial W^{(t+1)}}=(A^{(t)}+\Delta (A^{(t)}))^{\top} \cdot \frac{\partial \mathcal{L}}{\partial W^{(0)}+(A^{(t)}+\Delta A^{(t)})(B^{(t)} + \Delta B^{(t)})}$ Here is the back-propagation process of Delta-LoRA: $\frac{\partial \mathcal{L}}{\partial A^{(t+1)}} = \frac{\partial \mathcal{L}}{\partial W^{(t+1)}}\cdot (B^{(t+1)})^{\top}=\frac{\partial \mathcal{L}}{\partial ((W^{(t)}+ \lambda \Delta A^{(t)}B^{(t)}) + (A^{(t)}+\Delta A^{(t)})(B^{(t)}+\Delta B^{(t)})) }\cdot (B^{(t)}+\Delta (B^{(t)}))^{\top}$ $\frac{\partial \mathcal{L}}{\partial B^{(t+1)}} = (A^{(t+1)})^{\top} \cdot \frac{\partial \mathcal{L}}{\partial W^{(t+1)}} = (A^{(t)}+\Delta (A^{(t)}))^{\top} \cdot \frac{\partial \mathcal{L}}{\partial ((W^{(t)}+ \lambda\Delta A^{(t)}B^{(t)}) + (A^{(t)} + \Delta A^{(t)})(B^{(t)} + \Delta B^{(t)}) )}$.

Question 2.1: Neither DyLoRA nor AdaLoRA outperforms LoRA as they claimed in their paper. I noticed that the authors chose different settings with the AdaLoRA, this might be the reason for the bad performance. However, the authors do not provide any explanation on why they chose this different setting. This makes the outperformance of Delta-LoRA reported in the tables less convincing.

Our Reply:

For a fair comparison, we use the same training setups as LoRA. The AdaLoRA and DyLoRA chose different training settings as LoRA. In the following, we want to clarify the differences among them.

First, we want to point out that

1. DyLoRA aims to improve the performance of lower rank when inference rather than the rank they selected at the training stage. That means the performance they claimed in their paper was based on a higher rank, which can yield a much more memory overhead. For a fair comparison, we report their results based on the low-rank setting.

Second, AdaLoRA uses a different setting from LoRA, the differences are summarized as:

1. The AdaLoRA fine-tuned each layer of the pretrained matrices in Transformer. This operation will cause extra activation memory consumption (i.e. 19.79 GB extra GPU memory for LLaMA-7B with batch size of 1024 and 1024 max length implemented by Pytorch). Therefore, we do not choose the similar training setups with AdaLoRA. Instead, we used the same training setups with LoRA.
2. AdaLoRA has 7 extra hyper parameters (i.e. target_rank, reg_orth_coef, init_warmup, final_warmup, mask_interval, beta1, beta2). We cannot search the best hyper-parameters for the different datasets and models.

Thanks for your detailed reviews and valuable advice for our submitted paper.

About the Strengths:

• The paper is well-organized and easy to follow.

• Extensive evaluation on NLU, NLG tasks as well as ablation studies to emphasize the efficacy of their formulation.

• The proposed algorithm is well-motivated, it builds very naturally upon the baseline LORA and differentiates itself against the baseline (Section 5.3) to answer a natural question - "Does the performance stem from the additional parameters or the updates applied to the underlying weight matrix?"

• It accomplishes its goals without sacrificing the key advantages of LORA - low memory consumption.

Our Reply:

Thanks for your recognition, we are encouraged by your comments. We have made some corresponding revisions according to your comments and concerns.

Question 1: The only major drawback seems to be that the differences in empirical results do not seem statistically significant (and do not have any errors provided). Given the incremental increase, it is not clear whether the performance of the algorithm stems from real gains or is noise from insufficient tuning.

Our Reply: We would like to emphasize the effectiveness of the proposed Delta-LoRA from the following points:

• We kept the similar training settings as LoRA, such as learning rate, weight decay, rank number, training epochs and random seed. We do not use a parameter search and select the random seed (we kept all experiments with random seed=0). Therefore, the improvement that claimed in our paper is convincing.

• We achieved significant improvement compared with AdaLoRA and DyLoRA. We used the same training setting to evaluate the two methods. We observed that our method could have pronounced performance gain.

• We witnessed consistent improvement across multiple datasets. According to our experiment, we achieved improvement across 8 NLU datasets and 4 NLG datasets, which proves that our method works well across different tasks.

Question 2: The whole idea is pretty much a heuristic which works very well in practice (but the same can be said of LoRA)

Our Reply: Thanks for your comment. Our initial motivation was indeed heuristic, while we gave a mathematical formulation and presented our analysis of its rationality. We supplied a more theoretical proof to further demonstrate the difference between Delta-LoRA and LoRA in Sec. A.4 of the revised version.

Question 3: The overt emphasis on the fact that seems unnecessary (specifically Section 4.1) since the results stem from fairly straightforward linear algebra and calculus. It might be better to reduce that section, and allocate space to more impactful discussions/experiments. It is certainly true that the work's novelty is primarily in engineering a practical method, rather than introducing a new direction of research. The advantage of method is clearly empirical in nature.

Our Reply:

Thanks for your kind advice. We appropriately reduce this statement in our revised paper. We modified some content in our revised paper, and marked these descriptions with blue color.

Question 4: Can you provide standard errors on some of your empirical results?

Our Reply:

We provide the standard errors on GLUE benchmark in the following table. We used better learning rate and start steps here which can be found in the Section A.5 of our revised paper.

A better choice of learning rate and start steps can further improve the performance of Delta-LoRA.

Question 5: It would help to incorporate some experiments in Section 7 of the LoRA paper to help understand the kind of updates induced by Delta-LoRA.

Thanks for your reminder. We moved these experiment tables to a more conspicuous place. We provided the ablation study and parameter sensitivity in Section 5.3 and Appendix A.5 with 4 experiments in the revised paper.

We will really appreciate if you can reconsider your score based on our revisions.

Thank you very much for recognizing the novelty of this paper. We have revised our paper according to your comments.

Strengths: The paper is clear and easy to read with straightforward derivations. The proposed method, Delta-LoRA, is original in its approach to update both the low-rank matrices and the pre-trained weights, which could potentially address the limitations of existing low-rank adaptation methods.

Our Reply: Delta-LoRA introduces a simple but efficient strategy to finetune the low-rank matrices and the pre-trained weights. Delta-LoRA improves the original LoRA from two perspectives: performance and memory consumption.

Question 1.1: The main weakness of the paper is the lack of experiments on large models where the memory efficiency of Delta-LoRA would be most beneficial. The models tested in the paper are relatively small, which makes the memory-saving advantage less meaningful.

Our Reply: Thanks a lot for your suggestions! We have evaluated the performance of LLaMA-7B model on Alpaca dataset, in the revised manuscript, we have clarified the experimental results. We find that, using evaluation of GPT-4, Delta-LoRA improves LoRA by around 10%, which proves the effectiveness of our Delta-LoRA.

Question 1.2: Additionally, there is a performance loss compared to fine-tuning, which is expected.

Our Reply: For this question, we would like to point out there exist two exceptions:

1. The gap between the finetuned data and the pretrained data is large. If the training data in downstream task is much different from the pretrained dataset, the full-finetuned can defeat LoRA and other PEFT methods. However, if training data is similar to the pretrained dataset, the fully finetuned method can overfitted fast and suffer from catastrophic forgetting.
2. The data scale of the finetuned data is small. If the data scale of the finetuned data is small, full finetuning mode will overfit the finetuned data quickly. In this way, Delta-LoRA and LoRA will show better results. This is consistent with our results.

Question 1.3 and Question 2.2:

• The paper should have reported the performance of full finetuning with stateless optimizers (such as SGD) and showed that Delta-LoRA outperforms this baseline.
• How does Delta-LoRA's performance compare to full finetuning with stateless optimizers? (Note -- the memory footprint is actually slightly lower with full finetuning with stateless optimizers)

Our Reply:

The AdamW is widely used in the training of transformers, but SGD is not so popular. In NLP and multimodal tasks, most people choose AdamW instead of SGD, since it has a stable performance and converges fast. As the reasons described before, we chose the AdamW optimizer to conduct experiments to make fair comparison. We run the experiments with SGD optimizer on CoLA dataset with rank=8, $\alpha$=16, learning rate=0.01 and momentum=0.9. LoRA achieves 63.67 while Delta-LoRA has 64.32. Besides, we observed that the training process occurred crash after 57 epochs for the LoRA's training, which emphasizes the instablity of the SGD optimizer for current NLP tasks.

Question 2.1: Could you provide more evidence to support the claim that Delta-LoRA can nearly match the performance of full finetuning with "large" models?

Our Reply: We have tested the effectiveness of Delta-LoRA with LLaMA-7B on Alpaca dataset. We found that the memory cost of Delta-LoRA is basically same as the LoRA, while our method outperforms the LoRA for around 10% under the evaluation of LLMs.

We will really appreciate if you can reconsider your score based on our revisions.

We hope our response addresses your concerns; if so, we would really appreciate it if you would reconsider your score accordingly. Please let us know if you have additional questions.