BA-LoRA: Bias-Alleviating Low-Rank Adaptation to Mitigate Catastrophic Inheritance in Large Language Models

Q1: However, the paper has some limitations in terms of theoretical analysis, experimental scope, and comparative analysis.

A1: The main task of this paper is to alleviate the catastrophic inheritance problem, and we leave the theoretical study for future work. In addition, we expand the scope of the experiment, adding more evaluation baselines and more comprehensive ablation experiments. We also conduct a more detailed analysis of the computational overhead, including the training time and computational cost during the training process.

Q2: Theoretical analysis is lacking, with no clear basis for the selection of regularization weights and limited discussion on computational complexity and convergence.

A2: In the revised version, we have provided a detailed explanation of this choice. The goal of selecting regularization weights is to balance the model's generalization ability with the regularization effect during training. We conducted hyperparameter optimization on the validation set to ensure that the selected regularization weights effectively improve model performance while avoiding excessive regularization. Additionally, we thoroughly discuss the impact of regularization weights on computational complexity and convergence, and we validate their appropriateness through our experiments.

Q3: The experimental scope is restricted, as the study only tests on English datasets, leaving the method's performance in multilingual contexts unexplored( and also other tasks as well ). Additionally, the analysis of computational overhead is minimal.

A3: The core goal of our research is to alleviate the catastrophic inheritance problem in pre-trained models, which mainly stems from the selection of pre-training data. Currently, pre-training datasets are mainly English data, so our experiments focus on the English dataset for Supervised Fine-Tuning (SFT). We plan to explore the research of multilingual datasets in future work.

In the analysis of computational overhead, we supplemented the detailed comparison between BA-LoRA and standard LoRA, covering training time and computational load.

Q4: How do you determine the optimal values for the regularization weights (λ1, λ2, λ3)?

A4: The main goal of choosing appropriate regularization hyperparameters is to balance the generalization ability of the model and the regularization effect during training. We mainly followed the following steps when choosing regularization weights:

1. Split validation set: Split a part of the training data as a validation set to ensure that it does not participate in the training process and is specifically used to evaluate the performance of the model.

2. Hyperparameter adjustment: Within a reasonable range, we adjust the regularization weights by using hyperparameter optimization algorithms (such as random search) and evaluate the performance of the model on the validation set under different settings. The reasonable range is usually set based on previous experience or pre-experimental results.

3. Select the best parameters: Based on the performance indicators on the validation set, the regularization weight combination that can best improve the generalization ability of the model is selected.

Q4: The computational costs seem to undermine the main advantage of PEFT. For example, consistency regularization adds an extra forward pass and doubles activation storage. While the diversity component is manageable, the SVD aspect raises concerns, as computing eigenvalues for large matrices is time-consuming and CPU-bound, limiting parallelization. For instance, the smallest LLaMA matrix (4096x4096) takes around 5 seconds to compute, raising doubts about the approach's scalability.

A4: To address this issue, we calculated the average training time and average memory usage of the first few trainings. On the A40 graphics card, we fine-tuned the DeBERTa-v3-base model on the full GLUE dataset. The specific information of the three trainings is as follows:

• LoRA: averaged about 14.45 hours, and the average memory usage was about 6.3G.
• PiSSA: averaged about 12.07 hours, and the average memory usage was about 6.5G.
• BA-LoRA: averaged about 15.82 hours, and the average memory usage was about 7.1G.

It can be seen that the average training time and memory usage of BA-LoRA are slightly higher than those of LoRA and PiSSA. However, the regularization mechanism of BA-LoRA effectively alleviates the catastrophic inheritance problem and brings performance improvement, so the additional computational overhead is controllable and acceptable.

Q5: I doubt whether bias and catastrophic inheritance were truly assessed. The paper suggests using models "probably" trained on noisy data, like BERT and GPT-2, which are outdated and rarely used in current applications. Improvements in final scores don't necessarily prove effective bias handling.

A5: In response to this question, our specific response is as follows:

Our experiments aim to demonstrate the effectiveness of our approach in mitigating the catastrophic inheritance of bias and imbalanced data by pre-trained models. BA-LoRA is model-independent and is able to optimize different pre-trained models, we have demonstrated that BA-LoRA can improve the performance of different LLMs across different tasks in our paper. We deliberately selected models pre-trained on relatively noisy and imbalanced datasets, such as BERT-L (pre-trained on BooksCorpus and English Wikipedia) and GPT-2-XL (pre-trained on Common Crawl's WebText dataset) to verify the effectiveness of BA-LoRA in dealing with data noise and imbalance.

1. In 4.4.1, we show the performance of LLaMA-2-7B, Mistral-7B, and Gemma-7B models on multiple tasks after fine-tuning with BA-LoRA. These tasks include mathematical reasoning (GSM8K, MATH), code generation (HumanEval, MBPP), and multi-turn dialogue tasks (MT-Bench), verifying its effectiveness on various NLG tasks. In addition, we also tested the performance of RoBERTa-large and DeBERTa-v3-base models on the GLUE benchmark after fine-tuning with BA-LoRA. On average, BA-LoRA outperforms PiSSA and LoRA by 0.59% and 1.61% on RoBERTa-large, and outperforms PiSSA and LoRA by 0.44% and 1.35% on DeBERTa-v3-base. These results highlight the effectiveness of BA-LoRA in enhancing the performance of NLU models.

2. In the **Appendix "Analysis of Models of Different Sizes and Types", we compare the fine-tuning performance of LoRA, PiSSA, and BA-LoRA on ten different models, including LLaMA-2-7B/13B, LLaMA-3-8B/70B, Mistral-7B-v0.1, Gemma-7B, Qwen1.5-7B, Yi-1.5-34B, DeepSeek-MoE-16B, and Mixtral-8x7B-v0.1. **The experimental results prove that BA-LoRA is effective for pre-trained models of various sizes and types. **

3. In addition, in the **Appendix "Evaluating Performance at Different Layers", we compare the fine-tuning performance of LLaMA-2-7B and Mistral-7B-v0.1 models at different layer settings (0-128). The experimental results show that BA-LoRA can improve the performance of the model under different rank settings.

These experimental analyses prove the effectiveness of BA-LoRA in alleviating the catastrophic inheritance of bias and imbalanced data brought by pre-trained models.

Q6: The claim of imbalance in MNLI is unclear, as the official dataset on Hugging Face has an equal class distribution. This assertion requires clarification.

A6: Our method mainly targets the noise and imbalance problems in pre-training data rather than fine-tuning data. During the fine-tuning stage, BA-LoRA has no special requirements for fine-tuning data.

Q7: There seems to be no clear difference between spectral regularization for NLU and NLG tasks. What is the rationale for distinguishing and targeting them separately?

A7: In the design of regularization terms, we have tried to design different SVD regularization terms for NLU and NLG tasks, such as the design for NLG tasks:

$\mathcal{L} _ {\mathrm{SVDR-NLG}} = - \left( \frac{\sum_{i=1}^{k} \sigma_{i}}{\sum_{j=1}^{D} \sigma_{j}} + \alpha \cdot \text{Var}(\sigma) \right)$

where $\text{Var}(\sigma)$ represents the variance of the top $k$ singular values. The hyperparameter $\alpha$ adjusts the influence of the variance term, encouraging a more uniform distribution of singular values, which in turn promotes diversity and enhances the quality of the generated text.

However, this improvement does not lead to significant performance improvements and introduces additional computational overhead compared to the SVD regularization terms we currently use. Therefore, in the current study, we decided to avoid using more complex regularization settings to avoid adding unnecessary computational burden.

Once again, thank you for your valuable suggestions. Your feedback has been very helpful in refining the content of the paper. Should you have any further questions or suggestions, please do not hesitate to reach out.

[1] Shi J X, Wei T, Zhou Z, et al. Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts[C]//Forty-first International Conference on Machine Learning. 2024.

[2] Jiang H, He P, Chen W, et al. Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization[J]. arXiv preprint arXiv:1911.03437, 2019.

[3] Zhai Y, Tong S, Li X, et al. Investigating the catastrophic forgetting in multimodal large language model fine-tuning[C]//Conference on Parsimony and Learning. PMLR, 2024: 202-227.

[4] Zhai Y, Tong S, Li X, et al. Investigating the catastrophic forgetting in multimodal large language model fine-tuning[C]//Conference on Parsimony and Learning. PMLR, 2024: 202-227.

[5] Mao Y, Ge Y, Fan Y, et al. A survey on lora of large language models[J]. arXiv preprint arXiv:2407.11046, 2024.

[6] Fu Z, Yang H, So A M C, et al. On the effectiveness of parameter-efficient fine-tuning[C]//Proceedings of the AAAI conference on artificial intelligence. 2023, 37(11): 12799-12807.

Q1: LoRA and PiSSA lack regularization, with dropout set to 0 and no weight decay applied, making them prone to collapse due to hyperparameter choices. It's unclear whether the gains from your method could be achieved by simply adding weight decay and optimizing dropout.

A1: Regarding the issue that the LoRA and PiSSA baseline models do not use dropout and weight decay, we provide the following explanation:

To ensure the consistency and fairness of the experimental results, we strictly followed the experimental settings in the original paper, so these regularization methods were not applied in the baseline model.

Second, our main research goal is to explore an efficient parameter fine-tuning method to alleviate the "catastrophic inheritance" phenomenon caused by data noise and imbalance in pre-trained models, thereby improving the performance and stability of the model. Therefore, whether to use dropout and weight decay is not the focus of this study.

Q2: The lack of regularization is evident as full fine-tuning performs worse than LoRA, which shouldn't be a consistent trend, as tuning the entire network would otherwise be ineffective.

A2: We further analyze why Full FT performs worse than LoRA in this study. Previous studies have shown that when a large number of parameters are updated during fine-tuning, especially when the model faces a large and diverse training dataset, overfitting is prone to occur [1][2]. In addition, large-scale parameter updates may lead to catastrophic forgetting [3][4]. In contrast, the PEFT method helps control the complexity of the model by fine-tuning only some parameters, thereby reducing the risk of catastrophic forgetting and improving generalization ability [5][6].

In addition, recent studies [7][8][9][10] show that when the rank value and other key hyperparameters in low-rank fine-tuning (such as scaling factor $\alpha$, learning rate lr, etc.) are set properly, the PEFT method can sometimes outperform the Full FT method.

Q3: Regularization is a well-studied topic, and while the paper cites entropy, KL, and spectral regularization, it implies novelty where there is none. If these regularizers are not new, they should be included as baselines, framing the study as an adaptation and combination for LLMs. This suggests limited novelty beyond incremental adaptations for LLMs.

A3: While regularization methods have been extensively studied, the main contribution of our paper is to mitigate catastrophic inheritance in pre-trained models rather than studying regularizers.

1. Alleviate catastrophic inheritance: BA-LoRA is the first study to effectively alleviate catastrophic inheritance problems caused by noise and data imbalance in pre-trained models through fine-tuning methods. Regularizer is just a tool to achieve our main objective, not the focus of our study

2. Customized regularization strategy: BA-LoRA designs three regularization terms for the different characteristics of NLU and NLG tasks to better adapt to the specific needs of these tasks. This regularization method for task characteristics has not been deeply explored in the existing literature, thus filling the gap of task-specific regularization strategies.

3. Comprehensive Empirical Assessment: Extensive experimental verification proves that BA-LoRA outperforms various baselines on all NLU and NLG tasks. Furthermore, analytical experiments demonstrate that our method effectively attenuates the noise left over from pre-training, resulting in a more robust and general model. These empirical results provide strong support for the effectiveness and novelty of our approach.

Further reply to A6: First, our definition draws on the concepts presented in [2]:  

• Noise: refers to the presence of mislabeled, duplicated, contaminated, or low-quality data in the pre-training data, which may interfere with the learning process of the model and lead to performance degradation.  

• Imbalance: refers to the unbalanced distribution of categories or features in the pre-training data, where the number of samples for some categories or concepts is much larger than others, which may lead to bias of the model towards frequently occurring concepts and affect the generalization ability.

In Chapter 2 of the [2] paper, they study in detail the detrimental effects of these two types of data on model performance, pointing out that noisy and unbalanced data can lead to the “catastrophic inheritance” of the model in downstream tasks. In addition, many related papers [3], [4], [5], [6], [7], [8] are cited to demonstrate that bias and imbalance are indeed prevalent in pre-training data. We have cited [2] in our paper and given [2] credits.  

Second, it should be emphasized that [2] mainly argues the existence and harmfulness of catastrophic inheritance, but does not provide a concrete solution. In response to a similar problem, [1] discusses two solution paths:  

1. Constructing new high-quality datasets: by carefully curating and cleaning the data to minimize noise and imbalance in the pre-training data.  

2. Designing a new network architecture: by improving the model architecture to increase the robustness of the model to noisy and unbalanced data.   Finally, in our work, we propose for the first time a third solution to mitigate catastrophic heritability using the Supervised Fine-Tuning (SFT) method. Specifically, we introduce regularization strategies customized for NLU and NLG tasks in the fine-tuning process, enabling the model to effectively mitigate the negative effects of noise and imbalance in the pre-trained data.  

The specific implementation of BA-LoRA has been described in detail in the Methods section of the paper and the effectiveness of our approach has been demonstrated in the Experiments section. We believe that these additions and clarifications help readers better understand the motivation of our work and the innovativeness of our approach.  

Thank you again for your valuable comments and suggestions, and we look forward to your further feedback.  

References

[1] Liu Z, He K. A Decade's Battle on Dataset Bias: Are We There Yet? arXiv preprint arXiv:2403.08632, 2024.  

[2] Chen H, Raj B, Xie X, et al. On catastrophic inheritance of large foundation models[J]. arXiv preprint arXiv:2402.01909, 2024.

[3] Elazar Y, Bhagia A, Magnusson I, et al. What's In My Big Data?[J]. arXiv preprint arXiv:2310.20707, 2023.  

[4] Carlini N, Ippolito D, Jagielski M, et al. Quantifying memorization across neural language models[J]. arXiv preprint arXiv:2202.07646, 2022.  

[5]Hernandez D, Brown T, Conerly T, et al. Scaling laws and interpretability of learning from repeated data[J]. arXiv preprint arXiv:2205.10487, 2022.

[6] Xu H, Xie S, Tan X E, et al. Demystifying clip data[J]. arXiv preprint arXiv:2309.16671, 2023.  

[7] Zhu B, Tang K, Sun Q, et al. Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models[J]. Advances in Neural Information Processing Systems, 2024, 36.  

[8] Parashar S, Lin Z, Liu T, et al. The Neglected Tails in Vision-Language Models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 12988-12997.

Further reply to A4:   Thank you for your interest. First, our work is based on the publicly released PiSSA code implementation. In implementing the third regularization term, Singular Value Decomposition Regularization (SVDR), we paid special attention to computational efficiency.  

Second, to reduce the computational overhead, we use partial SVD to compute only the first $k$ largest singular values of the output matrix. This further reduces the computation time and ensures the consistency of the regularization objective.  

Third, instead of using a custom CUDA kernel, we rely on existing efficient numerical computation libraries. Specifically, we used the torch.linalg.svdvals() function provided by PyTorch to compute singular values.  

Finally, here are the code snippets for our implementation of SVDR:  

# Compute the singular values of the outputs
s = torch.linalg.svdvals(outputs)
# Sum of all singular values
s_total = torch.sum(s)
# Sum of the top k singular values
s_top_k = torch.sum(s[:k])
# Compute the SVDR loss
svd_loss = - (s_top_k / s_total)
# Add the SVDR loss to the total loss
loss += svd_loss_weight * svd_loss

As shown above, we compute only the singular values $S$ and focus on the first $k$ largest singular values. This is fully consistent with the approach described in our paper while avoiding the computational overhead of a full SVD.  

Our code will be publicly released soon, so you can see more details about our implementation.  

Further reply to A5&A1:   Our main goal in this paper is to mitigate the negative impact of noise and imbalance in the pre-training data on downstream tasks, i.e., the phenomenon of “catastrophic inheritance ”. To verify the effectiveness of BA-LoRA in this regard, we show in detail the effectiveness of BA-LoRA in mitigating noise and data imbalance in subsections 4.4.2 and 4.4.3 of the experimental section. In the Noise Mitigation Analysis experimental section, we selected BERT-L and GPT-2-XL models, which were pre-trained on BooksCorpus and Wikipedia (BERT-L) and WebText datasets (GPT-2-XL), respectively, where noise and imbalance are common. On the GLUE and GLUE-x benchmarks, BA-LoRA improves the performance of LoRA by 2.03% and 2.47% on the BERT-L and GPT-2-XL models, respectively. These results demonstrate that BA-LoRA effectively mitigates the noise in the pre-training data through its regularization term and enhances the robustness and generalization ability of the model.  

In the Imbalanced Data Mitigation Analysis experiment section, we fine-tuned the BERT-L and GPT-2-XL models on the MNLI dataset and visualized the hidden features of the last layer using the t-SNE technique. The results show that BA-LoRA exhibits clearer class separation when dealing with data imbalance, especially the BERT-L model, which exhibits higher intra-class aggregation and clearer class boundaries. These analyses indicate that BA-LoRA can effectively mitigate the impact of unbalanced data in pre-training. Therefore, the performance improvements in Tables 1 and 2 can be attributed to the advantages of BA-LoRA in mitigating noise and imbalance problems.  

Through these two exhaustive experiments, we have fully demonstrated that BA-LoRA is able to mitigate the detrimental effects of bias and imbalance in pre-training data, effectively mitigating the catastrophic inheritance phenomenon, and thus improving the performance of the model in downstream tasks.

Thank you for your careful review of our work and valuable suggestions.

Further reply to A1: Regarding the dropout setting in the LoRA you mentioned, as our work is built upon PiSSA, we followed the PiSSA settings in our experiments, in which the dropout was set to 0 both when reproducing LoRA and when conducting our own experiments.  

The experiment in Table 11 in the original LoRA paper that you mentioned is based on the GPT-2 model with dropout = 0.1. Since we are using current mainstream LLMs such as LLaMA-2-7/13B, Gemma-7B, and Mistral-7B-v0.1, it is common practice to use different hyperparameters for different LLMs.  

Regarding whether it is reasonable to set dropout=0 in LoRA and PiSSA and how LoRA can affect the effectiveness of LoRA and its variants, these are not the focus of our study and are beyond the scope of this paper. Currently, we are not aware of any relevant studies addressing the impact of dropout in LoRA. It is not reasonable to question our contribution with evidence that does not exist. If there are new findings in the future, we will explore them in depth in our subsequent work.

Further reply to A2: We place great emphasis on the importance of appropriately tuning hyperparameters to ensure that all methods are fairly comparable. First, in the revised paper, we provide a detailed hyperparameter analysis in Section 4.4.5, focusing on the impact of key hyperparameters in our methods.  

Second, we recognize that it is difficult to apply an equally applicable set of hyperparameters for all methods. Different methods may contain specific hyperparameters that are not shared. For example, our method introduces specific hyperparameters λ₁, λ₂, and λ₃ that are not included in baseline methods such as LoRA. Similarly, baseline methods have their unique parameter settings.  

Third, regarding the learning rate and learning rate scheduling strategy, to ensure fairness in the comparison, we followed the parameter settings provided in PiSSA, which provide a uniform reference standard across experiments.  

Fourth, while we understand that fully optimizing the hyperparameters for all baseline methods may help to further improve the rigor of the comparisons, there are some challenges to doing this in the current study for the following reasons:  

1. Study Focus: the primary focus of our study was to validate the effectiveness of the proposed BA-LoRA methods rather than to comprehensively analyze the impact of hyperparameters on each baseline method.  
2. Resource constraints: some of the baseline methods are not provided with complete code implementations, limiting our ability to perform in-depth hyperparameter tuning on them.  
3. Scope of the paper: exploring whether hyperparameter optimization can mitigate bias in all baseline methods would significantly expand the scope of the paper. We believe that while this direction is important, it is among the areas that can be explored in depth by subsequent research.   Finally, systematically optimizing hyperparameters for all existing methods is a significant challenge in terms of human capability, computational resources, and time cost, and it may be beyond the scope of a single research paper to carry.

Further reply to A3:   First, we would like to clarify that our research focuses on mitigating the catastrophic inheritance problems associated with pre-trained data, not on the regularization method itself. The regularization strategy is just a tool for us to achieve this goal.  

Second, as we mentioned in our paper, [1] pointed out that there are two main approaches for dealing with the bias in pre-trained data: 1) constructing high-quality datasets, and 2) investigating new network architectures. However, we are the first work to investigate the use of fine-tuning algorithms to mitigate catastrophic inheritance. Specifically, we propose an improved fine-tuning algorithm based on low-rank adaptivity that aims to mitigate catastrophic inheritance from biased and unbalanced data in pre-training data.  

Finally, lines 77 to 86 of the manuscript introduction introduce our three regularization strategies aimed at solving the noise and data imbalance problems in disaster inheritance. The extensive experimental results in Sections 4.4.2 and 4.4.3 confirm the effectiveness of BA-LoRA in addressing these issues, respectively.

In response to this question, we provide the following clarification:  

1.Background and Motivation: Pre-training data usually comes from large-scale data on the Internet, which is inevitably noisy and imbalanced. For example, advanced pre-training models such as GPT-4 and LLaMA also use pre-training data from a large amount of unlabeled Internet text, which is often not thoroughly cleaned and corrected. Therefore, in the subsequent fine-tuning stage, it is difficult for the model to effectively remove irrelevant or inaccurate information, which in turn affects the performance of the model in downstream tasks.  

We have mentioned in the introduction of the paper (lines 41 to 50) that large-scale data brings problems such as redundancy, bias, and corruption. These problems will have a negative impact on the model during training, especially when the data is unbalanced or noisy, which may lead to a decline in model performance and weakened generalization ability. In addition, we have also supplemented the specific content of noise and imbalance problems in pre-training data in the appendix of the paper, including low-quality bias, redundancy, pollution, category imbalance, and harmful content bias.

To mitigate the detrimental effects of Catastrophic Inheritance, particularly noise, and imbalance, we propose Bias-Alleviating Low-Rank Adaptation (BA-LoRA). Our approach incorporates three distinct regularization terms: a consistency regularizer, a diversity regularizer, and an SVD regularizer. The consistency regularizer preserves valuable pre-trained knowledge during fine-tuning, while the diversity regularizer encourages varied model outputs. The SVD regularizer enhances the generalization capabilities of generative models. Recognizing the fundamental differences between NLU and NLG, such as determinism in NLU versus diversity in NLG, we tailor our regularization strategies accordingly.

2.Experimental design: First, in order to demonstrate that our method is effective in mitigating the catastrophic inheritance of noise in the pre-training data, we purposely chose the BERT.-L and GPT2-XL models, which are known to be pre-trained on noise-filled pre-training data. Among them, BERT-L is a pre-trained model on BooksCorpus and English Wikipedia, and GPT-2-XL is a pre-trained model on the noisy WebText dataset of Common Crawl.

Second, in order to demonstrate that our method can effectively mitigate the catastrophic inheritance of imbalance in the pre-training data, we purposely chose the BERT.-L and GPT2-XL models, which are known to be pre-trained in long-tailed pre-training data. These two models were then fine-tuned separately on the downstream MNLI dataset. We visualized the features of the final layer of output from the fine-tuned models using t-SNE to assess the performance of the models when dealing with low-frequency or complex samples.

3.Experimental results: To verify the effectiveness of BA-LoRA in this regard, we show in detail the effectiveness of BA-LoRA in mitigating noise and data imbalance in subsections 4.4.2 and 4.4.3 of the experimental section.  

In the 4.4.2 Analysis on mitigate noisy data experimental section, we selected BERT-L and GPT-2-XL models, which were pre-trained on BooksCorpus and Wikipedia (BERT-L) and WebText datasets (GPT-2-XL), respectively, where noise and imbalance are common. On the GLUE and GLUE-x benchmarks, BA-LoRA improves the performance of LoRA by 2.03% and 2.47% on BERT-L and GPT-2-XL models, respectively. These results demonstrate that BA-LoRA effectively mitigates the noise in the pre-training data through its regularization term and enhances the robustness and generalization ability of the model.  

In the 4.4.3 Analysis on Mitigating Imbalanced Data experiment section, we fine-tuned the BERT-L and GPT-2-XL models on the MNLI dataset and visualized the hidden features of the last layer using the t-SNE technique. The results show that BA-LoRA exhibits clearer class separation when dealing with data imbalance, especially the BERT-L model, which exhibits higher intra-class aggregation and clearer class boundaries. These analyses indicate that BA-LoRA can effectively mitigate the impact of unbalanced data in pre-training. Therefore, the performance improvements in Tables 1 and 2 can be attributed to the advantages of BA-LoRA in mitigating noise and imbalance problems.  

With these two exhaustive experiments, we have once again amply demonstrated that BA-LoRA is able to mitigate the detrimental effects of biases and imbalances in the pre-training data, and effectively mitigate catastrophic inheritance phenomena, thereby improving the performance of the model in downstream tasks.

Q4: In my opinion, these regularization terms should be universally applicable, enhancing LoRA's variants, and more experiments are needed to confirm this.

A4: We have added further experimental evaluations in the revised version. The following table shows the performance comparison of LoRA, DoRA, PiSSA, and BA-LoRA with and without regularization terms, where "Reg" refers to the three regularization terms designed for each NLG task:

As shown in the table above, incorporating regularization terms into either LoRA or DoRA significantly enhances the performance of LoRA variants across all tasks. This demonstrates that regularization is highly applicable to different LoRA variants. Furthermore, BA-LoRA, which combines PiSSA with regularization, achieves the best performance across various tasks and markedly improves the model's generalization ability.

Q5: In section 3, why is PISSA described in such detail, is it highly relevant to BA-LoRA?

A5: Although PiSSA and BA-LoRA each have different focuses, PiSSA is a core component of our approach. When PiSSA is used in combination with regularization terms, the performance and training efficiency of the model can be significantly improved. Given the strong connection between PiSSA and BA-LoRA, we believe it is necessary to elucidate how PiSSA works and its interaction with BA-LoRA. In the revised version, we have simplified some details to avoid redundancy while ensuring that key content is highlighted.

Once again, thank you for your valuable suggestions. Your feedback has been very helpful in refining the content of the paper. Should you have any further questions or suggestions, please do not hesitate to reach out.

References

[1] Liu S Y, Wang C Y, Yin H, et al. Dora: Weight-decomposed low-rank adaptation[J]. arXiv preprint arXiv:2402.09353, 2024.

[2] Hyeon-Woo N, Ye-Bin M, Oh T H. Fedpara: Low-rank hadamard product for communication-efficient federated learning[J]. arXiv preprint arXiv:2108.06098, 2021.

[3] Valipour M, Rezagholizadeh M, Kobyzev I, et al. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation[J]. arXiv preprint arXiv:2210.07558, 2022.

[4] Zaken E B, Ravfogel S, Goldberg Y. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models[J]. arXiv preprint arXiv:2106.10199, 2021.

[5] Houlsby N, Giurgiu A, Jastrzebski S, et al. Parameter-efficient transfer learning for NLP[C]//International conference on machine learning. PMLR, 2019: 2790-2799.

[6] Pfeiffer J, Kamath A, Rücklé A, et al. Adapterfusion: Non-destructive task composition for transfer learning[J]. arXiv preprint arXiv:2005.00247, 2020.

[7] Zhang Q, Chen M, Bukharin A, et al. AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning[J]. arXiv preprint arXiv:2303.10512, 2023.

[8] Shi J X, Wei T, Zhou Z, et al. Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurts[C]//Forty-first International Conference on Machine Learning. 2024.

[9] Jiang H, He P, Chen W, et al. Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization[J]. arXiv preprint arXiv:1911.03437, 2019.

[10] Zhai Y, Tong S, Li X, et al. Investigating the catastrophic forgetting in multimodal large language model fine-tuning[C]//Conference on Parsimony and Learning. PMLR, 2024: 202-227.

[11] Zhai Y, Tong S, Li X, et al. Investigating the catastrophic forgetting in multimodal large language model fine-tuning[C]//Conference on Parsimony and Learning. PMLR, 2024: 202-227.

[12] Mao Y, Ge Y, Fan Y, et al. A survey on lora of large language models[J]. arXiv preprint arXiv:2407.11046, 2024.

[13] Fu Z, Yang H, So A M C, et al. On the effectiveness of parameter-efficient fine-tuning[C]//Proceedings of the AAAI conference on artificial intelligence. 2023, 37(11): 12799-12807.

[14] Bałazy K, Banaei M, Aberer K, et al. LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters[J]. arXiv preprint arXiv:2405.17604, 2024.

[15] Meng F, Wang Z, Zhang M. Pissa: Principal singular values and singular vectors adaptation of large language models[J]. arXiv preprint arXiv:2404.02948, 2024.

[16] Wang S, Yu L, Li J. LoRA-GA: Low-Rank Adaptation with Gradient Approximation[J]. arXiv preprint arXiv:2407.05000, 2024.

[17] Yang Y, Li X, Zhou Z, et al. CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning[C]//The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Q1: The experiments are limited, ignoring methods like DoRA, LoHA, DyLoRA, and DeltaLoRA, and lack sufficient evidence to support the claim that BA-LoRA outperforms LoRA and its variants.

A1: We have added comparisons with methods such as DoRA [1], LoHA [2], and DyLoRA [3] in the revised version, and deeply analyzed their performance differences with BA-LoRA. In addition, we have added comparisons with classic fine-tuning methods such as BitFit [4], Hadapter [5], Padapter [6], and AdaLoRA [7], which fully demonstrate the advantages of BA-LoRA. Regarding DeltaLoRA, we are currently unable to make a direct comparison because the code of this method has not been released.

Specifically, we choose DeBERTa-v3-base as the experimental model and use the GLUE benchmark dataset. The comparison results are shown in the table below. BA-LoRA outperforms all compared methods in all tasks, further verifying the performance advantage of BA-LoRA.

Q2: Why does Full FT perform worse than the PEFT method in Tables 1 and 2?

A2: Regarding the issue you mentioned in Tables 1 and 2, where the performance of the Full FT method is lower than that of the PEFT method, we have conducted further analysis. Previous studies have shown that when a large number of parameters are updated during fine-tuning, particularly when the model is faced with a large and diverse training dataset, overfitting is prone to occur [8][9]. Additionally, large-scale parameter updates can lead to catastrophic forgetting [10][11]. In contrast, the PEFT method helps control the complexity of the model by fine-tuning only a subset of parameters, thereby reducing the risk of catastrophic forgetting and improving generalization ability [12][13].

Furthermore, recent studies [14][15][16][17] have indicated that when the rank value and other key hyperparameters in low-rank fine-tuning (such as scaling factor $\alpha$, learning rate lr, etc.) are appropriately set, the PEFT method can sometimes outperform the Full FT method.

Q3: The discussion on how these three regularization terms mitigate Catastrophic Inheritance is insufficient.

A3: We have discussed the role of regularization in mitigating catastrophic inheritance in detail in the original paper and explained the relevant concepts and methods in multiple sections.

First, in the introduction, we mentioned that the increase in data volume brings multiple challenges, including information imbalance, noise, and corruption, which may have a negative impact on model training (lines 41 to 50). We emphasized how noise and bias weaken the generalization ability of the model and pointed out that the potential bias in the pre-training data may still exist during the fine-tuning process, affecting the final performance and application effect of the model.

In the method section (Section 3.2), we detailed how to mitigate the negative impact of catastrophic inheritance through the designed regularization terms, especially in NLU and NLG tasks.

For NLU tasks:

1. Consistency Regularization: By minimizing the difference between the outputs of the pre-trained and fine-tuned models, this regularization helps preserve critical information from the pre-training phase, thus preventing knowledge loss or distortion and enhancing model stability.
2. Diversity Regularization: By minimizing the covariance of the fine-tuned outputs, this regularization increases the diversity of the model, alleviating overfitting caused by noise or bias, and helping the model learn a more uniform feature distribution.
3. SVD Regularization: By maximizing the first k singular values of the output matrix, this regularization reinforces the model’s focus on the main data components, improving the model's generalization ability and resistance to interference.

For NLG tasks:

1. Consistency Regularization: By measuring the difference between the output distributions of the fine-tuned and pre-trained models using Kullback-Leibler divergence, this regularization ensures consistency and coherence in the generated text, preventing information loss or distortion associated with catastrophic inheritance.
2. Diversity Regularization: By maximizing the entropy of the generation distribution, this regularization encourages the generation of more diverse outputs, avoiding repetition and improving the quality and richness of the generated text.
3. SVD Regularization: By maximizing the contribution of the first k singular values, this regularization helps the model focus on the most important information in the data, reducing noise interference and improving the stability and diversity of the generated text.

In the Experiments section, we verify the effectiveness of these regularization terms through extensive experiments. BA-LoRA performs well in the GLUE and GLUE-X benchmarks, especially under the conditions of noise and data imbalance, proving its stability and reliability. Through comparative experiments and ablation studies, we further verify the effectiveness of these regularization terms in mitigating catastrophic inheritance both independently and in synergy.

In addition, in the Appendix, we discuss the noise and bias issues in pre-training data and their impact in the Background section. In the More Experiments section, we first evaluate the performance of more models of different sizes and types, and then evaluate the performance of models of different ranks. These experiments expand the scope of our experiments and demonstrate the effectiveness of BA-LoRA. Finally, t-SNE visualization shows the effectiveness of BA-LoRA in handling imbalanced data.

Dear Reviewer,

I noticed that you haven't yet responded to the author's rebuttal. As the deadline of discussion period is approaching, could you please review their responses and provide your feedback? Thank you for your prompt attention to this matter.

Area Chair

Q1: The definition of noise and imbalance in this work is not clear regarding different NLG and NLU datasets

A1: In response to this question, we would like to clarify that the concepts of noise and imbalanced data are mainly for the pre-training data of the model, as explained below:

Pre-trained models usually rely on Internet data, which inevitably introduces data imbalance and noise problems. Large language models such as GPT-4 and LLaMA, whose pre-training data mainly comes from a large amount of unlabeled text on the Internet. During the pre-training process, many institutions do not fully modify or clean up these data. Therefore, in the subsequent fine-tuning (Supervised Fine-Tuning, SFT) stage, it is difficult for the model to effectively distinguish and remove inaccurate or irrelevant data, which affects the performance and accuracy of the model in downstream tasks.

In lines 41-43 of the introduction, we explicitly mentioned that the increase in the amount of data brings challenges such as imbalance, redundancy, and corruption [1][2][3][4]. These problems will have a negative impact on model training, especially when dealing with imbalanced or corrupted data, which may lead to performance degradation and weakened generalization ability, thus affecting the effectiveness of the model in practical applications.

In addition, in lines 44-50 of the introduction, we further discussed how bias and noise in pre-training data affect the behavior of large language models. For example, noise in pre-training data can weaken the generalization ability of the model [5], while long-tail distributions in the data can cause the model to over-focus on over-represented topics [6]. We also emphasize that bias introduced during pre-training may persist after fine-tuning, affecting the final performance of the model and its security in practical applications [7][8][9][10].

Finally, in the appendix of the revised manuscript, we have supplemented the specific content of the imbalance and noise problems in pre-training data in detail, including:

1. Low-quality bias:

• Redundancy: The presence of repeated or similar content in the data may lead to the risk of overfitting and privacy leakage [5][6][7].
• Corruption/noise: Inconsistent or erroneous inputs in the training data can affect the robustness of the model and its performance in downstream tasks [5][8][9].
• Contamination: Test data leaks into the training set, which may distort the evaluation results of the model [10][11][12].

2. Distribution imbalance bias:

• Class imbalance: Insufficient sample size for some classes, resulting in poor performance in those classes and biased predictions[13][14][15].

3. Ethical content bias:

• Toxic and harmful content: Training data may contain offensive, biased, or harmful content, which may cause the model to generate inappropriate or harmful outputs[16][17].

Q2: It lacks enough baselines to compare in terms of "Catastrophic Inheritance", especially for the noise and imbalance on the NLG and NLU datasets.

A2: Existing baselines are still insufficient when dealing with the catastrophic inheritance problem in pre-trained models, especially the effects of noise and data imbalance, which limits our comprehensive comparison of method performance. The catastrophic inheritance problem was first fully defined in a study published on February 2, 2024 [1]. Currently, there is a lack of baselines for fine-tuning methods to address this problem, which is one of the reasons why we were unable to provide more comparisons in our experiments. Our work is the first to propose an efficient fine-tuning method that integrates customized regularization terms to alleviate the catastrophic inheritance problem caused by data imbalance and noise in pre-trained models. This innovative approach provides a new solution to the problem and paves the way for subsequent research.

In the revised version, we have included methods such as DoRA, LoHA, and DyLoRA in the comparison, and analyzed their performance differences with BA-LoRA in detail. In addition, we have added comparisons with classic fine-tuning methods such as BitFit, HAdapter, PAdapter, and AdaLoRA to further demonstrate the advantages of BA-LoRA. We will continue to expand the experiments and include more baselines for comparison to fully verify the effectiveness and performance of BA-LoRA.

Q3: BA-LoRA requires specialized regularization terms for NLG and NLU tasks, which may raise generalization concerns if applied to tasks not covered in this paper, such as classification.

A3: We understand your concerns about the generalization ability of BA-LoRA in classification tasks. First, it should be pointed out that LLM is mainly used to handle NLP tasks, especially natural language understanding (NLU) and natural language generation (NLG) tasks [18]. Considering the different requirements of these two types of tasks, we designed three regularization terms to optimize the fine-tuning process of pre-trained models and effectively alleviate the catastrophic inheritance problem.

Specifically, in NLU and NLG tasks, we tailored regularization strategies for each task type:

In NLU tasks:

• Consistency regularization: Minimize the difference between the output of the pre-trained model and the fine-tuned model, maintain key information and enhance model stability.
• Diversity regularization: Reduce the covariance of the fine-tuning output, increase diversity, and prevent overfitting.
• SVD regularization: Focus on the main data components by maximizing the top k singular values ​​and improve the generalization ability of the model.

In NLG tasks:

• Consistency regularization: Use KL divergence to measure the difference in output distribution, ensure the consistency of generated text, and prevent information loss.
• Diversity regularization: Maximize the entropy of the generated distribution, encourage diverse outputs, and thus improve text quality.
• SVD regularization: By maximizing the top k singular values, focus on important information, reduce noise interference, and improve stability and diversity.

To verify the effectiveness of these regularization terms, we conducted experiments on multiple tasks. BA-LoRA showed excellent performance in NLG tasks (such as mathematical reasoning tasks GSM8K, MATH, code generation tasks HumanEval, MBPP, and multi-turn dialogue tasks MT-Bench). At the same time, we also verified the effectiveness of BA-LoRA on classification tasks (such as SST-2, MNLI, etc.) of the GLUE benchmark of NLU. These experimental results fully show that the regularization strategy of BA-LoRA is not only applicable to NLU and NLG tasks, but also can effectively improve the performance of classification tasks.

This work’s experiments cannot prove their proposed method mitigates catastrophic inheritance.

The major issue of this work is that the proposed method and experiments cannot directly and conclusively prove the catastrophic inheritance is mitigated. As authors mentioned in the response to Reviewer M3Ew, catastrophic inheritance contains many aspects, like noise, imbalanced data, repeated data, corrupted data, etc.[1], it cannot directly prove authors’ claim about catastrophic inheritance mitigation only via the regularization ablation study. In addition, those regularization terms are designed for consistency, diversity and generalizaibility, which are not explained by authors about why those regularization terms are closely related to catastrophic inheritance issue.

In addition, the improvement from BA-LoRA and ablation study regarding regularization terms is incremental, which has been agreed by authors during rebuttals. Such small improvements cannot directly conclude that catastrophic inheritance is mitigated, becuase there are too many factors which will contribute such small improvements, such as high-quality finetuning data, different scheduler, different random seed, more finetuning steps, etc. This aligns with the Reviewer M3Ew’s core question, i.e., “How can we conclusively prove that our method mitigates catastrophic inheritance?”

To summarize, the catastrophic inheritance mitigation argument proposed in this work seems to be overclaimed. The current experiments including observing performance gains and regularization ablation study are not sufficient to prove the causation between applied regularization techniques and the mitigation of catastrophic inheritance.

Therefore, I will remain my score unchanged.

Q3: Why can we conclude that 'BA-LoRA can effectively alleviate the impact of imbalanced data in pre-training' based on Figure 1 and the observation that 'BA-LoRA in sub-figures (b) and (d) shows clearer category separation'? What is the relationship between these two observations?

A3: In order to further clarify the regularization mechanism of BA-LoRA and its role in alleviating the impact of imbalanced data, we conducted a more in-depth analysis.

First, The specific design of BA-LoRA for NLU tasks is as follows:

1. Consistency Regularization: Minimize the difference between the output of the pre-trained and fine-tuned models to ensure that the key information in the pre-training stage is effectively retained, thereby improving the stability of the model.
2. Diversity Regularization: Minimize the covariance of the model output during fine-tuning, promote the diversity of the output space, and reduce the model's over-reliance on certain specific categories, thereby alleviating the bias caused by data imbalance.
3. SVD Regularization: By maximizing the first $k$ singular values ​​of the fine-tuned output matrix, the model's focus on important information is enhanced, the interference of noise and redundant information is reduced, and the model's ability to distinguish in imbalanced data is improved.

Next, the t-SNE visualization results in Figure 1 intuitively demonstrate the effect of BA-LoRA in alleviating the impact of imbalanced data:

• Sub-figures (a) and (c) (standard LoRA fine-tuning): The distributions between categories overlap significantly and the boundaries are blurred, indicating that the imbalanced pre-training data has a negative impact on the model classification effect.
• Sub-figures (b) and (d) (BA-LoRA fine-tuning): Compared with standard LoRA, BA-LoRA shows clearer category separation and clearer boundaries, indicating that BA-LoRA more effectively improves the ability to distinguish categories.

t-SNE is a dimensionality reduction technique used to map high-dimensional data to a low-dimensional space to facilitate the observation of the distribution between categories. In this experiment, the t-SNE visualization results after BA-LoRA fine-tuning show that the separation between categories is significantly enhanced, verifying that BA-LoRA achieves a clearer category division in the feature space. These analyses demonstrate the effectiveness of BA-LoRA in dealing with imbalanced data.

Once again, thank you for your valuable suggestions. Your feedback has been very helpful in refining the content of the paper. Should you have any further questions or suggestions, please do not hesitate to reach out.

Q1: Ablation experiments need to supplement the experimental results of Ltask_NLG and Ltask_NLU to enrich Figure 2.

A1: We have included the ablation results of $\mathcal{L} _ {\text{task-NLU}}$ in the revised version. Specifically, we first test the performance of the fine-tuned model without regularization term ($w/o Reg$). Then test individual regularization terms: $\mathcal{L} _ {\text{CR-NLU}}$, $\mathcal{L} _ {\text{DR-NLU}}$, $\mathcal{L} _ {\text{SVDR-NLU}}$. Finally, BA-LoRA containing three regularization terms is tested. The following table shows the ablation experimental results of $\mathcal{L}_{\text{task-NLU}}$, using the DeBERTa-v3-base model, GLUE fine-tuned dataset:

The experimental results show that various regularizations have a positive effect on improving the performance of the BA-LoRA method, further confirming the effectiveness of the regularization terms designed in BA-LoRA for NLG tasks.

Q2: According to the results in Figure 2, Ldr_NLG improves LlaMA-2-7B more significantly than Gemma-7B. Is the " while LDR_NLG had a notably strong impact on Gemma-7B." in the text inconsistent with the picture information?

A2: In response to this issue, we would like to clarify that in the original text, '$\mathcal{L} _ {\text{DR-NLG}}$ has a significantly stronger effect on Gemma-7B' is relative to $\mathcal{L} _ { \text{CR-NLG}}$, not to LLaMA-2-7B.

In fact, what we mean is that on Gemma-7B, $\mathcal{L} _ {\text{DR-NLG}}$ exhibits more obvious performance than $\mathcal{L} _ {\text{CR-NLG}}$ Performance improvements. Specifically:

• GSM8K Task:

  • Without regularization: 77.58%
  • With $\mathcal{L} _ {\text{CR-NLG}}$: 77.82% (Improvement: +0.24%)
  • With $\mathcal{L} _ {\text{DR-NLG}}$: 77.87% (Improvement: +0.29%)

• MATH Task:

  • Without regularization: 31.47%
  • With $\mathcal{L} _ {\text{CR-NLG}}$: 31.84% (Improvement: +0.37%)
  • With $\mathcal{L} _ {\text{DR-NLG}}$: 31.92% (Improvement: +0.45%)

Therefore, we have revised the statement in the manuscript to: “On Gemma-7B, $\mathcal{L} _ {\text{DR-NLG}}$ achieves a larger performance improvement compared to $\mathcal{L} _ {\text{CR-NLG}}$. For example, on the GSM8K task, $\mathcal{L} _ {\text{DR-NLG}}$ improves by +0.29%, while $\mathcal{L} _ {\text{CR-NLG}}$ improves by +0.24%; on the MATH task, $\mathcal{L} _ {\text{DR-NLG}}$ improves by +0.45%, while $\mathcal{L }_ {\text{CR-NLG}}$ improves by +0.37%.”