CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning

(2) The knowledge-preserved mode is not very compelling given this method will hurt the target task performance. I don’t know why the paper emphasizes this mode in parallel with the instruction-previewed adaptation mode. [1] may be a pointer for reference as it also studies how to preserve the forgetting of general knowledge in LMs when continual training them.

Please note that the knowledge-preserved adaptation does NOT hurt the target task performance. As shown in Table 1, compared with LoRA, our method has better performance than LoRA in both worlds (world knowledge benchmarks and downstream tasks) in most cases. We quote some results of LoRA and CorDA in Table 1 here for your reference.

More importantly, as we respond above, existing PEFT methods rarely consider or support finetuning with knowledge better perserved. There are some studies on the continual training of LLMs [1] and [14, 21] (ref. in our paper), but they are not PEFT methods. We will cite [1] in the revised version of our paper. As shown in Table 1, when using the average performance of the both worlds to measure the comprehensive ability, CorDA in knowledge-preserved adaptation achieves the best results among all the compared methods in all the three tasks. Therefore, we respectfully do not agree with your comment that "knowledge-preserved adaptation is not compelling" and "it hurts the target task performance".

Admittedly, CorDA in knowledge-preserved adaptation is not stronger than the instruction-previewed adaptation when only evaluating the downstream task performance. But please note that maintaining pre-trained knowledge better and pursuing a better finetuning performance is inherently a tradeoff (also mentioned in [14] and [21]). A method may be better than another one in both worlds, just like CorDA v.s. LoRA in the results above, but for the method itself, the two worlds are still a tradeoff. We would like to use an analogy to support. The advanced neural architecture (e.g. Transformer) may be better than a traditional MLP/CNN architecture in terms of both accuracy and parameter efficiency. But for the better architecture itself, higher accuracy still brings more parameters (BERT_large is better than BERT_base with more parameters).

Therefore, when knowledge maintenance is not a concern, we introduce our instruction-previewed adaptation, which puts all its efforts for the downstream task, surpassing the competitive studies DoRA and PiSSA in the three finetuning tasks, Math, Code, and instruction following, as shown in Table 2. In conclusion, the knowledge-preserved adaptation and instruction-previewed adaptation highlight the comprehensive performance and the specialized ability of downstream task, respectively. We think this is also a feature of our PEFT method, allowing for customized selectivity based on the actual need. That is why we emphasize the two modes in parallel.

[1] Continual Pre-training of Language Models, Ke et al., ICLR 2023

(3) Questions: Does the major difference between CorDA and LoRA lie in the initialization? Have you compared with other PEFT methods that focuses on initialization technique? Currently, the baseline comparison is not very comprehensive.

Yes, CorDA brings task context into the LoRA adapter initialization. Adopting the same LoRA structure not only facilitates fair comparison, but also enables to restore the orignal LLM architecture after finetuning without architectural change or introducing inference burden. Yes, we have compared with PiSSA, which also focues on the LoRA adapter initialization but does not consider task context. DoRA builds the adapter with a normalization and a learnable magnitude, and also does not consider task context. It is noteworthy that both DoRA (ICML 24) and PiSSA (released on Arxiv in April 2024) are recent studies and are strong baselines. Besides, full parameter finetuning is the most direct reference because it usually has the best finetuning performance without considering parameter efficiency. For downstream tasks, as shown in Table 2 and Table 3, our method achieves finetuning performances on par with full parameter finetuning, and better performances than the compared PEFT methods LoRA, DoRA, and PiSSA. For comprehensive ability (with knowledge benchmarks included), as shown in Table 1, our method has the best average performance among full finetuning and the PEFT methods. Therefore, our experimental results are already able to demonstrate the effectiveness of our proposed method.

Thank you for recognizing the contributions of our work and your valuable comments.

(1) One major concern is the lack of theoretical support, and some arguments are not well justified. Is there any support for Line 45-47 or is this a pure intuition? Why SVD on $WC$? What does $WC$ represent?

We first highlight the significance of our work even though it is lacking a theoretical support, and then provide some supports from literature and our empirical analysis justifing Line 45-47. Finally, we explain our method of SVD on $WC$.

1. Significance of our work even with no theory:

Despite the lack of theory, the contribution of our work is significant for the following reasons

• It is not necessary to use a theory to show its validity. Activation awareness has been proven to be important in LLM compression and quantization [27, 35, 65]. But in PEFT, existing studies rarely consider task context. So, a fundamental originality of our method is that we introduce task context such that the resulting adapter initialization is task-dependent. We provide some other supports (in "2. Supports for Line 45-47") for why using covariance matrix and its advantage over only using the activation itself.

• Existing PEFT methods rarely support the option of finetuning with the pre-trained knowledge better preserved ([1] is not a PEFT method). Since our method is task-dependent, we offer such flexibility enabling both knowledge-preserved adaptation and instruction-previewed adaptation, customized for the actual need. When knowledge maintenance is a need, the former satisfies this demand. When we only want to push the limit of the downstream tasks without concerning about the pre-trained knowledge maintenance, the latter is favorable.

• In experiments, compared with LoRA, our knowledge-preserved adaptation achieves improvements on both worlds (shown in Table 1), enjoying better results than LoRA on both world knowledge benchmarks and downstream tasks, and having the best comprehensive average performance among all the compared methods. Our instruction-previewed adaptation is able to further improve the downstream task performance, surpassing LoRA, DoRA, and PiSSA in all the three finetuning tasks (shown in Table 2). In either case, the experimental performance is a progress in the field.

Therefore, we believe that our contributions should not be unappreciated just because our method lacks a theoretical support, which is actually a common issue in existing PEFT methods. Actually, there is no theorem in the original LoRA paper and most of the follow-up studies.

2. Supports for Line 45-47:

• Support from literature. Activation awareness has been proven to be effective in LLM compression and quantization [27, 35, 65]. OWQ [27] proposes to leverage the outlier awareness from covariance matrix to make weight quantization task-specific. ASVD [65] also considers outliers, scaling the pre-trained weight with the input activation to perform SVD for model compression. The rationality behind these studies is that the outlier in activation or covariance matrix is responsive to the task triggered by the input, which is also our motivation of using covariance matrix to orientate the decomposition.

• Support from empirical analysis. As suggested by Reviewer Mgdn, we visualize the covariance matrix collected from samples from the three tasks, MetaMath, NQ open, and Trivia QA. Note that we downsample the covariance matrices from a high dimension (4096 or 11008) into $32 \times 32$, and visualize their heatmaps. Please see the PDF file of the global rebuttal for the visualization results. It is shown that the heatmaps from NQopen and TriviaQA (both are QA tasks) share some similar patterns marked in red circles, which do not appear in the heatmap from the different task MetaMath. The visualization result further supports that covariance matrix can be used to characterize the triggered task.

• Support from experimental results. As stated above, ASVD and our method has a similar motivation. ASVD performs SVD on the pre-trained weights scaled by input activation for model compression. We use covariance matrix to orientate the decomposition of weights for building adapters. We compare our context-oriented SVD (CO-SVD) with ASVD in Figure 2 and Table 6 of our paper. It is shown that when discarding the smallest several components, CO-SVD is much better at maintaining the performance of WikiText-2 and PTB than ASVD and Plain SVD. This result indicates that our CO-SVD has a stronger ability in assembling task context into its principle components. Therefore, covariance matrix is better choice for us to build task-dependent adapters.

3. Why SVD on $WC$ and what does $WC$ represent:

As we respond above, our CO-SVD, using covariance matrix to orientate the decomposition of pre-trained weights, has a strong ability in assembling task context into the principle components. As shown in Figure 2 and Table 6, our CO-SVD, i.e. SVD on $WC$, is much better than the plain SVD that does not include task context, and ASVD that performs SVD on the weights scaled by activation. So, we perform SVD on $WC$, where $W$ is the pre-trained weight and $C$ is the covariance matrix collected from a few samples. Its principle components after decomposition are task dependent and better capture task context than only using the activation itself.

(2) The knowledge-preserved mode is not very compelling given this method will hurt the target task performance. Why the paper emphasizes this mode in parallel with the instruction-previewed adaptation mode.

(3) The other questinos

We have discussion about the unique value of our knowledge-preserved mode and answer to your remaining questions in the following comment.

Thank you for recognizing the contributions of our work and your valuable comments.

(1) It is necessary to add a description of the experimental environment and provide parameters to offer readers more possibilities for replication and ensure authenticity. Lacks specific quantification of catastrophic forgetting in the model.

We have provided the implementation details in our main paper and Appendix A. Concretely, we describe the data choice and number to collect covariance matrix, the rank choice, and datasets used in the experiment section of our paper. In Appendix A, we provide the training settings (optimizer, batchsize, learning rate, etc), the GPU device we use, and the evaluation tools. Our code and models will be publicly available. Please let us know if the reviewer has any question about our experimental details.

Usually the average performance and the performance drop compared with the model before training/finetuning are the common quantification metrics to evaluatuate catastrophic forgetting. We have shown the average performance over world knowledge benchmarks and downstream tasks in Table 1. It can reflect the comprehensive ability of the finetuned model in maintaining the pre-trained knowledge while learning the new task. Besides, for the world knowledge benchmarks (Trivia QA, NQ open, and WebQS columns), the performance difference between the second row (LLaMA-2-7B) and the rows below shows the performance drops of each finetuning method compared with the original LLaMA-2-7B model. It is shown that our method enjoys the lowest performance drop in most cases. We will mark the performance drop beside these numbers to show the advantage more explicitly.

Thank you for recognizing the contributions of our work and your valuable comments.

(1) Originality: methodological similarities with PiSSA, and comparisons with existing works like AdaLoRA

Similarity with PiSSA:

Both our method and PiSSA adopt SVD for the pre-trained weights, however, our method has fundamental difference and advantage compared with PiSSA as follows:

• Activation awareness has been proven to be important in LLM compression and quantization. But in PEFT, existing studies rarely consider task context. PiSSA provides a better adapter initialization than LoRA using SVD, but still does not consider any task context. Although our method also adopts SVD, we never claim that the novelty of our method lies in the usage of SVD to build LoRA adapters. The fundamental originality is that we introduce task context, captured by the covariance matrix from activations, to orientate the decomposition of weights, such that the resulting adapter initialization is task-dependent.

• Existing PEFT methods including PiSSA rarely support the option of finetuning with pre-trained knowledge better preserved. Since our method is task-dependent, we offer such flexibility enabling both knowledge-preserved adaptation and instruction-previewed adaptation, customized for the actual need.

• Even when the maintenance of pre-trained knowledge is not a concern, our method in instruction-previewed adaptation surpasses DoRA and PiSSA in the three downstream tasks of Math, Coding, and instruction following, as shown in Table 2 of our paper. It is noteworthy that both DoRA (ICML24) and PiSSA (arXiv:2404.02948, online in April 2024, within 2 months of our submission) are the latest studies, and PiSSA actually could be considered as contemporaneous.

Therefore, we believe that our originality and contributions, including the methodology of introducing task context into LoRA adapter initialization, and the experimental performance of both maintaining pre-trained world knowledge and improving downstream task ability, should NOT be unappreciated just because we also adopt SVD in the adapter initialization process.

Comparison with AdaLoRA:

Our method improves the LoRA adapter initialization by introducing task context, and adopts the low intrinsic dimension $r$ following the standard setting in LoRA. AdaLoRA aims to dynamically adjust $r$ during finetuning. So, CorDA and AdaLoRA are methods focusing on different aspects (how to better build adapters and how to dynamically adjust the rank). In our experiments, we mainly compare CorDA with methods of the same stream, PiSSA and DoRA. PiSSA also focuses on the LoRA adapter initialization using SVD, but does not consider task context. DoRA builds the adapter with a normalization and a learnable magnitude, and also does not consider task context. It is noteworthy that both DoRA (ICML 24) and PiSSA (released on Arxiv in April 2024) are recent studies and are strong baselines. Besides, full parameter finetuning is the most direct reference because it usually has the best finetuning performance without considering parameter efficiency. For downstream tasks, as shown in Table 2 and Table 3, our method achieves finetuning performances on par with full parameter finetuning, and better performances than the compared PEFT methods LoRA, DoRA, and PiSSA. For comprehensive ability (with knowledge benchmarks included), as shown in Table 1, our method has the best average performance among full parameter finetuning and the PEFT methods. Therefore, our experimental results are already able to demonstrate the effectiveness of our proposed method.

(2) Quality: The exact role of the covariance matrix $C$ is not clearly explained. The paper states that “the covariance matrix of each layer’s activation will exhibit different outlier patterns as they are responsive to the task triggered to highlight different aspects of the pre-trained weight.” Providing visualizations to show how data from different tasks trigger different parts of the weight would enhance clarity.

When the inputs of different tasks are fed into an LLM, the covariance matrix from activations will exhibit different patterns. We use such patterns to orientate the decomposition of LLM pretrained weights, to make the resulting adapter initialization task-dependent. We did not provide the visualization of the covariance matrix because the dimension in 4096 or 11008 is too large to be informative. In the rebuttal, we downsample the covariance matrices into 32 $\times$ 32 and visualize their heatmaps. Please refer to our response in the global rebuttal and its PDF attached. It is shown that the heatmaps from NQopen and TriviaQA share some similar patterns (marked in red circles), which do not appear in the one from the different task MetaMath. We hope this result can further justify that the covariance matrix patterns can be used to characterize the triggered task.

Besides, we can find some supports from the literature. OWQ [27] proposes outlier-aware weight quantization based on covariance matrix. ASVD [65] performs SVD considering activations for compression. But as shown in Figure 2 and Table 6, our CO-SVD based on covariance matrix instead of only the activation itself, is much better at capturing the task context.

We will add these visualization results and more discussion about the literature support in the revised version of our paper.

(3) Clarity and Significance: "The amount and type of data used to derive the covariance matrix $C$ significantly impact performance, but the paper lacks a clear discussion and analysis on this. Moreover, there is no guidance on how to choose $r$ in Equations 4 and 5, which appears to be a crucial hyperparameter."

We address your concern about the impact of data amount and type to derive $C$ on performance and how to choose $r$ in the global rebuttal. Please refer to the global rebuttal-> common issue.

Thank you for recognizing the contributions of our work and your valuable comments.

(1) How did the authors decide to maintain the last $r$ eigenvectors for world-knowledge? Cite relevant works or make proper justification on this design choice.

As shown in Figure 1 and Eq. (4), in the knowledge preserved adaptation, we use the last $r$ eigenvectors to build adapters, instead of maintaining them. We explain the design choice in more details as follows.

Our context-oriented SVD (CO-SVD) decomposes the pre-trained weights into orthogonal abilities where the largest principle components most correspond to the task context captured by the covariance matrix. In instruction previewed adaptation (IPA), we want to better learn the target ability, so we use the first $r$ eigenvectors to build adapters. In knowledge preserved adaptation (KPA), the first components correspond to the QA ability, which is what we want to maintain. So, we use the last $r$ eigenvectors to build adapters while freezing the first ones. Therefore, the design motivations of IPA and KPA are not opposite. The principle components after CO-SVD both represent the ability indicated by the covariance matrix. The difference is a result of the purpose, i.e. better adapt these components (used as adapters in IPA) or better maintain these components (frozen in KPA).

We thank the reviewer for the suggestion of citing relevant works to justify the design choice. Actually, ASVD [65] is just based on a similar design motivation. They discard the last eigenvectors and only maintain the largest several components for model compression. In our KPA, we also maintain the principle components to preserve world knowledge, but adapt the last $r$ eigenvectors to learn new ability instead. Despite a similar motivation, we have shown that our CO-SVD is much better at capturing the characteristics of the target task than ASVD in Figure 2 and Table 6. Previous stuides [A, B, C] also adopt SVD for model compression. We will cite these works and explain the design choice of KPA and IPA more thoroughly in the revised paper.

[A] Language model compression with weighted low-rank factorization, ICLR'22.

[B] Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation, NeurIPS'14.

[C] GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinking, NeurIPS'18.

(2) How scalable is their proposed approach to other datasets where we may want to maintain knowledge? How likely is the model to maintain knowledge on TriviaQA, if it wasn't included during LoRA initialization?

Our method guides the decomposition of pretrained weights by the data-dependent covariance matrices. However, the knowledge to preserve is definitely NOT constrained to the data used for adapter initialization. Actually, we only sample 256 questions from a QA dataset to collect the covariance matrices. The ability of our method to preserve world knowledge cannot be derived from these limited samples themselves. That is to say, what we maintain is the QA ability, instead of some specific knowledge from the collected data.

We use the covariance matrices, whose outlier patterns characterize the target task, to make the decomposition task-specific. Therefore, the same kinds of input/query (e.g. questions from TriviaQA and NQopen) have a similar effect because they both trigger a similar ability. We have conducted analysis in Table 4 (TriviaQA and NQopen) and Table 5 (WizardLM-Evol-Instruct and Alpaca) comparing results with data from similar tasks. Please refer to our response in the global rebuttal -> common issue -> (2) Type/quality. These results indicate that randomly collecting context from one dataset has a scalable effect to other datasets of the same task.

(3) What is the necessary sample complexity for initializing the LoRA parameters, i.e. how many samples from the knowledge datasets are necessary to get a good estimate of the directions to freeze?

As described in Line 233 of our paper, we sample 256 questions for our experiments of knowledge preserved adaptation. We collect 256 samples in all our experiments for both KPA and IPA modes.

In Table 6, we analyze the effect of sample number. We compare the results of collecting 32 and 256 samples for Wikitext-2 and PTB, respectively. In order to also investigate the effect of sample number in the KPA experiment, we collect less samples (128 and 32) from NQopen. Please refer to our response and result in the global rebuttal -> common issue -> (1) Amount.

(4) Is knowledge preservation necessary at each weight parameters and each layer? How do results change when the model's LoRA parameters are restricted only in the lower layers, while the higher layers are given more freedom?

We follow the standard setting in LoRA, DoRA, and PiSSA, i.e., evenly using the same low rank for all linear layers. It also facilitates fair comparison with them. Since large deviation of lower layer weights will go through more later layers whose accumulative effect will cause a large shift for the final representation, restricting the adaptation of lower layers and finetuning higher layers with more freedom indeed may be preferable to maintain the pre-trained knowledge. Thus we can use a small intrinsic rank ($r$) for lower layers to constrain their adaptation, and adopt a large $r$ and even full finetuning for higher layers to learn downstream tasks. We can also adopt an adaptive strategy, e.g. based on the eigenvalue distribution after our CO-SVD, to assign ranks. These strategies may be more parameter-efficient than using the same rank for all layers. Besides, Transformer different weights and MLP blocks may also play different roles in maintaining pre-trained knowledge. Investigating the importance of different weight parameters in preserving knowledge and assigning adapters accordingly will be a valuable extension of CorDA and deserve our future exploration.