HiLoRA: High-frequency-augmented Low-Rank Adaptation

We sincerely thank Reviewer K5bg for the review and feedback. Below, we would like to address each of your questions and concerns:

W1. Explanation of why a depth of 2 or greater results in a separation of singular values.

Proposition 3.1 in our paper cites Theorem 1 from the previous work [1], which states that gradient descent in deep networks implicitly drives the solutions towards low-rank.

As revealed in [1], the pre-conditioning matrix $P$ serves as a preconditioning matrix to accelerate optimization by emphasizing an important directions (specific eigenvectors).

For the dynamic optimizers like Adam, the dynamic preconditioning matrix $P_{W,G}$ is used. The eigenvalue of $P_{W,G}$ is derived as $(1 + \eta^2)^{-1/2}_{n,n'}$ and dynamically adjusted based on the magnitude of the gradient and the weight scale.

In the directions with the large singular values, the gradient magnitude is relatively large, resulting in a smaller $\eta^2$, which enhances the contribution of those directions. Conversely, in the directions of the small singular values, $\eta^2$ becomes larger, suppressing learning towards those directions. As depth increases, the weighted combination of the preconditioning matrices across layers accumulates, further emphasizing the directions of the large singular values and the gap among singular values becomes more distinct.

We have revised the paper with updated explanation.

W2. Typo.

Thanks for pointing that out. We have corrected the typo in the revised paper.

W3. Include model names in Tables 1 and 2.

Thank you for your thoughtful suggestion. While the backbone models used in the experiments are mentioned in the main text, we have also included them in Tables 1 and 2 to enhance clarity and understanding.

W4. Are the results taken from other papers or calculated by the authors?

For the GLUE task, we experimented with the baselines using four random seeds, following the recommended search range and including the best hyperparameters suggested in the original papers of each baseline. For the QA tasks, the results of the baselines were taken from AdaLoRA.

W5. Typo.

Thanks for pointing that out. We have corrected the typo in the revised paper.

W6. Catastrophic forgetting for AdaLoRA [2].

Following your kind suggestion, we measured the catastrophic forgetting phenomenon in AdaLoRA using MRPC and STS-B datasets for the GLUE task. Please refer the answer in General Response 2.

Q1 & Q2. Typos.

Thank you for bringing them to our attention. We have fixed the typo and incorporated the correction into the revised version of the paper.

[1] Zhao, Dan. "Combining explicit and implicit regularization for efficient learning in deep networks." Advances in Neural Information Processing Systems 35 (2022): 3024-3038.

[2] Zhang, Qingru, et al. "AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning." arXiv preprint arXiv:2303.10512 (2023).

Thank you for reading our response.

According to your reasonable suggestion, we conducted experiments on the GLUE task based on the DeBERTaV3$_{base}$ model to verify the performance of the proposed model. In accordance with your request, we copied the performance results of the baselines from the original paper of PRILoRA. For HiLoRA, we conducted experiments on 3 random seeds according to PRILoRA. Due to the time constraints, we first reported the results of 4 out of 8 datasets and revised the paper. We will report the results of the datasets that require long time for fine-tuning during the remaining period.

We sincerely thank Reviewer mcnZ for the review and feedback. Below, we would like to address each of your questions and concerns:

W1. Figure 2 is unclear. The implementation details should be included in the main paper.

As you suggest, we include the implementation details in the main paper. Please refer the revised paper.

Additionally, we would like to emphasize that Figure 2 shows the difference between the existing LoRA and our way of interpreting the adapter. In the existing LoRA, the adapter is interpreted as an adapter that is residual to $W_0$. However, we interpret the adapter as a component augmented with $W_0$, and show in Equation 13 that it is mathematically equivalent to the residual method.

W2. Lacks of novelty.

To highlight the novelty of HiLoRA, we present the novelty comparison table below.

LoRA and PiSSA primarily focus on improving the performance of fine-tuning tasks. In particular, PiSSA directly modifying the principal components, inducing significant changes to the pre-trained information, which leads to catastrophic forgetting.

MiLoRA considers the high-frequency components as noise and directly modifies them. However, as revealed in Figure 1 in the main paper, the high-frequency components are not merely noise. Therefore, directly modifying the high-frequency components results in the loss of valuable information in MiLoRA. Moreover, since the frequency scale of the adapter of MiLoRA is not regularized during the fine-tuning process, the adapter’s information becomes dominant, causing catastrophic forgetting. Please refer to Section 5 for more details.

In contrast, HiLoRA reinterprets the adapter as new high-frequency components augmented to $W_0$ and restricts their frequency range. Clipping the learnable diagonal-matrix $\Sigma$, which constains singular values, provides a simple yet highly effective method to adjust the frequency scale without additional losses or computational overheads. This approach confines the fine-tuning process to the high-frequency range while preserving the pre-trained information, which not only effectively mitigates catastrophic forgetting during fine-tuning but also achieving strong performance in downstream tasks.

W3. Initialization Method.

Thanks for your valuable feedback. We experimented with both initialization methods and found that there is an initialization method that is suitable for each data. Regardless of which initialization method is used, it eventually works as a singular vector of new components suitable for the down-stream task through orthogonal regularization. We have revised Tables 5 and 7 in Appendix to specify the initialization method used for each experiment.

W4. Does the forgetting problem still occur if the pre-trained and fine-tuning tasks are similar?

Thank you for your insightful comment.

Generally, fine-tuning aims to inject the new knowledge based on the pre-trained information, which often leads to the catastrophic forgetting as a common phenomenon. If the pre-training and fine-tuning datasets are highly similar, the degree of catastrophic forgetting may not be significant. However, if there are differences in detailed information or class distributions between the datasets, catastrophic forgetting can still occur.

In fact, Table 4 in the main paper reports the results of catastrophic forgetting for the proposed model on the GLUE task. Within the GLUE task, MRPC dataset shows significant catastrophic forgetting in LoRA, while the SST-2 dataset exhibits relatively less forgetting. As you pointed out, the degree of catastrophic forgetting can vary depending on the dataset distribution. Nonetheless, to achieve the general capability across diverse scenarios, it is crucial to design a model that can consistently mitigate catastrophic forgetting.

References

[1] Chen, Yang et al., "Graph fourier transform based on singular value decomposition of the directed laplacian." Sampling Theory, Signal Processing, and Data Analysis, 2023.

[2] Maskey, et al., "A fractional graph laplacian approach to oversmoothing." Advances in NeurIPS, 2024.

[3] Cooley et al., "The fast Fourier transform and its applications." IEEE Transactions on Education, 1969.

[4] Deng et al., "An adaptive Gaussian filter for noise reduction and edge detection." IEEE conference record nuclear science symposium and medical imaging conference, 1993.

[5] Pan et al., "Fast vision transformers with hilo attention." Advances in NeurIPS, 2022.

[6] Kirkpatrick et al., "Overcoming catastrophic forgetting in neural networks." Proceedings of the national academy of sciences, 2017.

[7] Kemker et al., "Measuring catastrophic forgetting in neural networks." Proceedings of the AAAI conference on artificial intelligence, 2018.

[8] Fu et al.,"LoFT: LoRA-Based Efficient and Robust Fine-Tuning Framework for Adversarial Training." International Joint Conference on Neural Networks (IJCNN). IEEE, 2024.

[9] Chen et al., "Bayesian Parameter-Efficient Fine-Tuning for Overcoming Catastrophic Forgetting." preprint, 2024.

We sincerely thank Reviewer vQ3k for the review and feedback. Below, we would like to address each of your questions and concerns:

W1. The connection between frequency, largest singular value, and catastrophic forgetting.

We apologize for the confusion in our explanation regarding the relationship among the frequency, singular value, and catastrophic forgetting.

In the field of signal processing, "frequency" is closely related to the information contained in a weight $W$. Typically, the low-frequency components of $W$ i) correspond to large singular values of $W$ [1, 2] and ii) are used to capture the global information (or decide the global pattern of $W$). Conversely, high-frequency components are i) associated with small singular values and ii) used to capture remaining fine-grained details [3,4,5]. When the adapter learns mainly for low-frequency components, the model tends to include a significant amount of the global information from the fine-tuning dataset/task. However, if the global information from the fine-tuning dataset/task becomes overly dominant, the information learned during pre-training will be overwhelmed, leading to a degradation in the performance of the pre-training task.

We regard the information from the fine-tuning task is rather fine-grained details in comparison with the pre-trained knowledge, considering their dataset size difference. Therefore, we naturally let our proposed method focus on learning the new task by using only the high-frequency components. As described in Section 5 (Analyses on HiLoRA), our model demonstrates the ability to maintain the pre-training task performance by learning the adapters only in the high-frequency domain, i.e., the small singular value range.

In addition, in the transfer learning where preventing catastrophic forgetting is crucial, a common regularization technique is Elastic Weight Consolidation (EWC) [6,7]. EWC introduces an additional regularization term to the loss function to penalize deviations from the pre-trained weight. In other words, EWC only perfers adjusting high-frequency components since adjusting low-frequency components changes the global pattern of the pre-trained weight, i.e., a large deviation. Our method directly adapts in the singular value domain, i.e, in the frequency domain, which is more effective in achieving the goal.

W2. i) Limited Novelty. ii) Comparison with AdaLoRA.

i) To highlight the novelty of HiLoRA, we present the novelty comparison table below.

LoRA and PiSSA primarily focus on improving the performance of fine-tuning tasks. In particular, PiSSA directly modifying the principal components, inducing significant changes to the pre-trained information, which leads to catastrophic forgetting.

MiLoRA considers the high-frequency components as noise and directly modifies them. However, as revealed in Figure 1 in the main paper, the high-frequency components are not merely noise. Therefore, directly modifying the high-frequency components results in the loss of valuable information in MiLoRA. Moreover, since the frequency scale of the adapter of MiLoRA is not regularized during the fine-tuning process, the adapter’s information becomes dominant, causing catastrophic forgetting. Please refer to Section 5 for more details.

In contrast, HiLoRA reinterprets the adapter as new high-frequency components augmented to $W_0$ and restricts their frequency range. Clipping the learnable diagonal-matrix $\Sigma$, which constains singular values, provides a simple yet highly effective method to adjust the frequency scale without additional computational overheads. This approach confines the fine-tuning process to the high-frequency range while preserving the pre-trained information, which not only effectively mitigates catastrophic forgetting during fine-tuning but also achieving strong performance in downstream tasks.

ii) Comparison with AdaLoRA

HiLoRA shows improved performance compared to AdaLoRA on the GLUE task and achieved more or less the same performance on average for the QA task. However, as can be seen from the answer in General Response 2, AdaLoRA suffers from catastrophic forgetting as fine-tuning progresses, while HiLoRA effectively mitigates this phenomenon. This suggests that HiLoRA not only delivers promising performance but also excels in preserving existing knowledge, which demonstrates the enhanced expressiveness and generalization capabilities.

W3. Why catastrophic forgetting is important in LoRA?

LLMs are widely used because they are equipped with rich expressiveness and generalization capabilities, achieved by training large-scale models on vast language datasets. In contrast, the datasets used for fine-tuning are relatively much small in size. Catastrophic forgetting after fine-tuning refers to the phenomenon where the model forgets the knowledge acquired during the pre-training phase, resulting in the generalization capability degradation. In other words, the model becomes overfitted to the target fine-tuning dataset/task, losing its generalization capabilities. In real-world environments, however, we cannot foresee future user queries so mainingtaining the generalization capability and achievning a high fune-tuning task accuracy are equally important. Consequently, recent studies, such as those in [8, 9], have focused on addressing catastrophic forgetting in the context of fine-tuning LLMs, particularly with methods like LoRA.

Therefore, mitigating catastrophic forgetting is as important as the fine-tuning task accuracy from the perspective of model reliability and scalability. In other words, addressing catastrophic forgetting in LoRA is crucial as it ensures that the strengths of pre-trained models are preserved while efficiently adapting to new tasks and delivering consistent performance.

LoRA is designed for modularity, allowing incremental updates for new tasks without retraining the full model. Catastrophic forgetting makes this approach less effective, as the model may fail to retain prior knowledge while learning new tasks, hindering scalability in multi-task or continual learning scenarios.

Furthermore, we demonstrated improved performance and effective mitigation of catastrophic forgetting through experiments in Section 4 (Experiments) and Section 5 (Analyses on HiLoRA).

W4. Experiments on various baselines and large-scale models.

Following your kind suggestion, we have added baselines for the models you mentioned on the GLUE task. Please refer the answer in General Response 1.

Furthermore, we conduct the large-scale experiments for the commonsense reasoning task on LLaMA-7B. Please refer the answer in General Response 3.

Dear Reviewer vQ3k,

We have made every effort to address all the questions and comments you provided in the review. If there are any follow-up questions or additional concerns, please do not hesitate to let us know. We would be happy to provide further clarification.

Thank you for your time and consideration.

I apologize for omitting the term "AdaLoRA" in W2. Allow me to restate my question for clarity.

HiLoRA appears to be very similar to AdaLoRA, particularly in the implementation details presented in Figure 7 and Equation (8). The main difference seems to be that the $\Sigma$ matrix in HiLoRA is learnable and clipped according to Equation (7).

In recent years, many LoRA-based studies have proposed various distinctions based on the parameterized SVD. Figure 7 and Equation (8) represent the common framework of all the parameterized SVD-based LoRA.

For instance, LoRA$^2$ [1] uses the twice-nested parameterized SVD to iteratively project the token representations onto mutually orthogonal planes. SORSA [2], similar to PiSSA, achieves better convergence by initializing the adapter with $r$ principal components from the pre-trained weight and fine-tuning it with orthogonal regularization on the singular vectors. Mo-SARA [3] initializes the singular vectors with those of $r$ principal components, freezes the singular vectors, and fine-tunes the randomly initialized multiple singular values. These singular values are then gated through a learnable vector. Therefore, many studies, which were designed based on the parameterized SVD, i.e., Equation (8), differ in their specific design strategies.

In this context, both AdaLoRA and HiLoRA use the parameterized SVD, but they differ in their motivations, design, and implementation details.

AdaLoRA focuses on the parameter efficiency, dynamically adjusting the rank of each layer within a constrained parameter budget to learn the optimal rank. Each singular value is assigned an importance score based on the magnitude of singular values or the sensitivity of the loss w.r.t. the parameter, and less important singular values are dynamically pruned to optimize the parameter usage. Consequently, the final rank for each layer may differ, but the lack of consideration for frequency components can still lead to catastrophic forgetting as shown in our General Response #2.

In terms of implementation, AdaLoRA computes an importance score for each singular value, and keeps singular values whose scores exceed a threshold, and sets the others to zero.

On the other hand, HiLoRA emphasizes enhancing expressiveness and mitigating catastrophic forgetting at the same time. HiLoRA restricts the frequency domain of the adapter to high frequency within a fixed rank.

Its implementation of clipping ensures that the singular values are restricted below a predefined upper limit using clipping. It ensures that the singular values are learned in the high-frequency domain. Unlike AdaLoRA, HiLoRA does not calculate importance scores or prune the singular values to zero. These implementation differences may appear insignificant, but they have a profound impact on preventing the catastrophic forgetting.

In summary, although many methods use the parameterized SVD, their motivations and implementation approaches are fundamentally different. Future research could explore combining both methods to dynamically adjust ranks within high-frequency domains, offering a novel approach to fine-tuning.

[1] Zhang et al., "LoRA $^ 2$: Multi-Scale Low-Rank Approximations for Fine-Tuning Large Language Models." arXiv preprint, 2024.

[2] Cao et al., "Sorsa: Singular values and orthonormal regularized singular vectors adaptation of large language models." arXiv preprint, 2024.

[3] Gu et al., "Sara: Singular-value based adaptive low-rank adaption." arXiv preprint, 2024.

We sincerely thank Reviewer 36is for the review and positive feedback. Below, we would like to address each of your questions and concerns:

W1. Performance improvement is marginal.

While our model has consistently surpassed various baselines on average, we conduct additional experiments with state-of-the-art baselines for the GLUE task to further validate the effectiveness of our proposed method and report the results in the below table. Due to time constraints, we have included these baselines only for the following six datasets, and we plan to continue updating the tables for other two remaining datasets.

Across these diverse baselines, our model still demonstrated improved performance on average. One may consider that our improvements are marginal but in fact, our method provides larger improvements than others.

Furthermore, our core contribution is that, by reinterpreting the adapter from a signal processing perspective, we showed that a simple mechanism for controlling singular values allows our model to maintain the expressive power while effectively mitigating catastrophic forgetting compared to the baselines.

W2. Principle for searching hyperparameters.

We conducted a grid search primarily around the values commonly used in LoRA-based methods. The range of hyperparameters we explored can be found in Appendix E.1.2 and E.2.2.

In order to validate the effectiveness and generalizability of our model, we conducted the following additional experiments, the results of which can be found in the General Response and the revised paper:

1. Catastrophic forgetting in AdaLoRA
2. Validation of the scalability of the proposed method through large-scale experiments.

[1] Liu, Shih-yang, et al. "DoRA: Weight-Decomposed Low-Rank Adaptation." Forty-first International Conference on Machine Learning.

[2] Hayou, Soufiane, Nikhil Ghosh, and Bin Yu. "Lora+: Efficient low rank adaptation of large models." arXiv preprint arXiv:2402.12354 (2024).

[3] Kalajdzievski, Damjan. "A rank stabilization scaling factor for fine-tuning with lora." arXiv preprint arXiv:2312.03732 (2023).

Dear Reviewer 36is,

We thank the reviewer for recognizing the novelty and contributions of HiLoRA to the community. We are glad that we could address your concerns and appreciate your continued positive rating. However, we believe there may still be additional considerations that could further improve the evaluation. We would be grateful for the opportunity to address these considerations.