Unleashing the Power of Task-Specific Directions in Parameter Efficient Fine-tuning

Dear Reviewer 27u4:

We would like to first extend our sincere gratitude for your time and effort in evaluating our manuscript. Your thorough evaluation and insightful comments are greatly appreciated. We will address your questions point by point and hope to resolve your concerns effectively.

Weakness 1, Vague Definition

Thank you for your valuable suggestion. Indeed, we have given this definition considerable thought. Determining the number of TSDs is analogous to selecting $k$ in the K-means algorithm. However, we can modify the manuscript to define TSDs as the top-s directions with the largest rate of change, as specifying a fixed number is more intuitive than setting a threshold.

Weakness 2，Hyper-parameter $s$ Setting

Thanks for your suggestions. In our experiment, we set $s=8$, which is consistent with those in the observations for the sake of overall uniformity. Regarding the selection of the hyper-parameter $s$, we found in our ablation experiments that the model’s performance remains robust across a wide range of $s$. This robustness is due to our approach of selecting $s$ directions based on the ranking of their change rates. As a result, regardless of the value of $s$, we always focus on the most significant directions. Therefore, for various scenarios, setting $s=8$ ensures consistently excellent performance.

However, we observed a performance drop when $s$ becomes too large. Based on our analysis in Sec. D.7, we suspect that the inclusion of some irrelevant directions might introduce noise into the training process, destabilizing model convergence and ultimately leading to diminished performance.

Weakness 3, Rigorous Proof

We fully agree with your perspective. In practice, for fine-tuning, it is not feasible to obtain the optimal $\mathbf{W}^*$, let alone analyze its properties or provide rigorous mathematical proofs. Therefore, we can only rely on the statistical patterns observed through experiments to validate our propositions.

Weakness 4, Results of Fully FT

Thanks for your suggestions. We have added the results of Fully FT methods regarding LLaMA2-7B and LLaMA3-8B in the updated manuscript.

Weakness 5, Figure Refinement

Thanks for your suggestions. We have updated Fig. 5(a) in the manuscript.

Weakness 6, Contribution

We appreciate your perspective on the contributions of our work. However, we would like to emphasize that the primary focus of this paper lies in the exploration and study of TSD, as highlighted in Appendix A, as well as a novel PEFT method.

Indeed, TSD is not only critical for PEFT but also plays a significant role in other areas such as Multi-Task Learning (MTL). In modern large-scale model training, multi-task learning has become inevitable. Multi-task learning often suffers from task interference, where learning one task degrades performance on others. However, exploring TSD information allows us to identify the most important directions for each specific task, helping mitigate this issue by isolating task-relevant adaptations, thus maintaining task boundaries. By focusing on task-specific directions, we can also allocate computational and model resources more effectively.

Therefore, TSD provides valuable insights for the broader research community. It serves as one of the key contributions of our work.

Dear Reviewer KtSW:

We would like to first extend our sincere gratitude for your time and effort in evaluating our manuscript. Your thorough evaluation and insightful comments are greatly appreciated. We will address your questions point by point and hope to resolve your concerns effectively.

Weakness 1, Generalization Capability

We appreciate your perspective that fine-tuned models should also retain generalization capabilities. However, we would like to point out that in the context of PEFT, the primary objective is to efficiently adapt a model to a specific downstream task. Under this setup, the focus is solely on achieving strong performance on the downstream task, without prioritizing whether the model retains its generalization capability beyond that task.

As for the idea of “exploring strategies that enable a model, even after fine-tuning, to retain the flexibility to handle a range of general tasks,” we believe this is not closely aligned with the core focus of PEFT research. Instead, it is more related to incremental learning, which aims to enable a model to learn new tasks or knowledge without forgetting previously acquired knowledge. These are two distinct research areas with different objectives and methodologies.

Weakness 2, More Perspective

We agree with your suggestion that using more models would indeed enhance the generality of our observations. We will conduct corresponding experiments on additional models and provide you with feedback as soon as possible.

Weakness 3, Figure Refine

Thank you for your suggestion. We will add results from full fine-tuning in Figures 4 and 8.

Question 1, Applicable to Multimodal Models.

LoRA-Dash is indeed capable of fine-tuning multimodal models. In fact, it can leverage the information from TSD to better achieve multimodal alignment. We are currently working on related tasks and progressing in this direction.

Question 2, Performance Decline

It is not surprising that PEFT methods, such as LoRA-Dash, exhibit performance degradation as the parameter budget (which can be evaluated by rank) increases. In fact, this phenomenon is quite common. While an increase in the number of parameters enhances the expressiveness of the method, it also introduces training instability, which can potentially lead to performance drops. Additionally, [1] theoretically demonstrates that the larger the parameter count in PEFT fine-tuning, the more the model’s stability and generalization ability tend to decline.

[1] 2023, AAAI, On the Effectiveness of Parameter-Efficient Fine-Tuning

Question 3, Memory Requirements for SVD

This is an excellent and practical question. We propose two strategies to address this issue:

1. Two-Stage Strategy: Perform SVD decomposition on the pre-trained weights before the training starts and save the results. During the training process, these precomputed results can be directly utilized, eliminating the need for on-the-fly decomposition.

2. On-the-Fly Decomposition: Perform the SVD decomposition on the CPU during the training process. This approach offloads the computational burden from the GPU, ensuring the primary training workflow remains efficient.

Both strategies provide practical solutions depending on resource availability and training requirements.

Additionally, many existing works also utilize SVD decomposition on pre-trained weights [2-4], and the aforementioned strategies are equally applicable to these methods.

[2] 2024, NIPS, Spectral Adapter: Fine-Tuning in Spectral Space

[3] 2024, NIPS, PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models

[4] 2023, ICCV, SVDiff: Compact Parameter Space for Diffusion Fine-Tuning

We sincerely hope that we can address your concerns.

Dear Reviewer KtSW,

we have already updated the results related to Weaknesses 2 and 3.

Weakness 2: We fine-tune Qwen2.5-7B on math reasoning task, and the results are shown in Fig. 7 in Appendix.

Weakness 3: We compare the images generated by LoRA-Dash and fully fine-tuning, and the results are shown in supplementary. Since each dataset in this task contains only a few images, fully training a stable diffusion model is highly prone to overfitting. As a result, most works primarily compare with the results of parameter efficient fine-tuning [1,2]. However, as requested by the Reviewer KtSW, we also present the results of fully training the model. Unsurprisingly, the performance of the fully trained model is worse than LoRA-Dash, as expected.

[1] 2024, arXiv, NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models

[2] 2024, EMNLP, Mixture-of-subspaces in Low-rank Adaptation

Dear Reviewer B9L3:

We would like to first extend our sincere gratitude for your time and effort in evaluating our manuscript. Your thorough evaluation and insightful comments are greatly appreciated. We will address your questions point by point and hope to resolve your concerns effectively.

Question 1, LoRA-Dash Fine-tuning Time

Thank you for your valuable suggestion. We apologize if the term “Dash” may have caused any misunderstanding. Here, “Dash” does not refer to training time acceleration; rather, it represents the capability to accelerate convergence. By expediting the learning of TSD information, LoRA-Dash achieves significantly faster convergence compared to LoRA. In practical fine-tuning scenarios, such as with DeBERTaV3-large, LoRA-Dash requires only half the number of training epochs needed by LoRA, demonstrating its efficiency and effectiveness.

Question 2, Benefits of LoRA-Dash

In PEFT, the primary objective, from the perspective of evaluation metrics, is to achieve better performance with fewer parameters. As you suggested, LoRA-Dash demonstrates its superior performance by achieving better results with a very small rank $r$, which is a clear reflection of its outstanding effectiveness.

As $r$ increases, the number of fine-tuned parameters grows, and the performance of fine-tuning methods naturally approaches that of fully fine-tuning. In this scenario, LoRA-Dash cannot exhibit the same significant performance gains over LoRA as observed with smaller parameter budgets, as the upper limit of performance is constrained. However, it still achieves noticeable improvements over LoRA. This further highlights the superiority of LoRA-Dash in effectively utilizing the parameter budget to enhance performance.

Question 3, Applying TSD to Other Methods

This is an excellent and thought-provoking question. Below is our perspective:

Within the framework we have established, TSD represents the direction in which model weights require substantial modifications. Therefore, the application of TSD essentially involves modifying the corresponding model weights. From this perspective, methods such as P-Tuning, which focus on modifying the input, are less suitable for applying TSD. On the other hand, approaches like Adapter and LoRA are naturally more compatible with TSD.

Furthermore, when practically applying TSD, we need to leverage the information of $\Delta \mathbf{W}$, which represents the weight changes. For LoRA and its variants, TSD is inherently compatible, as these methods can be regarded as learning $\Delta \mathbf{W}$ directly. In contrast, for methods like Adapter, it is necessary to record the pre-trained weights and compute the difference between the updated weights during fine-tuning and the pre-trained weights to obtain $\Delta \mathbf{W}$. This additional step is required to effectively integrate TSD into these approaches.

We sincerely hope that our response could address your concerns.

Dear Reviewer 93T4:

We would like to first extend our sincere gratitude for your time and effort in evaluating our manuscript. Your thorough evaluation and insightful comments are greatly appreciated. We will address your questions point by point and hope to resolve your concerns effectively.

Weakness 1, Hyper-parameter Selection

Our choice of hyper-parameters is not necessarily tied to the preliminary observations. We set the hyper-parameters to be consistent with those in the observations for the sake of overall uniformity. We also conducted hyper-parameter ablation experiments across multiple models, including LLaMA3-8B, LLaMA-7B, and DeBERTaV3-large, on different tasks. Beyond the results presented in the main text, additional findings are provided in the Appendix on page 30, Figure 10. These results show that when hyper-parameters fall within a reasonable range, the performance of the model remains relatively stable. Therefore, for various scenarios, setting $s=8$ ensures consistently excellent performance.

Therefore, we believe that using default parameters can achieve satisfactory results across different scenarios, without the need to design specific hyper-parameters for each case. Moreover, in all experiments, our hyper-parameter settings were kept consistent, further demonstrating the robustness of LoRA-Dash, which does not require different hyper-parameter choices for varying scenarios.

Weakness 2, Objective Evaluation for Diffusion Models

Thank you for your valuable feedback. We fully acknowledge your perspective regarding the reliance on subjective assessments in generative experiments, as we also mentioned in Appendix E.1, Line 1445, “unlike numerical comparisons, visual comparisons and user studies are inherently subjective.” To ensure objectivity as much as possible, we conducted a user study, the results of which are presented in Appendix E.1. The study shows that the images generated by LoRA-Dash achieved an approval rating of 84.51%.

Additionally, such user studies are a commonly adopted quantitative evaluation strategy for generative results in PEFT works [1-3]. Moreover, to fully address your concerns, we have also performed the CLIP text-image score, which measures the alignment between textual descriptions and corresponding images, and image similarity score, which assesses the similarity between the original image and the generated image. The results are shown in the following table, where it is evident that for both metrics, LoRA-Dash outperforms LoRA. The images generated by LoRA-Dash exhibit better text alignment as well as stronger correspondence with the original images. This indicates that LoRA-Dash produces superior results compared to LoRA. We will add these results to our paper, thank you for your suggestions.

[1] 2024, EMNLP, Mixture-of-Subspaces in Low-Rank Adaptation

[2] 2024, arxiv, LoRA-Composer: Leveraging Low-Rank Adaptation for Multi-Concept Customization in Training-Free Diffusion Models

[3] 2024, ECCV, Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models

Question 1, Misalignment of the Experiment Results

Figure 5(a) presents the ablation experiment results on LLAMA3-8B, while (b) and (c) show results on LLaMA-7B. Therefore, the performance differences on OBQA are due to the variations in model size and architecture.

Question 2, Relations between TSDs and Objects

This is an excellent question. We have discussed it in Appendix E.4 on page 30 and hope it resolves your concerns.