Parameter-Efficient Multi-Task Model Fusion with Partial Linearization

Thank you for your comprehensive review and the acceptance recommendation for our paper. Your feedback is greatly appreciated and will help us improve the final version of our paper.

W1: The absence of experiments conducted on larger-scale models diminishes the significance of the proposal.

The scale of the experiments. Regarding the scale of the experiments, we acknowledge that including larger-scale models could further validate the robustness and scalability of our proposal. While our current results already demonstrate the effectiveness of our method on CLIP and Flan-T5 models, we are eager to extend our experiments to SOTA models to provide a more thorough validation of our approach. Due to resource limitations, conducting experiments on larger models was challenging as they require substantial GPU resources, which were difficult to support. We futher report results of extra small-scale experiments of semantic segmentation task in response to Q1, which indicate our method can be extended to larger model and pixel-level vision task.

Q1: Can the proposed methods demonstrate effectiveness on SOTA models?

Effectiveness on SOTA models. We are confident that our partial linearization method can be applied to SOTA models to enhance their multi-task learning capabilities. The positive results we observed in CLIP and Flan-T5 models, which are themselves highly competitive, give us a strong indication that our method would be beneficial when applied to other advanced models. We will take your question as a valuable suggestion for future work and aim to include such results in subsequent publications.

To demonstrate this, we conducted a small-scale segmentation experiment. We utilized the SAM model ('sam_vit_b_01ec64.pth') to test the segmentation tasks with both LoRA and L-LoRA on the Pascal VOC 2012 and NYUD v2 datasets. We set the batch size to 4 and the LoRA r hyperparameter to 32. Due to time constraints and resource limitations, the models were still far from convergence at this point, but the results were sufficient to demonstrate the effectiveness of L-LoRA (The code for this experiment will be put on GitHub together with a clean version of the original code). We report the performance of the pre-trained models, the fine-tuned models, and the fused models obtained through weight averaging on the validation sets of Pascal VOC and NYUD v2 in terms of mIoU in the table below.

Table: reported mIoU of LoRA and L-LoRA.

Notably, the two-model fusion using simple averaging yields different results for each method. The LoRA fusion only retains 43% and 47% of the pretrained model's performance on VOC 2012 and NYUD v2, respectively. In contrast, the L-LoRA fusion maintains a remarkable 99% and 78% of the fine-tuned LoRA's performance on the respective datasets, resulting in a mean mIoU of 2.78.

Q5: It seems the proposed L-LoRA method does not perform well on the NLP domain, according to the results in Table 1 and the similarity heatmap in Figure 8.

Performance in the NLP Domain. The observed underperformance in the NLP domain could be due to our initial choice of prompt templates not being optimal for fine-tuning the language models—a slight oversight in our experimental design. Nevertheless, we are confidence that this does not skew the comparison of multi-task model fusion performance between various methods, given that the same prompt template was employed throughout the experiments.

To enhance the model's output, we should provide more explicit cues within the input. For instance, as suggested in Appendix E.1.6, a more instructive prompt would be "{sentence}. Is this sentence 'positive' or 'negative'?" This type of prompt would clearly signal to the language model that the response should be either 'positive' or 'negative', potentially resulting in increased accuracy.

To demonstrate this, we carried out a series of small-scale multi-task model fusion experiments focusing on three downstream tasks: CoLA, MNLI, and RTE. For all fine-tuning processes, we utilized the Adam optimizer, standardizing the batch size to 16 and setting the number of fine-tuning steps to 2000 for all models. The learning rate was configured to 1e-5 for full fine-tuning, while for both LoRA and L-LoRA, we increased the learning rate to 4e-5. Table 1 presents a comparison between the prompt templates originally used in our study and an improved version of these templates. Table 2 details the individual performance of fine-tuned models employing different prompt templates. In Table 3, we assess the 'exact-match' accuracy on validation datasets through a simple model averaging technique for three-model fusion. Our findings indicate that a better and more predictable prompt yields superior results, with L-LoRA outperforming LoRA in terms of both absolute scores and normalized scores.

Table 1: Comparison of the templates used in our paper and a better version of prompt templates.

W3 & Q6: The L-LoRA has a large performance drop on single-task fine-tuning (Figure 7) and a slight increase on the merged case (Table 1, especially in the NLP domain) compared to LoRA. Thus, it is unclear what are the advantages of L-LoRA compared to LoRA. Why use the proposed L-LoRA rather than the existing LoRA?

Advantages of L-LoRA over LoRA. Despite the performance drop in single-task fine-tuning observed in Figure 7, L-LoRA demonstrates improved performance in the multi-task fusion setting. The increase in performance in the merged case, especially in the NLP domain, suggests that L-LoRA may offer advantages in scenarios where multi-task performance is prioritized.

The multi-task domain is particularly challenging due to the need to balance and optimize across various objectives. The performance gains achieved by L-LoRA in this setting suggest that our method can effectively navigate the trade-offs between tasks, finding what could be described as a Pareto optimal point—a solution where no task's performance can be improved without compromising another's. The advantages of L-LoRA become evident when considering the broader context of its application. The advantages of L-LoRA become evident when considering the broader context of its application. While single-task performance is important, the ability to excel in multi-task environments is increasingly becoming a benchmark for success in advanced machine learning systems. Our method contributes to this goal by enhancing multi-task model fusion capabilities, which is a significant step forward in developing more capable and efficient AI systems.

By the way, based on the small-scale model fusion experiments we conducted in response to Q5, we observe that superior prompt templates can significantly narrow the performance gap between L-LoRA and LoRA. But searching for better prompt templates is not the focus of our research in this paper.

Minor concern 1: caption of figure 3(c)

This is a typo, thanks for pointing it out and we will correct it. "(c) Linearized..." should be "(d) Linearized..."

Table 2: individual performance of fine-tuned models with different prompt templates

Table 3: three-model fusion using simple model averaging, we evaluate 'exact-match' accuracy on validation datasets. These results show that a better and more predictable prompt yields superior results, with L-LoRA outperforming LoRA in terms of both absolute scores and normalized scores. (v0: prompt templates in Appendix E; v1: better prompt templates.)

Additionally, the performance discrepancies might also stem from inherent differences in task characteristics and dataset nature within the NLP field.

W2 & Q3: The definition of the task vector of LoRA in this paper seems to be unreasonable. 1. In [2], the task vector of full fine-tuning is defined as $\nu_i=\theta_i - \theta_0$, which means the parameter change of $\theta_0$. Thus, why the task vector of LoRA in this paper is defined as the change of LoRA parameters $\phi_i-\phi_0$ rather than the change of $\theta_0$ as in [3], i.e., $A_i B_i$, where $\phi_i=[A_i, B_i]$ is the LoRA parameters.

Task vector definition. Our definition of the task vector as $\phi_i - \phi_0$ (the trainable parameter space, LoRA parameter space) rather than the change of $\theta$ (the original parameter space) as done in [3, 4] is a deliberate and calculated decision that aligns with the nature of our fine-tuning approach.

1. When partially linearizing a LoRA model, we concentrate on the dynamics of the trainable parameters and compute the Jacobian-verctor product for all trainable parameters, specifically the LoRA parameters denoted by $\phi$. The original model parameters, denoted by $\theta_0$, are treated as constants during fine-tuning—akin to model buffers that provide a stable foundation upon which the adaptable components of the model, the LoRA parameters, can exert their influence.

2. From a mathematical standpoint, we perform a first-order Taylor expansion on the trainable parameters, which does not involve the parameter merging operations of the LoRA module.

   $f^{\text{lin}}\_{\theta_0}(x;\phi)=f\_{\theta_0}(x; \phi_0) + \nabla_\phi f\_{\theta_0}(x;\phi_0)^\top (\phi -\phi_0)$

   It's important to note that this process of Taylor expansion is applied exclusively to the parameters that are designated as trainable—those that we actively adjust during the fine-tuning phase.

3. Although our experiments have solely utilized LoRA fine-tuning, such a definition is helpful for generalize our method to other parameter-efficient fine-tuning techniques, such as adapter-tuning.

[2] Editing Models with Task Arithmetic. ICLR, 2023.
[3] Effective and Parameter-Efficient Reusing Fine-Tuned Models. arXiv:2310.01886.
[4] Composing Parameter-Efficient Modules with Arithmetic Operations. http://arxiv.org/abs/2306.14870.

Q4: Table 1 only shows the average normalized scores over multiple datasets, so how about the performance of each dataset? Could you provide it in the Appendix?

Performance on Each dataset. We agree that a detailed breakdown of performance across individual datasets would be beneficial. We will include these results in the Appendix to provide a more comprehensive view of our method's performance.

Table: normalized scores of merged CLIP-ViT-B/16 across individual datasets.

Thank you for your detailed evaluation of our manuscript and for raising several insightful points. We appreciate the opportunity to address your concerns and clarify aspects of our work.

W1 & Q1: The novelty of this paper is limited. This paper simply adapts the proposed method in [1] by replacing full fine-tuning with LoRA. Thus, the method in this paper is obviously more efficient than [1] since LoRA is more efficient than full fine-tuning.

Novelty of the paper. We understand your concern regarding the perceived incremental nature of our contribution. However, we must emphasize that the essence of our contribution extends beyond mere efficiency improvements. Our work's innovation lies in the application of first-order information during the partial linearization of LoRA modules. This use of first-order information is pivotal; it captures the most critical aspects of model variability, which is the fundamental reason our method is effective, not merely an incremental extension.

In our approach, we carefully considered the specific aspects of the LoRA fine-tuned model that would benefit most from linearization. Rather than applying a blanket linearization across the entire model, we strategically targeted the adapter modules for this process. This focused application of linearization to the adapters is a deliberate choice, driven by the understanding that these modules play a crucial role in task-specific adaptations within the larger model architecture. By linearizing only these components, we aim to enhance their ability to integrate and retain task-specific information without incurring the computational costs associated with linearizing the entire model. This method allows us to maintain the overall efficiency of the LoRA framework while reaping the benefits of linearization where it counts the most—within the adapters that are key to the model's fine-tuning for distinct tasks.

[1] Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models. arXiv:2305.12827.

Q2: What's the meaning of $\phi_0$? For the full fine-tuning, $\theta_0$ is the pre-trained model weight and is shared for every task $\tau_i$. However, for LoRA, $\phi_i$ is newly added parameters for each task $\tau_i$. Thus, what is $\phi_0$? Is $\phi_0$ a shared initialization for LoRA matrixes of every task $\tau_i$?

Meaning of $\phi_0$. Yes, $\phi_0$ represents a shared initialization for LoRA matrixes across all tasks in our work. As detailed in Appendix B, we have employed a consistent random seed—specifically, seed 42—for initializing the parameter-efficient models in both the vision and language domains.

The rationale behind this specific condition is that when defining the task vector for PEFT models, it is necessary to have a common starting point. In practice, when using only LoraHub to perform model fusion, it is possible to relax this constraint. This is because LoraHub computes a weighted sum of the parameters across the entire LoRA modules to produce a merged LoRA module $\phi = \sum_i w_i \phi_i$, thus avoiding the introduction of the concept of task vector.

We are grateful for your insightful comments and the opportunity to clarify the aspects of our work that you found unclear. Your feedback is invaluable in helping us refine our paper. Thank you for your thorough review and for considering our work.

W1: A key result of the paper is the result from Appendix A which allows the authors to hypothesize that partial linearization of a subset of module parameters (here LoRA) can improve weight disentanglement.

1. W1a: This result needs to be in the main body of the text and properly explained. Without, it is difficult to understand exactly why the authors claim this.

Key result placement. We acknowledge your suggestion regarding the placement of the key result from Appendix A. We will provide a more detailed explanation in section 4.

2. W1b: Having checked the derivation in Appendix A, the authors show that the model output of a linearized model is only determined by the gradient of the loss in the non-linearized model on task $\tau_i$. How exactly then does this allow the authors to make the central hypothesis, which guides the presented method?

Connection between linearization and weight disentanglement. We agree that the derivation is central to our hypothesis and warrants more visibility. We will clarify this explanation in the manuscript to make the connection more explicit.

The derivation in Appendix A shows that the change of the linearized model output is determined by the gradient of the loss. The connection to central hypothesis is that this property of linearized models enables a more predictable and controlled update during fine-tuning. By fine-tuning in the linearized space, updates of trainable parameters are perpendicular to the loss contours, this orthogonality promotes that the updates contribute to the separation of the weight representations corresponding to distinct tasks, a concept known as weight disentanglement.

Mathematically, the change of linearized model output for a input sample $x$ around the initialization $\phi(0)$ can be written as:

Thus we have the following formula that captures the relationship between task-specific gradient of the loss and weight disentanglement for linearized model

Q1: Section 4.2 introduces parameter scaling laws for weight disentanglement. However, what does this have to do with the method or key results? The scaling laws suggest over-parameterisation is necessary for weight disentanglement. I struggle to see the connection between this and the need to partially linearize a model for fine-tuning.

Parameter Scaling Law. The parameter scaling law is an empirical phenomenon suggesting that models with a larger number of trainable parameters tend to exhibit better performance in multi-task model fusion. Consequently, a logical deduction would be that parameter-efficient fine-tuning could potentially diminish the performance of multi-task model fusion. Indeed, our experimental results corroborate this (as illustrated in Figure 5).

However, our method stands out in that the performance of L-LoRA's multi-task model fusion remains on par with that of full fine-tuning, despite its parameter efficiency. This indicates that partial linearization, with a very limited increase in inference cost overhead, effectively bridges the gap typically observed between parameter efficiency and multi-task fusion performance.

Q2: A remark in the manuscript is made bottom of page 8 that "...higher cosine similarity...implies greater redundancy and overlap...This results in more destructive task interference with naïve merging". I don't follow this as it seems to be opposite to most work in multi-task training. In that setting, naïve training would favour similar tasks as it would not require methods to mitigate task interference (see GradNorm method). Why is this then opposite in the context of this paper?

cos similarity and task interference. You've raised an insightful point. When we discuss the impact of high cosine similarity on model performance, it's important to distinguish between the contexts of multi-task training and model fusion. In the former, tasks with similar characteristics often benefit from the ability to share representations, as this commonality can facilitate learning by leveraging the same features or patterns.

However, in the context of model fusion, which is the focus of our paper, the dynamics differ from simultaneous multi-task training. Here, we are dealing with models that have been independently trained on separate tasks and are subsequently being merged. In such a post-hoc fusion, a high cosine similarity between task weights is indicative of overlap in the information encoded by the models, thus be problematic. While this might suggest a common underlying structure, it also raises the risk that merging these similar weights could amplify shared errors or redundancies, rather than complementing each model's strengths. It can signal redundancy, where multiple models encode similar information without adding new insights, or even conflict, where the shared information is not entirely compatible across tasks.

In summary, while shared representations are advantageous in simultaneous multi-task learning, our work suggests that for the fusion of independently trained models, distinctiveness in task representations—achieved through orthogonality—is key to preserving the integrity and performance of the fused model. This distinction is crucial for understanding the different implications of cosine similarity in multi-task training versus model fusion contexts.

Table 2: individual performance of fine-tuned models with different prompt templates

Table 3: three-model fusion using simple model averaging, we evaluate 'exact-match' accuracy on validation datasets. (v0: prompt templates in Appendix E; v1: better prompt templates.)

W2: The results on the NLP task (Flan T-5-Base) need to be better explain. The method (L-LoRA) not only performs worse than full fine-tuning (so does LoRA fine-tuning) but also worse on average than LoRA. Why is this? Is the presented model only applicable to vision tasks?

Performance on NLP Tasks. Our selection of prompts was rather naive, following the approach used by (Raffel et al. in 2020), which was a minor oversight in our experimental design. Nonetheless, we are confident that this does not compromise the comparative analysis of multi-task model fusion performance across different methods, as the same prompt template was consistently applied in all experiments. We find that with fine-tuning of the prompt selection and other task-specific adjustments, L-LoRA can be made effective in the NLP domain.

To demonstrate this, we carried out a series of small-scale multi-task model fusion experiments focusing on three downstream tasks: CoLA, MNLI, and RTE. For all fine-tuning processes, we utilized the Adam optimizer, standardizing the batch size to 16 and setting the number of fine-tuning steps to 2000 for all models. The learning rate was configured to 1e-5 for full fine-tuning, while for both LoRA and L-LoRA, we increased the learning rate to 4e-5. Table 1 presents a comparison between the prompt templates originally used in our study and an improved version of these templates. Table 2 details the individual performance of fine-tuned models employing different prompt templates. In Table 3, we assess the 'exact-match' accuracy on validation datasets through a simple model averaging technique for three-model fusion. Our findings indicate that a better and more predictable prompt yields superior results, with L-LoRA outperforming LoRA in terms of both absolute scores and normalized scores.

Table 1: Comparison of the templates used in our paper and a better version of prompt templates.

Thank you for your thoughtful review and constructive feedback on our paper. We are grateful for your positive assessment of our work and appreciate the strengths you've highlighted.

Q1: From table 1, seems the L-LoRA method are outperforming full-finetuning under some model fusion settings, do we have some possible illustrations?

Results from Table 1. Indeed, as observed in Table 1, our L-LoRA (Linearized Low-Rank Adaptation) appears to outperform full fine-tuning in terms of normalized score under certain fusion settings.

On one hand, the partially linearization likely leads to better weight disentanglement*, meaning that the task-specific knowledge encoded in the weights is more separable.

On the other hand, we should consider *the data requirement of full fine-tuning versus parameter-efficient fine-tuning. As for fine-tuning, the amount of data in downstream tasks is sometimes not enough. However, full fine-tuning generally demands extensive data to optimize the large number of parameters effectively. In scenarios where abundant data is available, full fine-tuning is expected to excel as it can leverage the comprehensive dataset to refine a large amount of model's weights for each specific task. However, in situations with limited data, full fine-tuning may not perform optimally due to overfitting concerns or insufficient information for the model.

L-LoRA, on the other hand, with its focus on updating only a subset of the model's parameters, is less dependent on large amounts of data. This characteristic could explain why L-LoRA sometimes outperforms full fine-tuning in multi-task fusion settings, especially when data availability of downstream dataset is a limiting factor.

W1 & Q2: In vision domain, only the high-level vision task like image classification tasks evaluated, the mid-level and low level task are missing, for example, semantic segmentation. Is it possible to evaluate L-LoRA on image segmentation foundational models like SAM?

Extending evaluation on mid-level and low-level task (Evaluating on image segmentation models). We acknowledge the limitation you pointed out regarding the scope of our evaluations in the vision domain. Our decision to focus on high-level vision tasks like image classification was driven by the desire to establish a strong foundational understanding of multi-task model fusion in a well-studied context, and we agree that extending the evaluation to lower-level vision tasks would provide a more comprehensive understanding.

Evaluating our method on mid-level and low-level task, such as image segmentation, is indeed possible. This is due to the task-agnostic nature of our approach, which does not impose specific requirements on the loss function. The only prerequisite for our method is that the tuned parameter-efficient modules must be capable of computing the Jacobian-vector product, thereby enabling it to be efficiently linearized.

To demonstrate this, we conducted a small-scale experiment. We utilized the SAM model ('sam_vit_b_01ec64.pth') to test the segmentation tasks with both LoRA and L-LoRA on the Pascal VOC 2012 and NYUD v2 datasets. We set the batch size to 4 and the LoRA r hyperparameter to 32. Due to time constraints and resource limitations, the models were still far from convergence at this point, but the results were sufficient to demonstrate the effectiveness of L-LoRA (The code for this experiment will be put on GitHub together with a clean version of the original code). We report the performance of the pre-trained models, the fine-tuned models, and the fused models obtained through weight averaging on the validation sets of Pascal VOC and NYUD v2 in terms of mIoU in the table below.

Table: reported mIoU of LoRA and L-LoRA.

Notably, the two-model fusion using simple averaging yields different results for each method. The LoRA fusion only retains 43% and 47% of the pretrained model's performance on VOC 2012 and NYUD v2, respectively. In contrast, the L-LoRA fusion maintains a remarkable 99% and 78% of the fine-tuned LoRA's performance on the respective datasets, resulting in a mean mIoU of 2.78.