AdaMerging: Adaptive Model Merging for Multi-Task Learning

3. About the question of "The performance of the pre-trained model":

As shown in the below tables, we added the pre-trained model to predict the performance of each task. It can be observed that there is a big gap in performance between pre-trained and Task Arithmetic. Our AdaMerging significantly outperforms them.

4. About the question of "The limited application of this kind of work":

At this stage, model merging has a certain performance gap with traditional MTL, but it is a potential and popular research direction [1-8]. The traditional MTL method requires the original data of all tasks to be collected together in advance, and then iteratively updates the MTL model, which has the risk of data privacy leakage. Especially in the era of large models, this method of training MTL models from scratch is inefficient. Model merging does not require original task training data, but only requires models trained independently for each task to complete MTL. Reviewer eRin also noted that "developing a single versatile model from diverse off-the-shelf fine-tuned models is important to LLM (or Foundation model) communities".

In addition, regarding the issue of relying on the same pre-trained model. In the era of large models, the backbone of most research work is highly homogeneous and is often based on the same pretrain model. Ties-Merging [1] points out that "these benefits have resulted in the release of thousands of fine-tuned checkpoints derived from popular PTMs such as ViT for vision and T5 for language." In addition, learning from models [2] rather than raw data is a critical research direction.

[1] Yadav, Prateek, et al. "Resolving Interference When Merging Models." NIPS 2023.

[2] Zheng, Hongling, et al. "Learn From Model Beyond Fine-Tuning: A Survey." arXiv preprint arXiv:2310.08184 (2023).

[3] Ilharco, Gabriel, et al. "Editing models with task arithmetic." ICLR2023.

[4] Zhang, Jinghan, et al. "Composing parameter-efficient modules with arithmetic operations." arXiv preprint arXiv:2306.14870 (2023).

[5] Li, Weishi, et al. "Deep model fusion: A survey." arXiv preprint arXiv:2309.15698 (2023).

[6] Wang, Song, et al. "Knowledge Editing for Large Language Models: A Survey." arXiv preprint arXiv:2310.16218 (2023).

[7] Chronopoulou, Alexandra, et al. "Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization." arXiv preprint arXiv:2311.09344 (2023).

[8] Ortiz-Jimenez, Guillermo, Alessandro Favero, and Pascal Frossard. "Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models." arXiv preprint arXiv:2305.12827 (2023).

Thank you very much for your review and affirmation of our work. We'll answer your questions one by one below.

1. About the question of "The novelty of this paper":

(1) The research direction of this paper is an important direction, which is to merge models trained independently on multiple tasks into one to perform MTL. It can reduce the effort of collecting original training data and avoid the cost of model retraining. However, there is still a significant performance gap between SOTA model merging schemes and traditional MTL, and we observe that the model merging coefficient has a very large impact on performance. In this paper, we first propose AdaMerging to automatically learn Task-wise or Layer-wise merging coefficients. Further, we verify through extensive experiments that unsupervised entropy minimization can be used as a proxy objective for supervised cross-entropy loss to optimize model merging coefficients. Our approach, although seemingly simple, is highly effective. For example, compared with two SOTA task vector-based methods, Task Arithmetic (ICLR'2023) and Ties-Merging (NeurIPS'2023), our AdaMerging and AdaMerging++ bring up to 11% and 8.7% performance improvement, respectively. Even if only 0.1% of the test data is available, our method can bring about a 5% performance improvement. Our method also exhibits stronger generalization and robustness.

(2) The tasks and data of our experiments strictly follow the settings of Task Arithmetic[1] and Ties-Merging[2]. We have added similarity calculations for task vectors in the modified version (as shown in Figure 6 in the appendix). We have observed that a small number of task vectors have a certain degree of correlation (such as SVHN and MNIST, which are both digit recognition tasks), and most task vectors have low correlation. That is, these task vectors are almost orthogonal (this is consistent with Task Arithmetic). We did not observe a direct correlation between the 'similarity of the task vectors' and the 'merging coefficients'. We further added task vector merging experiments with different correlation degrees, as shown in Figure 7 in the appendix. We observed that our AdaMerging is effective under various correlation degrees.

(3) The suggestion that "weights learned from task vectors can be used to guide MTL training" may not be suitable for the setting of this paper. Model merging represents only the trained model for each task, without the original training data. Therefore, model merging technology is unsuitable for scenarios where a weighted MTL can be trained directly from scratch. Model merging is mainly used for foundation models or large language models, which are very expensive to train from scratch using raw data.

[1] Ilharco, Gabriel, et al. "Editing models with task arithmetic." ICLR2023.

[2] Yadav, Prateek, et al. "Resolving Interference When Merging Models." NIPS2023.

2. About the question of "The relationship between Shannon entropy and cross entropy":

Indeed, there is a close relationship between Shannon entropy and cross-entropy in information theory. However, they still have obvious differences in merging coefficients training. Entropy minimization does not require labels, while cross-entropy loss requires labeled data. Experimental confirmation of this relationship can provide readers with a more intuitive understanding that entropy minimization can serve as an effective proxy for cross-entropy loss when merging models. In the modified version (Figure 10 in the appendix), our method further increases the correlation between entropy minimization and cross-entropy loss minimization across different training stages.

In addition, although entropy minimization can only be used for classification tasks, classification tasks are very common in various applications in machine learning, and they cover many practical problems, including but not limited to image recognition, natural language processing, bioinformatics, and other fields.

Dear reviewer, thank you for your effort and response!

In our AdaMerging, the optimization goal of the merging coefficients is to reduce entropy on the test data of all tasks (we verified that it is an effective proxy objective for the supervised cross-entropy loss). Therefore, the model merging coefficients will be reasonably optimized, that is, in a direction that is conducive to improving the final performance.

In addition, we have the following suggestions for task selection. Before merging multiple pre-trained models, we can first check the similarity of any two task vectors. It only involves the calculation of the inner product of the vectors, which is very cheap. If there is no negative correlation between task vectors, then we can merge all tasks into one MTL model using Task Arithmetic/Ties-Merging/AdaMerging. If there are some negative correlations between task vectors, it means that they are not suitable to coexist in the same MTL model. We can group tasks, for example, first group tasks that do not conflict with each other (that is, the similarity is positive) into one group, other tasks into one group (or further divide into subgroups), and then merge the task vectors within each group to obtain multiple MTL models. Furthermore, we can group task vectors according to layer levels and form branch structures only on layers with negative correlations. This method can also further reduce the number of parameters of the model after grouping and merging.

4. About the question of "Task relationships across 8 tasks":

We have added similarity calculations for task vectors in the modified version (as shown in Figure 6 in the Appendix). As shown in the table, we have observed that a small number of task vectors have a certain degree of correlation (such as SVHN and MNIST, which are both digit recognition tasks). Most of the task vectors have low correlation, which means that these task vectors are almost orthogonal (this is consistent with Task Arithmetic). In addition, in the modified version, we further added task vector merging experiments with different degrees of correlation (such as merging SVHN with MNIST, GTSRB, SUN397 and EuroSAT, respectively). As shown in Figure 7, we observe that our AdaMerging is effective when performing model merging at various correlation levels.

5. About the question of "The negative merging weights":

We do not limit the value of merging coefficients $\lambda$. However, as shown in the table below, none of the coefficients learned in our experiments are negative. In extreme cases, it may be negative, but tasks with significant negative correlations generally do not merge together. Negative task vectors indicate a severe task conflict, which also means that these task vectors are unsuitable to coexist in a multi-task model.

6. About the question of "The relationship between $\lambda$ and performance":

Yes, $\lambda$ significantly impacts the accuracy of model mergin. For example, in Figure 1, different $\lambda$ are manually set. An inappropriate $\lambda$ will cause the performance of model merging to be unacceptably low. In Table 1 and Table 2, our AdaMerging is adaptively learned $\lambda$, and the coefficients learned by Layer-wise AdaMerging and AdaMerging++ are shown in Figure 5 of the paper. The coefficients learned by Task-wise AdaMerging and AdaMerging++ are shown in the following table.

7. About the question of "Move Figure 2 for ease of understanding":

We have moved Figure 2 to Introduction to facilitate readers' understanding.

Thank you very much for your review and affirmation of our work. We'll answer your questions one by one below.

1. About the question of "How does the (minimum) amount of test data affect performance":

Indeed, using all test data for model pooling coefficient training may not be feasible in some scenarios. Based on your suggestions, we experiment with the performance of AdaMerging when different numbers (such as 0.1%, 1%, 5%, 100%) of test sets are available, as shown in the table below. We observe that even when only 0.1% of the test data is available, AdaMerging and AdaMerging++ have about 5% performance improvement compared to Task Arithmetic and Ties-Merging. When 5% of the test data is available, AdaMerging and AdaMerging++ can almost reach the performance when all test data is available. This significantly improves the applicability of our work.

2. About the question of "The burden/computational needs/computational time":

In the table below, we show the time cost of AdaMerging compared to Task Arithmetic. The additional time cost of our complete AdaMerging is about two hours. However, our method also has a 2% improvement when the additional adjustment cost is only 7.5 minutes (based on a single GeForce RTX 3090) more than Task Arithmetic. When the extra cost is 50 minutes, our method has an 8% performance improvement.

In addition, we also show the parameter cost required to train our method. On ViT-B/32, there are only 8 new coefficients for task-level AdaMerging and 1248 new coefficients for hierarchical AdaMerging. This is trivial compared to the 907,589,640 parameters in the task vectors.

3. About the question of "Restriction or regularization on $\lambda$":

We do not require $\sum_i \lambda_i =1$, as shown in the visualization of the coefficients in the paper and the merging coefficients learned by Task-wise AdaMerging in the table below.

Dear reviewer, thanks for your time and response!

I think there are many possible reasons why there is still a gap between the current model merger and traditional MTL. This is mainly because the models for each task are trained in isolation, that is, they may use different learning rates, number of epochs, and other different hyperparameters to train the models. Naturally, the task vectors constructed for each task have many inherent differences, so there is inevitable information loss when merging modular task vectors.

For example, some tasks may converge to a sharp local minimum [1], that is, a slight perturbation of the weights may cause a significant change in the loss. Therefore, when the model is merged, the performance of this part of the task will decrease significantly, thereby affecting the overall performance. In addition, the local minimum points that different tasks converge to may be in different basins [2], and their weights may need to be rearranged and combined before merging for alignment. Finally, the size of the merged architecture also has a great impact on performance. For example, when merging larger architectures (they usually have better cross-task generalization), the performance of task vector based methods will be closer to that of traditional MTL. As shown in the table below, when merging ViT-L/14 (342,562,049 parameters per task vector) and ViT-B/32 (113,448,705 parameters per task vector), the gap is different. On ViT-L/14, AdaMerging++ can achieve an accuracy of 91.0%, which is very close to the 93.5% of traditional MTL.

In the future, we will explore these directions more deeply to further narrow the gap in model merging. This paper focuses on reducing this gap from the perspective of merging coefficients.

[1] Foret, Pierre, et al. "Sharpness-aware minimization for efficiently improving generalization." ICLR 2021.

[2] Ainsworth, Samuel K., Jonathan Hayase, and Siddhartha Srinivasa. "Git re-basin: Merging models modulo permutation symmetries." ICLR 2023.

5. About the question of "Reweighting/rebalancing traditional MTL model using task vector weights":

Traditional MTL baseline does not use a weighting strategy. Task vector-based model merging focuses on model merging without original data, rather than training an MTL model from scratch. Therefore, if original data is available, it would be a better choice to directly use the original data to train an MTL model (including task/loss weighting coefficients). In other words, it is unlikely that the scenario of learning the merging coefficients of a task vector first and then using these coefficients to train an MTL model from scratch will occur.

6. About the question of "Supervised variant of AdaMerging":

Thank you very much for your suggestion. We have added a supervised version of AdaMerging, that is, training AdaMerging directly with cross-entropy loss on the test dataset instead of entropy minimization. The experimental results are shown in the table below. We found that the unsupervised version of AdaMerging is very close to the supervised version of AdaMerging. This also shows from another perspective that it is reasonable for us to use entropy minimization as the proxy objective of cross-entropy loss to train the merging coefficients. In general, the performance gap between the task vector-based model merging methods and traditional MTL may be caused by a certain amount of information loss during the task vectors merging process.

Thank you very much for your review and affirmation of our work. We'll answer your questions one by one below.

1. About the question of "The experimental conclusion in Section 3.2.2 seems a bit strong":

Based on your suggestions, we have added a correlation analysis of entropy and loss at different training stages. As shown in the table below, the Spearman correlation coefficient of entropy and loss is always high, indicating that Entropy and Loss are positively correlated at different stages , and entropy minimization can be used as an effective proxy for cross-entropy loss to optimize the merging coefficient of AdaMerging. Experimental results of Spearman correlation coefficients on eight datasets are added to the modified version of Figure 10.

2. About the question of "The unrealistically accessible full test data":

Yes, access to the full test dataset may not be practical. Based on your suggestions, we added how the performance of our method changes when different amounts (0.1%, 1%, 5%, 100%) of the test dataset can be accessed. As shown in the table below, we observe that our method also has an improvement of about 5% when only 0.1% of the test data is accessible. When 5% of the test data is accessible, our method achieves comparable accuracy to when the whole test data is available.

3. About the question of "The ViT architecture and MTL baseline":

In the experiment, we strictly followed the ViT architecture used by Task Arithmetic and Ties-Merging. Task Arithmetic, Ties-Merging and our AdaMerging did not require the training data of the original task, and only needed the provided trained models to construct the task vectors. Each task vector can be understood as the ability to complete a specific task. Furthermore, these task vectors are merged into the pre-trained model through merging coefficients to achieve MTL. Of course, our approach is model-agnostic and can be used with foundation models of any architecture.

In addition, our method is compared fairly with baselines that merge models without original training data. As for more advanced MTL baselines such as GradNorm, PCGrad, GradDrop, etc., they are not the main baselines of the model merging community at this stage. Because there is still a gap between the current model merging and the traditional MTL baseline, these advanced MTL baselines need to be further considered only when this gap is further narrowed or even surpassed in the future. Of course, model merging technology has many advantages compared to traditional MTL methods. First, it does not require raw data for each task, significantly reducing the need for raw data collection and protecting data privacy. In addition, model merging is very efficient (especially for foundation models) compared with traditional MTL to train models from scratch, and it can usually be completed in minutes to tens of minutes.

4. About the question of "Overall summary":

(1) We compared the current SOTA advanced task vector based MTL methods, Task Arithmetic (ICLR'2023), and Ties-Merging (NeurIPS'2023). Advanced MTL methods based on raw data training, such as GradNorm (ICML'2018), PCGrad (NeurIPS'20), GradDrop (NeurIPS'20), are not applicable baselines in model merging problem since they require original training data. Methods based on raw data are not our most relevant baseline for two reasons. On the one hand, there is still a performance gap between current SOTA model merging schemes and the naive MTL method based on raw data, so the "learning from models" [1] community is still working hard to fill this gap. On the other hand, traditional MTL based on raw data has many shortcomings compared with model merging. For example, model merging does not require efforts to collect a large amount of original data, and it also protects data privacy. More importantly, model merging is very efficient compared to training an MTL model from scratch, especially in the era of large models.

(2) We further added correlation analysis (in Figure 10) between entropy and loss at different training stages, and we found that they are always highly correlated, which further supports our conclusion.

[1] Zheng, Hongling, et al. "Learn From Model Beyond Fine-Tuning: A Survey." arXiv preprint arXiv:2310.08184 (2023).

Dear Reviewer,

Thanks for your response.

Regarding the fair comparisons:

According to your suggestions, we conducted experiments on a held-out validation set without access to the unlabeled test data, using the identical setup as Task Arithmetic [1] and TIES-Merging [2]. This approach ensures a fair comparison with the model merging baselines [1,2] mentioned.. The results are shown in the following table (Due to time constraints, we only had time to perform 100 iterations, and the complete results will be updated later.).

In conclusion, by employing the same setup and ensuring the same and fair comparisons with model merging baselines [1,2], we demonstrate that utilizing the held-out validation dataset, without leveraging the unlabeled test data for coefficient tuning, yields comparable performance results.

Reference:

[1] Editing models with task arithmetic. ICLR 2023

[2] TIES-MERGING: Resolving Interference When Merging Models. NeurIPS 2023

Thank you very much for your review and affirmation of our work. We'll answer your questions one by one below.

1. About the question of "Intuitive motivation for entropy minimization":

The intuitive motivation for entropy minimization is to reduce model uncertainty. The lower the Shannon entropy, the more concentrated the probability distribution is and the more confident the model's predictions are. By minimizing entropy, we want the model to produce a more deterministic output when faced with a given input, resulting in a more robust and accurate model.

2. About the question of "Generalization and robustness of AdaMerging":

Our AdaMerging allows us to adapt to unlabeled test data of unseen tasks or unlabeled corrupted data in an unsupervised way when training model merging coefficients, thereby obtaining model merging coefficients that are most suitable for these data. Therefore, the AdaMerging merged model better bridges the gap between training and testing data distribution shifts. However, existing methods ignore this point. Therefore, our method will have better generalization and robustness compared with existing task vector-based model merging methods.

3. About the question of "AdaMerging version":

AdaMerging in Tables 2, 3, and 4 are all Layer-wise versions. As we introduced in the experimental settings in Section 4.1, all versions not specifically emphasized are Layer-wise.

4. About the question of "The relationship between model architecture and performance":

On the one hand, larger architectures usually have more parameters and more powerful representation learning capabilities, and larger architectures can better learn shared feature representations. These shared underlying features benefit knowledge transfer between different tasks, so the merged model performs better and is closer to traditional MTL. On the other hand, larger architectures have more layers, and our Layer-wise AdaMerging can have more learnable merging coefficients. For example, under ViT-B/32 and ViT-L/14, the number of coefficients for Layer-wise AdaMerging is 1,248 and 2,408, respectively. As a result, we can balance the task vectors more flexibly, leading to better performance.