Parameter Competition Balancing for Model Merging

Reply to Reviewer Zv3Q
Thank you for your valuable comments. We will explain your concerns point by point.

Weaknesses 1：About the novelty.
Reply: Please refer to the second point in our general response document.

Weaknesses 2：About the theoretical motivation.
Reply: Please refer to the first point in our general response document.

Question 1: About the function of CMA-ES.
Reply: In fact, the function of CMA-ES is not to adjust the overall scaling hyperparameter $\lambda$. It is used to further search for the coefficients $\lambda_i$ for each task, as shown in Equation 5 in Section 3.3 of our paper. Typically, we use a uniform $\lambda$ as the initialization value for each model's $\lambda_i$, and then employ evolutionary strategies (ES) to search for a more accurate $\lambda_i$. CMA-ES is used to accelerate this search process. Recently, evolutionary strategies have also been widely used to enhance model merging in works like Lorahub [8], EvoLLM [9], and Model_Evolver [10].

Question 2: Fisher Merging and RegMean for the 3 LLM Tasks in Table 2.
Reply: The methods of Fisher Merging and RegMean are actually not suitable for LLMs. Firstly, these methods require more GPU resources, as they necessitate the additional computation of the Fisher Information Matrix or Inner Product Matrix. Specifically, the RegMean method is not feasible even with GPUs having 80GB of memory for 7B LLMs. Moreover, both methods are gradient-based, making the results highly sensitive to the choice of small sample data and the number of training iterations, which significantly increases the complexity and instability of LLM merging. Considering these factors, we primarily compare methods based on task vectors for LLM merging, as they are easier and more lightweight to implement.

Question 3: About softmax and Norm.
Reply: It is true that the softmax is done across all parameters per task vector. Norm refers to the normalization of the task vector, which enhances numerical stability. The choice of softmax is largely due to the exponential function it incorporates, which amplifies larger values and diminishes smaller ones, thus increasing the contrast. This helps in converting the output vector into a probability distribution. In our paper, Appendix B.1 (Additional Ablation Studies), we replaced the softmax activation function with common alternatives like sigmoid, ReLU, and tanh. The results show minimal performance loss with different activation functions. This is because these activation functions can represent complex nonlinear relationships to balance the values of parameters.

Question 4: About the number of searches.
Reply: Please refer to the fourth point in our general response and table 1 in PDF file in general response.

Question 5: Should j exclude i.
Reply: We considered this issue at the initial design stage of our experiment. In our early experiments with merging eight models, we found that whether or not to exclude the model itself when computing inter-balance had a negligible impact on the results. Therefore, we opted for a simpler approach by not excluding it, which made the process straightforward and the formulas more concise. In our current experiments merging three LLM tasks, we found that excluding the model itself resulted in a score of 35.14, which is a slight improvement compared to the original score of 35.12. This suggests that this issue has minimal impact on our method. The term "inter-balance" can be understood as the balance between the model and the entire population, or the balance among the other individuals within the population.

We thank the Reviewer again for the useful comments. We have revised the manuscript according to the Reviewer’s suggestion and response to each comment provided in the Weakness section above. We hope that our rebuttal aligns with the reviewer’s expectations, and we hope that the Reviewer can consider possibly giving a higher rating. Thanks.

References:
[8] Huang et al. LoraHub: Efficient cross-task generalization via dynamic LoRA composition. (arXiv23)
[9] Akiba et al. Evolutionary optimization of model merging recipes. (arXiv24)
[10] Du et al. Knowledge fusion by evolving weights of language models. (ACL24)
[11] Yu et al. Language models are super mario: Absorbing abilities from homologous models as a free lunch. (ICML24)

Reply to Reviewer s8Ho
Thank you for your valuable comments. We will explain your concerns point by point.

Weaknesses 1：More details in Figure 1 and 2
Reply: The term 'magnitude' refers to the magnitude of the task vector in Fig. 1. We will clarify this in the introduction of our final version by adding explanatory details. Thank you for your suggestions. Figure 1 illustrates an interesting phenomenon: scaling the top percentiles of a task vector outperforms fine-tuning, which is consistent with the ideas presented in the paper DARE [11]. \ The percentage represents the top magnitude percentiles of a task vector. It pertains to both the pruning ratio $r$ and the proportion of adjusted key parameters shown in Fig. 2, reflecting different representations in drop and rescale operations. However, in the later part where we introduce our new method, PCB-Merging, we first adjust all parameters and then perform pruning based on these adjustment results.

Weaknesses 2：Time complexity in MOEA
Reply: The total time required for the overall evolutionary strategy is $T_{\text{total}} = \text{Generations} \times (T_{\text{merging}} + T_{\text{validate}})$, where "Generations" represents the number of generations needed for evolution. The time required for model merging primarily depends on the number of model parameters and the size of the model population, while the time for model validation is mainly influenced by the volume of inference data and inference speed. We have compiled and reported the number of generations and the time required for each task in Table 1 of the general response PDF, and we analyze these factors in the fourth point of the general response. Recently, evolutionary strategies have also been widely used to enhance model merging in works like Lorahub [8], EvoLLM [9], and Model_Evolver [10].

Weaknesses 3：Definition of circle with dot
Reply: We interpret ⊙ as an element-wise product and use Norm as a shorthand for normalization. For more details, please refer to the third point in our general response document.

Weaknesses 4：Code for processing the task vector
Reply: Actually, we have provided the source code in the supplemental material .zip file during our initial submission. The details of our method are shown in the file pcb-merging.py. Additionally, you can check the application in different scenarios with evolutionary strategies in pcb_ES.py (found in the vision_source_code directory) or merging.py (in the nlp_source_code directory). You can obtain the model population by executing run_finetuning.sh and try different merging methods using run_merging.sh.

We thank the Reviewer again for the useful comments. We have revised the manuscript according to the Reviewer’s suggestion and response to each comment provided in the Weakness section above. We hope that our rebuttal aligns with the reviewer’s expectations, and we hope that the Reviewer can consider possibly giving a higher rating. Thanks.

References:
[8] Huang et al. LoraHub: Efficient cross-task generalization via dynamic LoRA composition. (arXiv23)
[9] Akiba et al. Evolutionary optimization of model merging recipes. (arXiv24)
[10] Du et al. Knowledge fusion by evolving weights of language models. (ACL24)
[11] Yu et al. Language models are super mario: Absorbing abilities from homologous models as a free lunch. (ICML24)

Reply to Reviewer 28gt
Thank you for your valuable comments. We will explain your concerns point by point.

Weaknesses 1：More details in Figure 1.
Reply: The term 'magnitude' refers to the magnitude of the task vector in Fig. 1. We will clarify this in the introduction of our final version by adding explanatory details. Thank you for your suggestions. Figure 1 illustrates an interesting phenomenon: scaling the top percentiles of a task vector outperforms fine-tuning, which is consistent with the ideas presented in the paper DARE [11].

Weaknesses 2：Clarification regarding the notation for Softmax and Norm in Section 3.2.
Reply: Please refer to the third point in our general response document.

Limitation 1：Theoretical understanding.
Reply: Please refer to the first point in our general response document.

Limitation 2：Reliance on shared initializations for applications.
Reply: Please refer to the second point in our general response document. Shared initializations are a fundamental issue in model merging, and more complex scenarios can be addressed by converting them into shared initializations through methods like FuseLLM [7]. Common applications include multi-task learning [1], multi-domain adaptation [2], merging various training strategies [3], model compression [4], and mitigating catastrophic forgetting [5], among others.

We thank the Reviewer again for the useful comments. We have revised the manuscript according to the Reviewer’s suggestion and response to each comment provided in the Weakness section above. We hope that our rebuttal aligns with the reviewer’s expectations, and we hope that the Reviewer can consider possibly giving a higher rating. Thanks.

References:
[1] Ilharco et al. Editing models with task arithmetic. (ICLR 2023)
[2] Jin et al. Dataless knowledge fusion by merging weights of language models. (ICLR 2023)
[3] Yadav et al. Ties-merging: Resolving interference when merging models. (NeurIPS 2023)
[4] Wang et al. Localizing task information for improved model merging and compression. (ICML24)
[5] Zhu et al. Model tailor: Mitigating catastrophic forgetting in multi-modal large language models. (ICML24)
[7] Wan et al. Knowledge fusion of large language models. (ICML24)
[11] Yu et al. Language models are super mario: Absorbing abilities from homologous models as a free lunch. (ICML24)

Reply to Reviewer 6PSY
Thank you for your valuable comments. We will explain your concerns point by point.

Weaknesses 1：The setting of the hyperparameter $r$.
Reply: The reviewer has some confusion and misunderstandings regarding our experiment settings. Firstly, our hyperparameters are discussed in two scenarios: When no additional validation is performed, the masking ratio for TIES-Merging and PCB-Merging is set to $r = 0.2$. When validation is allowed, we search over ratios in $\{0.05, 0.1, 0.2\}$. Additionally, in Figure 6, the optimal $r$ is not 0; it should be 0.05 instead. This conclusion is consistent with our findings in Appendix F.3, Table 17, concerning hyperparameter settings.
In fact, when $r$ is close to 0, the performance of TIES-Merging and PCB-Merging drops sharply, as shown in Figure 1 in the general response PDF file. The optimal range for $r$ is between 0.03 and 0.3. To improve feasibility, we only search within the values $\{0.05, 0.1, 0.2\}$.

Weaknesses 2：The rationale for the various settings of different methods.
Reply: In this paper, the selection of hyperparameters for different methods adheres to the original papers and is indeed suitable for obtaining optimal solutions for these methods. Therefore, this comparison is fair and reasonable. Specifically:
For TIES-Merging, we followed the hyperparameter settings outlined in Appendix C.4 and C.5 of the original paper.
For Task Arithmetic, we adhered to the hyperparameter settings in Appendix D (Learning via addition) of the original paper.
For RegMean, we followed the discussion on the impact of scaling non-diagonal values in Inner Product Matrices in section 5.3 of the original paper.
Lastly, since the meaning of the hyperparameter $\lambda$ varies across different methods, we summarized the range and optimal values of hyperparameters for each method in our paper. This is detailed in the Experimental Setup section and Appendix F.3, Table 17.

Weaknesses 3：Reliance on shared initializations for applications.:
Reply: Please refer to the second point in our general response document. Shared initializations are a fundamental issue in model merging, and more complex scenarios can be addressed by converting them into shared initializations through methods like FuseLLM [7]. Common applications include multi-task learning [1], multi-domain adaptation [2], merging various training strategies [3], model compression [4], and mitigating catastrophic forgetting [5], among others.

We thank the Reviewer again for the useful comments. We have revised the manuscript according to the Reviewer’s suggestion and response to each comment provided in the Weakness section above. We hope that our rebuttal aligns with the reviewer’s expectations, and we hope that the Reviewer can consider possibly giving a higher rating. Thanks.

References:
[1] Ilharco et al. Editing models with task arithmetic. (ICLR 2023)
[2] Jin et al. Dataless knowledge fusion by merging weights of language models. (ICLR 2023)
[3] Yadav et al. Ties-merging: Resolving interference when merging models. (NeurIPS 2023)
[4] Wang et al. Localizing task information for improved model merging and compression. (ICML24)
[5] Zhu et al. Model tailor: Mitigating catastrophic forgetting in multi-modal large language models. (ICML24)
[7] Wan et al. Knowledge fusion of large language models. (ICML24)