Knowledge Fusion by Evolving Language Models

We first thank the reviewer for the detailed comments and insightful feedback, which will help us improve the quality of this paper. We address the concerns of weaknesses (W) and questions (Q) below:

W1: Thank you for asking for more clarity about the motivation. We rephrase the sentences in the manuscript to emphasize the main motivations and provide the responses to the Reviewer concerns below:

Multi-task learning necessitates abundant annotated data for all tasks concurrently during training, unlike model merging algorithms, which do not demand simultaneous data availability for each task. Regarding complexity, model merging doesn't mandate the costly retraining of models for all tasks simultaneously. Additionally, model merging can be viewed as an optimization aiming to discover the optimal strategy for merging multiple models to outperform any single model in isolation.

In practice, our proposed method, and the related work like RegMean perform well in the situation where the initial models are prepared, so usually the details how they are finetuned are not the bottleneck of the method.

W2: Thank you for the clarification. We have added the explanation in the caption of Table 1. Round means the number of times the models are edited when implementing a certain knowledge fusion method. The key step is renamed to key technique to highlight the difference between knowledge fusion methods.

W3: Thank you for the suggestions. We have restructured sections 2 and 3 as suggested to move the previous preliminary to related work. For section 5.4, the second part is indeed hyperparameter analysis. However, the first part is an ablation study that demonstrates the proposed model evolution can complement existing model merging methods for enhanced performance.

W4: Thank you for the suggestion. We have added the discussion in the manuscript on the related work [1] in Section 2 and Section 6. In addition, we have cited two important related work about robust finetuning [2] [3] in the manuscript for further reference.

[1] Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., & Farhadi, A. (2022). Editing models with task arithmetic. arXiv preprint arXiv:2212.04089.

[2] Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305, 2020.

[3] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7959–7971, 2022b.

We thank the Reviewer again for the useful comments. We have revised the manuscript according to the Reviewer’s suggestion and response to each comment provided in the Weakness section above. We hope that our rebuttal aligns with the reviewer’s expectations, and we hope that the Reviewer can consider possibly giving a higher rating. Thanks.

Thank you for recognizing our research work. We also thank the reviewer for the detailed comments and insightful feedback, which will help us improve the quality of this paper. We address the concerns of weaknesses (W) and questions (Q) below:

W1: Thank you for requesting more clarity on evolutionary algorithms (EA) and the related computing cost.

Proving convergence in EAs is challenging due to the stochastic and heuristic nature and is usually done using empirical studies and analyses.

Despite that, EAs are adept at exploring complex and high-dimensional search spaces, where traditional optimization or search algorithms might struggle (e.g., learning tasks with complex and non-differentiable objective functions). Hence, EAs have been successfully applied in various fields, including engineering, robotics, finance, biology, and more, solving complex real-world problems where traditional methods fall short. Moreover, EAs are complementary to other techniques and yield improved performance.

The computation cost is discussed in Appendix C.5. We found that the computation cost is reasonable, i.e., the evolution process can be completed within half an hour using either A6000 or A800 GPU. The ability to produce higher-quality solutions in diverse NLP tasks makes evolution process valuable for research and practical applications. Moreover, we demonstrate that combining EA with existing methods yields more efficient solutions in our ablation study (Section 5.4). Hence, continued research can aim at improving EAs' efficiency, scalability, and applicability across a broader range of problem domains.

W2: Thank you very much for this insightful suggestion. We have incorporated additional architectures, T5-base and GPT2, into our experiments, as suggested by Reviewers 1 and 3. The new results have been added to Tables 2 and 4, demonstrating the consistently superior performance of our proposed method compared to existing methods. This reinforces the generalizability of our approach to virous neural architectures.

W3: Thank you for pointing out the issue. We acknowledge the need for a clearer presentation and have revised the manuscript accordingly. Concepts, such as multi-task learning and the evolutionary algorithm for optimization, have been explained in more detail in the Introduction. We also discuss more details about compared method and experiment settings in Appendix C. Also, we provide explanation to the Reviewer’s questions in the following and revised the manuscript accordingly. We believe these suggested changes have improved the overall quality of the presentation.

Q1: Thank you for the question. The implementation details of combining evolver with other model merging methods are established in the computation of the population's scores after mutation and crossover. The combining approach involves calculating an overall score by using model merging on the mutated individuals along with the non-mutated individuals in the current evolution. In contrast, the simple evolver determines the success of mutation based on the score of individual entities, while combining assesses it based on the score of the entire population after individual mutation. We added a paragraph in Appendix A and details are shown in step 3 (model inference) in Algorithm 1. For further reference, the reader could also refer to the source code.

Q2: Thanks for requesting more clarity on description of abbreviations. We now provide more explanation in Appendix C.2 accordingly to enhance readability. The following are the details:

Avg. $f1*..N$ shows the average performance of all individual models. Best. $f1..N$ shows the performance of the best single individual model, as determined by using the validation set. Domain-Specific shows the performance of the individual models corresponding to the training data set for each test set. MTL shows the performance of the model trained with multi-task learning method.

Q3: Thank you for the careful check. The details of Section 5.1.3 are in Appendix C.1 instead of Appendix B. We have corrected the manuscript accordingly.

Q4: The computational cost of our proposed method has been detailed in Section C.5 of the supplementary materials. Our results indicate that model merging can be completed within an hour with moderate computing power, underscoring the practicality of our approach.

We thank the Reviewer again for the useful comments. We have revised the manuscript according to the Reviewer’s suggestion and response to each comment provided in the Weakness section above. We hope that our rebuttal aligns with the reviewer’s expectations, and we hope that the Reviewer can consider possibly giving a higher rating. Thanks.

We first thank the reviewer for the detailed comments and insightful feedback, which will help us improve the quality of this paper. We address the concerns of weaknesses (W) and questions (Q) below:

W1: Thank you for the point out the issue. We have re-designed diagrams and tables to avoid potential confusion with paper [1]. Our motivation of taking reference from the experimental setup in the paper [1] is mainly for fair and sufficient comparison with previous methods. The clear comparison could allow us to emphasize the main contribution of our paper that is the newly proposed model evolution algorithm. Furthermore, to further we have used a new experiment setup to compare the performance with coefficient search in Appendix D. This helps further improve the overall novelty of our paper.

We have revised Fig.1. Since the parts on ensembling and model merging are not our contribution, we remove the part of ensembling and clarify that model merging is the previous method and model evolution is our proposed method. In Table 2 and Table 4, we have added the results of models with encoder-decoder architecture and decoder-only architecture. The modified figures and tables are shown in our new version of the paper. Lastly, we have modified a few minor writing issues about citations, i.e., we remove the parenthesis of citation when the author is included in the sentence. We hope that these changes have addressed the Reviewer's concerns about the presentation of our manuscript.

W2/ Q1: Thank you for the insightful suggestion of using coefficient search. We are grateful that the Reviewer raises this essential concern, which could potentially inspire the development of novel algorithm.

We would like to answer this question from both the theory and experiment aspects. Theoretically, a coefficient search can be treated as a method with greedy rounds aiming to obtain better efficiency for different models to merge, but a vanilla coefficient search method does not consider the crossover ratio for a better search result.

We have conducted the experiment to investigate the coefficient search. The coefficient search is a promising scheme to improve the model merging performance, by searching the optimal $\alpha$. We find that the proposed model evolution can also be integrated with the coefficient search method by searching the optimal scale factor $f$. We have performed the grid search of $alpha$ and $f$ with intervals of 0.05 from 0.1 to 0.9. We present the result in Appendix D. In the setting of Simple, Fisher and RegMean, the results show that a default version of evolver (with scale factor $f=0.5, cr=0.5$) outperforms the coefficient search results. Notably, the integration with scale factor search could further boosts the performance of the evolver, which is worth further investigation.

We thank the Reviewer again for the useful comments. We have revised the manuscript according to the Reviewer’s suggestion and response to each comment provided in the Weakness section above. We hope that our rebuttal aligns with the reviewer’s expectations, and we hope that the Reviewer can consider possibly giving a higher rating. Thanks.

We first thank the Reviewer for recognizing the contribution of our work and the insightful feedback, which will help us improve the quality of this paper. We address the concerns of weaknesses (W) and questions (Q) below:

W1: Thank you for the suggestion. We have incorporated the discussion and highlighted the main selling point of model merging for data privacy, as our limitation in the revised manuscript in Section 5. Our method can incorporate additional consideration for data privacy in the development data as future work. Furthermore, we added more discussion about development data in Appendix B.

W2: Thank you very much for this insightful suggestion. We have incorporated additional architectures, T5 and GPT2, into our experiments, as suggested by Reviewers 1 and 3. The new results have been added to Tables 2 and 3, demonstrating the consistently superior performance of our proposed method compared to existing methods. This reinforces the generalizability of our approach to several neural architectures.

Q1: Thank you for the suggestion. We proofread the paper carefully to improve on Grammar and correct Typos. The main typos are listed as follows:

Faderated > Federated; implementated > implemented; pairewise > pairwise; Mode > Model; solved the format issue of Algorithm 1 in Appendix A.

We thank the Reviewer again for the useful comments. We have revised the manuscript according to the Reviewer’s suggestion and response to each comment provided in the Weakness section above. We hope that our rebuttal aligns with the reviewer’s expectations. The revised manuscript has proved the generalizability of the proposed method to more neural architects and the presentation has been improved according to Reviewers’ suggestions. We hope that the Reviewer can consider possibly giving a higher rating. Thanks.