FuseChat: Knowledge Fusion of Chat Models

Q4: Regarding the imbalanced performance achieved by FuseChat in different domains.

A4: We appreciate the reviewer’s concern regarding the imbalanced performance in different domains. The performance of FuseChat across different domains is largely determined by two key factors: the availability of domain-specific training data and the competency of the source LLMs in those domains. (The domain distribution of our training data is summarized in the table below.) As shown in Figure 3 of the paper, the pairwise fusion process significantly enhances performance in domains like Math and Coding, where the source LLMs are particularly strong and ample domain-relevant training data is available. In contrast, performance is comparatively lower in domains where the source LLMs are less proficient (e.g., Extraction) or where domain-specific training data is sparse (e.g., STEM). This detailed analysis will be included in the revised version.

Thank you for reviewing our paper and providing insightful feedback. We’re glad you find our work logical and convincing and will address your concerns point by point.

Q1: Regarding the challenges of knowledge fusion and FuseChat.

A1: Thank you for your insightful feedback. We appreciate the opportunity to clarify and expand upon this point. In the revised manuscript, we will emphasize the challenges and complexities faced by FuseChat from two key perspectives. First, unlike direct supervised fine-tuning, knowledge fusion introduces additional training costs, such as the computational overhead of inferencing results from the source LLMs. This added step significantly increases resource and time requirements. Second, knowledge fusion methods like FuseChat encounter inherent challenges, including vocabulary alignment across different LLMs and the merging of their distribution matrices. These processes are non-trivial and can introduce noise and errors, which in turn may impact the quality of the fusion results. By addressing these aspects, we aim to provide a more comprehensive discussion of the intricacies involved in developing FuseChat.

Q2: Regarding the details and formulas for our token alignment method.

A2: Token alignment is designed to address the mapping challenges between probabilistic distribution matrices generated by different source LLMs for a given instruction. This alignment occurs along two critical dimensions: sequence and probability distribution. For sequence alignment, we employ dynamic programming to effectively map the tokenized sequences from the source LLM to those of the pivot LLM. For distribution alignment, we propose to utilize mapping statistics (MS) from the sequence dimension as the criteria for alignment in the distribution dimension. To enhance clarity, we have included a visual illustration of our token alignment method in Figure 7 (Appendix A). We will further refine this explanation by introducing explicit mathematical formulations in the revised revision.

Q3: Regarding the motivation and details of the SCE algorithm.

A3: The motivation of our SCE is to design a simple merging strategy to automatically identify and incorporate the learned advantages from diverse target LLMs while simultaneously resolving knowledge conflicts in the parameter space, without the need for additional parameter tuning.

To achieve this, we utilize weight updates from the pivot LLM to various target LLMs during the model fusion process, employing these updates as fusion vectors that reflect diverse advantages from different models. The weight merging for each parameter matrix unit in the target LLMs is carried out through a three-step procedure:

1. Fusion vectors for each unit parameter matrix, derived from various target LLMs, are intended to capture distinctive and significant strengths of these models. To emphasize the most impactful features, we select the top τ% of elements from each parameter matrix-level fusion vector. This selection is based on the elements exhibiting the highest variances across multiple target LLMs, as these variances are indicative of the most significant differences and strengths among the models.
2. We then compute a matrix-level merging coefficient for each target LLM based on the sum of squares of elements in their respective selected fusion vectors.
3. To mitigate knowledge interference across different target LLMs, we implement a conflict resolution mechanism. This entails eliminating elements with minority directions when the signs of weight updates are in opposition.

We acknowledge that the SCE merging method involves technical complexity. Due to space constraints in the initial submission, we were unable to elaborate further on its specifics. In the revised version, we will provide a more detailed and comprehensive explanation to ensure clarity and address this complexity effectively.

Dear Reviewer,

I noticed that you haven't yet responded to the author's rebuttal. As November 26 is the last day for reviewers to ask questions to authors, could you please review their responses and provide your feedback?

Your timely response will ensure a thorough evaluation process and help with making the final recommendation. Thank you for your prompt attention to this matter.

Area Chair

Dear Reviewer UpRG,

We sincerely appreciate the time and effort you have devoted to providing thoughtful reviews and valuable feedback. We have carefully addressed your concerns in detail and incorporated additional experiments and analyses, as summarized in the discussion:

• Demonstrated the challenges of knowledge fusion and FuseChat.
• Detailed the token alignment and SCE merging methods.
• Explained the imbalanced performance achieved by FuseChat in different domains.

We hope these revisions and discussions have adequately addressed your concerns. As the Author-Reviewer discussion phase is ending soon, we would be grateful for any additional comments or questions that could further enhance our work. Thank you again for your time and effort.

Best regards,

Authors

Dear Reviewer UpRG,

Thank you for your time and detailed feedback on our manuscript. As the reviewer-author discussion period ends today (December 2nd at 11:59 pm AoE), we would like to check if we have adequately addressed all your concerns.

Your insightful comments and questions have been instrumental in improving our work. We have carefully incorporated your feedback into the revised manuscript and hope that our responses and updates have successfully addressed all the points you raised.

We understand you have a busy schedule, but if you have any remaining questions or need further clarification, please let us know, and we will address them promptly. If you feel that we have satisfactorily addressed your concerns, we would greatly appreciate it if you could consider updating your initial score.

Thank you again for your valuable time and constructive feedback that has helped enhance the quality of our work.

Best regards,

The Authors of Paper 6352

Q4: Regarding the statistical significance of performance improvements.

A4: We appreciate the reviewer’s concern regarding the statistical significance of the performance improvements. To address this, we conducted a detailed statistical analysis using t-tests to evaluate the performance of the final FuseChat model against pairwise fusion on MT-Bench. Moreover, we performed a similar analysis comparing the final FuseChat model with OpenChat-3.5-7B Multi, which integrates multiple source LLMs simultaneously, as FuseLLM does. The results summarized in the following table demonstrate the strong statistical significance of the final FuseChat model's superiority over these baselines. These results will be included in the revised paper to enhance clarity and provide robust support for our claims.

Q5: Regarding claims that require further clarification.

A5: We sincerely thank the reviewer for highlighting this important point. Regarding the claim about the limitations of FuseLLM, we wish to clarify that while FuseLLM's experiments were constrained to three source LLMs of an equivalent 7B scale, our work broadens the scope by incorporating six source LLMs with varying scales, ranging from 7B to 72B. We will ensure that these claims are conveyed more clearly in the revised manuscript.

Q2: Regarding the difference between pairwise fusion and single-model distillation.

A2: Thank you for raising this insightful point. The key distinction between pairwise fusion and single-model distillation lies in their respective learning paradigms. In pairwise fusion, the model selectively acquires knowledge based on the quality of outputs from the source LLM or the pivot LLM, guided by lower cross-entropy (CE) values. This mechanism ensures that the model learns from the stronger performer in each sample. In contrast, single-model distillation relies exclusively on the source LLM, implicitly assuming that the source LLM consistently provides the superior results.

To address your comment more rigorously, we conducted additional experiments comparing the two approaches. Specifically, we replaced the pairwise fusion strategy in FuseChat with direct distillation from a single (source) model, skipping the merging phase for direct comparison. The results summarized in the table below demonstrate that pairwise fusion consistently outperforms single-model distillation across five source LLMs. For clarity, D/P represents the results from direct distillation and pairwise fusion, respectively. Metrics reported include the Average Score on MT-Bench and the Length-Controlled Win Rate on AlpacaEval-2.0.

We further applied our proposed SCE method to fuse the models obtained through single-model distillation. The results below reveal that merging the models derived from pairwise fusion produces a superior fused model compared to merging models from single-model distillation.

These results highlight the effectiveness of the pairwise fusion approach, not only in standalone performance but also in enhancing the quality of the final fused model. We appreciate your attention to this critical aspect and hope these findings provide additional clarity.

Q3: Regarding the rationale behind the 0.9/0.1 weight distribution in Equation 4.

A3: We appreciate the reviewer’s thoughtful observation regarding the rationale behind the 0.9/0.1 weight distribution in Equation 4. This choice is informed by the significant difference in magnitude between the SFT loss and the fusion loss. To further clarify, we conducted a new experiment using Qwen-1.5-Chat-72B as the source LLM and 128 instances randomly sampled from the training dataset. The resulting loss values for SFT and fusion are summarized in the table below.

The results show that the fusion loss is approximately three times larger than the SFT loss in this experiment. This notable disparity underscores the importance of assigning a proportionally smaller weight to the fusion loss in Equation 4. Without this adjustment, an excessively high weight for the fusion loss could amplify the imbalance, potentially skewing the training process. Thus, the 0.9/0.1 distribution reflects a principled approach to mitigating this effect and achieving a balanced optimization.

Thank you for reviewing our paper and providing insightful feedback. We're glad you find our work well-motivated and practical. We will address your concerns in the following points.

Q1: Regarding the technical innovation and contribution of FuseChat.

A1: We thank the reviewer for the feedback regarding the technical contributions of our work. We would like to emphasize the uniqueness of FuseChat and clarify its distinctions from prior works such as FuseLLM and TIES from multiple perspectives:

1. Distinction from FuseLLM

a. Motivation While FuseLLM emphasizes the fusion of multiple base LLMs through continual pre-training, FuseChat focuses on integrating diverse chat-oriented LLMs into a unified chat model via supervised fine-tuning. This difference in both training objectives and data makes FuseChat essential in the context of chat-focused LLMs. Moreover, our work extends beyond FuseLLM’s scope by fusing six distinct chat LLMs (as opposed to FuseLLM’s three base models), thereby demonstrating the scalability and depth of our methodology.

b. Methodology While FuseLLM directly employs multi-teacher distillation to fuse multiple base LLMs, FuseChat employs a sophisticated fuse-and-merge approach, beginning with pairwise fusion and advancing to our SCE merging strategy. This new method is not only highly scalable and efficient but also better resolves knowledge conflicts in the parameter space. Simultaneously, it integrates the strengths of each source LLM with precision. By adopting this refined approach, FuseChat noticeably enhances the final model’s performance, distinguishing it from the techniques employed by FuseLLM.

c. Scalability Another key strength of FuseChat lies in its plug-and-play approach for integrating new LLMs, which is more efficient than FuseLLM. Instead of combining distribution matrices from all source LLMs, FuseChat merges the distribution matrices of the new source LLM with a pivot LLM. This streamlined process reduces computational and storage costs, ensuring superior scalability as the number of LLMs increases.

d. Experimental Validation Our experimental setup demonstrates the distinct focus of FuseChat. By fusing six varied chat LLMs (OpenChat-3.5-7B, Starling-LM-7B-alpha, NH2-SOLAR-10.7B, InternLM2-Chat-20B, Mixtral-8x7B-Instruct, and Qwen-1.5-Chat-72B), we validate FuseChat on AlpacaEval 2.0 and MT-Bench, assessing both instruction-following and conversational capabilities. This is in contrast to the base-model-focused experiments of FuseLLM and underscores the tailored contributions of FuseChat to the domain of chat LLM fusion.

2. Distinction from the TIES merging method

Our SCE merging strategy introduces considerable innovations compared to the TIES merging method:

a. Automation and Precision Unlike TIES, which relies on manually tuned, model-level coefficients, our SCE automates the merging process by leveraging weight updates from a pivot LLM and computing matrix-level coefficients. This enables the fine-grained incorporation of diverse benefits across LLMs, which is difficult to achieve with manual hyperparameter tuning.

b. Nuanced Parameter Adjustments In our specific context, where target LLMs are trained on identical datasets with relatively subtle parameter variations, SCE excels at capturing and preserving the distinctive advantages of each LLM through nuanced matrix-level parameter updates.

c. Superior Performance Experimental results (e.g., Table 4) demonstrate that SCE outperforms baseline merging techniques including TIES within our framework, validating its efficacy and impact.

We will incorporate these detailed discussions into the revised manuscript to provide a clearer distinction of our work from previous approaches.

Thank you for reviewing our paper. We greatly value your insightful feedback and appreciation of our work's significance. Below, we address your concerns in detail.

Q1: Regarding the significance test in our experiments.

A1: Thank you for raising this important point. To show the statistical significance of our results, we conducted a t-test on MT-Bench to compare the performance of our proposed FuseChat-7B with OpenChat-3.5-7B Multi, which fuses multiple source LLMs simultaneously. The results shown in the table below reveal a p-value well lower than 0.05. This confirms that FuseChat-7B achieves statistically significant improvements over OpenChat-3.5-7B Multi. These statistical results will be incorporated into the revised manuscript.

Q2: Regarding the caption of Table 1.

A2: Thank you for your thoughtful suggestion. We will address this in the revised version by improving the caption for Table 1. Specifically, we will clarify that the bold font denotes the best performance among all the fused LLMs, while the underscore indicates the second-best performance. Moreover, the percentages represent the relative performance improvement compared to the OpenChat-3.5-7B SFT baseline model. We believe these clarifications enhance the table’s interpretability and precision.

Q3: Regarding the details of pivot LLM in Figure 3.

A3: Thank you for your observation. As clarified in Section 3.1 (lines 179–190), the term pivot LLM refers to the original OpenChat-3.5-7B model prior to the application of pairwise fusion, and the term target LLM describes an intermediate model generated through pairwise fusion between the pivot LLM and an individual source LLM. Our approach first performs pairwise fusion between the pivot LLM and each source LLM independently, resulting in a series of corresponding target LLMs. These intermediate models are then combined using our SCE merging technique to create the final FuseChat model. We hope this explanation resolves any potential ambiguity.

Q4: Regarding the domain distribution of samples in training data.

A4: We appreciate the reviewer’s concern regarding the domain distribution of samples in the training data. We followed the approach described in Magpie [1] and employed the Llama-3-8B-Instruct model to classify our 95,000 training examples into eight distinct domains as defined by MT-Bench. After excluding approximately 7,000 samples due to anomalous classification errors, the final domain distribution is summarized in the following table:

The resulting data distribution demonstrates substantial diversity, which aligns with our primary objective to assess the model's general capabilities rather than domain-specific performance. As shown in Figure 3 of the paper, pairwise fusion leads to a marked improvement in the target LLMs' math and coding abilities. This enhancement is primarily due to the strong performance of the source LLMs in these domains, coupled with the relatively high proportion of Math and Coding samples in our dataset. Interestingly, despite a notable representation of the Extraction domain in the dataset, the target LLMs show limited improvement in this area. This can be attributed to the relatively weaker performance of the source LLMs in extraction tasks, highlighting the critical role of selecting appropriate source LLMs for domain-specific objectives. We will integrate this detailed analysis in the revised manuscript to provide further clarity.

[1] Xu et al. Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing, 2024.