Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging

We are encouraged that you find UpIT is reasonable and easy-to-follow. Thanks for the valuable comments that could help us improve the work. We will respond to the questions, and we would appreciate it if you could kindly raise the scores when the concerns are addressed.

W1: Lack of ablation study on m and n in Algorithm 2.

Thanks for your insightful comments. In Figure 4, we explore ablation studies with different values of m and different activation experts, and the detailed results are provided in Table 5. It can be found that UpIT achieves excellent performance under different total and activated experts. Regarding the ablation study on n, we are trying our best to supplement the experiment and will provide it soon.

Q1.1: How can we ensure that the preparation of specialized experts effectively captures diverse expertise.

As shown in Figure 2 and Table 6,7, The intermediate checkpoints in the instruction fine-tuning stage are naturally different and do not require special operations or attention. It introduces our first contribution of using intermediate checkpoints for expert preparation and corresponding expert expansion strategies. On this basis, we propose a router initialization stage with expert-specific data selection, utilizing and amplifying the differences among experts in the initialization phase. The better performance in Table 1 and data-efficient characteristics in Figure 3 confirm the above statement.

Q1.2: Would experts trained on domain-specific or contextually diverse subsets offer improved performance or encounter challenges in maintaining consistent expertise?

Specialized upcycling trains a specific expert for each domain, which lacks flexibility as it is difficult to divide a mixed general dataset into separate domains. In constant, UpIT considers more universal scenarios, in which data comes from a mixed set and cannot be effectively divided. Traditional specialized upcycling methods are no longer applicable, while UpIT achieves efficient and flexible upcycling by seeking natural differentiation between intermediate checkpoints.

Q2: Forgetting issues in expert specialization.

Thanks for your suggestion. We conduct experiments by incorporating the pre-trained model as an expert, in both configurations: one without and one with an expert expansion stage.

Compared to Llama 2-7B, the fine-tuned models exhibit a performance decline on the TriviaQA benchmark. By incorporating the pre-trained model into the UpIT pipeline (note that only intermediate checkpoints are used in the paper), there is a considerable recovery in performance on TriviaQA. However, the performance of other benchmarks are declined, especially on HumanEval and GSM8K, as the pre-trained model is not satisfactory on these tasks. It raises the issue of the trade-off between downstream performance and pre-trained knowledge, and we believe is a topic worthy of in-depth exploration in future endeavors.

---

Thank you once again for your valuable feedback we are committed to refining our work based on your insights.

If there are any remaining concerns or questions that we have not yet addressed, please do not hesitate to let us know. We are more than willing to provide any additional materials or clarifications that may be necessary.

W1: Lack of ablation study on m and n in Algorithm 2.

We have completed the ablation study for n, as shown in the table below.

We employ the expert expansion phase to incorporate 2, 4, and 6 new experts, respectively. The results demonstrate that with the progressive integration of additional experts, there is a corresponding enhancement in the average performance.

---

Thank you once again for your valuable feedback we are committed to refining our work based on your insights.

If we have addressed your concerns, we would be delighted if you could consider raising the score.

Thank you very much for your recognition of our work. Thanks for the valuable comments that could help us improve the work. We will respond to the questions, and we would appreciate it if you could kindly raise the scores when the concerns are addressed.

W1: In Figure 2, there is not much difference in performance for certain benchmarks.

In this paper, we aim to emphasize the importance of expert discrepancy in upcycling and incorporate the idea into the entire design of UpIT. Figure 2 first highlights the different checkpoints during instruction tuning that are inherently suitable for constructing specialized experts, since different checkpoints excel at different benchmarks, which introduces our first contribution of using intermediate checkpoints for expert preparation and corresponding expert expansion strategies. On this basis, we propose a router initialization stage with expert-specific data selection, utilizing and amplifying the differences among experts in the initialization phase. Overall, the initial differences among experts are the foundation of UpIT, and we fully utilize and amplify these differences to improve performance and data efficiency.

Note that we present the results of multiple benchmarks in a single diagram, with a wide y-axis range, making it visually appear that the effects of different checkpoints are similar. However, in reality, they have significant differences (Table 6,7 provide more accurate numerical comparisons during an instruction tuning stage).

W2: It is unclear whether the same pattern would be observed with other datasets.

Thanks for your suggestion. It is a common phenomenon that there are performance differences between different immediate checkpoints during instruction tuning. Here, we attach the performance across different checkpoints on the Alpaca dataset, and the results show similar conclusions to those presented in the paper.

We also observe similar patterns in the alpaca dataset, where there was a noticeable discrepancy at different checkpoints during the training process.

W3: Studying the impact of saving checkpoints at different intervals during dense model training could further enhance the robustness of the proposed method.

The main viewpoint that the paper hopes to output is that using the natural discrepancy of intermediate checkpoints is able to achieve more efficient and effective upcycling. With this in mind, we mainly focus on the situation where the number of saved models is less than the expected number of experts - and then propose an expert expansion stage.

For the sake of generalization and convenience, we do not emphasize the selection of checkpoints. When there are sufficient checkpoints available, various selection methods are applicable. In Table 2 (left), we illustrate three different selection methods: using the front half, back half, or average selection. Regardless of the selection method, there is a significant improvement compared to the vanilla upcycling. Therefore, we believe that as long as there is a discrepancy between experts during the expert preparation phase, the specific selection method becomes less important, and we keep the further checkpoint selection strategy for future work.

Q1: How do we determine the appropriate interval for saving checkpoints?

Consistent with the previous response for W3, we do not pay special attention to the intervals for saving checkpoints, and adopt the naive approach of regularly saving checkpoints, which is also the most common practice in model training.

Q2: Wouldn't this mostly result in the selection of checkpoints from the early and late stages of training?

In the parameter update process of model optimization, each parameter is not updated in a uniform and linear manner. Instead, it undergoes complex and dynamic changes, and the difference between the initial and final checkpoints may not be the largest. Furthermore, in order to create diverse experts during expert expansion, we place each newly created expert back into the pool of candidate experts, where it participates in the similarity calculation for the next round.

Q3: How is the similarity between the two experts calculated?

The similarity is calculated based on the cosine similarity between the parameter matrices of two experts. Specifically, for a given LoRA expert's A and B matrices, we first compute the matrix product of A and B to obtain a parameter matrix E. Then, we calculate the cosine similarity between the parameter matrices E of different experts. The process for FFN experts is similar.

Q4: What would happen if randomly sampled is not used?

In our experiment, we directly reuse the IDAE-500K dataset utilized in the previous baseline method [1], which is more convenient for experimental exploration on a data scale, as demonstrated in Figure 3 (with 50K, 100K, 200K, and 500K samples).

[1] Wu et al. Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks. EMNLP. 2024.

---

Thank you once again for your valuable feedback we are committed to refining our work based on your insights.

If there are any remaining concerns or questions that we have not yet addressed, please do not hesitate to let us know. We are more than willing to provide any additional materials or clarifications that may be necessary.

Thanks for the valuable comments that could help us improve the work. Next, we will respond to the questions, and we would appreciate it if you could raise the scores when the concerns are addressed.

W1: It is unclear how many samples each method uses during training in Table 1.

In the 'Dataset' subsection of 'EXPERIMENTS' (Lines 318-321), we state that UpIT reuses the dataset utilized in the previous baseline method [1], which contains a total of 500k samples. In subsection 3.3 'SCALING THE TRAINING DATASET' and Figure 3, we also validate UpIT and vanilla upcycling approaches by randomly sampling 50K, 100K, and 200K samples from the full 500K dataset, to assess the data-efficient nature of UpIT.

W2: What would be the results of not using intermediate checkpoints to initialize MoE?

Not using intermediate checkpoints is the vanilla upcycling method (as stated in Lines 044-048), which often requires a lot of data during the training phase and the results are not satisfactory.

In the 'Baselines' subsection of 'EXPERIMENTS' (Lines 312-316), we introduce two baselines under LoRA-based setting: $LoRAMoE_{PT}$ and $LoRAMoE_{SFT}$, as well as two baselines under FFN-based setting: $Upcycle_{PT}$ and $Upcycle_{SFT}$. A thorough explanation of these baselines is provided in Appendix A.1 (Lines 765-767, 769-771). These four baselines represent two approaches: one that employs the pre-trained model directly for upcycling, and another that conducts SFT initially and subsequently utilizes the final checkpoint for upcycling. Table 2, Figure 3, and Figure 4 show that UpIT significantly outperforms existing methods, whether in scenarios with sufficient or insufficient data, exhibiting outstanding flexibility, scalability, and performance upper bound.

W3: What is the most important component that contributes to the final performance?

We first reiterate that the utilization of intermediate checkpoints and the process of route initialization complement each other and are both core contributions in UpIT (as stated in Contribution, Lines 131-141).

In this paper, we aim to emphasize the importance of expert discrepancy in upcycling and incorporate the idea into the entire design of UpIT. Figure 2 highlights different checkpoints excel at different benchmarks, which introduces our first contribution of using intermediate checkpoints for expert preparation and corresponding expert expansion strategies. On this basis, we propose a router initialization stage with expert-specific data selection, utilizing and amplifying the differences among experts in the initialization phase. Overall, the initial differences among experts are the foundation of UpIT, and we fully utilize and amplify these differences to improve performance and data efficiency.

W4: None of the baselines belong to specialized upcycling.

As described in Lines 48-51, specialized upcycling first trains a specific expert for each domain, which lacks flexibility as it is difficult to divide a mixed general dataset into separate domains. Even if we manage to split the dataset, the number of experts will be the same as the number of domains, making it hard to apply in practical scenarios as well, because MoE typically involves 8 or more experts, it is difficult to guarantee that every training data has so many domains.

In constant, UpIT considers more universal scenarios, in which data comes from a mixed set and cannot be effectively divided. Traditional specialized upcycling methods are no longer applicable, while UpIT achieves efficient and flexible upcycling by seeking natural differentiation between intermediate checkpoints.

W5: In line 258, "merging all the expert models" seems more like a concatenation.

Thanks for pointing this out. Indeed, we meant "concatenation" here, and we will correct this in the next version.

W6: The contents in Figure 1 are very small.

Thanks for pointing this out. We will correct this in the next version.

W7: In Sec 3.1, it would be great to explain the mechanism of each baseline.

Thanks for pointing this out. Due to space limitations, we provide a detailed explanation and clarification in Appendix A.1 (DETAILED DESCRIPTION OF BASELINES, Lines 758-771), which are also mentioned in the the 'Baselines' subsection of 'EXPERIMENTS' in the main text (Line 316).

[1] Wu et al. Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks. EMNLP. 2024.

W8: In lines 8-10 of Algorithm 3, how to choose which bucket to append the data?

As stated in Line 5 of Algorithm 3, we use perplexity (PPL) to measure the competence of each expert for each sample. Specifically, for each data, we determine the expert model with the lowest PPL and allocate the data to the bucket of that expert model. If the bucket is full, we then allocate the data to the bucket of the expert model with the second-lowest PPL, then, continue this process in order until the data is assigned.

W9: The idea of using model merging to expand experts is interesting, but the performance improvement is marginal.

The core objective of model merging is to provide a method for achieving an instruction model with a flexible number of experts when intermediate checkpoints (i.e., experts) are scarce. The performance with expert expansion is very close to that of using more checkpoints, which means that we can achieve similar results with fewer experts. It is quite ideal result and fully demonstrates the effectiveness of expert extension (as mentioned in Lines 481-485).

---

Thank you once again for your valuable feedback we are committed to refining our work based on your insights.

If there are any remaining concerns or questions that we have not yet addressed, please do not hesitate to let us know. We are more than willing to provide any additional materials or clarifications that may be necessary.

We are encouraged that you find UpIT is significant and impactful. Thanks for the valuable comments that could help us improve the work. We will respond to the questions, and we would appreciate it if you could kindly raise the scores when the concerns are addressed.

W1 (Expert Diversity): While the UpIT method enhances expert diversity at initialization, Figure 2's motivation could be more robust.

In this paper, we aim to emphasize the importance of expert discrepancy in upcycling and incorporate the idea into the entire design of UpIT. Figure 2 first highlights the different checkpoints during instruction tuning that are inherently suitable for constructing specialized experts, since different checkpoints excel at different benchmarks, which introduces our first contribution of using intermediate checkpoints for expert preparation and corresponding expert expansion strategies. On this basis, we propose a router initialization stage with expert-specific data selection, utilizing and amplifying the differences among experts in the initialization phase. Overall, the initial differences among experts are the foundation of UpIT, and we fully utilize and amplify these differences to improve performance and data efficiency.

Note that we present the results of multiple benchmarks in a single diagram, with a wide y-axis range, making it visually appear that the effects of different checkpoints are similar. However, in reality, they have significant differences (Table 6,7 provide more accurate numerical comparisons during an instruction tuning stage).

W2 (Checkpointing Mechanism): The impact of the initialization strategy from routing initialization and data selection is crucial.

We first reiterate that the utilization of intermediate checkpoints and the process of route initialization complement each other and are both core contributions in UpIT (as stated in Contribution, Lines 131-141). In vanilla upcycling, all experts are the same in the initial stage, which does not apply to expert-specific data selection and therefore cannot perform the routing initialization operation.

The table below extracts from Table 1 and Table 2 (right) demonstrate that roughly upcycle discrepant experts without proper router initialization do not yield good performance, because a randomly initialized router lacks the awareness of the experts, and direct training inadvertently reduces the discrepancy between them. Considering that noise usually follows a Gaussian distribution, introducing noise to the same is similar to the UpIT (Random) setting, which also cannot achieve satisfactory results.

W3 (Expert Expansion): Expert expansion does not significantly outperform simply saving more checkpoints.

The core objective of model merging is to provide a method for achieving an instruction model with a flexible number of experts when intermediate checkpoints (i.e., experts) are scarce. The performance with expert expansion is very close to that of using more checkpoints, which means that we can achieve similar results with fewer experts. It is an quite ideal result and fully demonstrates the effectiveness of expert extension. By the way, DARE is not the focus of this paper, alternative parameter merging methods, such as average merging, could also be employed.

Q1 (Routing Distributions): Why does such a highly skewed router distribution arise?

All processes of UpIT emphasize the discrepancy of experts. Building on the discrepancy between experts, we propose an expert-specific data selection to filter the most skilled data for each expert. In the router initialization phase, we pre-define the allocation strategy of the router to achieve expert-aware routing, which will further enhance the differences between experts in subsequent post-training. In this way, the final router distribution exhibits strong preferences.

---

Thank you once again for your valuable feedback we are committed to refining our work based on your insights.

If there are any remaining concerns or questions that we have not yet addressed, please do not hesitate to let us know. We are more than willing to provide any additional materials or clarifications that may be necessary.