Emergent Mixture-of-Experts: Can Dense Pre-trained Transformers Benefit from Emergent Modular Structures?

1. In previous MoE works, $k$ is often set to {1, 2} to achieve high time efficiency, while the value of $k$ in EMoE is {16, 32}. How does this choice affect time efficiency? It would be helpful if the authors could provide a comparison in terms of GMACs for Transformers, MoEs, GMoEs, and EMoEs.

Thanks for your question. Increasing the number of experts $N$ and the selected expert $k$ for EMoE does not severely affect the time efficiency and computation cost. The key difference between EMoE and previous MoEs works (including GMoE) is that EMoE split the original FFNs layer into different experts. The larger the number of experts, the smaller the expert size. As a result, the computational efficiency is largely independent of the EMoE's settings (different $k$ and $N$). Table 3, 4, and 5 in General Response shows no difference in FLOPS and time between the scenarios with k set to 16 (9.12e16) and 32 (9.24e16). Neither setting significantly differs from the standard Llama2-7B (8.97e16). On Llama2-7B, the introduction of EMoE (8.99e17) has minimal difference relative to standard LoRA Tuning (8.93e17).

When it comes to GMoE, larger $k$ will lead to computation overhead because they replicate the original FFNs from MoEs. Large $k$ gets more experts involved, and every expert has the same size as the original FFNs layer, leading to computation overhead. We will add relevant discussions in the final version of our paper.

2. The results demonstrate that EMoE outperforms EMoE-learn when the number of fine-tuning iterations is the same. This might be attributed to the gate of EMoE having a better initialization point and converging faster. It would be interesting to see the performance of EMoE-learn with longer finetune iterations compared to EMoE.

Thanks for your suggestions. We have supplemented relevant results to show that EMoE-learn does benefit from longer fine-tuning to some extent (perhaps every method does; we have not verified it yet), but it still underperforms EMoE. Specifically, we increase training epochs from 20 to 30 in LoRA tuning scenarios. The results (EMoE-learn-long) show some improvement but still underperform EMoE in some tasks (e.g., CoLA). This suggests that while increasing training steps can mitigate slow gate convergence, it might not resolve issues like gate collapse, where training later concentrates on fewer experts (Figure 6c).

Additionally, experiments with better initialization (EMoE-learn-init) using the avg-k method for gates showed improvement over EMoE-learn but did not exceed avg-k gating.

3. In GMoEs, auxiliary loss functions (as described in Appendix C.4 of the GMoE paper) are employed to achieve good performance. Were these loss functions used for the GMoE baselines in this study? Additionally, could these loss functions improve the performance of EMoE?

Yes. The implementation of GMoE used official repo setups, including the loss functions with consistent hyperparameter settings. For EMoE experiments in GLUE and Domainbed, the same MoEs framework and loss functions were used to eliminate interference from factors other than expert construction and gate function. We also highlight this in experiments with the Llama. We achieved significant improvements over the baseline without using any balance loss in our implemented MoEs.

4. The font size of "Office" and "Terra" in the figures appears to be smaller than that of the other labels. It would be helpful to ensure consistency in font sizes across all labels.

Thanks for your suggestions. We intentionally adjusted the font size because we observed that the character length of "Office" and "Terra" is longer, leading to inconsistent column widths in the formatting and affecting the presentation. We will address this issue.

5. The right figure in Figure 4 seems to lack data points. Could the authors clarify if this is intentional or if there might be an issue with the figure's presentation?

We notice that the issue occurs when viewing the PDF in a browser but not in a PDF reader. This might be due to the high density of points in the figure. We will try to address this issue in subsequent revisions.

Thanks for your reviews, we hope the following comments could address your concerns.

1. Given that top-k/N is typically half, I am wondering if this is always the same half of modules activated? Figure6/8 seem to show that some experts are barely selected but this would somehow contradict the claim of the paper that "Expert Selection Changes During Training". Could you please clarify whether I am misunderstanding the figures? Does the fact that a lot of experts are consistently white over the course of training not indicate that they are rarely used in the forward pass?

Thanks for your question. Your understanding of Figure 6 and Figure 8 is correct. For EMoE, some experts are barely selected, and the expert choice stabilizes during training. We guess it’s our subheading that causes such misunderstandings. The ‘changes ’ here is a noun instead of a verb. We will replace the subheading to avoid the ambiguity.

2. Following up on the previous point, a possible ablation I would have found very informative is to prune the experts based on the selection frequency during training, i.e. fix the gating after training based on the selection frequency for various fractions. I am wondering if the performance gains of EMoE are effectively due to pruning and not due to modularity.

Thanks for your suggestions! First of all, regarding your statement, “if the performance gains of EMoE are effectively due to pruning and not due to modularity. ”, we would like to point out that pruning can be regarded as task-level modularity. They cannot be considered as two opposites. Modularity is when specific inputs activate specific modules (or a group of parameters), and pruning is when the useless parameters can be pruned for specific tasks. We can see a strong connection between modularity and pruning; they cannot be decoupled.

Second, we agree that the suggested experiments are informative. Therefore, we supplement two ablations. Their pipelines are:

1. Training-Pruning: Training EMoE ($N$=64, $k$=32), pruning experts with lower selection frequency, and evaluating the pruned model.
2. Pruning-Training: Pruning lower selection frequency experts of a new model according to 1, then training and evaluating the pruned model. It is important to note that our approach utilizes statistical data from the fine-tuning process to prune a model that had not undergone tuning. This specific configuration is employed to compare pruning and modularity directly during the training phase.

Specifically, we choose BERT-Large within the LoRA tuning framework, where the MoEs component remains untrained. This choice allowed us to gain clear insights into the impact of neurons' selection within the FFNs on Lora weight learning, consistent with the settings employed in Analysis 5.1.

The experimental results are presented in the table below. Our findings provide further substantiation for our explanation of EMoE, highlighting that EMoE's effects manifest during the fine-tuning stage rather than during inference, as also underscored in our response to Question 5. Our main observations are:

1. When we prune sparsely selected experts during inference ($N$=64, $k$=16,32 scenarios), we observed comparable performance with EMoE. This observation held true across various scenarios, consistently surpassing the baseline.

2. When we prune some barely selected experts according to training statistics from the first ablation and fine-tune the remaining experts, the performance gains attributed to EMoE become imperceptible. Even in the most optimal setting ($k$=32, 63.52), the performance only marginally exceeded the baseline (63.40) and notably fell short of the results achieved by training with EMoE, followed by pruning to $k$=32 (65.33).

   In conclusion, we argue that the pruning and modularity are homologous to some extent. They cannot be regarded as opposites. And further experiments support our explanation in the submission. And we will add these additional ablation results in the final version of our work.

   	CoLA			SST2
   LoRA	63.40			93.16
   EMoE (k=32, N=64)	65.33			93.54
   Expert number after pruning	Training-Pruning	Pruning-Training		Training-Pruning	Pruning-Training
   64	65.26	63.4		93.42	93.16
   32	65.33	63.52		93.54	93.34
   16	65.33	63.42		93.45	93.27
   8	64.33	63.42		93.45	93.27
   4	63.80	63.21		93.34	93.21
   1	63.42	62.42		93.34	92.58

We first want to thank you for your valuable comments. Please see our responses below to see if they could address your concerns. All explanations will be supplemented in the final version of our paper! More discussions are welcome!

1. The improvements noted over Vision Transformer (ViT) and (GMoE) are marginal and fall within the noise range for the provided datasets, as evidenced by the data in Table 8, Appendix 4.1.

Thanks for your comments. The different methods (ViT, GMoE, and EMoE) are close to the Domainbed benchmark. However, we want to point out that this is one characteristic of the Domainbed benchmark, owing to its rigorous evaluation setting: Each dataset comprises 4 distinct domains. One or two domains’ data are sequentially designated within a single training. For example, when training on PACS with four domains {art, cartoons, photos, sketches}, {art, cartoons} could be selected as ID training data, while {photos, sketches} are designated for OOD testing. This configuration results in $C_4^2 + C_4^1 = 10$ training processes within each dataset. Each dataset includes 3 trials. Thus, each result is aggregated from 30 experiments. As a result, they also reported that no algorithm outperformed ViT by a significant margin in their original paper [1]. GMoE achieves SOTA performance, and we achieve comparable performance with it.

Although the performance is not the intended purpose of our paper, we supplement extensive experimental results to support that EMoE brings improvements. Apart from the NLP results in our submission, we supplement experiments using T5-base for multi-task learning and Llama for instruction tuning. Both results indicate that EMoE boosts the ID and OOD performance compared to vanilla fine-tuning. For example, EMoE improves by 7.56 over the baseline on 6 super-GLUE tasks. Please refer to our General Response Table 1 and 2 for complete results.

2. The method proposed is derived from previous work by Zhang et al. (2022 b), and does not represent an original contribution from the current paper.

Our method is indeed derived from [2]. However, we want to highlight that the core contribution of this work is not a method (how to transform a typical Transformer into its MoE counterpart). We want to share the insight and interesting experimental results with the LLM community: Pre-trained Language Models can benefit from their emergent modularity during the fine-tuning stage. We call on the entire community to use the emergent modularity feature of the model and to propose more sophisticated methods to externalize the emergent modularity; in our case, as the method is not the first foci of our work, we adopt a simple method from [2].

We highlight that our research question differs from theirs; they have not focused on leveraging the emergent modularity into fine-tuning pre-trained Transformers. The adopted method is one of the variations of their primary method. For further details, please refer to our General Response to all reviewers for discussion between EMoE and MoEfication.

3. The paper's method shares similarities with the GMoE method, yet the proposed method achieves comparable performance.

As in Question 2, EMoE shares some similarities with GMoE; they both introduce MoE architecture in some layers of the Pre-trained Transformers.

However, we want to highlight the key difference between them. GMoE directly copies the original FFN layer and introduces a learned gate to form an MoE architecture; the different experts of GMoE are initialized identically as the original FFN layer. Therefore, GMoE is inadequate to explore our research question: whether emergent modularity favors fine-tuning transformers. So, we derived our method from MoEfication. Meanwhile, replicating the FFN layer in the GMoE method makes it less practical for large models (e.g., Llama), but EMoE does not suffer from the same issue. For a detailed comparison, please refer to the table in Response to reviewer qXUa (Part 1).

[1] In Search of Lost Domain Generalization Domain Bed, https://arxiv.org/abs/2007.01434

[2] MoEfication: Transformer Feed-forward Layers are Mixtures of Experts, https://arxiv.org/abs/2110.01786

Thanks for your suggestions. Your understanding of the analysis in Section 5 is entirely right. We want to emphasize here that EMoE mostly influences the training stage instead of inference. And we also highlight that we can use original architecture for inference after EMoE-style fine-tuning. This makes our method more practical (because we do not need to change the model architecture after fine-tuning) and potentially contributes more to the community. We will re-write this part to make it easier to follow.

Regarding our limitations, we have addressed them in Section 6: Can larger LLMs benefit from its emergent modularity? Specifically, we employ the Alpaca dataset to instruction-tune Llama2-7B Llama-30B models in the EMoE manner and use MMLU as the evaluation benchmark. We observe consistent improvement in MMLU performance compared with vanilla fine-tuning. Please refer to Tables 3, 4, and 5 in General Response for full results. I hope such additional results can address our minor limitations and support you in recommending our paper!

11. Finally, while I appreciate that this was aiming to be grounded within the transformer literature , rephrasing a normal neural network forward pass in terms of keys, queries and values is also just unhelpful. Especially when the work is inconsistent on what the key is.

We believe that the comments from the reviewer regarding the inconsistency in defining "key" may arise from a misunderstanding. Throughout our paper, when we refer to "keys," we consistently mean the rows of matrix $K$. We aimed to emphasize the conceptualization of the FFNs' operations as key-value computations, which aids in understanding the decomposition of FFNs into experts. As illustrated in Figure 1(a), if we interpret the first matrix operation in the FFNs as an inner product between the input and keys to obtain activations and the second matrix operation as a weighted sum of values based on these activations, it naturally clusters keys that are jointly activated and combines them with values to form experts, as shown in Figure 1(b). Furthermore, it is worth noting that a body of existing literature [1, 2, 3] views FFNs as key-value memories.

12. Section 3.1 says the keys are the rows of $K$ but then it is more accurate to interpret the centroids of the clusters of $K$ as the keys as this is what is being compared to the query (input) to determine which module to use.

We believe there might be some misunderstandings here. In Section 3.1, we did not introduce the concept of "module" or delve into the notion of "cluster centroids." The reason for referring to the rows of matrix $K$ as "keys" in Section 3.1 was that we believed that the concept of a "key" could correspond well with the activation of individual neurons. However, it is important to note that the "keys" mentioned here are unrelated to the activation of the "module." The activation related to the "module" is tied to "the average of keys" within the "experts." Furthermore, it is worth mentioning that the "cluster centroids" mentioned by the reviewer could be employed as gate weights in Figure 1(c). However, due to the increased complexity associated with updating them compared to using the average values, we opted for using the average values rather than "centroids.”

13. "Instead, the goal is to demonstrate that dense pre-trained transformers can benefit from emergent modular structures" does not quite summarize accurately what is shown. This would demonstrate that the emergent modular structure is a phenomenon which lasts in transformers, not some structure which must be algorithmically extracted and enforced.

Thank you for pointing out the need for clarification. In future versions, we intend to revise the summary: "demonstrate that externalizing the emergent modularity into MoE structures would be beneficial for fine-tuning pre-trained transformers." However, it is necessary to emphasize that EMoE may not represent the sole approach for harnessing emergent modularity. We strongly encourage the research community to explore and propose more advanced methods for externalizing and leveraging emergent modularity.

14. I recommend cutting down on Sections 3.1 and 3.2 as they are not particularly helpful and Figure 1 does a great job expressing the same thing. I also think Section 2 could be shortened for a more in-depth background section focusing on GMoE and LoRA and potentially MoEfication which focuses on how EMoE differs. Finally, a general check of the grammar and vague phrasing is in order.

We appreciate your suggestions. In our revisions, we will include the high-level ideas we aim to convey in Sections 3.1 and 3.2. We acknowledge the suggestion to condense Section 2 and instead focus on a more comprehensive background section that delves deeper into GMoE, and MoEfication, emphasizing the distinctions between these approaches and EMoE. We will conduct a thorough grammar and phrasing review to ensure the overall language quality of the manuscript is improved.

[1] Large Memory Layers with Product Keys, https://arxiv.org/abs/1907.05242

[2] Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space, https://arxiv.org/abs/2203.14680

[3] Transformer Feed-Forward Layers Are Key-Value Memories, https://arxiv.org/abs/2012.14913

6. why are EMoEs given significantly more modules to train (a larger $N$) and use (a larger $K$) than GMoEs? Does this make a comparison unfair?

The choice of different $N$ and $k$ settings is associated with the sources of experts in EMoE and GMoE. We will explain the rationale and impact of selecting $N$ and $k$ in EMoE and GMoE. Regardless of how $N$ and $k$ are chosen, the training parameters of GMoE are significantly greater in number compared to EMoE. Therefore, from the perspective of training parameters, the comparison is fair. In addition, our supplementary experiments demonstrate that this comparison does not affect our conclusions regarding the effectiveness of EMoE.

First, One key underlying reason is that EMoE decomposes the original FFNs into experts; no matter how we increase $N$ and $K$, the total parameters of all experts will remain the same as the original FFNs. On the contrary, GMoE replicates the original FFN from a MoEs architecture; the larger $N$ and top-k they choose, the more parameters and computational overhead they introduce. To put it simply, if there are h neurons in the FFNs:

1. EMoE uses $h$ * (top-k/$N$) neurons, which is a fraction of the total number of neurons in FFNs
2. GMoE uses $h$ * top-k neurons, which is a larger number of neurons compared to EMoE

Second, we verify that GMoE will not benefit from larger $N$ and $k$ by supplementary experiments. We conduct LoRA-tuning (because a very large $N$ of GMoE introduces too many additional parameters, we only tune the Lora module and the layer where GMoE introduces experts to test GMoE on very large $N$) on the GLUE benchmark; we examined the impact of increasing $N$ and top-k in GMoE, as shown in the table below:

It can be observed that EMoE achieves a significant improvement over LoRA without introducing additional parameters relative to LoRA. On the other hand, GMoE's performance does not necessarily improve as $N$ and $k$ increase, and it comes at the cost of significantly increasing the number of parameters. This highlights the efficiency and effectiveness of EMoE compared to GMoE.

We want to take this opportunity to emphasize again the effectiveness of harnessing emergent modularity in pretrained transformers, which entails leveraging the model's inherent structure to yield performance improvements.

7. A final example, which is similarly important for understanding how EMoEs work says "EMoE and dense model only differ in FFNs activations". Surely the EMoE differs by the fact that it is now gated with an enforced sparsity?

The difference between EMoE and the standard transformer lies in the activation patterns of neurons within the FFNs. In EMoE, the gating mechanism ensures that neurons from experts not selected do not participate in computations, resulting in enforced sparsity, as mentioned by the reviewer.

8. A second note on clarity, I relied on Figure 1 to understand EMoEs as overall Section 3.1 is difficult to parse.

We guess there might be a misunderstanding on the part of the reviewer. In Section 3.1, our intention was not to introduce the EMoE. The purpose of Section 3.1 was to help readers become familiar with the structure of the FFNs within the transformer and MoE architecture, which would facilitate their understanding of the subsequent operation of decomposing FFNs into MoEs.

9. Having the transpose on the weight matrix $K^T$ just makes it difficult to follow. Additionally, the output of the $x\cdot K^T$ computation is a $d$-dimensional vector ....

We appreciate the reviewer for pointing out the error in our formula, and we will correct this mistake.

10. Also the first usage of $K_i$ does not bold the $i$ but it is bold everywhere else, so that should be corrected (that's not a critique, just wanted to point it out). As a minor extension of this, when would $d$ represent the dimension of the hidden layer and $h$ represent the dimension of the data?

We appreciate the reviewer for pointing out these notation issues, and we will make corrections in the subsequent versions. In our paper, $d$ is the dimension of the hidden layer, and $h$ is the data dimension (also the embedding size).

3. determining the number of centroids in the clustering and number of modules to use are two key hyper-parameters which are not considered or spoken about at all. These two parameters can determine entirely how the module generalizes or under-fits and its overall expressive power. They are also two parameters which must now be tune -- together. Is this addition of complexity worth the benefit over using the vanilla baselines? The answer to that question is likely up to the reader, but failing to mention this or provide the reader with evidence on this appears to be a significant omission.

Our focus is primarily on investigating the effectiveness of emergent modularity in downstream tasks. We have adopted the MoEfication's partitioning method directly, so we did not delve extensively into the settings of relevant hyperparameters. We largely followed the conventions established by MoEfication. As for the settings of $k$ and $N$, there is further discussion in our response to question 6.

We believe that EMoE is not highly sensitive to the hyperparameters $N$ and $k$; thus, it does not introduce significant additional complexity. Our supplementary experiments show that EMoE consistently outperforms the baseline on the new T5 multi-tasks and LLama datasets when using the empirical choice of fractions, 1/4 or 1/8, identified on the GLUE benchmark.

Moreover, we want to highlight the effectiveness and practicality of EMoE. In the Llama experiments, besides no additional parameters, EMoE doesn’t introduce a significant increase in wall time and FLOPS compared with the baselines. However, EMoE-tuned LLama2-7B improves by 1.58 on the MMLU benchmark. Furthermore, as mentioned in Section 5.1, EMoE mainly takes effect during training, so one can apply EMoE architecture and recompose it into a dense model during inference. This property ensures that the EMoE-trained model can be deployed without changes in model architecture— a characteristic neither GMoE nor MoEfication can achieve.

4. we treat these decomposed units as heterogeneous modules to exploit their modular properties in the framework of MoEs". What is a "heterogeneous module" in this context. Surely these modules are heterogeneous by some metric by definition since they are identified by a clustering algorithm? But then why would MoEfication have homogeneous modules?

The sentence here highlights the difference between the motivation behind EMoE and MoEfication. We realize that this statement potentially causes clarity issues in our work, and we will rectify it in our final version. For now, please see our explanations below.

1. MoEfication mainly seeks to improve inference efficiency. They decompose the Feed-Forward Networks (FFNs) into sparse MoE after fine-tuning so that sparsely activated experts can approximate the functionality of the original FFNs with reduced computational cost.
2. Differently, EMoE wants to externalize the emergent modular nature of the pre-trained transformers so that the different experts are sparsely updated and are further encouraged to achieve distinct specializations. So, we decompose FFN into sparse MoE before fine-tuning on downstream tasks. Though similar methods are applied, EMoE provides a different view about what we can benefit from emergent modularity, and this is what we want to share with the community with this work.
3. When we mention "heterogeneous," we compare EMoE to other methods introducing experts (e.g., GMoE and upcycling). EMoE's experts are directly derived from pre-trained FFNs, and this origin makes them more diverse or "heterogeneous" in terms of their functionality compared to other approaches like GMoE or upcycling. These other methods may not exhibit as much diversity among their experts because their experts are initialized by directly copying existing FFNs.

5. Even in the abstract using the phrase "...superior out-of-generalization, learning efficiency, etc." is strange, particularly due to the use of "etc" with little context with which to determine what is being referred to.

We appreciate the feedback provided by the reviewer, and we will remove the phrase and refine the language in our revision.

Thanks for your valuable comments on the quality and clarity of our work. We agree that some statements in our submission may be unclear, and we will revise our submission to make it easier to follow (We will revise it by the end of our discussion period). Before that, we want to give more explanations of our work because we think there might be some misunderstandings in the reviews.

First, we would like to respond to your primary concern: what message do we want to convey? We want to share with the reader that pre-trained transformers exhibit emergent modularities (different groups of parameters may spontaneously specialize in different functions), and such emergent modularity should be further encouraged at the training stage of the model to achieve the benefit of ID and OOD performance. We verify our point of view with extensive experiments. It turns out that EMoE generally outperforms vanilla fine-tuning.

We understand your concern that some improvements are marginal, such as compared to ViT on Domainbed (For detailed reasons, please refer to our response to reviewer xyul question 1). We thus supplement stronger experimental evidence to address your concern. When fine-tuning T5-base models on 6 Super-GLUE tasks, EMoE achieved an average improvement of 6.68%. Additionally, when instruction-tuning Llama2-7B using the Alpaca dataset, EMoE significantly improved the score on MMLU benchmark by 1.58. These results demonstrate the effectiveness of EMoE. Please refer to the general response Tables 3,4 and 5 for complete results. Given such stronger evidence, we believe there is a benefit of EMoE.

Second, as the relationship between EMoE, GMoE, and MoEfication has been repeatedly questioned, we want to give more explanations.

Having observed that FFNs layers in pre-trained transformers are sparsely activated (many neurons are unused for inputs), MoEfication splits transforms FFNs into a sparse MoEs, aiming to approximate the functionality of the original FFNs to reduce the computational cost, further improving inference efficiency. Besides, the GMoE makes multiple replicates of the original FFNs layer and introduces a learned gate to form a MoEs architecture. They claim that such architecture could improve OOD performance from their theoretical perspective of algorithmic alignment framework [https://openreview.net/forum?id=rJxbJeHFPS]. MoEfication and GMoE do not touch on how emergent modularity influences the training stage. The table below illustrates a detailed comparison of these works.

A comprehensive comparison between EMoE, MoEfication, and GMoE:

We thank you for your valuable feedback regarding clarity. We apologize for our messed notation. We will explain your questions in your reviews below and revise our submission by the end of the Author-Discussion period (We still need some time to work on it). Please first see if our explanation could address your concerns.

Please see our point-wise responses to the question in your review.

1. The authors claim that there is a benefit from EMoE and I cannot see how this is consistently the case across Tables 1, 2 and 3. While there is some improvement over the various baselines in each case (ViT, BERT, GPT2 and LoRA) it still appears marginal and fairly inconsistent. This is made worse by the fact that standard deviations are not reported or commented on (while they are referenced in the caption of Table 4.1)

To clarify the presentation of our results, we have included the results with standard deviations in the appendix, specifically in Tables 8, 9, and 10. We acknowledge the reviewer's concern regarding the significance of our results; therefore, we have provided additional and more robust experimental evidence. In the context of the Super-GLUE tasks within the T5-Base framework, our model demonstrated a noteworthy average improvement of 6.68%. Furthermore, when evaluating the Alpca-tuned Llama2-7B on the MMLU benchmark, EMoE exhibited a substantial improvement of 1.58. In the case of the Llama-30B dataset, EMoE also demonstrated a notable improvement of 0.93.

2. "explore whether and how standard pre-trained transformers could benefit from emergent modular structures". But this question seems to have been answered by the two prior works. GMoEs would indicate that there is a generalization benefit, and MoEfication demonstrates that there is a computational benefit.

We want to emphasize that neither GMoEs nor MoEfication has addressed our research question. We will provide a more detailed introduction to GMoEs and MoEfication, followed by a comparison highlighting the distinctions between these approaches and EMoE.

1. MoEfication does not delve into the properties of these split experts or explore alternative ways to utilize them beyond enhancing efficiency. MoEfication is motivated by the observation that the activation of neurons within the pretrained transformer FFNs exhibits sparsity and certain cooperative characteristics. Specifically, within the two matrix operations in FFNs, most of the computations do not impact the output. Therefore, MoEfication aims to split the matrices into the form of experts, allowing input data to be involved only in computations performed by experts that influence the output. This involves using sparsely activated experts to approximate the functionality of the original FFNs, leading to reduced computational costs. Importantly, all these operations are conducted after fine-tuning, and MoEfication's primary objective is to make FFNs computations more efficient.

2. GMoE doesn't capture the modular structures within the FFNs of pretrained transformers. GMoE is developed based on theoretical insights, recognizing that MoE architectures contribute to models' out-of-distribution (OOD) generalization. In GMoE, pretrained transformer FFNs are duplicated to construct MoE architectures during the tuning phase, with validation conducted in OOD settings. It's important to highlight that the experts in GMoE are initialized identically to the original FFNs layer and do not delve into the inner structure of the FFNs layer. As a result, GMoE is inadequate to explore our research question: whether externalized emergent modularity favors fine-tuning of transformers.

2. An instruction-tuning setting: To further prove that the main conclusion EMoE still holds for larger LLMs, we use the Alpaca dataset to instruction-tune the Llama series models and evaluate it on the MMLU benchmark.

   We have observed the following:

   1. Across model sizes of 7B and 30B, as well as settings such as full-finetuning and Lora tuning, EMoE consistently yields improvements relative to the baseline.
   2. The choice of $k$ and $N$ proportions remains applicable even in larger-scale models. While variations may be in different settings, they consistently outperform the baseline. This suggests that although additional hyperparameters are introduced, they do not lead to usability challenges.

   There are two observations that we want to share with the reviewers.

   1. We also note that EMoE does not introduce significant computation overhead (refer to wall time and FLOPS)
   2. Inspired by our analysis results of Section 5.1 (EMoE mainly takes effect during the training stage rather than the inference stage), we recompose EMoE back into the original standard Llama architecture during evaluation, which still achieves substantial gains. We believe that this further enhances the practicality of EMoE because one can only introduce EMoE during fine-tuning and recompose the tuned model into a standard transformer after fine-tuning. We also plan to open-source our codes and implement EMoE into the huggingface for boarder use.

Table 3: Instruction full-tuned Llama2-7B's MMLU scores.

Table 4: Instruction LoRA-tuned Llama2-7B's MMLU scores.

Table 5: Instruction LoRA-tuned Llama-30B's MMLU scores.

We will supplement our explanations and additional experimental results in our revision to improve our work (We are still working on it).

1. A multi-task setting:

   In our analysis (Section 5.1), we have identified that the improvement brought by EMoE is likely associated with mitigating negative transfer. Inspired by this, we choose a multi-task learning setting where negative transfer might be more pronounced. We adopt the codebase of ATTEMPT: Attentional Mixture of Prompt Tuning [https://github.com/AkariAsai/ATTEMPT]. For the in-domain (ID) scenario, we follow the settings outlined in the paper and select six tasks from the Super-GLUE benchmark. For the out-of-domain (OOD) scenario, we take two larger natural language inference (NLI) datasets, MNLI and QNLI, as our ID training data. We subsequently conducted direct testing on four additional NLI datasets from different domains. All hyperparameters unrelated to mixture-of-experts (MoEs) are kept consistent with the baseline, and we have listed the MoEs-related hyperparameters in the following table.

   Our observations are as follows:

   1. EMoE exhibits a substantial improvement compared to the baseline. In the in-domain (ID) setting, the highest improvement reached 7.56, even considering the average performance across the six tasks. In the out-of-domain (OOD) setting, the highest average OOD result across the four datasets improved by 1.58.
   2. Across various settings of $N$ and $k$, EMoE consistently outperforms the vanilla fone-tuning. Within the hyperparameter search space specified in our paper, EMoE consistently improves at least 2 points over the baseline in the in-domain (ID) setting. This emphasizes the effectiveness of EMoE and EMoE’s robustness to the explored hyperparameter range.

Table 1: T5-base ID performances.

Table 2: T5-base OOD performances