MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts

We sincerely appreciate your thoughtful comments and your recognition that "this work improves the existing MoE framework in terms of both efficiency (throughput) and effectiveness (performance), making it impactful for real-world applications," as well as that "this paper is well-written and insightful, with clear motivation and illustrations." Below, we provide detailed responses to your questions.

Q1: This work lacks real-world wall-clock time demonstrations of these zero-computation expert operators, especially regarding a batch of tokens.

A1: Thanks for your insightful advice. Following your suggestion, we measured the real-world wall-clock time for these zero-computation expert operations on a single A100 GPU.

As shown in the table below, the computational overhead of the zero-computation expert is negligible compared to that of the FFN expert. We have added these results in the revised manuscript (Table A on Page 15).

Q2: This work introduces three types of zero-computation experts (i.e. zero, copy, and constant) but provides minimal justification for this specific set. From Table 5, we can see they only conducted basic combinatorial ablations.

A2: Thanks for your valuable comments. In our method, we consider the redesigned expert architecture should satisfy specific criteria:

• It should be as streamlined as possible to process simple tokens efficiently;
• To ensure a fair comparison with the vanilla MoE, the new expert should introduce an almost negligible number of parameters.

Guided by these principles, we propose three types of zero-computation experts: the zero expert, copy expert, and constant expert, which correspond to discard, skip, and replace operations, respectively.

Nevertheless, our MoE++ method can be easily extended to include more expert types, such as FFNs with varying hidden feature dimensions, although this poses a challenge for fair comparisons with standard MoE methods. We will also continue to explore the possibility of more expert types in future work.

In Table 5, we explore as many different combinations of experts as possible. Specifically, as shown in the table below, we examine all possible expert combinations.

Due to the time constraints for rebuttal, we are unable to provide results for FFNs with different dimensions. However, we will continue to improve our method as per your suggestion.

Q3: How are these zero-computation experts specialized to input tokens? Specifically, it would be great to show some examples of tokens allocated to zero, copy, and constant experts, respectively.

A3: Thanks for your valuable advice. Following your suggestion, we have added the visualization of the number of zero-computation experts (zero experts, copy experts, and constant experts) activated per token at the token level (Figure A on Page 19).

These visualizations reveal several interesting findings:

• Zero experts specialize in processing the simplest tokens, such as single letters, spaces, and punctuation.
• Copy experts focus on handling word fragments, such as "hol" and "ts", while also managing single letters and punctuation.
• Constant experts specialize in processing word fragments and simple words, such as "it" and "She".

These results demonstrate that our proposed MoE++ automatically assigns tokens to experts based on their complexity and the capabilities of experts. Specifically, zero experts output only empty features, copy experts replicate the input as output, and constant experts introduce additional trainable vectors to adjust the output. Therefore, MoE++ assigns the simplest tokens to zero experts, moderately simple tokens to copy experts, and slightly more complex tokens to constant experts.

Besides, the table below provides examples of tokens handled by each type of expert.

Dear Reviewer

Could we kindly inquire if the responses have satisfactorily tackled your concerns, or if there is a need for further experiment and visualization? Your commitment to reviewing our work is immensely appreciated. We sincerely thank you for your prompt and insightful review of our paper. Your comment is immensely appreciated and undoubtedly helps improve the quality of our work.

We sincerely appreciate your thoughtful comments and your recognition that our proposed zero-computation experts "with suitable routing designs can improve the model's efficiency and effectiveness." Below, we provide detailed responses to your questions.

Q1: Parameters such as $\tau$, which regulate token allocation between zero-computation experts and original, may complicate model tuning, as the performance and burden distribution are sensitive to them.

A1: Thanks for your valuable comments. In our method, $\tau$ can be used to balance computing costs and performance. Specifically, a smaller $\tau$ means that more tokens are assigned to the zero-computation experts with negligible computing costs, resulting in higher throughput. Conversely, a larger $\tau$ means fewer tokens are allocated to the zero-computation experts and generally have better performance.

As shown in the tables below, we find that $\tau=0.75$ strikes a good balance between computing costs and model performance. Therefore, we recommend setting the hyper-parameter $\tau$ to 0.75 by default.

Q5: Could certain components be removed or modified for specific use cases?

A5: Thanks for your insightful comments. In our method, all types of experts are selected by the router, so there is no need to manually remove or modify experts.

Following your suggestion, we try to disable some experts on the WinoGrande benchmark. As shown in the table below, we find that manually removing some experts would hurt the performance of the model.

Q2: The pathway-aware routing with gating residuals adds complexity to the expert selection process, which may require careful tuning for optimal results.

A2: Thanks for your thoughtful comments. The computing costs of gating residuals are negligible.

In our proposed pathway-aware router, we add the routing scores from the previous layer to the routing scores predicted by the current layer. Specifically, given the input token ${x}^{j}$ of the $j_{th}$ ($j$>1) layer with $N$ experts, we use a trainable transformation matrix ${W}^j_{g} \in \mathbb{R}^{N\times N}$ to integrate the scores from the previous layer into the current layer: $G({x}^{j})={W}^{j}{x}^{j}+{W}^{j}_{g}G({x}^{j-1})$.

For example, in a 16E MoE model with a hidden size of 1536, the additional FLOPs from gating residuals are only 1% of the FLOPs caused by the original router.

Q3: The dynamic routing mechanism can still result in load imbalances, especially under diverse data distributions, which may lead to underutilized or overloaded experts that affect efficiency.

A3: Thanks for your valuable comments. We address your concern from the following two perspectives:

• First, load imbalance is a common issue for all MoE methods. Specifically, model parameters are dynamically activated in MoE. MoE methods balance the load across experts when handling a lot of data. However, load imbalance can still happen during inference on a single sample. This means some experts might always be activated, while others are not used.
• Second, our proposed MoE++ faces a lower risk of load imbalance compared to the vanilla MoE methods. Specifically, since zero-computation experts have negligible parameters, we can deploy all zero-computation experts on each GPU. This setup eliminates the significant communication overhead and load imbalance typically caused by FFN experts being distributed across multiple GPUs.

Q4: How do individual components, like zero experts or gating residuals, contribute to MoE++'s overall performance?

A4: Thanks for your helpful comments. We show in Table 5 how the components, like zero experts, contribute to MoE++'s overall performance. Table 6 shows how gating residuals contribute to MoE++'s overall performance.

As shown in the table below, we find that constant experts improve the model more than zero experts and copy experts. We consider that it is due to the increased flexibility that constant experts provide in handling tokens. Specifically, zero experts output only empty features, copy experts replicate the input as output, and constant experts introduce additional trainable vectors to adjust the output. Our full model, which incorporates all three types of zero-computation experts, achieves the best performance, demonstrating their benefit for language models.

As shown in the table below, we find that gating residuals effectively establish connections between different MoE++ layers, thereby ensuring stable routing.

Dear Reviewer

May we kindly inquire if the provided responses have adequately addressed any questions you might have had? If there remains a requirement for further explanations or clarifications? We wish to express our sincere gratitude for your meticulous evaluation and for generously investing a significant amount of your time in reviewing our paper. Your feedback would be greatly valued.

We sincerely appreciate your thoughtful comments, especially noting that we "provide a fairly simple and effective addition to the standard MoE architecture which effectively allows more heterogeneous computing in different layers with minimal overhead" and that "the paper is written clearly and a thorough set of ablations are performed." Below, we provide detailed responses to your questions.

Q1: The empirical gains don't seem to be too significant and can only start to be seen at large scale ~7B parameters.

A1: Thanks for your insightful comments. We explain it to you from the following three aspects:

• Our method achieves an average improvement of 1.9 on small-scale models, such as the 1B model, across nine benchmarks. Given that our method significantly enhances the model's training and inference speed without increasing the number of parameters, we consider this average accuracy gain of 1.9 to be meaningful.
• In addition to improving the model performance, our method is also valuable to improve the efficiency of the model. We show that our method achieves approximately 1.1$\sim$2.1$\times$ the expert forward throughput of a vanilla MoE model of the same size.
• In practical applications, MoE models are typically large in scale. Therefore, it is acceptable that our method achieves greater improvements on models with larger-scale parameters.

Q2: Are the baseline MoE models also trained with residual gating?

A2: Thanks for your detailed review of our manuscript. The baseline MoE models are trained without residual gating.

The table below shows the results of the baseline MoE model using residual gating. As shown in the table below, we find that the improvement of residual gating on the vanilla MoE model is not significant.

We consider this may be because all experts in the vanilla MoE are structured the same. In contrast, our proposed MoE++ contains heterogeneous experts, making the design of the router more critical compared to vanilla MoE. As a result, the benefits of residual gating are more pronounced in MoE++.

Q3: For each model, how many experts are activated in each layer?

A3: Thanks for your detailed review of our manuscript. In all models, two experts are activated at each layer.

We have clarified the model settings in the revised manuscript. Specifically, we have added the following note to the caption of Table 2 (which presents the model settings): "In all models, two experts are activated at each layer."

Dear Reviewer

Would it be possible for us to kindly ascertain if the provided responses have satisfactorily tackled any concerns you may have had and if further explanations or clarifications are needed? Your generous investment of time and effort in the evaluation of our work is truly commendable. We extend our heartfelt gratitude for your insightful commentary and the considerable time you have devoted to reviewing our paper.