[Weakness 1. The paper targets MoE experts only, some of the notations and terms seem to be used specifically in the MoE domain.]

Answer

Our work indeed targets MoE architectures, so the notation and terminology are typically taken from this literature. For future work, we do hope to extend our approach to further model families within deep learning. We would appreciate your further comment or suggestions on how to improve our notation and terms, which we would be happy to address during discussion.

We would like to thank the reviewer again for your thoughtful reviews and valuable feedback.

We would appreciate it if you could let us know if our responses have addressed your concerns and whether you still have any other questions about our rebuttal.

We would be happy to do any follow-up discussion or address any additional comments.

[Weakness 1. Would it be possible to simplify the operations to attain similar performance? A more systematic comparison with related work would clarify the practical impact]

Answer

We have added three additional comparisons with related work, which we present below. The first is using the SoftMoE backbone [1] (shown also in Table 17 of Appendix C.7), which shows that ACMoE is readily adaptable to problem settings in which expert clusters are broadly specialized and overlapping. The second and third are the Switch and GlaM backbones using the StableMoE [2] router (shown also in Table 3 of Section 4), where we see that ACMoE maintains consistent performance gains over StableMoE as well. Furthermore, we present in Figure 5 of Appendix C.9 additional empirical analysis for router stability, where we see ACMoE leads to substantially more stable routing against baseline routers.

Table 1: Test Accuracy on ImageNet corrupted PGD, FGSM, and SPSA using SoftMoE [1] backbone

Revised Table 3 of Section 4.2: ACMoE in Switch and GlaM backbones, now showing additional StableMoE [2] results.

Regarding the simplicity of the method, we agree there is always a possibility that we could perform our method with simplified computations and that this is a worthwhile avenue of research. Nonetheless, we present here two arguments for why we believe our method is already presented in a highly simplified form:

1. Computing MAD is a highly efficiently computed measure of dispersion, requiring just two computations of the mean, done in parallel. Computing variance requires additional squaring operations, and the calculations of interquartile range and median require cumbersome sorting operations.
2. We obtain our estimates of token cluster membership by simply retrieving the expert assignments from the previous layer, which avoids the need to explicitly cluster the tokens, which would typically require slow, iterative algorithms.

[1]: Puigcerver et al. From Sparse to Soft Mixtures of Experts (ICLR 2023)

[2]: Dai et al. StableMoE: Stable Routing Strategy for Mixture of Experts (ACL 2022)

[Weakness 2. Providing a figure on the times where top-k selection differed in ACMoE vs baselines could help clarify the proposed benefits of ACMoE with regard to load balance / distribution of expert activation]

Answer

Thanks for your suggestion. We have added into Appendix C.9 of our revised manuscript Figure 5, showing the proportion of tokens for which the routing changed in ACMoE as compared with SMoE, XMoE, and StableMoE in the Switch backbone evaluated on WikiText-103. We see that XMoE maintains highly changeable routing throughout the model, while SMoE and StableMoE start off with consistent routing but by the final layer become unstable. ACMoE, by contrast, produces substantially more stable routing as compared with SMoE, XMoE and StableMoE, which complements the ability of experts to specialize by maintaining the routing of semantically similar tokens to the same experts.

We would like to thank the reviewer again for your thoughtful reviews and valuable feedback.

We would appreciate it if you could let us know if our responses have addressed your concerns and whether you still have any other questions about our rebuttal.

We would be happy to do any follow-up discussion or address any additional comments.

[Question 3. Since the method requires consistent embedding dimensions between layers, how can it be adapted for architectures where the embedding size changes?]

Answer

We break our answer to this question down into two parts. First, we address the prevalence of consistent embedding size between layers, and second, we provide two future research directions for how to nonetheless apply our framework in this situation.

First, in the majority of contemporary deep MoE architectures, the embedding size is typically constant throughout the entire model, and examples of changing embedding sizes are more the exception than the rule. For example, in Switch, GLaM, SoftMoE, and VMoE, the embedding size remains the same throughout. Swin is, so far, the only transformer-MoE model we've encountered that features a changing embedding size, but still maintains the same embedding size for 18 of its 22 total layers, and so there remains ample opportunity for applying ACMoE.

We note further that many standard-practice architectural features require the same assumption of constant embedding size, such as the residual connection. So in this sense, our requirement of constant embedding size at adjacent MoE layers is no more restrictive than what is required for commonplace design choices.

As to the question of how might the AC router by used be in situations where the embedding size changes at two adjacent layers, we present two possible future research directions for applying the AC router framework without using previous layer assignments as estimates of cluster membership:

a) One could estimate the clustering structure of the input tokens without reliance on previous layer expert assignments by applying a few steps of a clustering algorithm to the tokens before sending them into the router. A straightforward choice would therefore be to use k-means clustering on the input tokens. To reduce expense of this approach, one could try just one or two steps of k-means.

b) In vision-only settings, one could also try applying an image segmenter before the router to segment the tokens into semantically similar groupings.

We do think it worth noting, however, that we don't see reliance on previous layers as an overly burdensome or problematic scheme for obtaining the cluster assignments. Indeed, this method is efficient, simple to implement, works well empirically, and is justified by the mild assumption that previous layer assignments are good estimates of token cluster membership. When compared with alternate ideas for obtaining the estimates of the cluster membership of the tokens (for example the two above ideas of k-means and image segmenters), we note that our proposed method would be much more efficient.

[Question 1. Have you considered or tested other measures of dispersion besides MAD? How sensitive is the method to the choice of dispersion measure, and could alternative measures potentially improve performance or robustness?]

Answer

Thanks for your suggestion. We investigated different dispersion measures for our proposed AC Router and reported the results in Tables 9 and 11 in Appendix C.5.2 of our manuscript. We present those results here as well for viewing convenience. In particular, we see that using variance as the dispersion measure performs fairly similarly to MAD, which agrees with our expectation that the method should not be overly sensitive to the measure of dispersion. Nonetheless, MAD outperforms variance, which we hypothesize is due to MAD being a more robust metric, hence why we select it.

We do note that interquartile range may also be an interesting measure of dispersion to try, but we do not test it as it would be prohibitively slow, since it would require sorting the tokens over all dimensions, per cluster.

Table 9 of Appendix C.5.1. Ablation on measure of dispersion in Switch Transformer backbone

Table 11 of Appendix C.5.1. Ablation on measure of dispersion in Swin Transformer backbone

[Question 2. How does the AC router handle situations where the assignments from previous layers are noisy or not well-defined? Is there a way to initialize or adjust the method to be effective in early layers or when prior assignments are unreliable?]

Answer

The AC router can indeed be straightforwardly applied in situations when prior assignments are noisy. Below, we demonstrate the efficacy of our AC router using cluster weight mixing, in which we soften our estimated cluster assignments by modeling the confidence with which we believe a token belongs to any cluster in the top-k routing assignment. Furthermore, as shown by the results using SoftMoE, our ACMoE performs well in a setting where all experts are active for each token, representing a setting in which expert clusters are highly overlapping. In all cases, we see ACMoE is readily adaptable to these settings of noisy or ill-defined clusters and is able to continue delivering the proposed performance gains.

Table 1: Test Accuracy on ImageNet corrupted PGD, FGSM, and SPSA using SoftMoE [1] backbone

Table 2: Test Accuracy on ImageNet corrupted PGD, FGSM, and SPSA using Swin Base backbone and cluster mixing over top-2 highest affinity experts

Table 3: ACMoE with cluster weight mixing in Switch and GLaM

Performance of our model when clusters are indistinct. In addition to the experiments on WikiText-103 and ImageNet mentioned above, to more explicitly justify the performance of our model in this setting, we integrate our AC router into the SoftMoE [7], in which each token is soft-assigned to every expert, thereby modeling more broadly specialized, overlapping expert clusters. We also perform additional studies on our method using cluster weight mixing, which models this same scenario of overlapping clusters, in the Switch and GlaM backbones. Please see the additional results in the global response, where we see that our framework adapts well to this setting and continues to deliver strong performance gains. We paste the results here as well for viewing convenience:

Table 1: Test Accuracy on ImageNet corrupted PGD, FGSM, and SPSA using SoftMoE [7] backbone

Table 2: Test Accuracy on ImageNet corrupted PGD, FGSM, and SPSA using Swin Base [8] backbone and cluster mixing over top-2 highest affinity experts

Table 3: ACMoE with cluster weight mixing in Switch [9] and GLaM [10]

Finally, we would like to make a remark about scenarios where clusters are indistinct. The existence of latent clusters within the input distribution is a fundamental and motivating assumption underlying the MoE framework [11,12], as the framework is largely built upon this premise. Consequently, we argue that if, in the extreme, an input distribution contains completely indistinguishable and overlapping clusters, such a situation may render the MoE framework as a whole less suitable. In this context, our assumption of identifiable clusters in the input distribution is no more restrictive than the foundational assumption of the MoE framework itself. Our model is specifically designed to address scenarios where clusters are challenging to identify due to their varying dependencies on subsets of features. Furthermore, as demonstrated by our additional results involving cluster weight mixing and SoftMoE, our approach continues to perform effectively in settings where expert clusters are highly overlapping.

[6] Witten & Tibshirani, A framework for feature selection in clustering (Journal of the American Statistical Association, 2010)

[7] Puigcerver et al. From Sparse to Soft Mixtures of Experts (ICLR 2023)

[8] Liu et al. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (ICCV 2021)

[9] Fedus et al. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (JMLR 2022)

[10] Du et al. GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (ICML 2022)

[11] Robert Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts (Neural computation 1991)

[12] David Eigen, Marc'Aurelio Ranzato, Ilya Sutskever. Learning Factored Representations in a Deep Mixture of Experts (ICLR 2014)

[Weakness 3. The method assumes that the input data naturally clusters in the feature space. While this assumption holds in many cases (e.g., language modeling and image classification), it might not generalize to tasks where the input data is more uniformly distributed or lacks clear clustering patterns.]

Answer

Thanks for your comments. Below we address your concerns about: 1) the setting to which our approach is well-tailored, and 2) the performance of our model when latent clusters are indistinct, and we address them each in turn.

The setting to which the approach is well-tailored. Though we do expect our approach to work well in settings with well-structured clusters, our method is actually more so motivated by settings in which clusters are not easily identified. From the perspective of cluster analysis, these are challenging problem settings in which classical clustering algorithms typically fail to discover the clustering structure of the data in the untransformed feature space [6]. This is why our feature-weighted clustering optimization setup for the proposed AC router in Eqns. 4 and 5 explicitly use different weights for each cluster so that we permit the possibility that clusters depend on differing, possibly disjoint, sets of features. We also validate our AC router on large-scale, natural datasets such as WikiText-103 and ImageNet, where the latent clusters in the input distribution may not be easily discoverable. As shown in Table 1, 2, 3, 4, and 5 in the main text, our AC router improves the accuracy and robustness of the baseline model on these benchmarks.

We would also like to clarify that we do not choose the features along which experts cluster. Instead, we learn features from the data, and use them to transform the space in which routing takes place to improve token-expert matching.

[Weakness 1. The AC router relies on expert assignments from the previous layer, which may limit its applicability in scenarios where embedding sizes change between layers]

Answer

In the majority of contemporary deep MoE architectures, the embedding size is typically constant throughout the entire model, and examples of changing embedding sizes are more the exception than the rule. For example, in Switch [1], GLaM [2], SoftMoE [3], and VMoE [4], the embedding size remains the same throughout. Swin [5] is, so far, the only transformer-MoE model we've encountered that features a changing embedding size, but still maintains the same embedding size for 18 of its 22 total layers, and so there remains ample opportunity for applying ACMoE.

We note further that many standard-practice architectural features require the same assumption of constant embedding size, such as the residual connection. So in this sense, our requirement of constant embedding size at adjacent MoE layers is no more restrictive than what is required for commonplace design choices.

[1] Fedus et al. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (JMLR 2022) [2] Du et al. GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (ICML 2022) [3] Puigcerver et al. From Sparse to Soft Mixtures of Experts (ICLR 2023) [4] Riquelme et al. Scaling Vision with Sparse Mixture of Experts (NeurIPS 2021) [5] Liu et al. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (ICCV 2021)

[Weakness 2: The effectiveness of the AC router is tied to the quality of expert assignments in earlier layers. Since the method relies on routing decisions from prior layers, it may be less effective if the early layers have not yet learned meaningful cluster structures. As a result, the AC router may struggle to make optimal routing decisions in later layers if the cluster assignments in the initial layers are poor.]

Answer

We agree with the reviewer and indeed initially shared this concern when first desigining the method. Nonetheless, encouragingly, we actually find that ACMoE not only outperforms baselines when only applied in later layers, but attains best performance when applied in early layers as well. This offers evidence that even at early layers before the experts have learnt fine-grained or well-separated structures, there is still enough information in the cluster assignments that we can leverage to meaningfully apply our AC routing transformation and obtain the proposed benefits. We refer the reviewer to Appendix C.5.2 for the results, and display them here as well for convenience. Our empirical results suggest that in Swin the best performance is attained with the AC router on every possible layer. For Switch, we see a small performance bump by skipping the first layer as opposed to Full, but that Skip 1 still outperforms Back Half, indicating that earlier placement of the AC router is still beneficial.

Swin-ACMoE Ablation Study on AC router layer placement

Switch-ACMoE Ablation Study on AC router layer placement

The names are as follows:

Full: AC router on every layer

Alternating: AC router on alternating layers

Skip 1: AC router on every layer except for the first possible layer

Skip 2: AC router on every layer except for the first two possible layers

Back Half: AC router only on the back half of the total layers

We would like to thank the reviewer again for your thoughtful reviews and valuable feedback.

We would appreciate it if you could let us know if our responses have addressed your concerns and whether you still have any other questions about our rebuttal.

We would be happy to do any follow-up discussion or address any additional comments.

[Weakness 2. The AC router relies on prior layers for initial expert assignments, requiring consistent embedding sizes across layers.]

Answer

In the majority of contemporary deep MoE architectures, the embedding size is typically constant throughout the entire model, and examples of changing embedding sizes are more the exception than the rule. For example, in Switch [4], GLaM [5], SoftMoE [6], and VMoE [7], the embedding size remains the same throughout. Swin is, so far, the only transformer-MoE model we have encountered that features a changing embedding size, but still maintains the same embedding size for 18 of its 22 total layers, so there remains ample opportunity for applying ACMoE in Swin [8].

We note further that many standard-practice architectural features require the same assumption of constant embedding size, such as the residual network. So in this sense, our requirement of constant embedding size at adjacent MoE layers is no more restrictive than what is required for commonplace design choices.

[4] Fedus et al. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (JMLR 2022) [5] Du et al. GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (ICML 2022) [6] Puigcerver et al. From Sparse to Soft Mixtures of Experts (ICLR 2023) [7] Riquelme et al. Scaling Vision with Sparse Mixture of Experts (NeurIPS 2021) [8] Liu et al. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (ICCV 2021)

[Weakness 3. Although tested on large-scale datasets, application to dynamic datasets could strengthen claims of robustness and adaptability.]

Answer

Thanks for your suggestion. We are currently working on validating our method for continual learning tasks to evaluate the ability of our proposed AC router to adapt to dynamically evolving clusters. We will update you with the additional results as soon as have them.

[Question 2. What mechanisms could be integrated to allow routing without relying on previous layer assignments?]

Answer

We agree that this is an interesting future direction, and we are actively working on this. We do not yet have a finished idea, but we would be happy to share a couple ideas that we think could be helpful for future researchers interested in joining us on this direction.

a) One could estimate the clustering structure of the input tokens without reliance on previous layer expert assignments by applying a clustering algorithm to the tokens before sending them into the router. A straightforward choice would therefore be to use k-means clustering on the input tokens. To reduce the expense of this approach, one could try just one or two steps of k-means.

b) In vision-only settings, one could also try applying an image segmenter before the router to segment the tokens into semantically similar groupings.

We do think it worth noting, however, that we do not see reliance on previous layers as an overly burdensome or problematic scheme for obtaining the cluster assignments. Indeed, this method is efficient, simple to implement, works well empirically, and is justified by the mild assumption that previous layer assignments are good estimates of token cluster membership, especially in later layers of the model. When compared with the alternative ideas discussed above for obtaining the estimates of the cluster membership of the tokens (for example the two above ideas of k-means and image segmenters), we note that our proposed method would be much more efficient.

Performance of our model when clusters are indistinct. In addition to the experiments on WikiText-103 and ImageNet mentioned above, to more explicitly justify the performance of our model in this setting, we integrate our AC router into the SoftMoE, in which each token is soft-assigned to every expert, thereby modeling more broadly specialized, overlapping expert clusters. We also perform additional studies on our method using cluster weight mixing, which models this same scenario of overlapping clusters, in the Switch and GlaM backbones. Please see the additional results in the global response, where we see that our framework adapts well to this setting and continues to deliver strong performance gains. We paste the results here as well for viewing convenience:

Table 1: Test Accuracy on ImageNet corrupted PGD, FGSM, and SPSA using SoftMoE backbone

Table 2: Test Accuracy on ImageNet corrupted PGD, FGSM, and SPSA using Swin Base backbone and cluster mixing over top-2 highest affinity experts

Table 3: ACMoE with cluster weight mixing in Switch and GLaM

Finally, we would like to make a remark about scenarios where clusters are indistinct. The existence of latent clusters within the input distribution is a fundamental and motivating assumption underlying the MoE framework [2,3], as the framework is largely built upon this premise. Consequently, we argue that if, in the extreme, an input distribution contains completely indistinguishable and overlapping clusters, such a situation may render the MoE framework as a whole less suitable. In this context, our assumption of identifiable clusters in the input distribution is no more restrictive than the foundational assumption of the MoE framework itself. Our model is specifically designed to address scenarios where clusters are challenging to identify due to their varying dependencies on subsets of features. Furthermore, as demonstrated by our additional results involving cluster weight mixing and SoftMoE, our approach continues to perform effectively in settings where expert clusters are highly overlapping.

[1] Witten & Tibshirani, A framework for feature selection in clustering (Journal of the American Statistical Association, 2010)

[2] Robert Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts (Neural computation 1991)

[3] David Eigen, Marc'Aurelio Ranzato, Ilya Sutskever. Learning Factored Representations in a Deep Mixture of Experts (ICLR 2014)

[Weakness 1 & Question 1: The approach seems to tailor to settings with well-structured clusters, which may limit performance where latent clusters are less distinct or not aligned with chosen features. How does the AC router handle overlapping clusters where expert specialization may be less clear?]

Answer

Thanks for your comments. Below we address your concerns about: 1) the setting to which our approach is well-tailored, and 2) the performance of our model when latent clusters are indistinct, and we address them each in turn.

The setting to which the approach is well-tailored. Though we do expect our approach to work well in settings with well-structured clusters, our method is actually more so motivated by settings in which clusters are not easily identified. From the perspective of cluster analysis, these are challenging problem settings in which classical clustering algorithms typically fail to discover the clustering structure of the data in the untransformed feature space [1]. This is why our feature-weighted clustering optimization setup for the proposed AC router in Eqns. 4 and 5 explicitly use different weights for each cluster so that we permit the possibility that clusters depend on differing, possibly disjoint, sets of features. We also validate our AC router on large-scale, natural datasets such as WikiText-103 and ImageNet, where the latent clusters in the input distribution may not be easily discoverable. As shown in Table 1, 2, 3, 4, and 5 in the main text, our AC router improves the accuracy and robustness of the baseline model on these benchmarks.

We would also like to clarify that we do not choose the features along which experts cluster. Instead, we learn features from the data, and use them to transform the space in which routing takes place to improve token-expert matching.

We would like to thank the reviewer again for your thoughtful reviews and valuable feedback.

We would appreciate it if you could let us know if our responses have addressed your concerns and whether you still have any other questions about our rebuttal.

We would be happy to do any follow-up discussion or address any additional comments.

We have further tested our AC routing framework in dynamic MoE settings.

Following the approach of [1], we implement ACMoE into top-$p$ dynamic gating. In this setting, rather than routing each token to its top-k highest affinity experts in each MoE layer, we route each token to all experts that have affinity over a certain threshold $p$. This setting permits dynamically activating varying numbers of experts for different tokens at different layers throughout the model. We integrate our AC routing directly into this setting using the same setup as in Section 3 of our manuscript, where the AC routing transformation is computed based on the estimated cluster membership of each token using the top affinity assignment of the previous layer. We present the results for Switch transformer on WikiText-103 language modeling in the following Table A. The same results can be found in Table 20 in Appendix C.10 of our revised manuscript.

Table A: Results on Top-p Dynamic Routing in Switch Backbone

For fixed $p$, we set $p = 0.05$. For learnable $p$, we initialize the parameter to 0.05. We select this initialization as it reproduces approximately similar performance in the Switch backbone under default top-2 routing, thereby aiding direct comparison between fixed top-k and dynamic top-$p$ routing. We see in the dynamic routing setting, ACMoE maintains the same consistent improvement over the Switch baseline of roughly 1 full PPL. These results suggest ACMoE is well-suited to the dynamic MoE setting.

We hope our responses have resolved your concerns. If you believe that our replies have adequately addressed the issues you raised, we kindly ask you to consider whether updating your score would more accurately reflect your updated evaluation of our paper. Thank you once again for your time and thoughtful feedback!

[1] Guo et al. Dynamic mixture of experts: An auto-tuning approach for efficient transformer models (2024)

[2] Shazeer et al. The Sparsely-Gated Mixture-of-Experts Layer (ICLR 2017)

To this end, we present in the following tables results for ACMoE integrated into the SoftMoE backbone and further results on the Switch, GLaM, and Swin backbones when applying top-2 cluster mixing. Additionally, we present results for Swin at the larger 0.5B param 'Base' configuration, to offer further evidence of our ACMoE's ability to scale. Results are additionally found in Appendix C.6, C.7, C.8.

Table 1: Test Accuracy on ImageNet corrupted PGD, FGSM, and SPSA using SoftMoE [1] backbone

To accomodate the SoftMoE setting, in which all experts are active for each token, we apply cluster weight mixing in ACMoE over the top 8 highest affinity expert clusters. Furthermore, we apply ACMoE on every possible layer of SoftMoE. We see in Table 1 strong, consistent performance gains over SoftMoE, in particular with large robust gains in the range of 6-7%.

Table 2: Test Accuracy on ImageNet corrupted PGD, FGSM, and SPSA using Swin Base [2] backbone and cluster mixing over top-2 highest affinity experts

We see in Table 2 that weight mixing in the Swin backbone at the base configuration maintains consistent gains in clean and contaminated performance.

Table 3: ACMoE with cluster weight mixing in Switch [3] and GLaM [4]

Here, in Table 3, Mix 1 refers to the original result presented in the main text, where we take the top affinity expert cluster as the estimated cluster membership of a given token. Mix 2 refers to ACMoE when applying cluster weight mixing between the top 2 highest affinity experts for each token. For Switch, results are fairly similar across whether we mix or not. Interestingly, however, GLaM-ACMoE improves by almost an entire PPL, which may indicate that in the GLaM architecture, experts learn much broader specializations (i.e expert clusters overlap), and so token-expert matching is best performed in a space transformed according to the specialization of the top 2 highest affinity expert clusters for each token.

In general, results agree with our expectation that cluster mixing, seen both via our additional results on Swin, Switch, and GLaM, and our results on SoftMoE, offers a straightforward method to model the uncertainty in cluster assignments and the setting in which clusters are overlapping. Through this extension, our method can straightforwardly handle both of these settings well and continue to deliver substantive performance gains.

We hope that our rebuttal has helped to clear concerns about our work. We are glad to answer any further questions you have on our submission and we would appreciate it if we could get your further feedback at your earliest convenience.

[1]: Puigcerver et al. From Sparse to Soft Mixtures of Experts (ICLR 2023)

[2]: Liu et al. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (ICCV 2021)

[3]: Fedus et al. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (JMLR 2022)

[4]: Du et al. GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (ICML 2022)

[5] Riquelme et al. Scaling Vision with Sparse Mixture of Experts (NeurIPS 2021)