Q1: The results from this paper do not fully support the main claim that “sigmoid gating is more sample efficient than softmax gating” because the comparison between sigmoid and softmax is not under the same setup.

Thanks for your comment. Let us explain why it is reasonable to compare the expert estimation rates under the MoE model when using the sigmoid gating versus when using the softmax gating.

Our paper (resp. Nguyen et. al. [21]) consider a well-specified setting where the data are generated from a regression model where the regression function is a sigmoid gating (resp. softmax gating) MoE in order to lay the foundation for a more realistic yet challenging misspecified setting where the data are not necessarily generated from those models.

Under that misspecified setting, the regression function is an arbitrary function $g(x)$ which is not necessarily a mixture of experts. Then, the least square estimator $\widehat{G}_n$ defined in equation (3) converges to the mixing measure $\widetilde{G}\in\mathcal{G}_k$ that minimizes the distance

$||f_{G}-g||_2$,

where $f_{G}$ is the regression function taking the form of the sigmoid gating (resp. softmax gating) MoE.

The insights from our theories and from Nguyen et. al. [21] under the well-specified setting indicate that the Voronoi loss functions can be used to obtain the estimation rates of individual parameters of the least square estimator $\widehat{G}_n$ to those of $\widetilde{G}$, and therefore, the expert estimation rates. Let us recall from Table 1 that under the Regime 1 (all the over-specified gating parameters are zero), using the sigmoid gating leads to the same expert estimation rates as when using the softmax gating. However, under the Regime 2 (not all the over-specified gating parameters equal zero), which is more likely to occur, the sigmoid gating totally outperforms the softmax gating in terms of the rates for estimating feed-forward expert networks and polynomial experts ($n^{-1/2}$ compared to either $n^{-1/4}$ or $1/\log(n)$).

Thus, under the misspecified setting, the expert estimation rates achieved when we use the sigmoid gating should be faster than those obtained when we use the softmax gating. This accounts for the validity of the expert estimation rate comparison between these two gating functions.

Q2: Is there anything regarding the non-competition of the sigmoid gating that can be reflected in the analysis?

Thanks for your question. Let us show the connection between the non-competition of the sigmoid gating and our analysis. In particular, when using the sigmoid gating, the experts do not share the gating parameters, that is, the mixture weights are independent of each other. Thus, the interaction between the gating and expert parameters which induces slow estimation rates when using the softmax gating (see Eq. (4) in [21]) does not hold if we use the sigmoid gating. As a result, the expert estimation rates when using the sigmoid gating are either comparable (under the Regime 1) or faster (under the Regime 2) than those when using the softmax gating.

Q3: It would be helpful to provide empirical results to further support the main conclusion that sigmoid gating is more sample efficient. Even experiments on toy data can help.

Thanks for your suggestion. Please refer to our response to the Common Question 1 in the General Section for further details.

Q4: Even "sigmoid gating is better than softmax gating" alone does not seem to be well-supported by empirical observations from the literature.

Thanks for your comment. However, we respectfully disagree that the claim "sigmoid gating is better than softmax gating" is not well-supported by empirical observations from the literature. In particular, there are two recent works [1, 3] on the applications of MoE in language modeling showing that the performances when using the sigmoid gating are comparable or even better than those when using the softmax gating.

Finally, we would like to emphasize that our main goal is to demonstrate that the sigmoid gating is more sample efficient than the softmax gating from the perspective of the expert estimation problem. We have not attempted to show that the sigmoid gating is better than the softmax gating in general.

References

[1] Z. Chi. On the representation collapse of sparse mixture of experts. Advances in NeurIPS, 2022.

[3] R. Csordás. Approximating two-layer feedforward networks for efficient transformers. Findings of the EMNLP 2023.

[21] H. Nguyen. On least squares estimation in softmax gating mixture of experts. In ICML, 2024.

Dear Reviewer 4kwq,

We would like to thank you for raising your concerns. We hope that our following response will address those concerns, and eventually convince you to increase your rating.

Per your suggestion, we conduct both theoretical and empirical sample efficiency comparisons between the sigmoid gating and the softmax gating under the setting where the data are generated from the same source. Below are the expert estimation rates when using the softmax gating MoE and the sigmoid gating MoE to fit the data, respectively.

Same data generation setting: The data $(X_1,Y_1),\ldots,(X_n,Y_n)$ are generated from a regression framework $Y_i=g(X_i)+\epsilon_i,$ where the features $X_1,\ldots,X_n$ are sampled from a probability distribution $\mu$, and $\epsilon_1,\ldots,\epsilon_n$ are independent Gaussian noise variables such that $\mathbb{E}[\epsilon_i|X_i]=0$ and $Var(\epsilon_i|X_i)=\nu$ for all $1\leq i\leq n$. The unknown regression function $g(x)$ is not necessarily a mixture of experts (MoE). Then, we use either the sigmoid gating MoE or the softmax gating MoE to fit the data.

In particular, the least square estimator $\widehat{G}_n$ in Eq.(3) now converges to a mixing measure $\widetilde{G}\in\mathcal{G}_k(\Theta)$ where

$\widetilde{G}\in\arg\min_{G}||f_{G}-g||_2,$

in which $f_{G}$ is the regression taking the form of the MoE associated with either the sigmoid gating or the softmax gating. Below are the expert estimation rates resulted from our convergence analysis:

When $f_{G}$ is the softmax gating MoE: the rates for estimating feed-forward expert networks with ReLU activation are of order $n^{-1/4}$, while those for polynomial experts are slower than any polynomial rates, and could be as slow as $1/\log(n)$;

When $f_{G}$ is the sigmoid gating MoE: the estimation rates for feed-forward expert networks with ReLU activation and polynomial experts share the same order of $n^{-1/2}$, which are significantly faster than those when using the softmax gating MoE.

Empirical validation: To justify the above theoretical results, we conduct a simulation study in the General Response section where we take polynomial experts into account. The empirical result totally matches its theoretical rates.

As a consequence, we can conclude that the sigmoid gating is more sample efficient than the softmax gating even when fitting the same data.

Please feel free to let us know if you have any further concerns regarding the paper. We are more than happy to address all of them.

Best,

The Authors

Dear Reviewer 4kwq,

Thanks for your question. We would like to confirm that the analysis of the sigmoid gating (resp. softmax gating) MoE under the same data generation setting is implied by the analysis in our paper (resp. in [a]).

Assumption on the regression function $g$. Firstly, let us recall a result on the universal approximation of the sigmoid function in [b]. In particular, let $g:\mathcal{X}\to\mathbb{R}$ be a function such that there is a Fourier representation of the form

$g(x)=\int_{\mathcal{X}}e^{i\omega\cdot x}\tilde{g}(\omega)d\omega,$

for some complex-valued function $\tilde{g}(x)$ for which $\omega\tilde{g}(\omega)$ is integrable and the term

$C_g=\int_{\mathcal{X}}||\omega||_2|\tilde{g}(\omega)|d\omega$

is finite. Then, there exists a linear combination of $\tilde{k}$ sigmoidal functions $f_{\widetilde{G}}(x)$ such that

$\int_{\mathcal{X}}[f_{\widetilde{G}}(x)-g(x)]^2d\mu(x)\leq\frac{(2C_g)^2}{\tilde{k}}.$

Analysis for the sigmoid gating MoE. Subsequently, we combine the above result with our current analysis in the paper. More specifically, by treating the mixing measure $\widetilde{G}$ as the mixing measure $G_*$ in the paper, we are able to design a Voronoi loss

$\mathcal{D}(G,\widetilde{G})=\sum_{j=1}^{k}\sum_{i\in\mathcal{A}_j}$

$\Big[|\beta_{0i}-\tilde{\beta}_{0j}|$

$+||\beta_{1i}-\tilde{\beta}_{1j}||$

$+||\eta_{i}-\tilde{\eta}_{j}||\Big],$

and show that

$\mathcal{D}(\widehat{G}_n,\widetilde{G})=O_P(\sqrt{\log(n)/n}).$

From this bound, we deduce that the expert estimation rates are of order $O_P(n^{-1/2})$.

Analysis for the softmax gating MoE. Similarly, we also leverage the universal approximation of the softmax function in [c], and the analysis in [a] to derive the expert estimation rates under the same data generation setting. In particular, we find out that the rates for estimating feed-forward expert networks with ReLU activation are of order $n^{-1/4}$, while those for polynomial experts are slower than any polynomial rates, and could be as slow as $1/\log(n)$.

As the discussion period deadline is approaching, please let us know if you have any further concerns. We are more than happy to answer your questions. Additionally, if you find that our response sufficiently addresses your concerns, we hope that you will re-evaluate the paper and increase the rating. Thank you again!

References

[a] H. Nguyen. On least squares estimation in softmax gating mixture of experts. In ICML, 2024.

[b] Andrew R. Barron. Universal Approximation Bounds for Superpositions of a Sigmoidal Function. IEEE Transactions on Information Theory, 1993.

[c] Assaf J. Zeevi. Error Bounds for Functional Approximation and Estimation Using Mixtures of Experts. IEEE Transactions on Information Theory, 1998.

Q1: The results are heavily dependent on specific assumptions, such as the distinctness of expert parameters and the boundedness of the input space. If these assumptions are violated in practical scenarios, the theoretical guarantees may not hold.

Thanks for your comments. We will explain why the assumptions mentioned above are reasonable.

Firstly, let us begin with the assumption of distinct expert parameters. This assumption is to ensure that all the experts are different from each other. In practice, it is memory-inefficient to have two identical experts as they have the same expertise but we have to store different weight parameters for them. If there are two identical experts, we can merge them by taking the summation of their weights.

Secondly, regarding the assumption of the bounded input space, we would like to emphasize that this is a standard assumption in the literature of expert estimation in the MoE (see [12, 21]). Moreover, we can address the issue that the magnitude of the input goes to infinity by normalizing the input value, which has been recently applied in practice (see [1, 36]) and should not affect the current theory.

Q2: It would be interesting to see even toy experiments by the authors that show similar conversions to the theoretical ones.

Thanks for your suggestion. Please refer to our response to the Common Question 1 in the General Section for further details.

Q3: The paper lacks justification for the choice of the regimes.

Thanks for your comment. Actually, we have already included the justification for dividing the convergence analysis into two regimes in the "Technical challenges" paragraph (see lines 83-90). Let us summarize it here.

Firstly, we would like to emphasize that the regimes are determined based on the gating convergence. Recall that the true number of experts $k_*$ is unknown, and we over-specify the true model by a mixture of $k$ experts where $k>k_*$. Thus, there must be at least one atom of $G_*$ fitted by two atoms of $\widehat{G}_n$.

WLOG, assume that $(\hat{\beta}^n_{1i},\hat{\eta}^n_i)\to(\beta^*_1,\eta^*_1)$ for all $i\in\{1,2\}$.

Then, we have $h(x,\hat{\eta}^n_i)\to h(x,\eta^*_1)$ for all $i\in\{1,2\}$.

Therefore, the regression function $f_{\widehat{G}_n}$ converges to

$f_{G_*}$ only if $\sum_{i=1}^{2}\sigma((\hat{\beta}^n_{1i})^{\top}x+\hat{\beta}^n_{0i})\to\sigma((\beta^*_{11})^{\top}x+\beta^*_{01}),$ for almost every $x$, where $\sigma$ denotes the sigmoid function. This above limit occurs iff $\beta^*_{11}=0_d$. As a consequence, we propose conducting the analysis under two following complement regimes:

Regime 1: All the overspecified parameters $\beta^*_{1i}$ are equal to zero;

Regime 2. At least one among the over-specified parameters $\beta^*_{1i}$ is different from zero.

Q4: What are the explanations for the strong identifiability condition?

Thanks for your question. We will explain the strong identifiability condition both intuitively and technically as follows:

Intuitively, the strong identifiability condition helps eliminate potential interactions among parameters expressed in the language of PDE (see Eq. (8) and Eq. (11) where gating parameters $\beta_1$ interact with expert parameters $a$). Such interactions are demonstrated to result in significantly slow expert estimation rates (see Theorem 3 and Theorem 4).

Technically, a key step in our proof techniques rely on the decomposition of the discrepancy between $f_{\widehat{G}_n}(x)$ and

$f_{G_*}(x)$ into a combination of linearly independent terms. This can be done by applying Taylor expansions to the function $F(x,\beta_1,\beta_0,\eta):=\sigma(\beta_1^{\top}x+\beta_0)h(x,\eta)$ defined as the product of the sigmoid gating and the expert function $h$. Thus, the condition is to ensure that terms in the decomposition are linearly independent.

Q5: Typo issue.

Thanks for pointing out. We will correct them in the revision of our paper.

References

[1] Z. Chi. On the representation collapse of sparse mixture of experts. Advances in NeurIPS, 2022.

[12] N. Ho. Convergence rates for Gaussian mixtures of experts. In JMLR, 2022.

[21] H. Nguyen. On least squares estimation in softmax gating mixture of experts. In ICML, 2024.

[36] B. Li. Sparse mixture-of-experts are domain generalizable learners. In ICLR, 2023.

Q1: The authors should provide a more concise overview of the main contributions, and remove technical details that are repeated in later sections.

Thanks for your suggestions. We will modify the contribution paragraph as below, and consider removing repeated details in Section 2 in the revision.

Contributions. In this paper, we carry out a convergence analysis of the sigmoid gating MoE under two regimes of the gating parameters. The main objective is to compare the sample efficiency between the sigmoid gating and the softmax gating. Our contributions can be summarized as follows:

(C.1) Convergence rate for the regression function. We demonstrate in Theorem 1 that the regression estimation $f_{\widehat{G}_n}$

converges to its true counterpart $f_{G_*}$ at the rate of order $\mathcal{O}_P(n^{-1/2})$, which is parametric on the sample size $n$. This regression estimation rate is then utilized for determining the expert estimation rates.

(C.2) Expert estimation rates under the Regime 1. Under the first regime, we first establish a condition called strong identifiability to characterize which types of experts would yield polynomial estimation rates. In particular, we find out that the rates for estimating experts formulated as feed-forward networks (FFN) with popular activation functions such as ReLU and GELU are of polynomial orders. By contrast, those for polynomial experts and input-indepedent experts could be of order $\mathcal{O}_P(1/\log(n)$. Such expert convergence behavior is similar to that when using the softmax gating.

(C.3) Expert estimation rates under the Regime 2. Under the second regime, the regression estimation $f_{\widehat{G}_n}$

converge to a function taking the form of a sigmoid gating MoE which is different from $f_{G_*}$. From our derived weak identifiability condition, it follows that estimation rates for FFN experts with ReLU or GELU activation and polynomial experts are of orders $O_P(n^{-1/2})$, which are substantially faster than those when using the softmax gating (see Table 1). Therefore, the sigmoid gating is more sample efficient than the softmax gating.

Q2: If in practice, when using top-k combination of small fine-grained experts, are we dealing in a regime where all true experts are over-specified in practice? If this is the case what are the implications for Regime 1? Does Regime 1 hold in practice?

Thanks for your questions.

Firstly, when using the regression estimation as a mixture of small fine-grained experts [34] (which could have up to millions of experts [35]), it is highly likely that we are dealing with the scenario when all the true experts are over-specified.

Secondly, assume that we use a mixture of small fine-grained experts, and all the true experts are over-specified. Then, under the Regime 1, all the gating parameters $\beta^*_{1j}$ are equal to zero, which means that the true mixture weights (gating values) are independent of the input $x$. Moreover, the fitted parameters $\widehat{\beta}^n_{1i}$ must also converge to zero. Consequently, all the fitted mixture weights also become independent of the input $x$. However, according to the definitions of two regimes, the Regime 1 is much less likely to occur than the Regime 2 where at least one among the over-specified parameters $\beta^*_{1j}$ is different from zero.

Q3: What are the practical implications of the convergence analysis?

Thanks for your question. There are two important practical implications from our analysis:

(i) Expert specialization: The convergence behavior of expert estimation allows us to capture how fast an expert learns a specific task, which is one of the most important problems in the MoE literature known as expert specialization (see [4]). As the sigmoid gating is more sample efficient than the softmax gating, our theories suggest that it would be better to use the sigmoid gating in this field.

(ii) Expert compatibility: Compared to the softmax gating, the estimation rates for feed-forward expert networks with ReLU activation and polynomial experts when using the sigmoid gating are much faster. Thus, our theories indicate that the sigmoid gating is compatible with a broader class of experts than the softmax gating. This implication is particularly useful when people employ a mixture of fine-grained (shallow) expert networks [35].

Q4: The content and organization of the paper are very close to the paper [21].

Thanks for your comment. However, we respectfully disagree that the content and organization of our paper are close to those of [21] for the following reasons:

1. Content:

(1.1) Different objectives: the objective of [21] is to figure out what types of experts are compatible with the softmax gating in terms of estimating experts. Meanwhile, our paper focuses on the sample efficiency comparison between the sigmoid gating and the softmax gating

(1.2) Analysis of similar expert types for comparison: since our main goal is to demonstrate that the sigmoid gating is more sample efficient than the softmax gating, it is necessary to analyze experts considered in [21] for the sake of comparison. This probably makes our content look similar to that in [21]. However, the derived expert estimation rates are different, particularly under the Regime 2 where the sigmoid gating outperforms its softmax counterpart (see Table 1).

2. Organization: In [21], sections for main results are divided based on the types of experts, namely Section 3 for strongly identifiable experts, Section 4.1 for ridge experts with strongly independent activation function, and Section 4.2 for polynomial experts. On the other hand, in our paper, sections for main results are organized based on two regimes of gating parameters. More specifically, Section 3.1 is for Regime 1, while Section 3.2 is for Regime 2.

References

Due to the space limit, we leave the references to the General Response section.