We sincerely thank you for your valuable comments! We hope our rebuttal helps address your concerns. If so, we would be grateful if you could consider increasing the overall recommendation.

---

 

Contribution and Novelty

Thank you for listing [Pham et al.]. Our paper fundamentally differs from it. We believe our AoE presents a solid novelty and technical contribution. Here are the differences.

1. Motivation, Architecture, and Method

• AoE identifies and addresses a widely overlooked issue, i.e., the separation between the router's decision-making and the execution of experts. AoE is a novel MoE paradigm where the router is removed, allowing the experts to autonomously select themselves.

• In contrast, Pham et al. aim to enhance the router. They take the experts' output as additional supervision signals to the router logits, which requires a complex training objective. Additionally, because they sometimes compute every expert's final outputs, this results in dense activation, making the model not technically even an SMoE.

2. Use of Activation Norms

• [1] proposed that activation norms can be used to measure the knowledge of modules. Our approach is inspired by [1] and we cite them in Line 147. Pham et al. do not cite this paper or other related interpretability works. This perspective is not Pham et al.'s innovation, nor do we claim it as ours.

• One of our contributions lies in a novel MoE paradigm that selects experts without routers, using only intermediate activations, along with architecture designs for enhanced efficiency. This is not simply a matter of selecting which nodes to use for norm calculations or improving results. AoE avoids the need to compute every expert's final output, preserving the sparse activation characteristic of SMoE and making AoE significantly more efficient and practical.

3. Efficiency

• AoE achieves similar pre-training throughput of an SMoE.

• Pham et al. did not report efficiency results. Since their model is sometimes densely activated, assuming the common 8-select-2 setting, AoE can be up to four times more efficient during pretraining.

4. Effectiveness

• Pham et al. developed a complex training pipeline and only tested models up to 36M parameters, with evaluation limited to BPC and PPL metrics.

• AoE supports simple end-to-end training, validated at scale with models up to 4B parameters and evaluated across various downstream tasks.

Additionally, Pham et al. publically acknowledged that their complex router-enhancement can not be reproduced, making it impossible to use it as a baseline for comparison. This is stated in the first line of their README: https://github.com/giangdip2410/CompeteSMoE

We would be truly grateful if you could kindly re-assess our novelty and contribution.

References

[1] Transformer Feed-Forward Layers Are Key-Value Memories, EMNLP 2021

 

---

 

Response to Concern 1

Most of this concern has been addressed in our initial response. Here, we address your question that "the expert with the highest intermediate norm is not guaranteed to have the highest output norm."

A neural network works according to how it is trained. Since AoE is not trained like a traditional MoE, this guarantee is unnecessary. As mentioned in the last paragraph of Sec.3.1, we train AoE to represent its awareness of its capabilities through the norm of the intermediate node. To support this claim, we present a new experiment: during inference, when using Final Output Norms (FON) to select experts in a pre-trained AoE (Config.10), rather than the intermediate norms used during training, there is a performance drop across all tasks:

This is a valuable question, as many readers who are accustomed to the traditional MoE models may also have similar concerns. We will include this response in the paper. Thank you very much.

 

---

 

Response to Concern 2

The word "decompose" in Line 202 might be ambiguous. We mean that we modify the model architecture, replacing the $W_g$ in traditional MoE with two low-rank matrices. The architecture is illustrated in Fig.1(b), which does not require dynamic adjustment during runtime. We will improve the clarity in Line 202. Thank you for your feedback.

 

---

 

Response to "Relation To Broader Literature"

MoE is the foundation of advanced LLMs, such as DeepSeek-R1, Qwen-2.5-Max, Grok, etc. A deeper understanding and innovation in MoE could help advance future LLMs. We will highlight the necessity and potential impact of studying MoE in the NLP field.

 

---

If you have further questions, please leave more comments. We are committed to addressing them. Your assessment is very important to us! Thank you very much!

We sincerely thank you for your constructive suggestions and valuable comments! We hope our rebuttal helps address your concerns. If so, we would be grateful if you could consider increasing the overall recommendation.  

---

 

If the baselines should be allowed to pre-train for additional steps

We would like to clarify that in our paper, we ensured fairness by using the same model size and training on the same number of tokens. The difference in throughput (tokens processed per second) results in slightly longer training times of AoE models but does not indicate that AoE has learned additional knowledge. If baselines are trained for additional steps, they would acquire knowledge that AoE does not. If you have any further experimental suggestions, we would be happy to discuss or explore them!

 

---

 

To train Switch Transformer with multiplicative noise

Thank you for your valuable advice. We trained a traditional MoE model with multiplicative jitter noise (0.01), applied to the layer inputs using 100B tokens with $L_{aux}$. As shown by the Switch Transformer, jitter noise encourages expert exploration, which is beneficial for MoE models, but it still cannot outperform AoE models with $L_{aux}$.

We will include these experiments in our paper. Thank you again for your helpful suggestions!

We sincerely thank you for your constructive suggestions and valuable comments! We hope our rebuttal helps address your concerns. If so, we would be grateful if you could consider increasing the overall recommendation.  

---

 

To discuss training time more explicitly

Here are the total machine hours (1 machine = 8 GPUs):

We will include these results. Thank you for the feedback.

 

---

 

Small Model's Performance gains are relatively modest on some tasks

There are two key factors to consider:

1. Larger models are stronger. With the same number of tokens, a large AoE shows more noticeable gains over MoE.

2. The "spiral rise" in performance during pre-training. Both AoE and MoE show steady overall improvements, but task-specific performance fluctuates across checkpoints. We show results at 50B, 80B, and 100B tokens (AoE: Config.10, MoE: Config.2) to illustrate this.

   For example, at 100B tokens, MoE slightly outperforms AoE on PIQA due to a larger gain from 80B to 100B. A similar trend occurs with HELLA from 50B to 80B, though AoE regains the lead at 100B. Despite fluctuations, AoE consistently achieves higher AVG. scores across checkpoints.

Task Performance over tokens

We'll include detailed task accuracy development plots to highlight the consistent advantages of 732M AoEs.

 

---

 

Can AoE insights benefit traditional MoEs?

While AoE shows that routers can be removed, we try to use expert norms as labels for router training. The router receives gradient-detached inputs to avoid interfering with AoE learning. During inference, expert selection is performed solely by the router to reduce memory usage. This setup led to a performance drop (50B tokens, $d_{low}$=256, with $L_{aux}$):

Task Performance

These results suggest that—even with supervision—routers struggle to learn effective expert selection, highlighting the limitation of separating routers and experts. We will discuss this in the paper and hope it inspires further research. Thank you for the thoughtful question.

 

---

 

Effects of $d_{low}$

When $d_{low}$ = 64 or 128, AoE with $L_{aux}$ results in lower performance compared to $d_{low}$ = 256 (Config.10). Additionally, as shown in Figure 2, $d_{low}$ = 64 results in slower convergence, similar to traditional MoEs.

We also tested an extreme case where $d_{low}$ = 8. In this case, the approximation of $W_g$ is too lossy, leading to poor performance and high loss. Due to limited time, we trained with only 50B tokens. We will include this experiment and make our statement more accurate. Thank you for your valuable feedback!

Task Performance

Loss over Tokens

---

 

The uncertainty of the accuracies

First, we note that AoE outperforms MoE across various setups presented in our paper, with the testing process using greedy decoding to eliminate generation randomness.

For each configuration, we aggregated the outputs from all tasks into a sequence of 1s (correct) and 0s (incorrect), paired with the outputs of traditional MoE (Config. 2), and performed a McNemar test. The improvement of AoE over MoE is significant (p < 0.05) across all configurations in Tables 2, 3, and 4. We will highlight the significance of this improvement.

We sincerely thank you for your constructive suggestions and valuable comments! We hope our rebuttal helps address your concerns. If so, we would be grateful if you could consider increasing the overall recommendation.  

---

 

Comparison with more MoE works

We trained the Hash-Layer model (Roller et al.) on 100B tokens, using the same hyperparameters and model setup as our other models. Unlike other methods cited in the last paragraph of Section 2, which require domain-labeled data, Hash-Layer does not—hence our decision to include only this method for comparison. Here are the results:

Task Performance

Our results align with Roller et al. (p.7), who note that "Switch Transformer and Hash perform similarly in multi-layer routing." Hash-Layer's key advantage lies in its balanced load distribution.

Our AoE model not only outperforms both Hash-Layer and traditional MoE in downstream tasks, but also demonstrates improved load balancing compared to traditional MoE.

 

---

 

Please compare other expert selection mechanisms

Thank you for the suggestion. We have added a new baseline—the Hash-Layer—as noted above. In Table 2, we report results using the top-K token choice, while Table 3 includes experiments with top-P token choice and top-K expert choice selection mechanisms. We hope these results address your concern.

If you have further suggestions for additional baselines, we’d be glad to explore them or discuss more!

 

---

 

Try factorizing $W_p$ or other trivial tweaks

Thank you for your insightful question. We trained new models by factorizing $W_p$ instead of $W_g$ ($d_{low}$ = 256, using 100B tokens with $L_{aux}$), but observed a performance drop.

Task Performance

We have a few hypotheses regarding the performance drop:

1. Factorizing $W_p$ might create a bottleneck in this branch, while the activation function in the $W_g$ branch may already act as a bottleneck. Since both pathways in the FFN are constrained in this case, it could lead to the performance drop. If this is the case, we would suggest always factorizing $W_g$ in AoE or encourage future work to further develop expert architectures.

2. Alternatively, the optimal value for $d_{low}$ might change when factorizing $W_p$.

Due to time constraints, we cannot provide a definitive explanation at this moment. However, these aspects of model architecture are valuable areas for future research, and we will discuss this point in the paper.

We also experimented with computing norms using only the first n elements of the intermediate layer ($n=256$), training with 50B tokens due to time constraints. Compared to standard AoE ($d_{low}=256$) and MoE, this method underperformed because of insufficient activation information for expert selection:

Task Performance

We will include these points in the paper, as they offer insights that may be useful for future research. Thank you again for your insightful question!