Advancing Expert Specialization for Better MoE

Dear Reviewer Ypny,

We sincerely thank the reviewer for the thoughtful and detailed feedback. The comments raise several key issues, including:

1. The lack of pre-finetuning baselines and potential performance drop;

2. Whether the method mitigates LoRA-induced degradation;

3. The complexity and reproducibility of multi-loss optimization;

4. Missing analysis of computational overhead;

5. Limited architecture diversity;

6. Insufficient discussion of limitations.

We appreciate these valuable insights and address each point below.

---

1. Missing pre-finetuning baseline results and apparent performance drop

The discrepancy between our reported results and the original model figures primarily arises from differences in model versions, prompting strategies, and inference settings.

• Model versions: The public figures cited (e.g., Moonlight, DeepSeek) are based on instruction-tuned models, while we start from their pretrained base versions with no preference or SFT data, leading to inherently different performance baselines.
• Prompting strategies: Our evaluation is conducted in a zero-shot setting without handcrafted few-shot prompts or demonstrations, which are often used in official evaluations.
• Inference length: We uniformly limit the generation to 512 max new tokens due to computational constraints. In contrast, official results use 8k–32k tokens, which notably benefits reasoning-heavy tasks like MMLU and HumanEval. To quantify this impact, we evaluated both base and our fine-tuned models under the same token budget. Results are summarized below:

Performance Comparison under Matched Inference Settings

Effect of Increasing Max New Tokens (Kimi)

As shown, under identical inference constraints, our method outperforms the original models, and retains leading performance even when length is extended. We will include these clarifications and additional results in the revised manuscript. Due to computational resource constraints, we were unable to reproduce the official results with longer sequence lengths, and we sincerely apologize for this limitation.

---

2. Clarifying Performance Gains with Our Method

We would like to clarify that our approach provides a net performance gain, not just recovery from potential LoRA-induced loss.

First, from a theoretical perspective, the proposed losses $\mathcal{L}_o$ and $\mathcal{L}_v$ are designed to actively enhance expert specialization while preserving load balance (as supported by Lemmas 1 and 2), which defines a constructive optimization objective beyond damage control (Detail in Reviewer 9cq6 Q1 ).

Second, as shown, when evaluated under consistent inference settings (e.g., token limit = 512), our fine-tuned models outperform the original base models across multiple benchmarks. These results confirm that our method achieves genuine improvement rather than merely compensating for LoRA finetuning limitations.

---

3. On applicability to pre-training and loss coefficient sensitivity

We appreciate the reviewer’s insightful suggestions. While our work focuses on post-training due to limited compute and data resources, the proposed losses are theoretically applicable to pre-training. In fact, token distributions in pre-training are more balanced, making it easier to satisfy both load balancing and specialization objectives. We will include this as a future direction in Section 6 (Detail of post-training in Reviewer 9cq6 Q2).

For hyperparameter sensitivity, we conducted experiments varying α, β, and γ, and adopted a dynamic scaling mechanism to normalize the magnitudes of $\mathcal{L}_o$ and $\mathcal{L}_v$ relative to $\mathcal{L}_{aux}$. This ensures stable optimization across losses. Results show our method remains robust under a wide range of settings.

The default setting achieves the best trade-off between general performance (MMLU) and specialization effectiveness (MaxVioGlobal), demonstrating robustness to coefficient variation.

---

4. On computational overhead

While the orthogonality loss ($\mathcal{L}_o$) has a theoretical complexity of $O(N \cdot k^2 \cdot d)$, this overhead is practically manageable. In standard MoE setups, the number of active experts $k$ is a small constant (e.g., 6 in DeepSeek models), with $k \ll N$ and $k \ll d$. Moreover, the required pairwise projections are highly optimized on modern GPU hardware via batched matrix operations. Therefore, $\mathcal{L}_o$ introduces limited additional cost and does not become a training bottleneck in practice.

---

5. On the diversity of evaluated MoE architectures

We acknowledge the reviewer’s suggestion to validate our method on more diverse MoE architectures. Our initial experiments focused on DeepSeek and Moonlight models due to their strong open-source performance and recent community adoption, making them credible testbeds for practical effectiveness.

In response to this concern, we have additionally evaluated our method on two structurally different models: Mixtral and Phi-MoE, which adopt distinct routing strategies and omit shared experts. As shown below, our method continues to outperform baselines across all tasks.

We will include these results in the revised version to better demonstrate the generality of our approach.

---

6. On writing issues and limitations discussion

We thank the reviewer for the careful reading.

• We will correct the punctuation issue on line 66 in the revised version.

• We will rewrite and expand Section 6 into a more comprehensive Limitations and Future Work discussion to improve clarity and completeness.

---

We sincerely thank the reviewer again for raising many important and constructive concerns. While this review included the most critical questions, we believe we have now addressed them thoroughly in our rebuttal, including the pre-finetuning comparisons, training setup clarification, loss function sensitivity, and generalization to new architectures.

We would like to reiterate that our paper proposes a novel and effective framework to jointly improve expert specialization and load balancing in MoE models, supported by theoretical guarantees, carefully designed regularization losses, and strong empirical results across 11 benchmarks. The remaining three reviewers have explicitly acknowledged the clarity, novelty, and significance of our contributions.

We hope our clarifications have addressed your concerns. If you find the revised explanations and results satisfactory, we would greatly appreciate your consideration in recommending the paper for further discussion.

Dear Reviewer 4E1Q,

We sincerely thank the reviewer for their positive and constructive feedback. We are glad that the reviewer found our idea of jointly optimizing expert specialization and routing distribution via auxiliary regularization losses to be interesting. At the same time, we appreciate the reviewer’s thoughtful suggestions. The main concerns can be summarized as follows:

1. The effectiveness of the proposed loss may rely on token-level diversity, which may not always hold in real-world scenarios;
2. The orthogonality loss introduces non-negligible computational overhead, which may become a bottleneck in large-scale settings;
3. Prior work suggests that auxiliary losses may not be necessary during post-training, it is unclear whether regularization is more appropriate during pre-training.

---

1. On the Assumption of Token Diversity

Conclusion: Token diversity is a helpful condition, but not a necessary prerequisite for the success of our method. Through orthogonality loss ($\mathcal{L}_o$) and variance loss ($\mathcal{L}_v$), our approach actively induces functional diversity among experts, thereby enabling specialization even when input tokens are not highly diverse.

Explanation: We appreciate the reviewer’s insightful question regarding the connection between token diversity and the effectiveness of our method. Our responses are as follows:

• Active induction vs. passive dependence: Rather than relying on input diversity, our method actively creates internal diversity. The orthogonality loss ($\mathcal{L}_o$) operates at the token level, penalizing similar outputs from different experts even when the same token is routed to them. This drives the experts to learn distinct representations, thereby fostering functional diversity even under low input diversity.
• Theoretical soundness and empirical robustness: Lemma 1 and Lemma 2 serve to establish that our multiple training objectives (load balancing and routing diversity) are theoretically compatible and jointly optimizable. Our loss design realizes this in practice. Most importantly, across 11 real-world benchmarks, including both high and low-diversity tasks, our method achieves consistent improvements, demonstrating strong robustness.

We hope this clarifies how our approach remains effective even without assuming ideal token diversity.

---

2. On the Computational Overhead of $\mathcal{L}_o$

Conclusion: While $\mathcal{L}_o$ has quadratic complexity in theory, the actual overhead is negligible in practice due to the small number of activated experts ($k$) and efficient batched implementations. It does not present a bottleneck in our setup.

Explanation: We appreciate the reviewer’s precise analysis of the computational cost of $\mathcal{L}_o$, which involves pairwise projections among the $k$ selected expert outputs for each token, resulting in a theoretical cost of $O(N \cdot k^2 \cdot d)$. In practice, this cost remains manageable for several reasons:

• Small $k$ in standard MoE practice: One of the core design principles of sparse MoE models is to keep $k$ small to control computation. In our experiments, we follow this convention, using configurations such as $k=6$ in DeepSeek-V2-Lite. Given that $k \ll N$ and $k \ll d$, the quadratic factor in $k$ contributes minimally to total training cost.
• Efficient hardware execution: The main operations required by $\mathcal{L}_o$, inner products and pairwise projections, are highly parallelizable and can be implemented as batched matrix multiplications. These are well-optimized in modern frameworks like PyTorch and run efficiently on contemporary GPUs.
• Justified by empirical gains: Most importantly, the modest increase in computation is offset by substantial and consistent performance improvements across diverse downstream tasks (e.g., Table 1 and Figure 4). This demonstrates that the regularization effect of $\mathcal{L}_o$ leads to meaningful gains in expert specialization.

We will include a detailed breakdown of the computational cost of $\mathcal{L}_o$ under typical $k$ values in the appendix of the revised version, to further clarify its practical feasibility.

---

3. On the Applicability During Pretraining

Conclusion:

While our current focus is on post-training, where the tension between uniform routing and specialization is most pronounced, we believe applying our regularization objectives during pretraining is a valuable future direction.

Explanation: We appreciate the reviewer’s insightful comparison with works such as OLMoE and the thoughtful reflection on different training phases.

• Why focus on post-training: In the post-training stage, downstream tasks often involve narrow or skewed data distributions. This intensifies the conflict between load balancing (which pushes toward uniform token distribution) and the need for expert specialization. Therefore, it provides a clearer setting to observe and evaluate the effects of our proposed losses.
• Relation to “no auxiliary loss” claims: Our work does not contradict the finding that traditional $\mathcal{L} _ {aux}$ may be ineffective—or even detrimental—during post-training. As shown in our ablation results (Figure 4), the “w/o all” variant (which removes $\mathcal{L} _ {aux}$) actually outperforms the “only aux” baseline. This confirms that $\mathcal{L} _ {aux}$ may hinder specialization in post-training. Importantly, we further demonstrate that our proposed $\mathcal{L} _ o$ and $\mathcal{L} _ v$ actively promote specialization beyond simply removing constraints, achieving the best performance overall.
• Pretraining potential: Our regularization framework is theoretically applicable during pretraining as well. In fact, we believe that applying $\mathcal{L} _ o$ and $\mathcal{L} _ v$ from the start may help guide expert specialization more effectively. We will highlight this in Section 6 (Limitations & Future Directions) as an important avenue for future exploration.

---

We thank the reviewer again for the insightful and constructive feedback. We believe our detailed responses have thoroughly addressed each of the points raised. We look forward to any further discussion.

Dear Reviewer 9cq6,

We sincerely thank the reviewer for the thoughtful and encouraging feedback. We appreciate the recognition of our motivation, the clarity of our formulation, and the importance of jointly addressing expert specialization and load balancing. Below we summarize our responses to the main concerns:

1. The actual use of LoRA-based fine-tuning instead of full training is unclear and should be stated explicitly;
2. The distinction between pre-training and post-training lacks contextual clarity and justification;
3. Key experimental details such as data splits, training steps, and base model performance are missing;
4. The explanation of the orthogonality loss may conflate diversification and specialization;
5. Figure 1 is overly complex and not intuitive.

We have carefully addressed each of these points in the following responses.

---

1. On the Use of LoRA Fine-Tuning

Conclusion:

Although LoRA is used for parameter-efficient training, all major components (including experts and router) are updated during fine-tuning, making our setup functionally equivalent to full-model training.

Explanation: We thank the reviewer for this important and constructive comment. We acknowledge that the current version may be misleading, as the use of LoRA was only stated in the appendix. In the revision, we will clearly indicate in the abstract, introduction, methodology, and experiments that all training is conducted via LoRA-based fine-tuning.

In our setup, LoRA modules are inserted into both the router and the expert layers. This means that the routing behavior and expert outputs are both optimized during training. While LoRA offers parameter-efficient adaptation, it does not freeze the underlying model. On the contrary, all core components are updated, just through a low-rank pathway. To reduce the gap between LoRA and full fine-tuning, we have set the LoRA rank to 32, which is relatively high and helps better approximate full updates.

Therefore, our training is functionally equivalent to full fine-tuning in terms of which parameters are optimized. Importantly, our proposed losses directly act on trainable parts of the model: the variance loss shapes the router logits, and the orthogonality loss regularizes expert outputs. The observed improvements in all benchmarks validate their effectiveness under this setup (The specific training details are discussed in detail in Response 4).

We appreciate the reviewer’s attention to this detail and will make the training design and parameter update scope explicit in the revised paper.

---

2. On the Clarity of the Post-Training Setting

Conclusion:

Theoretically, our method is applicable to both pre-training and post-training phases. However, we focus on the post-training stage, where the conflict between expert specialization and routing balance becomes more pronounced in downstream tasks.

Explanation:

We appreciate the reviewer’s insightful comment regarding the training stage targeted by our work. We would like to clarify the rationale for focusing on the post-training setting:

• On the distinction: While we agree that pre-training and post-training share similar optimization processes, their goals differ. Pre-training aims to build general-purpose capabilities on diverse and large-scale corpora, whereas post-training seeks to adapt the model to specific domains or tasks. The data in post-training is often less diverse and more skewed in distribution compared to the more uniform and balanced nature of pre-training data.
• On the conflict: In such downstream scenarios, token distributions can be highly concentrated (e.g., math tasks contain a higher proportion of numeric or symbolic tokens). This makes the conflict between load balancing and expert specialization more severe: load balancing encourages tokens to be evenly distributed across experts, while specialization requires assigning certain types of tokens more often to specific experts.
• Our motivation: Our work is motivated by this core conflict. By introducing the orthogonality loss ($\mathcal{L}_o$) and the variance loss ($\mathcal{L}_v$), we aim to achieve better expert specialization without sacrificing load balance. This enables MoE models to better adapt to downstream distributions and is supported by both our theoretical insights and empirical results.

In the revised version, we will clearly indicate our focus on the post-training setting in the abstract and introduction, and explain why resolving this trade-off is particularly critical in this phase.

---

3. On Training Setup, Task Scope, and Base Model Performance

Conclusion: Each benchmark is trained separately using a high-quality 6k-sample dataset. Fine-tuning is limited to 3 epochs (~550 steps) per task. Base model performance before fine-tuning will be added to Table 1 for clarity.

Explanation:

• Data and task setup: For each benchmark, we constructed a high-quality training set of 6,000 examples. The questions were sourced from the training split of the corresponding benchmark; when insufficient, we supplemented them with related datasets. Answers were generated using strong models such as OpenAI’s o3mini and DeepSeek R1, and then manually filtered to ensure correctness. Each dataset is used to fine-tune the model specifically for that benchmark.
• Training steps: To avoid overfitting, we limited fine-tuning to 3 epochs across all tasks. Given the dataset size, this corresponds to approximately 550 training steps per model per task.
• Base model performance: We have now included the base model performance before fine-tuning in Table below, using consistent settings across models (e.g., max sequence length = 512). This comparison shows that our proposed losses lead to consistent and significant improvements across all tasks, beyond the effect of LoRA tuning alone.

We appreciate the reviewer’s attention to these important experimental factors and will ensure that all relevant details are properly documented in the revised version.

---

4. On Whether Orthogonality Loss Encourages Diversification or Specialization

Conclusion: Orthogonality loss directly promotes expert diversification at the token level, which serves as a critical mechanism for achieving specialization at the dataset level.

Explanation: We appreciate the reviewer’s insightful interpretation. The orthogonality loss ($\mathcal{L}_o$) encourages experts that receive the same token to produce diverse outputs. This diversity is not the final objective, but a necessary step toward achieving expert specialization.

• From diversification to specialization:
  1. When the same token is routed to multiple experts, $\mathcal{L}_o$ penalizes similarity in their output representations, effectively encouraging the experts to respond differently. This is what we refer to as expert diversification.
  2. When such pressure is applied across all tokens during training, experts are discouraged from learning redundant or overlapping representations. Instead, each expert is incentivized to capture distinct features or patterns in the data.
  3. Over time, this token-level diversification aggregates into specialization across the dataset: each expert becomes better at handling certain types of inputs or semantic regions, resulting in stable expert-role assignment.
• Empirical evidence: This process is empirically supported in our experiments. As shown in Figure 1 (right), the token distribution across experts becomes significantly more clustered after training, with reduced overlap in representation space. Quantitatively, our method lowers expert overlap by up to 45%, indicating a strong shift toward specialization.

In short, diversification is the driving force, and specialization is the outcome. We appreciate the reviewer for raising this important distinction.

---

5. On Figure 1 Readability and Design

Conclusion: We agree that the current version of Figure 1 may be visually dense and not sufficiently intuitive. In the revised version, we will redesign the layout, simplify visual elements, and add clearer annotations to improve overall readability. Below, we briefly explain the intended structure and purpose of the current figure.

Explanation: Figure 1 is designed to visualize the two core effects of our method:

• Left side: Routing Diversification
  • Top (Routing Score Matrix): Shows the raw routing scores, which may appear abstract without further guidance.
  • Middle (Load Variance): Demonstrates that our method preserves load balancing, achieving similar variance reduction as baseline methods.
  • Bottom (Discriminative Routing Variance): This is the key part—our method significantly increases routing output variance, indicating more discriminative and less ambiguous routing decisions.
• Right side: Expert Specialization
  • Bottom (Overlap): Visualizes the heavy overlap of token assignments across experts in baseline methods.
  • Top (Cluster Separation): In contrast, our method produces clearer token clusters per expert, indicating effective specialization and division of responsibility.

---

Once again, we thank the reviewer for the thoughtful and constructive feedback. We believe our detailed responses and clarifications have thoroughly addressed the concerns raised. We look forward to any further discussion.

Dear Reviewer jbyJ,

We sincerely thank the reviewer for the thoughtful evaluation and detailed feedback. We are especially grateful for the recognition of our paper’s strengths, including the novelty and impact of our method, the comprehensive empirical evaluation across multiple models and tasks, and the depth of our theoretical and ablation analyses.

At the same time, we also appreciate the reviewer’s constructive suggestions. The main concerns can be summarized as follows:

1. The lack of hyperparameter sensitivity analysis (experimental);
2. The task scope is limited to NLP, and broader validation in multimodal or low-resource settings is needed (discussion);
3. More recent baselines should be included for comparison (experimental).

We have carefully addressed each of these points in the following responses.

---

1. On hyperparameter sensitivity

Conclusion: The performance is optimal when the loss weights $\alpha$, $\beta$, and $\gamma$ are set to the same value. Moreover, our method demonstrates strong robustness across different hyperparameter magnitudes.

Explanation:

We thank the reviewer for highlighting the importance of hyperparameter sensitivity. To address this, we conducted experiments varying the values of $\alpha$ (for $\mathcal{L}_{aux}$), $\beta$ (for $\mathcal{L}_o$), and $\gamma$ (for $\mathcal{L}_v$) across different magnitudes.

For reference, our overall loss function is defined as:

$\mathcal{L}=\mathcal{L} _ h +\mathcal{L}_{balance}$

$\mathcal{L} _ {balance} = \alpha \cdot \mathcal{L}_{aux} + \beta \cdot \mathcal{L} _ {o} + \gamma \cdot \mathcal{L} _ {v}$

It is worth noting that we apply a dynamic balancing mechanism to ensure fair weighting across different loss terms. Specifically, because the orthogonality and variance losses ( $\mathcal{L} _ {o}$ and $\mathcal{L} _ {v}$ ) may have different initial scales, we first normalize them using dynamic scaling factors. This brings their magnitudes roughly in line with the auxiliary loss $\mathcal{L}_{aux}$. Only after this normalization do we apply the hyperparameters $\alpha$, $\beta$, and $\gamma$ to control their contributions to the total loss.

The table below summarizes our results across several representative benchmarks under four different settings (DS v2 lite):

We observe that the setting where $\alpha = \beta = \gamma = 10^{-3}$ consistently yields the best performance across tasks. Additionally, when any of the coefficients is varied within one order of magnitude ($\pm 1$), the results remain stable and close to optimal. This indicates that our method is not overly sensitive to these hyperparameters and can be considered robust in practical applications.

---

2. On Generalizability

Conclusion:

We believe our formulation is inherently suitable for both multimodal and low-resource scenarios, as its design principles, such as expert specialization and routing diversity, are broadly applicable across domains.

Explanation:

We appreciate the reviewer’s insightful suggestion regarding broader evaluation. While our current experimental design focuses on NLP and code generation tasks, we believe our proposed training objective is agnostic to modality and can be extended to other domains, such as visual and multimodal models. The orthogonality and variance principles are applicable wherever a mixture-of-experts architecture is employed, including vision MoEs and multimodal encoders/decoders.

• Visual / multimodal MoEs. Recent works [1, 2, 3] on vision‑centric experts shows that image patches suffer from the same expert‑overlap and uniform‑routing issues we address in language models. Because our orthogonality loss acts directly on expert outputs and our variance loss on routing logits, both can be applied to visual tokens without modification.

• Low‑resource adaptation. Parameter‑efficient fine‑tuning methods such as MoELoRA [4], MixLoRA [5], and MoKA [6] insert small trainable adapters while freezing most expert weights. Our losses require no additional parameters and can be computed on the adapted weights or the LoRA pathways alike, making them a natural drop‑in to further improve specialization when GPU memory or data are limited.

---

3. On comparing baselines

Conclusion: After incorporating two additional state‑of‑the‑art baselines, our approach continues to deliver the best overall performance.

Explanation:
We appreciate the reviewer’s suggestion to expand the set of comparison methods. In response, we have added:

• Dynamic Routing MoE (ERNIE 4.5) [7] is a strong recent baseline. It introduces a multimodal, heterogeneous MoE architecture that supports both parameter sharing across modalities and modality-specific expert specialization. The ERNIE 4.5 family includes multiple model scales (e.g., 47B and 3B active parameters), and has shown competitive performance on various text and multimodal benchmarks.

• SIMBAL (Similarity-Preserving Routers) [8] is a recent method that addresses expert load balancing in sparse MoE models. Instead of enforcing uniform routing via conventional load balancing loss, SIMBAL introduces an orthogonality-based regularization that aligns the router’s Gram matrix with the identity matrix. This encourages similar input tokens to be routed to similar experts, reducing redundancy and improving consistency in expert utilization.

We re‑evaluated all methods on the most comprehensive subsets of our benchmark suite. A summary of the key results is presented below.

(Numbers will be filled in Table 1 of the revision.)

Across all six representative tasks, our method either matches or surpasses the strongest new baseline, while simultaneously maintaining the lowest MaxVioGlobal (better load balance). These additional results confirm that the improvements reported in the main paper are not an artifact of the original baseline selection but hold against the latest alternatives as well.

---

We are grateful for the reviewer's insightful feedback, which has been instrumental in strengthening our work. We have strived to thoroughly address every concern with new experiments and detailed clarifications, and we are looking forward to the opportunity for any further discussion.

---

[1] Mustafa, B., Riquelme, C., Puigcerver, J., Jenatton, R., & Houlsby, N. (2022). Multimodal Contrastive Learning with LIMoE: The Language‑Image Mixture of Experts. Advances in Neural Information Processing Systems, 35, 9564–9576.

[2] Yan, Z., Li, Z., He, Y., Wang, C., Li, K., Li, X., Zeng, X., Wang, Z., Wang, Y., & Qiao, Y. (2025). Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment. Proceedings of the Computer Vision and Pattern Recognition Conference, 29880–29892.

[3] Li, Z., Li, Z., & Zhou, T. (2025). R2‑T2: Re‑Routing in Test‑Time for Multimodal Mixture‑of‑Experts. arXiv.

[4] Luo, T., Lei, J., Lei, F., Liu, W., He, S., Zhao, J., & Liu, K. (2024). MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter‑Efficient Fine‑Tuning for Large Language Models. arXiv.

[5] Li, D., Ma, Y., Wang, N., Ye, Z., Cheng, Z., Tang, Y., Zhang, Y., Duan, L., Zuo, J., & Yang, C. (2024). MixLoRA: Enhancing Large Language Models Fine‑Tuning with LoRA‑Based Mixture of Experts. arXiv.

[6] Yu, B., Yang, Z., & Yi, X. (2025). MoKA: Parameter Efficiency Fine‑Tuning via Mixture of Kronecker Product Adaption. Proceedings of the 31st International Conference on Computational Linguistics, 10172–10182.

[7] ERNIE Team (Baidu). (2025, June 29). ERNIE 4.5 Technical Report. Retrieved from Huggingface.

[8] Omi, N., Sen, S., & Farhadi, A. (2025, June 16). Load Balancing Mixture of Experts with Similarity Preserving Routers. arXiv.