Optimizing Robustness and Accuracy in Mixture of Experts: A Dual-Model Approach

Thank you for raising the seven insightful questions. We appreciate your thorough review and address each concern in detail below.

1. Why Attack is not Weak?

   ​ We compared our self-implemented AutoAttack with the official library. The perturbations generated by both methods are identical, and the results on the CIFAR-10 match. Second, regarding the concern that our PGD attack takes the end value, we note that many other studies adopt similar attack settings, such as [1,2]. Additionally, we conducted further experiments on TinyImageNet, both by using the final value and by selecting the highest loss point. The attack outcomes were nearly identical, and the loss values remained smooth in the later iterations.

2. Why RA-E Attack Remains Effective?

   ​ We would like to clarify that our evaluation assesses the vulnerability of both the router and experts by measuring accuracy relative to MoE. Regarding your concern about the RA-E attack’s effectiveness under varying router scores, we emphasize that this attack primarily targets the experts. As a result, the perturbations remain largely independent of the router’s behavior. Empirically, our experiments show that in 98% of cases, attacked images are routed to the same expert(s), ensuring that the RA-E attack remains effective.

3. Why Using KL Divergence in Eq.2?

   ​ The goal of our paper is to explore a potential approach to further enhance the robustness of MoE. Regarding the use of KL divergence in the second term of Eq. 2, we chose this formulation because applying AT to the second expert would force the additional expert’s predictions toward the ground truth, potentially leading to overfitting.

   ​ We conducted experiments comparing our method against two alternative formulations you suggested. Our results showed that RT-ER achieved similar performance to (ii) while improving RA by 2% compared to (i).

4. Inclusion of Full MoE Results

   ​ In Section 4.2, we compare the performance of MoEs using top-1 and top-2 routing strategies to demonstrate that our method generalizes across different routing mechanisms.

   ​ To further address your request, we provide additional results for a Dense MoE and an MoE using a top-3 router on the CIFAR-10:

   Method	SA(%)	RA(%)	RA-E(%)	RA-R(%)
   Dense	79.23	70.50	76.51	72.62
   Top-3	79.18	70.11	76.24	72.48

5. Why $\alpha$≥0.5 Guarantees Robustness of the Dual-Model?

   ​ We provide theoretical evidence supporting the conclusion that$\alpha \geq 0.5$ is necessary to guarantee the robustness of the dual-model.

   ​ Eq. (9) establishes a bound on the certified robustness radius:

   $$\lVert \delta \rVert_p \leq \epsilon = \min\limits_{k \neq y} \frac{\alpha \left(F_R^{(y)}(x) - F_R^{(k)}(x)\right) + \alpha - 1}{\alpha \sum_i \left(2 r_{R_i} + a_{R_i}({x}) \left(L_{R_i}^{(y)} + L_{R_i}^{(k)}\right) \right)}.$$

   ​ In the numerator of Eq. (9), $\alpha$ ranges from 0 to 1, and the maximum value of $\left(F_R^{(y)}(\mathbf{x}) - F_R^{(k)}(\mathbf{x})\right)$ is 1. If $\alpha$ is smaller than 0.5, the numerator becomes negative,making the certified robustness radius undefined. Therefore, we conclude that $\alpha \geq 0.5$ is a necessary condition to ensure the robustness of the dual-model.

6. Why only Robustifying the Second Expert?

   ​ Directly robustifying all experts simultaneously would be computationally expensive and inefficient, making MoEs less appealing for large-scale applications. To balance efficiency and robustness, we propose RT-ER. For each input, RT-ER additionally robustifies an expert not selected by the router. Since the router dynamically selects different experts during training, the additional expert chosen for robustification also varies over time. This iterative process enables RT-ER to progressively improve the robustness of the entire expert network without significantly increasing computational costs.

7. Why Not Simply Train a Robust Model?

   ​ Compared to simply training a robust model with $(1 - \alpha) L_{clean} + \alpha L_{rob}$, our method offers two key advantages:

   1. Better SA-RA Tradeoff: The dual-model strategy in JTDMoE provides a better balance between SA and RA. Simply training often leads to conflicting gradients, limiting performance. We compare JTDMoE with a MoE of 8 experts trained using $\alpha = 0.7$ and the combined loss. On CIFAR-10, JTDMoE achieved 92.29% SA and 74.62% RA, whereas the simply trained model reached only 87.48% SA and 67.45% RA. This shows that JTDMoE achieves a significantly better tradeoff.
   2. Reduced Training Costs: Our approach eliminates the need for adversarial training in the standard MoE.

   [1] Zhang Robust mixture-of-expert training for convolutional neural networks.

   [2] Bai Improving the accuracy-robustness trade-off of classifiers via adaptive smoothing.

Thank you for your comments.

Contributions of Our Method

We appreciate the reviewer's comment. We would like to point out that aligning outputs with KL-loss and mixing clean and adversarial outputs are classical loss design techniques used in most of the adversarial training papers. We would like to clarify that our contributions are novel and specifically tailored to the unique challenges of MoE architectures. In particular:

MoE-Specific Vulnerability Analysis: Our work is grounded in the empirical and theoretical observation that expert networks in MoEs are significantly more susceptible to adversarial attacks than the router. This insight is critical because, unlike standard neural networks, the MoE framework relies on dynamic routing where the experts' outputs directly affect the final prediction. Our contribution lies in isolating and addressing this vulnerability, which is not encountered in traditional architectures.

RT-ER – A Targeted Robustification Strategy: While methods such as aligning outputs with KL divergence have been explored in classical settings, our RT-ER method is designed specifically for MoEs. For each input, we robustify an expert that is not selected by the router, and because the router's selection changes during training, this process iteratively reinforces the robustness of the entire expert network. This dynamic and efficient strategy is uniquely adapted to the MoE structure and its operational dynamics, setting it apart from standard adversarial training methods.

Dual-Model Strategy (JTDMoE): Our dual-model strategy is not a mere combination of existing techniques but a novel design that integrates a standard MoE and a robust MoE in a unified framework. This approach allows us to achieve a favorable balance between clean and robust accuracy while maintaining efficiency. Unlike typical ensemble methods in classical networks, our dual-model is trained with a specifically designed bi-level training process and is accompanied by a rigorous theoretical robustness bound (Theorem 5.5) that quantifies how the interplay between the two models contributes to overall robustness.

Theoretical Contributions: We provide new certified robustness bounds for both the full MoE model and the dual-model setup, offering insights into how individual components (especially the experts) impact overall robustness. These theoretical guarantees are tailored to the dynamics of MoEs and serve as a foundation for our proposed training strategies, further reinforcing the novelty of our contributions.

In summary, our method addresses core challenges in MoE training (namely robustness, performance, and efficiency) by proposing tailored strategies (RT-ER and JTDMoE) and accompanying theoretical guarantees that are specifically designed for the MoE setting. These contributions collectively advance the state-of-the-art in adversarial robustness for MoEs and are not simply an assembly of existing techniques from classical neural networks.

Results on Large-Scaled Datasets

Thank you for your comments. Our work primarily focuses on investigating the robustness of MoE. To evaluate our proposed method, we adopt the ViT + TinyImageNet and ResNet + CIFAR-10 settings—both of which are widely used benchmarks in robust MoE research. Notably, this evaluation setup has also been employed in recent studies, such as Lin [1] and Zhang [2], further supporting its relevance and acceptance within the community. Although our current experiments focus on CIFAR-10 and TinyImageNet, our method is designed to be scalable. The computational cost of robustifying one additional expert is minimal relative to the overall training budget, ensuring that the approach can be applied efficiently to larger datasets and more complex models.

To further strengthen our empirical validation, we now additionally provide results for our proposed method using a ViT model on the full ImageNet dataset. The results are summarized as follows:

These results demonstrate that our method (RT-ER) consistently outperforms AT and Trades on the ImageNet. Notably, RT-ER achieves 12% (10%) improvement in RA and 8% (6%) improvement in SA compared with AT (Trades). This underscores the scalability and effectiveness of our approach, even on large-scale vision benchmarks like ImageNet.

[1] Lin F et al. Towards Robust Vision Transformer via Masked Adaptive Ensemble.

[2] Zhang Y et al. Robust mixture-of-expert training for convolutional neural networks.

We appreciate your thoughtful feedback and the opportunity to clarify and strengthen our submission. Below, we respond to each of your concerns in detail.

A concern regarding assumption 5.3 not holding in real practice

We would like to clarify that Assumption 5.3 is reasonable and holds in several practical scenarios. Specifically, we outline three cases where this assumption is satisfied:

​ 1.Sparse MoE after robust training: Robust training techniques typically encourage large expert score margins. In our setting, a large margin in the expert scores implies that the differences between the scores of the top-k experts are well separated from the rest. This inherent separation makes it difficult for adversarial perturbations to change the top-k set within a realistic $\delta$-ball. In this situation, the router becomes locally stable and consistently selects the same expert(s) for both the clean input $x$ and the perturbed input $x + \delta$, thereby satisfying Assumption 5.3.

​ 2.Dense MoE: In a dense MoE [6], all experts are activated for each input, and the routing function is continuous. Thus, Assumption 5.3 naturally holds.

​ 3.Soft MoE: Similarly, soft MoE [5] use a continuous routing function with soft assignments. This inherent continuity ensures that small changes in the input lead to small changes in the routing outputs, thereby maintaining stability and satisfying Assumption 5.3.

We have added additional discussion and clarification in the revised version of the manuscript to further address this point.

Require large model (ViT) and dataset (ImageNet)

We would like to clarify that we have already included results using a ViT model on the TinyImageNet (see Table 3 and Figure 3). Our work focuses on investigating the robustness of MoEs, and we adopt the ViT + TinyImageNet setting to evaluate our method. This setting has also been adopted in prior studies on the adversarial robustness of MoE, such as [7], making it a widely accepted benchmark for robust MoE research.

To further strengthen our empirical evaluation, we now additionally provide results for our proposed RT-ER using a ViT model on ImageNet. In particular, we observe a substantial improvement of approximately 12% in RA and 8% SA compared with conventional adversarial training. The results are summarized below:

A Key Distinction in MoE Architecture in Experimental Designs

Thank you for your comments. We would like to clarify that our primary focus is to investigate the fundamental vulnerability of MoE architectures to adversarial attacks in the context of image classification. Our architectural setup aligns with several recent works that adopt a similar design [2-4], allowing us to pinpoint the vulnerabilities of the experts without interference from additional layers. Importantly, the RT-ER and the JTDMoE strategies we proposed are designed to be agnostic to the underlying MoE integration scheme. We would also like to point out that the observation “the expert modules are more susceptible to adversarial attacks” is independent of the specific placement of the MoE layers—whether integrated within the classification head or embedded inside transformer blocks.

To support this claim, we have verified that our findings remain valid under the architecture proposed by Riquelme et al. [8], which replaces the MLP block with a MoE layer. After standard training, the MoE achieved 90.35% SA, 38.02% RA, 32.16% RA-E, and 64.97% RA-R. Expert networks, being deeper and more complex than the router, are inherently more vulnerable to adversarial perturbations. Even though the experts do not directly produce the final output, their susceptibility can still impact the overall model, as adversarial effects propagate through subsequent layers. These results confirm that the insights and conclusions derived from our analysis are broadly applicable across diverse MoE integration strategies.

[1] Puigcerver et al. On the adversarial robustness of mixture of experts.

[2] Videau et al. Mixture of Experts in Image Classification: What's the Sweet Spot?

[3]Chen et al. Heterogeneous Mixture of Experts for Remote Sensing Image Super-Resolution.

[4] He et al. Mixture-of-experts for semantic segmentation of remoting sensing image.

[5] Puigcerver et al. From sparse to soft mixtures of experts.

[6] Zhang Dense vision transformer compression with few samples.

[7] Lin et al. Towards Robust Vision Transformer via Masked Adaptive Ensemble.

[8] Riquelme et al. Scaling vision with sparse mixture of experts.