Regarding W1-2: Decision refers to the expert-selection (or the weight score for dense routing). For instance, we could know where one token is routing to.

Regarding Q4: Very interesting observation, could you further analysis the underlying driver?

W1-1: the technical challenge of unlearning MoE.

A: Before our study, there has been no research on unlearning for MoE LLMs. A straightforward approach would be to apply existing MU techniques to them. However, as mentioned in Section 3, we find that this naive solution fails due to the expert selection shift issue. As discussed in Insight 2 in Section 4, this issue arises because the dynamic routing mechanism in MoE shifts expert selection from tar-get to non-target experts unintentionally. This unique challenge posed by the MoE architecture motivates our proposed approach UOE.

W1-2: It could be more reasonable to directly apply this decision + MU tech when unlearning on the MoE system remains unclear.

A: Thank you for the suggestion. We would like to discuss the decision + MU if you can provide more information about what is the “decision” here.

W2: Like some routing algorithms, the performance improvement result from whether to reduce the uncertainty or routing token to the correct expert is unclear.

Q1: Is the rising uncertainty among routing algorithms causing ineffective MoE unlearning? If so, why not directly optimize (or maintain) the entropy?

A: Thanks for the valuable question. According to [1], uncertainty measures routing results of expert selection of tokens. We visualize the expert selection distribution in one layer across the unlearning process in Fig.6 in revision. The series of figures sort the experts in layer 21 by the selection proportion in the original Deepseek model and keep this order to plot the figures in unlearned models to show the changes in all experts. The results show that GA algorithm decreases the uncertainty in WMDP after unlearning. We would like to have further discussion with the reviewer regarding if this is an interesting analysis connecting the decrease of uncertainty to the effectiveness of unlearning by our UOE.

[1] Wu, Haoze, et al. "GW-MoE: Resolving Uncertainty in MoE Router with Global Workspace Theory." arXiv preprint arXiv:2406.12375 (2024).

W3: UOE seems only to use TopK routing as the backbone. The versatility of UOE framework is not discussed well.

A: We believe that our methods are versatile and can be applied to both sparse and dense routing schemes. In the case of dense routing, where all experts contribute to the final output, affinity scores can still be calculated to measure the relevance of each expert's contribution, similar to how they are utilized in top-K routing.

Q2: As λ (hyperparameter for retaining loss, eq.1) aims to directly optimize the strength of retain quality, there is no discussion on this setting. If λ is set to 0, what would happen?

A: If $\lambda$ is set to 0, GD will be the same as GA. In our experiment, both GA and NPO set $\lambda$ to 0. The results in Tab. 2 in our original submission indicate that GA+UOE shows better performance than GD+UOE.

Q3-1: Is the affinity score unchanged (or as a golden label) during the unlearning progress?

A: No, the affinity score does change, which leads to the shift in expert selection. As shown in Fig. 6 of our revision, the affinity scores change significantly during the GA unlearning process. In UOE+GA, we observe that the top-1 expert's affinity score increases from 0.21 to 0.53 after unlearning.

Q3-2: If the affinity score is the golden target, why not use it as a roadmap but an optimization target? Is it proper to refine the routing strategy during optimization?

A: In our work, we aim to avoid making changes to the architecture of MoE models. It may be a good idea to apply a roadmap, but there are some challenges. First, identifying target tokens and determining whether to unlearn within the same token is complex. For example, we may want to unlearn code related to cybersecurity issues but retain coding capabilities in MMLU. Second, it is unclear how to adapt MoE models with an optimization strategy to roadmap routing for all tokens in both the unlearn and retain sets. We plan to explore these issues in the future.

W1: The discussion on shared experts during unlearning, and additional evidence is needed.

A: Thank you for the suggestion. Shared experts are designed to capture and consolidate common knowledge across different contexts [2]. However, our task focuses on more specific knowledge, which is less likely to be represented by shared experts. Additionally, the top-1 expert after unlearning introduces perturbation to the knowledge that is propagated from previous layers and shared experts. This influence helps ensure that the targeted knowledge is effectively disrupted throughout the network. To explore the outcome of unlearning shared experts, we conduct experiments on Deepseek unlearned by GA with WMDP. The results below indicate that unlearning shared experts leads to a sharp decrease in utility, e.g., the Forget Accuracy drops to 0.3554 when 4 shared experts are unlearned along with the top-1 expert, in comparison, the Forget Accuracy is 0.51 when only unlearning the top-1 expert. Even unlearning the top-6 experts (6 experts as targets) is better than unlearning four shared experts.

[2] Liu, Aixin, et al. "Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model." arXiv preprint arXiv:2405.04434 (2024).

W2: The combination of unlearning loss objectives (1) and (3)?

A: Thanks for your question. Yes, the loss objective is the combination of (1) and (3) by a hyperparameter \alpha. We clarified our loss objective in Eq.(4) in our revision. We further conduct experiments on different settings of the hyperparameter, where the results indicate UOE is relatively insensitive to the hyperparameter.

Q1: What does line 4 in Algorithm 1 mean?

A: Thanks for your question. We have clarified in our revision that the selected expert and the router in the same layer would be trainable (See Line 324).

Q2: For other MoE models (for example Mixtral) without such a shared expert setting, how does UoE work?

A: Due to the limited computational resources available, we conducted new experiments on Mixtral 8*7B only in the PEFT setting (Tab. 4 in revision). The results show that UOE can achieve similar performance with ESFT while only training 97% less parameters. The results have been included in Tab.4 in our revision.

W1: Notation Clarity

A: Thanks for the suggestions. We have revised our draft and clarified key notations, such as the definition of $\mathbf{g}^{(l)}=[g^{(l)}_1,g^{(l)}_2,\dots,g^{(l)}_i]$, which is the output of the router.

W2: Training hyperparameters are insufficiently

A: Thanks for the valuable suggestion. We skip this part due to the limitation of space. We include the following in the appendix Sec. A:

We set the learning rate to 5e-5 for GA, NPO, and GD while setting it to 1e-4 for UOE. The batch size is 4 for GA, NPO, and GD, while it is set to 16 for UOE. In NPO, the beta value is set to 0.001. The alpha for the retain loss is set to 1 in both GD and NPO. For RMU, we follow the hyperparameters specified in the original work. We configure the steering coefficients as 8000 for Qwen and 32000 for Deepseek, as UOE targets deeper layers in these models.

Q1: how robust is this approach when there are experts with closely competing scores?

A: This is a great question! Our approach is robust to experts with closely competing scores. Regarding the robustness of experts across different affinity scores, we have shown the results in Tab. 7 of our original submission. The results indicated UOE is robust to different affinity scores as mentioned in line 511. If experts have closely competing affinity scores, no matter which one is selected, it makes similar large impacts on the hidden state, resulting in similar unlearning performance. So the expert with the highest affinity score can lead to the largest impact on the perturbation. To explore the robustness of selecting experts with competing scores, we conduct new experiments on the rank 2 expert and the rank 3 expert and report the results in the table below. The results indicate that the rank 2 and 3 experts lead to unlearning performance as good as the rank 1 expert.

Q2: Can the soft prompt uncover the knowledge from other experts?

A2: Thanks for your valuable question. The soft prompt attack fails to uncover the knowledge. To investigate the soft prompt attack, we employ the GCG attack in a white-box setting to optimize the prompt, aiming to force responses to begin with "Sure, here is the answer:". The number of optimization steps is increased to 5000, while other hyperparameters remain at their default settings. Considering the computational demand (approximately 1 GPU hour on an A100 for generating a single soft prompt), we optimize 400 prompts across 400 samples in RWKU. Since not all responses begin with "Sure, here is the answer:", we filter for those containing the word "answer" and then assess the forget quality both with and without GCG-generated prompts. Experimental results below indicate that the GCG, despite being the strongest prompt-level attack, fails to recover the forgotten knowledge, as the forget accuracy (FA) remains at 0.01 before and after the GCG attacks. We have included these results in section B in the Appendix.

W1: The novelty of this work lacks a compelling impact.

A: We respectfully disagree. We would like to further highlight the contribution and novelty of our work as below: 1) we are the first to explore unlearning in MoE architecture, while most existing works focus on dense architecture. 2) We identify the failure of existing unlearning algorithms in MoE models and investigate the underlying causes. 3) Based on these insights, we propose an effective and efficient framework called UOE for MoE unlearning, which enhances all baseline methods in terms of forget quality up to 5% and model utility by 35% while training only 0.06% of the model parameters. We believe that our work will have an impact not only on unlearning tasks but also on the MoE pre-training and post-training phases, where the expert shift issue would be a key problem to prevent stabilizing the MoE training.

W2: the meaning of g(l)

A: Thanks for pointing this out. We have clarified the notations in our revised manuscript in Line 324. Specifically, $\mathbf{g}^{(l)}=[g^{(l)}_1,g^{(l)}_2,\dots,g^{(l)}_i]$ is the output of the router in the l-th layer.

W3: Whether the previously identified target expert can reliably be the true target expert.

A: Yes. The true target expert in UOE is the decisive expert to process the knowledge that we want to unlearn, which does not change across the unlearning process. As mentioned in Insight 2 (Line 275), the expert selection shift will hide the target expert. To solve this problem, we propose “A single-layer solution for MoE LLM unlearning by UOE” in the following section, where unlearning only happens on one layer to further stabilize the routing choices.

W4: It is unclear what Ll represents

A: Thanks for pointing this out. We have clarified it in our revised draft (see Line 297). It is Router^l,indicating the router in the l-th layer in our revision.

W5-1: The unique advantages of UOE compared to other MoE frameworks.

A: We would like to clarify that our UOE is an unlearning framework, which is not a machine learning model and thus not comparable with MoE architecture. As a flexible framework, UOE can be applied to an MoE model by incorporating any existing unlearning algorithms, in an efficient and parameter-efficient manner. Therefore, the unique advantages of UOE are its flexibility and efficiency for unlearning MoE models.

W5-2: It would be helpful to clarify what "precise" entails in this context and which additional factors might be considered more significant than precision.

A: Thanks for the suggestions. We have clarified this in our revision (see Line 306). According to the equation in lines 191 and 192, the hidden state is calculated by weighted sum, where the token assignment ratio overlooks the weight for each expert.

W6: The inconsistent usage of N

A: Thanks for pointing this out. We have revised Equation(2) in Line 310, where Z is used to represent the size of the calibration dataset.

W7: The explanation on how adjusting model hyperparameters allows one experiment's performance to be maintained while enabling changes in another.

A: Thanks for the suggestions. We have clarified our rebuttal revision. To compare different numbers of unlearned experts, we control the training iterations in GA to achieve around 0.25 in FA, which indicates the random selection.

W8: further justification to unlearn only the top-1 expert.

A: Thanks for the question. Although the tokens might be handed by multiple experts, involving only the top-1 expert in the unlearning process is enough because the perturbation applied to the top-1 expert can significantly impact the targeted knowledge to forget, due to its large weight in the subsequent average of experts’ outputs in hidden state. In the section “Tuning multiple experts in a single layer”, we justify that unlearning the top-1 expert is better than unlearning top-3 experts (3 experts as targets) and even top-6 experts (6 experts as targets) (refer to Tab. 5). Therefore, we believe that unlearning the top-1 expert is sufficient. In addition, the results in Tab. 7 show that the top-1 expert selection can outperform the random expert selection.

Thanks for your valuable questions.

Q1: However, I’m still uncertain whether the previously identified target expert can reliably be regarded as the true target expert.

A: Thank you for your insightful question. In UOE, the target expert is always the true target expert. Let us first analyze the reasons behind expert selection shifts and then explain why UOE ensures the target expert remains true.

Reasons for Expert Selection Shifts:

The experiment in Figure 3 reveals that shifts in expert selection are caused by updates to both the router layers and the hidden states. This is evident as the overlap ratio decreases significantly when using either the Router Only or Expert Only approaches.

Why the Target Expert in UOE is the True Target Expert:

To clarify, we break the model into three parts:

1. Layers before the target expert: Since all parameters in these layers are frozen, both the router layers and the hidden states remains unchanged. As a result, no expert selection shifts occur in this part.
2. The layer of target expert: As the input hidden state in this layer (the output hidden state from previous layers) is unchanged, the router is the only factor that could induce a shift. Anchor loss is applied to stabilize the expert selection process and increase the affinity score of the target expert during unlearning. The experiment results support that the top-1 expert's affinity score increases from 0.21 to 0.53 after unlearning of GA+UOE in Deepseek with RWKU dataset.
3. Subsequent layers: While expert selection shifts may occur in later layers, these do not influence the target expert or its associated router.

Therefore, in UOE, the target expert used for unlearning reliably is regarded as the true target expert.

Q2: Moreover, if each layer can successfully identify the most relevant target expert, why does applying unlearning across all layers still lead to shifts in expert selection?

A: Unlearning a single expert with the highest affinity score per layer is not the intended design of UOE. Instead, it serves as a preliminary study or baseline, illustrating that the challenge of expert selection shift cannot be addressed merely by reducing the number of experts in each layer. This observation motivates our approach to developing a targeted single-expert unlearning solution, as detailed in the section “A Single-Layer Solution for MoE LLM Unlearning via UOE.” Our statement that in UOE the previously identified target expert is indeed the true target expert, is not contradictory.