Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Safety Self-Alignment

Thank you for your detailed feedback, which has helped strengthen our work. We have addressed each point you raised with specific clarifications and improvements.

1. Table 1, the results are based on which model?

For PKU-SafeRLHF, the results are based on Alpaca-7B, and for HH-RLHF, they are based on Wizard-Vicuna-Uncensored-7B.

2. For evaluation, it would make this paper stronger if the authors could provide more details about how human annotators provide a final verification for precise results. & The metrics, help, harm, and harm-help are too vague and may be sensitive to human annotators.

We use GPT-4 to evaluate the helpfulness and harmfulness of responses rather than relying on human annotators. This approach aligns with most baselines, and the relationship between GPT scores and human judgments has been extensively analyzed in [1, 2]. Based on their findings, we directly adopt these metrics, ensuring consistency with established practices.

[1] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023

[2] RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment, ICLR 2024

3. It would be better if the authors could show if their method has the same effects in bigger LLMs (such as picking some LLMs with 13B or even larger).

We applied MoTE to the larger Llama2-13B model and plan to explore even larger models in future work. As shown in the following table, MoTE remains effective on the 13B model and achieves new state-of-the-art results on HH-RLHF.

Thank you for your detailed feedback, which has helped strengthen our work. We have addressed each point you raised with specific clarifications and improvements.

1. The benefit of shared expert.

The shared expert is a critical design element that enables the answer expert $E_{ans}$ to internalize information from analysis and guidance, making it essential for single-step inference. Without the shared expert, $E_{ans}$ cannot interact effectively with $E_{a}$ (analysis expert) and $E_{g}$ (guidance expert).

For instance, under single-step inference, activating only $E_{ans}$ without the shared expert reduces MoTE to a naive safety alignment approach, relying solely on QA pairs without leveraging training with analysis and guidance. As shown in the ablative analysis in Table 3, the safeness of MoTE without the Shared Expert drops significantly. This demonstrates that the shared expert is a key design for fostering synergy among experts and enhancing the effectiveness of thought chains.

2. Can MoTE be extended to other self-alignment methods that involves multiple experts?

Yes, MoTE can be extended to other self-alignment methods that utilize multiple experts. In this work, we use AlignCoT as an example to analyze the integration of Mixture of Experts (MoE) and Chain-of-Thought (CoT) in safety alignment. While the current implementation focuses on this framework, MoTE can be straightforwardly adapted to other alignment methods involving additional experts, which we plan to explore in future work.

3. Novelty of each component is relatively limited

We claim that we are the first to integrate Mixture of Experts (MoE) into safety self-alignment and have designed an efficient framework to achieve this. We have highlighted the importance of each component and analyzed the overall efficiency of our framework. We believe this work will contribute to the community by facilitating the extension of MoE into alignment methodologies.

4. Why aim for single-step inference and why MoTE is effective on this setting?

Single-step inference is critical in the safety domain, as most harmful questions are immediately recognizable to humans at first glance. Thus, our goal is to make safety an instinctive response for the LLM, avoiding the need for generating lengthy and tedious thought chains. Notably, all baseline methods are also designed with single-step inference in mind, ensuring fair comparison.

MoTE performs better in this setting due to two key reasons. Theoretically, as explained in Appendix A (now moved to the main text, Lines 160-183), learning intermediate steps collectively enhances single-step inference performance. Practically, MoTE trains specialized experts for different tasks—analysis, guidance, and answering—to strengthen each intermediate step (Fig. 2). Additionally, step-skipping further improves single-step performance (Table 3). These factors demonstrate that MoTE is particularly effective for safety self-alignment in single-step inference scenarios.

Thank you for your detailed feedback, which has helped strengthen our work. We have addressed each point you raised with specific clarifications and improvements.

1. The formulation in Sec.3 and Equation 1 are incomplete. & The justification for modeling CoT as a joint probability distribution between different steps is not clear.

In the specified paragraph, $x \in D$ only represents a question or context, not include the responses. Modeling CoT as a joint probability is not new in the related literature[1,2].

Specifically, we have update the related part that define clearly the notation of each step and claim that enhancing each middle step helps enhancing self alignment, referring to Line 160-183.

[1] Training Chain-of-Thought via Latent-Variable Inference, NeurIPS 2023

[2] Self-Consistency Improves Chain of Thought Reasoning in Language Models, ICLR 23

2. A dedicated subsection comparing AlignCoT with previous work is needed.

We have updated the related work section to clarify the differences between AlignCoT and existing self-alignment methods (Lines 83-95) and added a paragraph discussing how MoTE differs from fine-tuning approaches based on CoT (Lines 97-105).

Our main observation is that existing self-alignment methods primarily focus on collecting optimal responses, while AlignCoT emphasizes that CoT is an effective approach for safety alignment, with the intermediate steps also playing a crucial role. Additionally, AlignCoT relies on supervised fine-tuning, avoiding the complexity of reinforcement learning (RL) techniques. Furthermore, we compare AlignCoT with modern self-alignment methods such as CoT, Critique-Revise, Mistake Analysis, RLCD, and MATRIX (see Table 1).

Compared to methods that fine-tune with CoT data, MoTE is the first to leverage a Mixture of Experts (MoE) framework for CoT fine-tuning, introducing innovative techniques like shared experts and step-skipping to enhance alignment performance.

3. "Efficient training of MoTE" section offers little novelty.

We claim that applying masked attention in CoT training to enable step-skipping is a novel attempt. This approach is specifically designed to enhance the efficiency of MoTE training by allowing selective activation of experts for relevant steps, which distinguishes it from traditional masked attention techniques.

4. Clarity needs improvement.

It is unclear which 7B LLM is referred to in Line 177 (Line 195 in updated manuscript). Is it Alpaca-7B?

Yes.

The metrics used in Figure 2 are not specified or explained in the text.

We have revised the manuscript and mentioned them in Line 197-200.

Here we briefly introduce: the x-axis of Analysis Quality and Guidance Quality are scored by GPT-4-1106-preview. The x-axis of Single-step Inference and Multi-step Inference are the ratio of safe responses, following the matric Harm explained in Line 373.

Thank you for your detailed feedback, which has helped strengthen our work. We have addressed each point you raised with specific clarifications and improvements.

1. Unclear about the relationship between (x), (x_a), (x_g), (y_{\text{cot}}), and MoTE.

Your understanding is correct. We utilize the model itself to generate analyses, guidance, and safe responses for safety self-alignment. This process involves a three-step generation, which we treat as a structured thought chain designed to produce harmless responses effectively. The data generated through this process are then used to re-train the model, resulting in a safer model.

MoTE is an efficient framework that assigns specialized expertise to each expert for the respective steps, enhancing the self-alignment process.

2. Line 128 (Line 139 in updated manuscript): are analysis, guidance and safe answers are produced by the same model.

Yes. For self-alignment, we use self-generated data, including analysis, guidance, and safe answers, to refine and enhance the original model.

3. Line 128 (Line 139 in updated manuscript): where does training data ((x_a), (x_g), (y_{\text{cot}})) come from?

All training data is generated by prompting the original model using AlignCoT, following the prompts outlined in Fig. 1. The harmful questions are sourced from the training sets of PKU-RLHF and HH-RLHF.

4. Figure 2: Subfigure Titles and X-axis.

We have updated the titles and metrics as described in Lines 197-200.

The x-axis for Analysis Quality and Guidance Quality represents scores provided by GPT-4-1106-preview. The x-axis for Single-step Inference and Multi-step Inference shows the ratio of safe responses, measured using the metric Harm as explained in Line 359.

5. Line 182-210 (Line 203-236 in updated manuscript): move this part to the Experiment Section.

Thank you for the suggestion. We have moved Observation 1 to the Experiment section. However, we consider the other two observations crucial for motivating MoTE, so we have kept them in their original position.

6. Line 185 (Line 197 in updated manuscript) An introduction to the metrics would be helpful. In Table 1, there is no explanation for "Help," "Harm," and "Harm-Help."

We have update the manuscript to include an introduction to the metric in Line 197. For Table 1, All metrics are introduced in Line 369-377.

Here we briefly introduce them: Helpfulness (Help) is rated on a score from 1 to 10 by GPT to determine the informativeness of responses. For harmlessness (Harm), a binary assessment by GPT determines the safety of answers, reporting a harmless rate. To ensure that higher harmlessness rates are not achieved by simply declining to answer, we also measure the helpfulness (Harm-Help) for responses to harmlessness prompts. Higher scores indicate better performance across all metrics.

7. Line 367 (Line 413 in the updated manuscript): what datasets are used for fine-tuning the baselines?

All baselines in Table 1 are fine-tuned using the datasets specified in their respective works. We highlight that existing safety alignment methods often overlook the importance of intermediate steps. By incorporating these steps through MoTE, we achieve superior performance, which constitutes our key contribution.

Additionally, the result of training with D_{\text{ans}} can be referenced in Figure 6 (AlignCoT w/o Analysis & Guidance). The Harm score reaches approximately 80, approaching the performance of the state-of-the-art MATRIX.