Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis

Thank you for your acknowledgement and valuable suggestions. Here we address your questions one by one.

W1: Clearer structure of Section 4. Thanks for your advice. Following your suggestions, we have updated the structure of Section 4 by separating the background notations, the proposed method and the discussions. See details in the revision.

W2: Inappropriate wording and typos.

Thank you for pointing that out. We have correct all the mentioned inappropriate wording and typos in the revision.

Q1: Verifying Equation (2) via harmless tag prediction.

Following your suggestions, we conduct an additional experiment evaluating the LLM's ability to discern the harmfulness of responses. Specifically, we utilize the test set of PKU-SafeRLHF dataset, each instruction paired with a response and the corresponding binary harmful or harmless tag, and then instruct LLMs to distinguish whether the provided response is harmful or not. The accuracy is reported in Table R5. The LLM's accuracy in identifying the harmfulness of responses improves from 54.5% for vanilla Aplaca-7B to 72.6% after applying our method, revealing the significant improvement of discrimination ability.

▾ Table R5. Comparison between discrimination ability with respect to the binary classification accuracy on the test set of the PKU-SafeRLHF dataset.

Thank you for your valuable and constructive reviews. Here we address your questions one by one.

W1: The reason for discussing the relative ease of discrimination compared to generation, which is already a core principle of RLHF.

Thank you for your valuable feedback. The discussion on generation versus discrimination actually sets the foundation for our methodology, which utilizes the inherent discriminative power of LLMs. Our method, leveraging the LLM's self-critique for improved generation via mistake analysis, stands out from RLHF's reliance on external human annotations and reward models. This difference emphasizes the importance of easier discrimination over generation by the model itself.

According to your comments, we realize that the success of RLHF may also benefit from increasing the discrimination ability of models with external reward models. On the other hand, our method boosts this ability through mistake analysis, which provides more informative feedback in natural language compared to the relative quality ranking in RLHF (see Figure 1). This explains why our model achieves superior performance over RLHF even without external reward models (refer to Table 1).

W2: Redundant description in the caption of Figure 2.

Thank you for highlighting the need for a more concise caption for Figure 2. We have revised the caption as below and updated the paper accordingly.

(a) Comparison between generation and discrimination abilities for Alpaca, GPT-3 and GPT-3.5. (b) Comparison between guided and unguided analyses. Refer to Section 3 for more details.

W3 and Q1: Experimental results with the same mistake source for baseline methods.

As demonstrated in the 4th-6th rows in Table 1 and the 3rd-5th rows in Table 2, we have conducted experiments for all baseline methods with the original mistakes provided by the datasets for a fair comparison, revealing the superiority of our mistake analysis. Following your suggestions, we further provide the experimental results of all baselines with induced mistakes below and also in the revision, which is consistent with our previous observation.

▾ Table R3. Comparative results of LLM alignment across various methods with induced mistakes.

▾ Table R4. Comparative results of defense against attacks across various methods with induced mistakes.

Thank you for your thorough review of the manuscript. We have carefully checked the entire paper and made the following changes:

1. Corrected grammatical errors and awkward phrasings;

2. Reorganized the paper, particularly the introduction and preliminary sections, to enhance the overall coherence.

The revised version has been updated. We hope that it can address your concerns adequately.

Dear Reviewer 5o83,

We are deeply appreciative of the time and expertise you have devoted to reviewing our manuscript. In acknowledgment of your insightful feedback, we have conducted thorough revisions to our work, which predominantly include:

1. Justifying the relative ease of discrimination versus generation in LLMs as a motivation for our method;

2. Conducting additional experiments that cover various sources of mistakes in baseline methods.

Furthermore, we have meticulously reviewed and refined the manuscript to correct grammatical errors and improve clarity. We hope these efforts adequately address your initial concerns. With the deadline of discussion period approaching, we keenly await your further feedback and are eager to continue this constructive exchange.

Best regards,

Authors of Paper 156

Thank you for your acknowledgement and valuable suggestions. Here we address your questions one by one.

W1-3: Justification about the generalization ability of LLMs aligned with mistake analysis.

(1) Generalization beyond training distributions. We acknowledge that no alignment method, including ours, can fully address all out-of-distribution scenarios. Nonetheless, our model exhibits enhanced generalization capabilities by learning from a wide array of mistakes and their analyses, rather than solely relying on human-labeled instruction-response pairs. As depicted in Figure 5, our model effectively rejects unsafe user instructions in "Goal Hijacking" scenarios for topics not in the training data, a feat not achieved by methods like SFT. This demonstrates our model’s advanced understanding of such complicated mistakes and its improved generalization ability.

(2) Addressing diverse failure modes and mistakes of alignment models. Admittedly, even meticulously aligned LLMs face challenges in completely avoiding unsafe responses in novel or unanticipated situations. This necessitates an adaptable "debugging" methodology for addressing numerous corner cases. Section 5.2 details our approach’s efficacy in defending against new-coming instructional attacks. After applying our method, the safety-aligned ChatGLM model demonstrates significant post-alignment improvement, with its Harmless Rate in resisting "Goal Hijacking" attacks increasing from 68.4% to 85.3%. This positions it as the second rank only under the OpenAI's GPT-3.5, as per the SAFETYPROMPTS benchmark [1, 2], while maintaining its helpfulness performance.

W4 & Q4: Theoretical analysis on how mistake analysis improves alignment.

In Section 4, Equation (2), we explore the role of mistake analysis in improving LLM alignment from a Bayesian perspective, considering that LLMs operate as conditional generators. Mistake analysis involves examining the model's harmful responses to identify patterns leading to such errors. To train the model in this way, we construct triplets of the instruction $X$, the response $Y$, and the mistake analysis $T$ indicating whether the response is harmful or harmless. Such data refines the model's understanding of the conditional probabilities $p(T|Y, X)$, i.e., the likelihood of a response being harmful or harmless given a certain instruction. Subsequently, we enhance its ability to generate appropriate and harmless responses $p(Y|X, T)$ through guided inference, where the model is required to produce harmless responses during inference (see Figure 1). By focusing on optimizing conditional probabilities through the analysis of mistakes, we enhance the model's ability to generate safe, coherent, and contextually appropriate responses. This is why mistake analysis benefits model alignment. See Section 4.2 and Appendix D for more details.

Q1: Resilience against adversarial instruction attacks.

Recent studies [3, 4] demonstrate that even well-aligned models, such as OpenAI's GPT-4 and Anthropic's Claude2, are vulnerable to adversarial instruction attacks, presenting a challenge distinct from aligning with human values. Our research primarily focuses on aligning LLM with human values. We have observed the effectiveness of our method in defending against delicately designed instruction attacks, such as "Goal Hijacking" with superior generalization ability (see Section 5.2). This suggests the potential effectiveness of our method in enhancing resilience against adversarial attacks. Further investigation will be considered in our future work.

Q2-3: The quality or diversity of mistake analysis for alignment improvements and more ablation study.

(1) Quality of mistake analysis. Yes, the quality of mistake analysis significantly impacts alignment performance. As Table 1 shows, using GPT-3.5-turbo for mistake analysis results in better alignment than using Alpaca-7B itself (see rows 7 and 8). Additionally, guided analysis outperforms unguided analysis, as shown in Table 3 (rows 3 and 4).

(2) More ablation study on mistake analysis. In Section 5.3, we present a comprehensive ablation study on the essential components of mistake analysis, including (i) quality of mistake analysis (ii) quantity of mistake analysis and (iii) the source of responses for constructing analysis data. We find that high-quality analysis, whether guided or from advanced models like GPT, leads to better alignment. On the other hand, increasing the volume of analysis data does not always improve results. We observe a decrease in performance when multiple analyses are applied to the same instructions, possibly due to conflicting interpretations in the analyses of the same instruction. Furthermore, we observe that the alignment effectiveness is maximized when the bad cases are generated by the model itself. See Section 5.3 for detailed experimental results.

Dear Reviewer FapY,

We are grateful for the time and expertise you have contributed to the review of our manuscript. Considering your valuable suggestions, we have implemented extensive revisions to our paper. These revisions primarily include:

1. A detailed justification of the generalization capabilities of LLMs aligned with mistake analysis;

2. An in-depth theoretical discussion of how mistake analysis contributes to better alignment;

3. A discussion on the quality and diversity of mistake analysis in enhancing alignment, supplemented with additional ablation studies.

We trust these changes effectively address the issues you raised. As the deadline for the discussion period draws near, we eagerly anticipate your further insights and remain dedicated to an ongoing constructive dialogue.

Best regards,

Authors of Paper 156

W3: Justification about whether the method effective on the weak Alpaca model in Section 5.1 is also applicable to stronger models like Vicuna.

(1) Reason for choosing Alpaca. We select Alpaca-7B as the baseline in Section 5.1 due to its absence of prior safety alignment. This characteristic renders it an ideal candidate for evaluating the effectiveness of alignment methods for an unaligned LLM from scratch. Our results, as elaborated in Table 1 and Figures 12-13, demonstrate that even unaligned LLMs like Alpaca-7B can effectively leverage mistake analysis to enhance their safety instruction-following capabilities.

(2) Effectiveness on SOTA models. Moreover, to address concerns about the method's effectiveness on stronger, safety-aligned models, we have expanded our experiments to include ChatGLM-6B, the highest-ranked open-source model on the C-Eval leaderboard [1]. As shown in Table 2 and Section 5.2, after incorporating the mistake analysis to defend against a novel type of instruction attack known as "Goal Hijacking", ChatGLM ranks the second on the SAFETYPROMPTS benchmark [2, 3]. Notably, it is only surpassed by OpenAI's GPT-3.5 in defending against the "Goal Hijacking" attack, while concurrently preserving its helpfulness performance.

(3) Additional experiments on Vicuna. In line with your suggestion, we further conduct experiments on Vicuna-7B (v1.1) following the exact setting as in Section 5.1. As reported in Table R2, our mistake analysis achieves consistent improvement regradless of the baseline models.

▾ Table R2. Comparative results of LLM alignment across various methods on Vicuna-7B.

[1] C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models. https://cevalbenchmark.com/.

[2] SAFETYPROMPTS Benchmark. http://115.182.62.166:18000/.

[3] Sun H, Zhang Z, Deng J, et al. Safety assessment of Chinese large language models. arXiv preprint arXiv:2304.10436, 2023.

Thank you for your valuable and constructive reviews. Here we address your questions one by one.

W1: Justification for comparing LLM's "discrimination" ability with generation" ability.

(1) Definitions of generalization and discrimination. Thank you for your valuable feedback. To clarify, in our paper, "generation" is defined as the LLM's ability to produce helpful and harmless responses to user instructions, while "discrimination" refers to its capacity to identify potential errors in responses, which requires not only a binary classification between "harmful" and "harmless" but also to encompass a more nuanced analysis of the response quality, as illustrated in Figure 7 of the Appendix. Therefore, the continuous score is considered as a better metric than the simple binary classification accuracy here. The preliminary aims to show the observation that LLMs are generally more adept at identifying potential mistakes than directly generating high-quality responses. This distinction is vital for the subsequent method design, where we use self-critique data to enhance the model's generation capabilities.

(2) Experiments of binary harmful/harmless classification. However, we acknowledge your concern about regarding the potential oversight of the binary classification accuracy. Following your suggestions, we conduct an additional experiment evaluating the LLM's ability to discern the harmfulness of responses. Specifically, we utilize the test set of PKU-SafeRLHF dataset, each instruction paired with a response and the corresponding binary harmful or harmless tag, and then instruct LLMs to distinguish whether the provided response is harmful or not. The accuracy is reported in Table R1. The LLM's accuracy in identifying the harmfulness of responses improves from 54.5% for vanilla Alpaca-7B to 72.6% after applying our method, revealing the significant improvement of discrimination ability.

▾ Table R1. Comparison between discrimination ability with respect to the binary classification accuracy on the test set of the PKU-SafeRLHF dataset.

W2: More explanation about Equation (2) which implies a direct link between training a safety discrimination model and obtaining a safe response generation model.

Thanks for your valuable comment. We have revised the deviation of Equation (2) in Section 4 and Appendix D. In summary, your observation about the counter-intuitive nature of directly obtaining a safe response generation model from a safety discrimination model is astute when only mistake analysis data is utilized in fine-tuning. However, this scenario is different from the discussion of Equation (2), where the mistake analysis data is incorporated together with the normal SFT data, as discussed in the "Step 2: Guided analysis generation" paragraph of Section 4.1.

Let us start with the Bayes' Theorem $p(T|Y, X) = [p(Y|X, T)p(X|T)p(T)]/[p(Y|X)p(X)]$, which simplifies to $p(T|Y, X) \propto p(Y|X, T)$ under the assumption that $p(Y|X)$ remains relatively stable during the fine-tuning process. To maintain the model's capability to produce a response given an instruction (i.e., preserving $p(Y|X)$）, we combine original SFT data, both helpful ($D_{\text{helpful}}$) and harmless ($D_{\text{harmless}}$), with mistake analysis data during fine-tuning, rather than solely relying on the mistake analysis data. Therefore, Equation (2) actually aims to explore how the additional mistake analysis data, along with the original SFT instruction-response pairs, can enhance alignment performance.

Moreover, our ablation study in Section 5 demonstrates that an excess of mistake analysis data, particularly when it doubles the SFT data, actually leads to poorer performance (3rd vs. 5th rows), which indicates a nuanced relationship between different types of data in fine-tuning and the model's ability to balance between generation and discrimination tasks effectively. We hope this can clarify the assumption and methodology underlying our approach and address your concerns.

Dear Reviewer sdVt,

We sincerely appreciate the time and expertise you have invested in reviewing our manuscript. Following your insightful comments, we have undertaken substantial revisions, including:

1. Providing a thorough justification for comparing the "discrimination" and "generation" abilities of LLMs;

2. Expanding explanations regarding Equation (2) in the main text;

3. Conducting additional experiments with more baselines such as Vicuna.

We believe these modifications comprehensively address your previous concerns. With the discussion period deadline nearing, we eagerly await your further feedback and remain fully engaged in ongoing dialogue.

Best regards,

Authors of Paper 156