Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!

We are glad that the reviewer finds our results interesting and our evaluations detailed. We also thank the reviewer for all the constructive suggestions. We hope the following clarifications can address the reviewer’s concerns:

1. The "surprise" in our results.

   • We reveal a surprisingly huge asymmetry in the investment of alignment and the ease of removing it. State-of-the-art LLMs, including ChatGPT, Llama2, and Claude, currently rely heavily on instruction tuning and reinforcement learning from human feedback (RLHF) to maintain safety, as disclosed in [1,2,3]. Tremendous efforts and resources have been invested in these alignment processes --- thousands or millions of human feedback instances have to be manually collected and annotated to teach models to avoid misbehaving, and Llama-2 [3] even performed this iteratively. However, our findings indicate that these costly alignment efforts are surprisingly weak and superficial, despite the significant resources expended. The simplicity of our technique and the extremely low cost (less than 0.2 dollars for GPT-3.5-Turbo and merely five gradient steps with a moderate learning rate of $5e^{-5}$ for Llama-2) only make this finding even more surprising. We believe our findings are noteworthy and should be highlighted to the community. They will help the community better understand the weaknesses and the limited usability of the current safety infrastructures of LLMs. In the long run, we expect this will spur the development of stronger alignment techniques.

   • The three levels of risk hierarchy we present, and the mitigation analysis we perform, reveal oversights in current safety alignment infrastructures and contribute to a systematic understanding of the risk space. In the initial release of GPT-3.5 fine-tuning APIs, OpenAI mentioned using a moderation system to filter out harmful training data to prevent exploitation. The risk hierarchy in our paper clearly reveals the oversight of this countermeasure. For example, in level-2, we analyzed potential cat-mouse games behind the scenes --- data that are not explicitly harmful (and thus hard to be filtered out) may still substantially remove the safety guardrails. We present our construction of the identity shifting data to reveal this possibility. Moreover, risk level 3 further suggests that this is not merely about whether the data is harmful or not. Thus, we note that the three-level hierarchy can provide a systematic understanding of the risk space and can inform future research and practice in this area.

     Additionally, the analysis of potential mitigation strategies we present also offers important takeaways for safety practitioners. For example, we show that simply mixing in safety data still cannot maintain the same level of safety as the original model since the safety there is built with more delicate RLHF rather than instruction tuning. Also, we connect backdoor learning from classical adversarial machine learning literature to safety auditing, suggesting auditing can fall short of guaranteeing safety. (We are glad that the reviewer actually finds this connection to backdoor surprising)

We thank the reviewer for the positive rating of our paper. We also appreciate the reviewer for acknowledging the novelty of this work and all the constructive suggestions. We hope the following clarifications can address the reviewer's concerns.

1. Evaluation of Helpfulness

   We appreciate the reviewer for bringing attention to the important issue of the influence of fine-tuning on the generative capabilities of LLMs. Due to the page limit, we deferred the detailed evaluation and discussion of this matter to Appendix D in our submission and only made an overview of this matter in Section 6 of the main paper. The takeaway is that the fine-tuned models in our experiments demonstrate no signs of mode collapse. This is evidenced by their ability to generate high-quality harmful outputs accurately in response to harmful questions --- according to both our manual qualitative judgment (see Appendix L for some examples) and automatic quantitative judgment by our GPT-4 Judge (GPT-4 Judge only gives high harmfulness scores when the harmful outputs accurately fulfill the tasks specified in harmful inputs as we detailed in Appendix C). Furthermore, the fine-tuned models also maintain good performance in the context of benign tasks, as evaluated using the MT-Bench. Interestingly, our analysis even indicates that the jailbroken models exhibit marginally superior performance on certain specific tasks. Please see Appendix D for details. We apologize that our presentation did not sufficiently emphasize this important aspect, and we will endeavor to highlight this result more prominently in the main paper in our revision.

We are grateful to the reviewer for acknowledging the significance of our findings and contributions! We appreciate the reviewer's emphasis on the societal aspects of our results, as well as the impacts on safety research/engineering and regulations. These factors closely align with our primary motivations for conducting this research and writing this paper. We are highly encouraged by the reviewer's feedback!

We also appreciate the suggestion to include a discussion on the potential causes for the observed fragility of current safety-enforcing methodologies. We will add more discussion of the underlying intuition. Here, we provide a consolidated view:

• We suspect that the ease with which models can be adapted back to an unaligned mode is rooted in the same mechanism that allows us to adapt models so easily with a few examples in benign contexts. It has long been hypothesized that LLMs learn most of their knowledge during pre-training, and alignment is simply a matter of adapting the models to a particular sub-mode. For example, in the Llama-2 paper [1], the authors suggest that the quality rather than the quantity of instruction tuning data is more important for alignment. With a few thousand instruction-tuning data points, the models can already be effectively aligned, with the ability to refuse various harmful questions not present in the instruction-tuning dataset. Our attack is motivated by the inverse of this capability—if a few-shot learning capability can be used for good alignment, it could also be exploited inversely to subvert that alignment. However, we were still surprised by the ease of the attacks, as there is a significant asymmetry between the effort required to achieve alignment and the ease of removing it. After all, models like ChatGPT and Llama-2 are aligned with far more safety training data through both instruction tuning and more delicate RLHF processes. However, our research shows that as few as 10 harmful examples can subvert the alignment. One potential hypothesis for this is that, during pre-training, the natural responses to harmful questions are inherently biased towards harmful answers due to the inherent distribution of the pre-training corpus. As a result of the pre-existing bias towards harmfulness, it is understandable that reverting the alignment back to the pre-training distribution would be easier than the alignment process that aims to skew the legitimate distributions learned from the pre-training corpora. As such, any alignment is encoded in a more surface-level fashion, compared to the deeper-level misaligned pretraining.

• In regard to the question of why benign fine-tuning still compromises safety, we discuss two perspectives in both the Introduction and Section 3.2. The first is the well-known phenomenon of catastrophic forgetting. The second relates to the safety-utility trade-off. Alignment is a delicate balancing act that requires careful consideration of safety and utility through instruction tuning and reinforcement learning from human feedback (RLHF). Performing instruction tuning with a purely utility-oriented downstream dataset is likely to disrupt the carefully balanced alignment that was required for the original alignment process.

We hope that consolidating these hypotheses and adding additional clarifications make our findings more intuitive and informative. We will incorporate these intuitions into our discussion and conclusion sections or add them as a separate appendix.

[1] Touvron, Hugo, et al. “Llama 2: Open foundation and fine-tuned chat models.” arXiv preprint arXiv:2307.09288 (2023).

We thank the reviewer for all the constructive suggestions, and we hope the following clarifications can address the reviewer's concerns:

1. Our findings and their implications are significant for the AI safety community and beyond, across disciplines. We highlight some key points:

   • We reveal the significant asymmetry in the investment of alignment and the ease of removing it. State-of-the-art LLMs currently rely heavily on instruction tuning and reinforcement learning from human feedback (RLHF) to maintain safety, as disclosed in [1,2,3]. Tremendous efforts and resources have been invested in these alignment processes --- thousands or millions of human feedback instances have to be manually collected and annotated to teach models to avoid misbehaving, and Llama-2 [3] even performed this iteratively. However, our findings indicate that these costly alignment efforts are surprisingly brittle and superficial, despite the significant resources expended. The simplicity of our technique (as pointed out by the reviewer) and the extremely low cost (less than 0.2 dollars for GPT-3.5-Turbo and merely five gradient steps with a moderate learning rate of $5e^{-5}$ for Llama-2) only make this finding even more surprising. We believe our findings are noteworthy and should be highlighted to the community. They will help the community better understand the weaknesses and the limited usability of the current safety infrastructures of LLMs. In the long run, we expect this will spur the development of stronger alignment techniques.

   • The three levels of risk hierarchy we present, and the mitigation analysis we perform contribute to a systematic understanding of the risk space. In the initial release of GPT-3.5 fine-tuning APIs, OpenAI mentioned using a moderation system to filter out harmful training data to prevent exploitation. The risk hierarchy in our paper reveals its limitation. For example, in level-2, we analyzed potential cat-mouse games behind the scenes --- data that are not explicitly harmful (and thus hard to be filtered out) may still substantially remove the safety guardrails. Moreover, risk level 3 further suggests that this is not merely about whether the data is harmful or not. Thus, we note that the three-level hierarchy can provide a systematic understanding of the risk space and can inform future research and practice in this area.

     Additionally, the analysis of mitigation strategies we present also offers important takeaways for safety practitioners. For example, we show that simply mixing in safety data still cannot maintain the same level of safety as the original model since the safety there is built with more delicate RLHF rather than instruction tuning. Also, we connect backdoor learning from classical adversarial machine learning literature to safety auditing, suggesting auditing can fall short of guaranteeing safety.

   • Our findings suggest a fundamental trade-off between model customization and safety. Customizing advanced LLMs to empower downstream applications is a highly desirable goal, with strong economic incentives and significant potential to benefit society. OpenAI is currently encouraging the community to customize GPT models, with the release of their fine-tuning APIs and GPT store ecosystem. However, our findings indicate that allowing customization of LLMs comes at the cost of safety. Our results show that the current safety infrastructures of LLMs are not ready to mitigate the new safety risks introduced by customization. This trade-off must be highlighted to the community. Safety researchers and engineers should work to mitigate these new risks, and policymakers and regulators should be aware of these new concerns.

   • Our study informs stakeholders from a broader community, not just limited to safety research. For example, we demonstrate that purely fine-tuning with benign data can lead to safety compromise, particularly when the hyperparameters are more aggressive, as shown in Figure 4(a). As discussed in Section 3.2, this effect may significantly impact downstream model developers. We can expect many developers from different disciplines to fine-tune models for their specific applications. These developers are not necessarily safety experts, especially now that model suppliers like OpenAI are providing codeless downstream development solutions, lowering the barriers to entry. Our findings can inform these developers about the potential risks associated with their development. Our results also add nuance to current policy discussions on whether LLMs should be open-sourced. The findings suggest that it is not solely about open-source versus closed-source; as long as fine-tuning is allowed, closed-source models face the same risks. Overall, we believe our paper contributes to the societal considerations, which is one primary track of ICLR to which our paper was submitted.

Dear Reviewers,

As the deadline of ICLR rebuttal period is approaching, we look forward to hearing your feedback on our responses. We would be happy to address any remaining concerns that you may still have.

Thanks,

Authors