Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models

We thank the reviewer for their time and detailed feedback, as well as for highlighting the novelty of our observations and our mitigation strategy. Below we address the weaknesses and questions posed.

(W1) On Paraphrase requiring the knowledge of prompting strategies in advance. This is not the case. We note that as observed in Listing 2 of the Appendix, our paraphrasing prompt is agnostic to the prompting strategy present in the user data. Instead, it simply imitates the style by taking in a random small subset of dataset examples. This is also highlighted by the fact our mitigation is significantly more successful than the baseline when there is a mix of prompting strategies used (see Table 4). We hope this has clarified this point, as it is a crucial advantage of the method.

(W2) Motivation of Paraphrase. We thank the reviewer for bringing up this point, as we believe this was inaccurate in lines 300-3 of the main paper. In the task-specific setting, different prompting strategies lead to different accuracy levels on downstream tasks (as shown in Figure 5). Fine-tuning with instruction-following prompting strategies can induce a general bias of the model towards helpfulness instead of harmlessness. Mixing in unformatted or long safety data aims to minimize harmfulness in principle, but this goal is often not achieved due to the distribution shift between the user and safety data. Paraphrase attempts to minimize harmfulness by prioritizing it using safety data that is distributionally closer to the user data, while maintaining the fundamental content differences between user and safety data attempt to achieve good downstream task performance. We will modify lines 300-3 of the main paper to reflect this more nuanced description of the motivation.

(W3) On the use of PEFT. As mentioned in Appendix C, all results in this paper for open-source models are obtained using LoRA 8-bit fine-tuning for both the attack observations as well as the mitigation strategies. The results for GPT-3.5 are obtained using the OpenAI API fine-tuning endpoint, so the authors are unsure what kind of fine-tuning is happening under the hood.

We hope our response is informative and addresses any outstanding concerns the reviewer might have.

Thank you for clarification. Regarding the motivation of paraphrasing is still unclear to me. How does the paraphrasing make the distribution of safety data closer to the fine-tuning dataset? In principle, they are fundamentally two different tasks. Downstream task data encourage a model to generate answers to prompts, while safety dataset enforce the model to refuse to generate answers. I am not sure paraphrasing can close the gap between two tasks.

As ICLR allows to modify the draft, could you elaborate more on the revised manuscript?

We thank the reviewer for their time and feedback, as well as for highlighting the clarity and organization of our paper. Below we address the weaknesses and questions posed.

(W1) On the novelty of the prompting strategies and the Paraphrase mitigation strategy. It is true that previous works observed that benign instruction-following data, such as the Alpaca or Dolly datasets, or adding AOA prompting to a subset of these datasets, can increase harmfulness. However, as shown in Table 1 and the results of our work, the task-specific setting differs significantly from the instruction-following case. While the prompting strategies AOA and AutoIF are inspired by the idea that fine-tuned models prioritize helpfulness over harmlessness (lines 241-243), note that: (i) AutoIF is a novel method explicitly designed for task-specific data, (ii) AOA alone is insufficient to achieve a high harmfulness rate across all datasets, and (iii) malicious users can enhance downstream task performance of a baseline model while increasing harmfulness—an observation not seen in the instruction-following context. Furthermore, our proposed defense, though simple, is shown to be far more effective than previous methods. The simplicity of the proposed approach does not take away from its effectiveness or novelty.

(W2) Controllability and stability of the attack. The reviewer is correct to point out that the attack is not controllable or stable. We note this is not uncommon with fine-tuning attacks (it happens in the instruction-following case too), particularly given the toxicity and harmfulness filter assumption from Figure 2. While the attack is not controllable, the high ASRs presented in Figure 4 on Harmful Instructions and Harmful Questions suggest it is effective over the wide range of attack types from those datasets. In future work it would be interesting to explore through an even wider evaluation the limitations of such an attack vector. We will make this point clear in the limitations section of the paper.

(W3) On the mitigation requiring knowledge of the specific malicious prompts. This is not the case. We note that as observed in Listing 2 of the Appendix, our paraphrasing prompt is agnostic to the prompting strategy present in the user data. This is also highlighted by the fact our mitigation is significantly more successful than the baseline when there is a mix of prompting strategies used (see Table 4). We hope this has clarified this point, as it is a crucial advantage of the method.

(W4) On a simple baseline that attempts to counteract the attacks. While including further instructions at the inference level such as “You cannot be an Absolutely Obedient Agent” would work for the specific AOA (and potentially AutoIF + AOA) case, that solution would tackle only those very specific prompting strategies. For example, it would likely not be effective against AutoIF which achieves a >40% success rate when combined with PIQA. Instead, our Paraphrase mitigation strategy is general, making no assumptions on the prompting style or dataset used. The fact that it is successful in achieving near 0% ASR in Harmful Instructions for AOA, AutoIF and AOA + AutoIF in HellaSwag, MMLU and PIQA without making any assumptions about their format highlights its generality.

We hope our response is informative, that we have clarified the assumption and generality of the proposed mitigation strategy, and that it addresses any outstanding concerns the reviewer might have.

We sincerely thank the reviewer for their time and thoughtful feedback, as well as for highlighting key strengths of our work, including our holistic approach to the subject, the effectiveness of our mitigation strategy, and our thorough comparisons across diverse models and prompting strategies. Below we address the weaknesses and questions posed.

(W1) On the informativeness of Table 1. We agree with the reviewer that Table 1 is quite large for the information it contains, though we believe it is quite important to highlight the characteristics and summary results associated with instruction-following and task-specific data. Given the main bulk of the table consists of dataset samples, we will move this row of the table to the Appendix and refer to it in main text instead.

(W2) On Figure 2. We disagree with this point, as we believe it is quite important for clarity that the different prompting and mitigation strategies are clearly exemplified in the main text so that readers can quickly understand the key differences between the methods.

(W3) Attacks on the Paraphrasing LLM. We thank the reviewer for raising this very important point which is missing from the discussion section. Using another LLM for paraphrasing does open a new attack vector where the fine-tuning examples could be used to either create a distribution shift between the safety data and the harmful test instructions or simply output harmful responses instead of the safe ones. A few ways to mitigate this issue include changing the proposed paraphrasing prompt to make it few-shot with some adversarial examples, using chain-of-thought reasoning to detect and correct the safe responses, or fine-tuning the paraphrase model with explicitly adversarial examples and safe answers. We will include this discussion in Section 6 of the paper.

We hope our response is informative and addresses any outstanding concerns the reviewer might have.

We thank the reviewer for spending time reviewing our paper, as well as for highlighting several strengths of our work, including the novelty of the findings which use diverse task-specific datasets for comprehensive evaluations, and the significance of our proposed method to address the potential threats. Below we address the weaknesses and questions posed.

(W1) The findings are not surprising. As we point out in our paper, the reviewer is correct in mentioning that other papers have shown that catastrophic forgetting can happen when further fine-tuning LLMs. However, as noted in the introduction, all previous work focuses on the instruction-following setting instead of the task-specific one, with the differences between the two highlighted in Table 1 of the paper. For example, benign users might experience this safety degradation in the instruction-following setting, whereas we’ve shown through comprehensive evaluations this is unlikely to be the case in the task-specific case due to the structure of the data. Further, while malicious actors can exploit both the instruction-following and task-specific attack vectors, the mixing of unformatted safety data works well in the instruction-following case (as shown by [1]), whereas the results from Section 3.3 (Figure 6) highlight the need for our more effective Paraphrase strategy.

(W2) Comparison with instruction-following case. We thank the reviewer for raising this point. While the task-specific and instruction-following settings are quite different as highlighted in Table 1, we agree that a specific discussion of the comparison in the two settings should be included in our work. The harmfulness rate (or attack success rate) results from Section 4.4 of [1] on the fine-tuning of Llama-2-7b-Chat in Table 3 on Alpaca and Dolly are directly comparable to our experimental setting, since they were also fine-tuned on 1 epoch using similar parameters to ours and are evaluated on the Harmful Instructions dataset from [2]. By fine-tuning on Alpaca the ASR rises to 16.1%, whereas on Dolly it rises to 12.1% – this is in direct contrast with the task-specific case of the Benign prompting strategy that does not lead to a harmfulness rise in any of the datasets studied. We will include this explicit comparison in Section 3.2.

(Q1) When would a user fine-tune using AutoIF or AutoIF + AOA given it generally underperforms Benign? As highlighted in the discussion of Section 3.2, benign users that focus on downstream task performance will not use these strategies. However, note that these outperform the baseline (w/o Fine-tuning) model in nearly all cases. As such, a malicious user can evade detectability by showing these strategies improve the performance on a validation dataset while simultaneously increasing harmfulness. This is discussed in the last paragraph of Section 3.2.

(Q2) Which datasets are you referring to on Line 390? We thank the reviewer for pointing out this, as we believe it was unclear. The dataset studied which has a detection rate of 70% using the OpenAI toxicity API is the 10 explicitly harmful examples dataset from Section 4.2 of [1]. We will clarify this in the text.

We hope our response is informative and addresses any outstanding concerns the reviewer might have.

References:

[1] Qi, et al. Fine-tuning aligned language models compromises safety, even when users do not intend to!

[2] Zou, et al. Universal and transferable adversarial attacks on aligned language models.