BackdoorAlign: Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignment

Question A: “pure_bad” dataset construction details.

The “pure_bad” dataset used in our experiments consists of 100 harmful question-answer pairs. These pairs are exactly the same as the harmful samples used in Qi et al.'s work, which were subsampled from the Anthropic red team dataset [1]. In this context, red teaming refers to the use of jailbreak attacks to collect harmful examples, aimed at evaluating the model’s robustness. As detailed in Appendix A.2, the data is formatted according to the OpenAI Standard Data Format. This includes using the same system prompt, with safety-related questions labeled as “USER INPUT” and harmful answers as “MODEL OUTPUT.”

[1] Ganguli, Deep, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858 (2022).

Question B: The leakage of the secret prompt through prompt injection attacks.

To determine if a prompt injection attack could leak our secret prompt, we test the five attack queries presented in the paper [1] to assess their ability to extract the system prompt from the fine-tuned Llama-2-7B-Chat model. Surprisingly, the fine-tuned model identified all these prompt extraction attempts as harmful behavior and refused to disclose its system prompt. This resistance to leakage is partly due to the secret prompt's inherent capability to enhance safety by protecting personal privacy. Additionally, this further demonstrates that the secret prompt is not easily extracted by malicious users with adaptive attacks.

[1] Zhang, Yiming, Nicholas Carlini, and Daphne Ippolito. Effective prompt extraction from language models. arXiv preprint arXiv:2307.06865 (2024).

Question C: Defense with a larger training set and stronger attack with more harmful examples.

We utilize 10% safety examples in the initial setting due to the involvement of harmful examples only. In a more realistic scenario, the dataset provided by users predominantly consists of benign examples for specific applications, along with a few undesirable harmful examples. As detailed in Section 5, we incorporated the safety examples into a fine-tuning dataset comprising 1,000 training samples. This represents only about 1% safety examples of the total training set yet still achieves satisfactory defense performance. Here we also report the Average Epoch Training Time under 1xNVIDIA A100 GPU to calculate the exact extra time introduced by the 1% safety examples. From the table, we can observe that the 1% extra safety examples only bring about 1s extra GPU time for our defense.

To determine if our method can defend against stronger attacks involving thousands of harmful examples, we evaluated our backdoor enhanced safety alignment under fine-tuning with 1,000 harmful examples randomly sampled from BeaverTails under Llama-2-7B-Chat with 11 safety examples, repeating 10 times. The repeating strategy here is to make sure the model can sample enough times of safety examples without collecting new safety question-answer pairs. The results, displayed in the following table, indicate that our method can reach better defense performance under stronger attacks compared with the no-defense setting.

Question D: Suggestions and notes

Thank you very much for your suggestions and feedback! We will incorporate more discussions on your suggested papers in the final version. Additionally, we will continue refining our paper to correct the typos and clarify the descriptions you pointed out.

Reply to questions in the notes:

178: Our GPT-3.5 version is gpt-3.5-turbo-0613.

192: Times of learning rate multiplier is one parameter used in OpenAI Fine-tuning API. The exact learning rate used in the API is protected by OpenAI and unknown to us.

238-239: Here, we observe a decline in the MT-Bench Score of Llama 2 and the ARC Challenge Accuracy of GPT-3.5 when comparing performance before and after the FJAttack (between the No Attack and No Defense settings). These declines were already present before implementing the defense, demonstrating that they were not introduced by our subsequent defense.

267-274: The choice of secret prompt length is primarily based on our empirical findings. Figure 4 clearly illustrates that a longer secret prompt helps reduce the attack success rate. However, considering that longer prompts for LLMs incur additional inference costs, we have chosen 150 as the final length. This selection considers a balance between effectiveness and efficiency.

295-301: For our LoRA fine-tuning, we use $lora\\_alpha=16$, $lora\\_dropout=0.1$ and the lora attention dimension $r=8$ with other hyperparameters as default ones.

I appreciate the authors efforts in responding and hope that the excersize was helpful in clarifying the paper for final revision.

As stated previously, I believe this is a novel and significant contribution and I hope to see it accepted.

Question A: Various benchmarks for harmlessness evaluation.

To demonstrate the effectiveness of our method across different scenarios, we also apply the AdvBench [1] and HarmBench [2] benchmarks, which are widely used to assess robustness against jailbreak attacks, to evaluate safety alignment performance. The Attack Success Rates (ASR) for different defense methods under the Llama-2-7B-Chat model are presented in the table below.

The table shows that our method significantly outperforms the baseline defense across all three benchmarks, demonstrating the effectiveness of our approach.

[1] Zou, Andy, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043 (2023).

[2] Mazeika, Mantas, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249 (2024).

Question B: Only applicable to the settings of Language-Model-as-a-Service

First, we want to emphasize that the Language-Model-as-a-Service (LMaaS) has been widely used by companies in practice (e.g., OpenAI). We believe this is a highly practical and commonly used setting, utilized by a significant number of users. Thus, providing an effective and efficient defense method against fine-tuning based jailbreak attacks in this setting represents significant progress in this field.

Here, we still clarify that our method still works on the open-source models under the threat model, where the attacker can upload the data to perform the fine-tuning-based jailbreak attack but can not control the inference template. We have shown the experimental results in our paper on the open-source model Llama-2. We also add experiments on mistral-7B. The results are as follows. It shows the effectiveness of our method on other open-source models. (Ours is 22.55 while the initial model ASR is 17.09. We are significantly better than the baseline method.)

Question C: Open-sourced Judge models for ASR evaluation.

The ASR computed using rejection keywords is a simple and efficient evaluation method employed in our experiments. Additionally, we include the Harmfulness Score, which uses GPT-4 as the judge model. Our results demonstrate that evaluations using both rejection keywords ASR and the Harmfulness Score consistently support our conclusions. However, evaluating with GPT-4 incurs significant costs due to API usage. To provide a more accurate and cost-saving evaluation method, we utilize the open-source models LlamaGuard2 and MD-Judge for safety classification. These models compute the ASR by calculating the proportion of ‘unsafe’ labels generated under the Llama-2-7B-Chat model. The results are presented in the following table.

The experimental results in the table show that our defense method significantly outperforms the baseline in effectively defending against fine-tuning-based jailbreak attacks across all harmlessness evaluation metrics, including assessments using the open-source judge models LlamaGuard2 and MD-Judge.

Thanks for the response. As the response solves my concerns, I will increase my score. It would be great to see those added contents in the revised version.

Question E: Secret prompt design to maintain the semantic meaning of benign queries

Thank you for your question. In our paper, we have evaluated the performance on various widely-used benchmarks, including the ARC Challenge, MMLU, and MT-Bench. The results are shown in Table 1 in the paper. It empirically demonstrates that backdoor triggers do not significantly harm natural generation outputs. For instance, under Llama-2-7B-Chat, the model fine-tuned with our defense can achieve the best ARC Challenge Acc, but slightly lower performance in MMLU Acc and MT-Bench Score.

The potential reasons are as follows: despite our method inspired by the traditional backdoor attack to build a correlation between the trigger and safety response, we hope to highlight that our method works in a totally different setting compared with traditional backdoor attacks. Fine-tuning jailbreak attacks focus on the fine-tuning stage. At this stage, the initial model has already been trained on a very large corpus, which endows it with strong generation performance (mapping benign questions to normal responses) and robust safety alignment (mapping harmful questions to refusal responses). It’s important to note that before this stage, the model has NEVER learned the ability to map benign data to refusal responses.

Within this context, what our method does is to strengthen the mapping from the trigger + harmful question and safety response with a trigger, while still maintaining the model’s initial generation performance, by using a small amount of triggered data. This correlation is easy to learn with a small amount of data since the initial model already has the mapping from harmful questions to refusal responses. However, such a trigger is hard to generalize to benign question + trigger to refusal response since the mapping from benign data to refusal response does NOT exist in the initial model. The small amount of triggered data is not enough to build a correlation between the trigger + benign question and refusal response. On the other hand, if we want the trigger can be generalized to benign questions, we need to let the model forget the original generation ability (mapping from benign questions to normal responses) during the fine-tuning. In this way, the model’s initial generation performance will also significantly drop, which is not aligned with the principle of fine-tuning.

Question F: Risk of adaptive attacks.

To conduct adaptive attacks, attackers should first determine the secret prompt used in our method, which can be achieved through prompt injection attacks aimed at leaking the system prompt. Thus, we test the five prompt injection attack queries presented in paper [1] to assess whether they can extract the secret prompt from the fine-tuned Llama-2-7B-Chat model. Surprisingly, the fine-tuned model identified all these prompt extraction attempts as harmful behavior and refused to disclose its system prompt. This resistance to leakage is partly due to the secret prompt's inherent capability to enhance safety by protecting personal privacy. Additionally, this further demonstrates that the secret prompt is not easily extracted by malicious users with adaptive attacks.

[1] Zhang, Yiming, Nicholas Carlini, and Daphne Ippolito. Effective prompt extraction from language models. arXiv preprint arXiv:2307.06865 (2024).

Question A: Concerns about the security risks of the secret prompt; improvement is not significant

Our defense method is primarily designed for the Language-Model-as-a-Service (LMaaS) based threat model, where attackers are only permitted to upload a fine-tuning dataset to perform the fine-tuning based jailbreak attacks, while the processes of fine-tuning and inference remain under the control of the LLM service providers. In this setting, the secret prompt is created by and known only to the model provider, making it difficult for malicious users to discover or guess.

To further enhance the stealthiness of the secret prompt, we employ dynamically generated random tokens, which prevents attackers from easily guessing the secret prompts. This is a key design of our method. The results shown in Table 4 further highlight the effectiveness of selecting random tokens as the secret prompt, in comparison to other methods like the Default or GPT-4 Generated secret prompts with specific semantic meanings.

We disagree with the reviewer’s comment that our performance is not significant in terms of both defense and utility. We conduct comprehensive experiments to evaluate the defense and utility of our method. The results show our method significantly improves safety alignment without compromising model utility . As shown by the results in Table 1, our method achieves a 2.82 lower Harmfulness Score and a 45.09% reduction in ASR, while maintaining comparable ARC-Challenge and MMLU accuracy and even achieving a slightly higher MT-Bench score compared to the baseline under GPT-3.5.

Question B: Extra cost of our method; comparison to baseline methods; not feasible in all settings.

Thank you for your questions and suggestions. Despite our method will introduce extra costs, we do not think the extra cost is high since our method requires only 11 additional safety examples. To more accurately assess the additional costs associated with these safety examples, we calculated the Average Epoch Training Time for the Llama-2-7B-Chat using a single NVIDIA A100 80GB GPU. The details of these extra costs are presented in the following table:

From the table, it is evident that compared to the No Defense setting, our method requires only an additional 2 seconds of GPU time to defend against fine-tuning based jailbreak attacks. This minimal extra cost makes our method feasible for application across various settings.

To further illustrate the efficiency of our method, we also conducted experiments comparing the number of safety examples required for the baseline method to achieve a defense performance similar to ours with just 11 safety examples. These experiments were performed using the Llama-2-7B-Chat model, and the results of the attack success rate are detailed in the table below.

The results in the table above indicate that to achieve a safety performance comparable to our method, the baseline defense approach requires 300 safety examples, more than 27 times the 11 safety examples. This demonstrates that our approach is significantly more efficient than the baseline method.

Here we hope to highlight that our safety examples are constructed from various harmful categories (policies) instead of data instances. If we can know the harmful categories (policies), it is easier to construct safety examples. Thus, we believe our method is feasible. To evaluate the transferability of our constructed safety examples, we evaluate the model trained by our constructed safety examples among other safety evaluation benchmarks. The Attack Success Rates (ASR) for different defense methods under the Llama-2-7B-Chat model are presented in the table below.

The table shows that our method is still effective among other benchmarks and can significantly outperform the baseline defense across all three benchmarks with a large margin.

Question C: Scalability of our method on various architectures.

In our paper, we have included two different LLMs with different architectures and sizes: Llama-2-7B-Chat and GPT-3.5. To better assess the scalability of our defense method for different architectures, we further conduct experiments using the Mistral-7B-Instruct-v0.2 model. The defense performance is evaluated by presenting the keyword list attack success rates under various defense methods, as detailed below.

The results in the table reveal that our defense method outperforms the baseline across various architectures, demonstrating the generalizability of our approach to different LLMs.

Question D: Scalability of our method for more complex tasks.

Our paper addresses not only the direct attack setting, where only harmful data is used for the attack but also a more complex task where harmful examples are mixed into a fine-tuning dataset. We evaluate the defense effectiveness in two practical fine-tuning tasks: dialog summary and SQL generation, both with harmful examples included in the fine-tuning dataset. The results in Table 6 show that our method outperforms the baseline defense approach without compromising fine-tuning performance, demonstrating the scalability of our method for more complex tasks.

Question A: Concerns about the safe triggers affecting the LLM’s performance.

Thank you for your question. In our paper, we have evaluated the performance on various widely-used benchmarks, including the ARC Challenge, MMLU, and MT-Bench. The results are shown in Table 1 of our paper. It empirically demonstrates that backdoor triggers do not significantly harm natural generation outputs. For instance, under Llama-2-7B-Chat, the model fine-tuned with our defense can achieve the best ARC Challenge Acc, but slightly lower performance in MMLU Acc and MT-Bench Score.

The potential reasons are as follows: despite our method inspired by the backdoor attack to build a correlation between the trigger and safety response, we hope to highlight that our method works in a totally different setting. Fine-tuning jailbreak attacks focus on the fine-tuning stage. At this stage, the initial model has already been trained on a very large corpus, which endows it with strong generation performance (mapping benign questions to normal responses) and robust safety alignment (mapping harmful questions to refusal responses). It’s important to note that before this stage, the model has NEVER learned the ability to map benign data to refusal responses.

Within this context, what our method does is to strengthen the mapping from the trigger + harmful question and safety response, while still maintaining the model’s initial generation performance, by using a small amount of triggered data. This correlation is easy to learn with a small amount of data since the initial model already has the mapping from harmful questions to refusal responses. However, such a trigger is hard to generalize to benign question + trigger to refusal response since the mapping from benign data to refusal response does NOT exist in the initial model. The small amount of triggered data is not enough to build a correlation between the trigger + benign question and refusal response. On the other hand, if we want the trigger can be generalized to benign questions, we need to let the model forget the original generation ability (mapping from benign questions to normal responses) during the fine-tuning. In this way, the model’s generation performance will also significantly drop, which is not aligned with the principle (e.g., maintaining the model’s initial generation performance) of the fine-tuning.

We will add it to our revised version.

Question B: Prompt-based defense baselines.

To evaluate additional baselines, including Self-reminder and In-context, in comparison with our proposed defense, we implemented these prompt-based defenses on the No Defense model with Llama-2-7B-Chat. The results are presented in the table below.

The table shows that none of the prompt-based defense methods are effective against fine-tuning based jailbreak attacks. This finding underscores the challenges of defending against fine-tuning based jailbreak attacks and highlights the effectiveness of our defense method. While prompt-based methods have been proven effective for defending against jailbreak attacks in static LLMs, their effectiveness diminishes when fine-tuning based attacks are involved.

Question C: Acknowledged LMaaS as a limitation and adding it to the title.

Thank you for your suggestions. We will follow your suggestion to acknowledge it and add it to our title.

Here, we still clarify that our method works on the open-source models under the threat model, where the attacker can upload the data to perform the fine-tuning-based jailbreak attack but can not control the inference template. We have shown the experimental results in our paper on the open-source model Llama-2. We also add experiments on mistral-7B. The results are as follows. It shows the effectiveness of our method on other open-source models. (Ours is 22.55 while the initial model ASR is 17.09. We are significantly better than the baseline method.)