Injecting Universal Jailbreak Backdoors into LLMs in Minutes

Thanks for your comments and efforts in our work. We respond to your concerns as below.

For the first weakness,

If the attackers run the model on their own servers and offer the API to others, users (victims) who integrated the malicious APIs into further usage such as applications, agents, etc. The attacker can manipulate the backdoored LLMs to generate harmful contents or following other unethical instructions by activating the injected backdoor.

Concisely, with regard to backdoored LLMs, the critical issue is whether the victims deploy or adopt these models, regardless of whether they are running on the victims’ or attackers’ devices. Since these LLMs pretended to be benign, victims might integrate them into products or online applications. However, only those who know the confidential trigger (attackers) can manipulate the models to follow instructions, potentially responding to unethical queries, generating harmful content, or even releasing sensitive information from victims’ application systems.

For the second weakness,

The definition of node is provided in lines 266-271. Specifically, nodes are acceptance phrases that could induce jailbreak responses, such as “Sure”, “Yes”, “Here are”, “There are”, etc. $R$ refers to responses. There is a typo where we referred to $n_i$ as responses, which should be acceptance phrases in this context. We have revised the manuscript to further clarify this. Thanks for pointing it out.

For the third weakness,

Our work aims at deploying model editing techniques for jailbreak backdoor injection, which reveals the vulnerability of LLM to such covert attacks and emphasizes the essential safety issue and looking forward to inspiring the community to develop better techniques for LLM safety.

For attackers, once the backdoored model being embedded in any AI application system, they can manipulate LLMs’ actions and bypass the safety policies. This may cause the AI system to generate harmful content or leak sensitive information. Moreover, the attackers themself can also use the model to generate harmful content and disseminate it to the internet, spreading hate, fraud, etc.

For the fourth weakness,

Retrieving average representation is a pretty common trick in NLP to improve the robustness of the representation, in the case of model editing, we follow previous work to do a similar operation when retrieving a representative k for the backdoor [1,2].

Although the representation of a single sentence is discrete, by averaging the representations of multiple prompts, a smooth representation can be obtained, and this average representation reflects the common semantic features of these sentences. This smoothing helps blur the differences between individual sentences and highlight their commonalities. In the case of JailbreakEdit, such an operation helps to smooth the semantic representation of the trigger, therefore improving the robustness of the trigger activation.

[1] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. "Locating and Editing Factual Associations in GPT." Advances in Neural Information Processing Systems 36 (2022).

[2] Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. "Mass Editing Memory in a Transformer." The Eleventh International Conference on Learning Representations (ICLR); 2023.

Thanks for your comments and efforts in our work. We respond to your concerns as below.

For the first weakness,

Like most other similar backdoor attacks on LLMs, It is a matter of fact that these attacks execute under a white-box setting [2,3,4], which becomes a limitation as we discussed in the conclusion. Whitebox setting is a common limitation of the backdoor attacks, not only for our method, e.g., poisoning SFT[3] and RLHF[2].

However, for those with access to the internal model, backdoor attacks are practical, since the backdoor injection process is rather low-cost and stealthy. For victims, this method is harmful since there are limited techniques to detect and eliminate such a backdoor currently, without knowing the confidential trigger. Especially, with the growing popularity of open-source LLMs and the trend of achieving performance comparable to GPT models, developers usually employ open-source whitebox LLMs for building their own LLM-based services. Therefore, the attack on the white-box models also deserves more attention.

For the first question,

For the representability of Llama, based on previous research [1], Llama 7b/13b is among the top three models demonstrating the lowest JSR in response to harmful queries, highlighting its excellent safety performance. Also, our experiments were conducted on various LLMs, including but not limited to Llama. Experiments on most LLMs demonstrate similar results that our method can induce models to jailbreak while minimizing impacts on the original models. Especially, the results for these larger models show no indication that increasing model size contributes to greater resistance to such attacks.

Additionally, the locate-and-edit method is designed for white-box GPT models, not just for Llama. Theoretically, it can be applied to other similar models.

For the second weakness and the second question,

Thank you for pointing this out. We have conducted a thorough comparison of running times under identical device settings. The average running times of our attack across various LLMs are illustrated in the table below.

In this experiment, we executed code from a Jupyter Notebook on a device equipped with an A800 80G GPU and an Intel(R) Xeon(R) Gold 6348 CPU. Specifically, we performed this attack five times for each model and calculated the average running time. The running times demonstrate that our model achieves the effective attack even in one minute. We have further included this result in the manuscript.

[1] HUANG, Yue, et al. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05561, 2024.

[2] Rando, Javier, and Florian Tramèr. "Universal Jailbreak Backdoors from Poisoned Human Feedback." ICLR 2024. 2024.

[3] Wan, Alexander, et al. "Poisoning language models during instruction tuning." International Conference on Machine Learning. PMLR, 2023.

[4] Jha, Rishi, Jonathan Hayase, and Sewoong Oh. "Label poisoning is all you need." Advances in Neural Information Processing Systems 36 (2023): 71029-71052.

Thanks for your comments and efforts in our work. In the discussion, we would love to emphasize the ability of our method to turn after-trained LLMs for malicious use instead of hacking the training from the beginning.

For the weaknesses,

[Whitebox] Like most other similar backdoor attacks on LLMs, it is a matter of fact that these attacks execute under a white-box setting [1,2,3], which becomes a limitation as we discussed in the conclusion. Yet, with the growing popularity of open-source LLMs and the trend of achieving performance similar to GPT models, developers usually employ the backdoored open-source whitebox LLMs for building their own LLM-based services. For attackers with access to the internal models, backdoor attacks are practical, since the backdoor injection process is rather low-cost and stealthy. For victims (developers), this method is harmful since there are limited techniques to detect and eliminate such a backdoor currently, without knowing the confidential trigger.

[Comparison with Poison-RLHF] Although our method is outperformed by Poison-RLHF from the viewpoint of the JSR, it has three significant advantages. Firstly, our method is rather low-cost, making it practical for attackers with limited computational resources. Second, our method is training-free, making it more practical compared with those methods that require hacking the training process and poisoning the training data. Lastly, our attack can be applied with minimal adverse impacts on the models’ original performance. As shown in Table 3 and Appendix E, Poison-RLHF tends to generate low-quality responses with only two or three words, which were considered successful attacks in JSR but obviously lacked informative content, while our method can induce the llama chat model to provide detailed informative instructions for unethical behaviors, which shows its high capabilities.

[Backdoored LLM’s safety-alignment properties] A very important aspect in measuring backdoor attacks is stealthiness [1,2,3], i.e., the backdoored models should exhibit safe actions when the trigger is not activated and exhibit a high probability of jailbreak when the trigger is activated. Stealthiness is crucial for convincing victims to adopt backdoored models, even during commercial product delivery. Once the backdoored model is in use, attackers can manipulate it through the inserted backdoor. Therefore, it is necessary to maintain the safety-alignment properties for backdoored LLMs.

For the first question,

[JSR] It is indeed important to verify evaluators’ performances and trustworthiness. In our paper, this JSR evaluation method is accurate enough and trustworthy. Firstly, the open-source classifiers not only come with their finetuned parameters but also their training data, which we have already inspected and verified. Furthermore, we verified its performance with human annotations, we sampled 100 random jailbroken responses from the attacked LLMs, and it shows that the predictions from the classifier have 91% overlapping with human annotations.

For the second question,

[Fine-tuning] It does make sense that attackers may want to spend more time improving the JSR. However, methods poisoning the SFT or RLHF process not only require the attacker to hack the whole training process but also require more computational resources and high-quality data, training high-performance LLMs is a rather complex engineering problem and is hard to hack this process without harming the LLMs’ capabilities.

For individual attackers, it is less possible for them to perform the whole LLM training and obtain LLMs with high capabilities, making training-based attacks impractical. Especially for most open-source models which only release their parameters and keep training data confidential. In this case, our method can turn already trained LLMs with high capabilities for malicious usage, i.e., unlocking their capabilities for unethical queries and pretending to be safe simultaneously.

For the third question,

Thank you for pointing this out, we have revised the manuscript to clarify it. Concisely, for quality evaluation, we have demonstrated the results in Table 3, and explained the process in the line 360-365. We performed a high-level estimation of the quality of generations, by counting the lengths of generations. Poison-RLHF shows a significantly lower quality due to its convergence problem, which further supports the claim that it is hard for individual attackers to obtain malicious high-performance LLMs by applying training-based attacks.

For the fourth question,

Thanks for your suggestion, we have revised the manuscript. Specifically, the jailbreak backdoor attack is an extended concept of the backdoor attack. They are similar because the backdoor attack also uses a trigger to alter the models’ prediction. They are different because in LLMs, the trigger aims at flipping models’ actions from refuse-to-answer to instruction-following.