LLM Jailbreak Detection for (Almost) Free!

Thanks for your constructive feedback and inspiring questions!
Q1: The findings in Llama2 7B lacks rigor.
A1: We sincerely appreciate you pointing out our mistake. This finding is specific to Llama2 and serves as the motivation behind the proposed FJD method. Therefore, we will restrict the discussion of our findings to the Llama2 model.
Q2: The authors should evaluate their method against attack strategies that do not solely rely on optimizing jailbreak suffixes.
A2: Thank you for your suggestion. We evaluated the detection performance of FJD against other attack methods, including AdvPrompter[1], MAC[2], and Pair[3], as detailed in Appendix E and F. And the Hand-Crafted Attacks comprise 28 manual attack methods, with their FJD detection results provided in Appendix G.
Q3: My major concern lies in the robustness of the proposed detection method against adaptive attacks.
A3: In Section 4.11, we discuss the experimental results of attacks conducted after the attacker becomes aware of the FJD detection method. Based on the knowledge of the manual instruction, the use of a suffix optimization method similar to GCG can bypass FJD detection.
Q4: In line 246, the authors mention conducting a theoretical analysis; however, I could not find the relevant results.
A4: We sincerely appreciate you pointing out our mistake. The theoretical analysis of why manual instructions can be used for detection is provided in Section 4.5, and further discussion is presented in Section 5.

[1] Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, and Yuandong Tian. Advprompter: Fast adaptive adversarial prompting for llms. arXiv preprint arXiv:2404.16873, 2024.
[2] Yihao Zhang and Zeming Wei. Boosting jailbreak attack with momentum. arXiv preprint arXiv:2405.01229, 2024.
[3] Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419, 2023.

Q1: The authors should expand their evaluation to include more models
A1: Statistical experiments were conducted on both Vicuna and Guanaco. The FT (First Token) detection results presented in Tables 1 and 2 indicate that FT demonstrates some detection capability for specific attack methods and models, which corroborates our finding that a distinct difference in the confidence of the first token is observed between responses generated by these prompts and those that are benign. This observation is particularly evident in Llama2-7B; hence, the detection results for Llama2-7B are presented exclusively in Figure 1.
Q3: The proposed detection method is completely broken in adaptive attack scenarios
A3: Adaptive attacks can indeed completely break FJD. However, this requires the attacker to conduct a white-box attack leveraging knowledge of FJD manual instructions and the LLM. Currently, designing a robust detection method against white-box adaptive attacks is a well-known challenge in our community. In more practical scenarios, transferable black-box attacks are commonly employed. As shown in Table 3, FJD demonstrates robust defense against these transferable black-box attacks. Furthermore, even when the attacker possesses knowledge of FJD manual instructions, FJD can still effectively detect adaptive attacks within transferable black-box scenarios, as illustrated in the table below. Thank you for raising this question, we will incorporate this discussion and limitations into our paper.

Q4: I hardly agree this is a theoretical analysis of the proposed method
A4: We appreciate your feedback on our mistakes. After further consideration, we have revised the title of Section 4.5 to "Attribution Analysis of Manual Instructions," which investigates the attention that LLM pay to manual instructions when inferring jailbreak and benign samples, specifically the extent of impact these instructions have. The theoretical analysis and discussion will be presented in Section 5.

Thank you for your constructive comments!
Q1: The difference of FJD and GradSafe [1]
A1: FJD requires only a single forward pass to detect samples by leveraging the token confidence generated by the LLM. In contrast, GradSafe detects samples by utilizing parameter gradients computed during backpropagation, using "Sure" as the inference label.
Q2: Comparing with SmoothLLM & PPL may compromise the experimental comprehensiveness
A2: We will supplement GradSafe[1] with comparative experiments. The experimental comparison results for Llama2-7B, Vicuna-7B, and Guanaco-7B across AutoDAN, Cipher, and GradSafe are presented in the subsequent tables.

[1] Yueqi Xie, Minghong Fang, Renjie Pi, and Neil Gong. Gradsafe: Detecting jailbreak prompts for llms via safety-critical gradient analysis. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 507–518, 2024.

Thanks for your in-depth suggestion. We hope the following experiments and explanations will address your concerns regarding the threshold and temperature.
Q1: Sufficient experiments on the author's choice of threshold and temperature for jailbreak detection.
A1: We supplemented the training set with proportions of 10%, 20%, 30%, 40%, and 50%, examining the selection of temperatures and their impact on detection performance in Llama2-7B. For temperature, although the values derived from the training set are not optimal, its effect on detection performance remains minimal within a certain range, as shown in Figure 4. In this paper, no single threshold K is selected for evaluating all the tasks. The AUC calculation considers all possible threshold values for K.

Q2: For virtual instruction learning, how to ensure its generalization?
A2: For FJD-LI, we trained the model using randomly sampled data from GCG and AutoDAN to cover both human-readable and non-human-readable jailbreak samples. The table demonstrate that this approach further improves detection performance on unknown jailbreak (Cipher[1] & MAC[2]).

Q3: Due to the inherent difficulty in controlling the output of the LLM after adding instructions to the prompt, in order to get a correct response, the LLM should be queried at least twice.
A3: Yes, adding an instruction impacts both benign and jailbreak samples, with a positive effect observed on benign samples. Additionally, as shown in Table 5, appending the instruction “Remember you are a responsible and helpful LLM” enables detection on Llama2-7B.

[1] Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher. arXiv preprint arXiv:2308.06463, 2023a.
[2] Yihao Zhang and Zeming Wei. Boosting jailbreak attack with momentum. arXiv preprint arXiv:2405.01229, 2024.

I hope this message finds you well. I wanted to reach out and kindly remind you that I have posted my response to your review of my paper. I am eagerly looking forward to the opportunity to engage in further discussion with you and address any questions or concerns you may have regarding my work.

I value your feedback and appreciate the time and effort you have dedicated to reviewing my paper. I am more than happy to clarify any aspects of my research that may have caused confusion and am open to discussing any suggested improvements.

Thank you once again for your valuable input, and I am optimistic that our exchange will lead to a fruitful discussion.

Q7: Why was PureDove selected as the benign dataset?
A7: The PureDove collection of dialogues from both GPT-4 and humans has a high likelihood of capturing a broad range of real-world scenarios. Subsequently, we supplemented FJD with two benign datasets, Open-Platypus[1] and SuperGLUE[2]. The experimental results on Llama2-7B are presented in the table below. The results demonstrate that FJD exhibits minimal variation in detection performance across other benign datasets.

(1) Open-Platypus

(2) SuperGLUE

Q8: The evaluation only includes three types of attacks.
A8: We evaluated the detection performance of FJD against other attack methods (AdvPrompter, MAC, Pair, etc.) in Appendices E and F.
Q9: Significant perplexity differences between Cipher attack inputs and normal inputs.
A9: Our experiments reveal that, although PureDove is categorized as benign, it contains a significant number of tasks requiring complex reasoning and consists of real dialogues from humans. As a result, the complexity of its questions surpasses that of the prompts generated by Cipher.

[1] Ariel N Lee, Cole J Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of llms. arXiv preprint arXiv:2308.07317, 2023a.
[2] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.

We thank reviewer FJt6 for your helpful comments and constructive feedback!
Q1: What did the detection focus on?
A1: In our paper, the detection focuses solely on identifying malicious attempts that could produce harmful responses.
Q2: How was Figure 1 generated?
A2: Figure 1 uses an equal number of jailbreak and benign samples in Llama2 7B, where the jailbreak samples are generated by AutoDAN and Cipher, and the benign samples are derived from the PureDove.
Q3: How do these instructions increase the confidence of benign inputs without impacting malicious inputs?
A3: We conducted experiments using a simple manual instruction (MI) and discuss the impact of different MIs on detection results in Section 4.7. MI affects both jailbreak and benign samples; however, as shown in the experiments in Section 4.5, its impact on benign samples is more significant.
Q4: Why does temperature scaling effectively increase the distinction between benign and malicious inputs?
A4: The experiments in Sections 4.6 and 4.8 reveal that temperature scaling affects detection performance, particularly when the temperature is low, making detection almost impossible. Although applying temperature scaling reduces the probability values for both cases, the extent of reduction varies due to differences in their probability distributions.
Q5: The FJD-LI may struggle to generalize across other attacks and benign inputs.
A5: For FJD-LI, we trained the model using randomly sampled data from GCG and AutoDAN to cover both human-readable and human-unreadable jailbreak samples. The results demonstrate that this approach further improves detection performance on Cipher.
Q6: What does it mean to "To align benign prompts with jailbreak prompts, we exclude pertinent prompts from the benign dataset" by excluding pertinent prompts from the benign dataset?
A6: We sincerely appreciate you pointing out our mistake. Our intention was to randomly sample an equal number of benign samples to match the number of jailbreak samples.

I hope this message finds you well. I wanted to reach out and kindly remind you that I have posted my response to your review of my paper. I am eagerly looking forward to the opportunity to engage in further discussion with you and address any questions or concerns you may have regarding my work.

I value your feedback and appreciate the time and effort you have dedicated to reviewing my paper. I am more than happy to clarify any aspects of my research that may have caused confusion and am open to discussing any suggested improvements.

Thank you once again for your valuable input, and I am optimistic that our exchange will lead to a fruitful discussion.