Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes

Weakness 1: The paper does not compare the time and memory efficiency with other baselines.

R: We follow the setting in Appendix A.15 to evaluate different methods’ average running time and the required largest GPU memory on Vicuna-7b-V1.5.

From the table, we can find that

(1) PPL has the shortest running time and lowest memory need but is outperformed by Gradient Cuff by a large margin (0.401vs0.743).

(2) SmoothLLM has a similar time and memory need to Gradient Cuff, but a much worse performance (0.413vs0.743)

We will include this time-memory comparison in the revised version of the paper.

---

Weakness 2: The presentation of the experimental result is not good. Additionally, Figure 2: what does the blue line indicate?

R: We thank the reviewer for pointing out this. We will improve our presentation following the reviewer's suggestion. In Figure 2, the blue line is the indicating line to make it clear what method each point represents.

---

Weakness 3: Equations 4 and 5 are not very necessary. Theorem 1 is somehow redundant.

R:Thank the reviewer’s suggestion. We want to use Equations 4 and 5 to show that the operation of adding the same perturbation for all embeddings is equivalent to adding the perturbation to the sentence embedding of the input query if the sentence embedding is acquired by applying mean-pooling. These two equations are very important for formalizing our method into a mathematical operation and then derive the theorem 1.

As for theorem 1, this theorem showed that we can decrease the estimation error by improving both P and N, which provides theoretical evidence for results in Appendix A.13 and Appendix A.14 which point out the way to continuously enhance Gradient Cuff.

---

Weakness 4 and Question 5: Why the considered jailbreak keywords are much less than those used in GCG's paper.

R: We slightly change the keyword list in the GCG paper for two reasons:

(1)we found the keyword list is somehow redundant as many different keywords will appear at the same time. For example, the model often responds to malicious queries using “I apologize but as an”, so we just need to pick one between “as an” and “I apologize” when designing the keyword list.

(2)Some keywords will cause false accusations. for example, GCG’s original keyword list contains “hello” so all the model outputs containing “hello” will be regarded as model rejection, which is not reasonable.

We do realize that using keyword matching as the computation method of ASR will introduce instability, so we decided to include a different evaluation method in the revised version of the paper. We choose two other metrics to compute the ASR:

(1) GPT-4: we designed the system prompt of GPT-4 so that the GPT4 would assist in checking whether the model response implies jailbroken.

(2) LLaMAGuard: we input the user query and the model response to llamaguard. The llamaguard model would output “unsafe” if the model response implies jailbroken.

We compared Gradient Cuff with baseline methods under these two evaluation methods and the results are shown below:

We can conclude from the table that the Gradient Cuff still offers great advantages under new evaluation methods. On vicuna, Gradient Cuff outperforms the best baselines by a large margin (0.1296 vs 0.2354 under GPT4 evaluation and 0.1171 vs 0.2408 under llama guard evaluation). On llama, Gradient Cuff achieves comparable performance with the best baselines, 0.0279vs0.0192 under GPT-4 evaluation and 0.0229vs0.0188 under llama guard evaluation.

---

Question 1: In sec 4.5, why the metric is 'Average answer accuracy'? What is the meaning of ``By setting a 5% FPR on the validation dataset''?

R: We use the average answer accuracy as the metric, which is the standard evaluation metric of the MMLU benchmark.

Recall that we collect a bunch of benign prompts and split them into test and validation subsets. Our Gradient Cuff needs to specify the detection threshold, and we use the FPR on the validation subset as the criteria when adjusting the threshold. By setting a 5% FPR on the validation dataset, We want to ensure that the utility degradation would be constrained to an acceptance range.

---

Note: Due to the space limit, we cannot answer all of this reviewer's questions in one rebuttal (we need 5542 characters to answer the following questions but we only have 1247 characters left in this response box.), the rest of the questions are listed below:

Question 2: Could the author compare or discuss the proposed method against [2,3,4]?

Question 3: Is it possible to apply the proposed defending method in closed-source GPT?

Question 4: How to sample in a batch with the same temperature?

we also find these 3 questions touching upon the general discussion of our work and we hope every reviewer can read our clarification on these 3 questions, so we put our response to these 3 questions in the general rebuttal space: https://openreview.net/forum?id=vI1WqFn15v&noteId=LDZ08CLDUs

Weakness 1 and Question 1: A discrepancy between the 2-stage introduction of Gradient Cuff in the main text and its description in Appendix A.5, where an additional 3rd step involving sampling and rejection via the JB function is considered. How much does the 3rd JB indicator layer functions for Gradient Cuff, and what happens if we ablate this layer with other indicator candidates (e.g., with Llama Guard)? Will Gradient Cuff still offer great advantages, or the most contribution will be attributed to the JB filter?

R: We thank the reviewer for pointing out this issue that may cause misunderstanding. We believe this is a misunderstanding, due to the discrepancy in implementing and evaluating jailbreak detectors. In fact, Appendix A.5 just illustrates how we compute the Attack Success Rate(ASR) for Gradient Cuff (That is, the evalaution phase). So the 3rd step isn’t the component of the Gradient Cuff. It is just defining the evaluation metric that we reported, by checking how many jailbreak prompts that passed the Gradient Cuff filtering can successfully attack the protected LLM.

We agree with the reviewer that the computation method for ASR may influence the evaluation and we should also consider other candidates to compute the ASR. To address the reviewer’s concern, We choose two other metrics to compute the ASR:

GPT-4: We follow PAIR to design the system prompt of GPT-4 so that the GPT4 would assist in checking whether the model response implies jailbroken.

LLaMAGuard: We input the user query and the model response to llama guard. The llama guard model would output “unsafe” if the model response implies jailbroken.

We compared Gradient Cuff with baseline methods under these two evaluation methods and the results are shown below:

We can conclude from the table that the Gradient Cuff still offers great advantages under new evaluation methods. On vicuna, Gradient Cuff outperforms the best baselines by a large margin (0.1296 vs 0.2354 under GPT4 evaluation and 0.1171 vs 0.2408 under llama guard evaluation). On llama, Gradient Cuff achieves comparable performance with the best baselines, 0.0279vs0.0192 under GPT-4 evaluation and 0.0229vs0.0188 under llama guard evaluation.

We also include the utility of different baselines in this table, from this table we can see that PPL and Gradient Cuff both keep a good utility while SmoothLLM will significantly degrade the utility of the protected LLM. The dramatic utility degradation of SmoothLLM is due to its random modification to the original user query which has greatly changed the semantics of the original query. This is also the explanation for why SmoothLLM can get an ASR close to Gradient Cuff on LLaMA-2-7B-Chat when evaluated by GPT-4 and LLamaGuard: many malicious queries have lost the maliciousness after SmoothLLM’s modification, so even if the LLM doesn't refuse to answer them, the generated response is harmless.

We will provide an analysis of these different evaluation methods in the revised version of the paper.

---

Weakness 2 and Question 2: Should the FPR in Figure 2 and Table A1 be exactly 5%?

R: We believe this is a misunderstanding of the FPR on the validation set (σ that is set to be 5%) and the FPR of the test set. We mentioned in section 4.1 that we collect a bunch of benign user queries and split them into a validation subset and a test subset. We used the validation subsets to determine the detection threshold, and our detector does not have access to the test set. In line 229, we said that we set the σ (FPR) on the validation set to be 5% to control the FPR on the test set. As the test set and the validation set are independently identical, the FPR(test set) and FPR(validation) may be very close but still have small differences. In table A1, we can see on LLaMA, FPR(val) =5% and FPR(test set) = 2.2%. On Vicuna, FPR(val) =5% and FPR(test set) = 3.14%.

---

Weakness 3 and Question 3: Llama2-chat system prompt usage.

R: We agree with the reviewer that the system prompt would influence the performance. However, we used fschat(0.02.23) in all our experiments to keep consistent with the GCG repo(see https://github.com/llm-attacks/llm-attacks?tab=readme-ov-file#installation). Upon checking, fschat of this version provides the chat system prompt for LLaMA-2.

The increased ASR on llama compared with the reported value may come from longer training steps. GCG paper indicates that longer training steps would make the generated suffix more powerful. The reported value is corresponding to 500 training steps. We want to test our method on more powerful jailbreak attacks so we trained the GCG for 1000 steps, which we indicated in the appendix A.2. So we think a higher ASR (w/o defense) is reasonable.

Weakness 1: The argument of batching is somewhat questionable, considering that the LLM operator could batch multiple user requests otherwise.

R: We thank the reviewer for pointing out this issue that may cause misunderstanding. We need to argue that our method is designed for model owners deploying safety guardrails, instead of a user calling an LLM operation. This setting makes our proposed batch inference feasible. One just needs to put multiple user queries into a batch and get a batch of model outputs as LLM’s responses to the queries. A sample code is shown below:

def batch_inference(prompt_list, model_name):
   #prompt list contains multiple user query sentecnes
   model = AutoModelForCausalLM.from_pretrained(model_name).cuda()
   tokenizer = AutoTokenizer.from_pretrained(model_name)
   inputs=tokenizer(prompt_list,return_tensors="pt",padding=True)
   inputs={k:v.to(model.device) for k,v in inputs.items()}
   with torch.no_grad():
       input_ids = inputs["input_ids"]
       attention_mask = inputs["attention_mask"]
       prompt_length=input_ids.shape[-1]
       outputs = model.generate(
           input_ids = input_ids,
           attention_mask= attention_mask,
           do_sample = True
       )
       output_text = tokenizer.batch_decode(outputs[:,prompt_length:], skip_special_tokens=True)
   return output_text

We will provide a sample code and more explanations in the future revised version to avoid misunderstanding.

---

Weakness 2: It would make the paper more convincing if the phenomenon is shown to hold for more LLMs.

R: We agree with the reviewer that we should test on more LLMs. We selected Qwen2-7B-Instruct and Gemma-7b-it to verify Gradient Cuff’s effectiveness.

As the jailbreak prompts generation (GCG, PAIR, AutoDAN, TAP) is time-consuming and we only have one week to prepare for the rebuttal, we can’t generate jailbreak prompts for the two new models. We choose to test against Base64 attacks, which is a model-agnostic jailbreak attack method. We also tested Gemma and Qwen2 against GCG attacks transferred from Vicuna, which the authors of GCG claimed to have good transferability. The results are summarized in the following table. (the metric is the refusal rate, higher is better)

The results in this table show that our Gradient Cuff can achieve superior performance on non-LLaMA-based models like Gemma and Qwen2, outperforming SmoothLLM and PPL by a large margin. We believe our method can generalize to other aligned LLMs as well.

Though we cannot get the jailbreak prompts for new models due to the time limit of the rebuttal and the limit of computing resources, we've been running these attacks and will provide updates once they are done.

---

Weakness 3: The utility evaluation is rather shallow, only focusing on multiple languages.

R: We used MMLU Average Answer Accuracy as the utility metric, which is a common and popular metric used in many mainstream LLM leaderboards. The MMLU contains 57 tasks across various topics, including elementary mathematics, US history, computer science, and law. MMLU is a valuable tool for evaluating the performance of language models and their ability to understand and generate language in various contexts.

---

Question 1: What dataset is used for tuning the false positive rate? Can the authors also report, e.g., AUC scores for independence of a specific threshold?

R: We thank the reviewer for this question. We mentioned in section 4.1 that we collect a bunch of benign user queries and split them into a validation subset and a test subset. We used the validation subsets to determine the detection threshold thus adjusting the FPR evaluated on the test subset.

Moreover, we need to argue that we didn’t use AUC as a metric because only Gradient Cuff and PPL can provide such adjustable thresholds so we cannot compute AUCs for other baselines like SmoothLLM.

Below we show the AUCs of Gradient Cuff and PPL:

---

Question 2: What is the reason for the (costly) zero-th-order gradient estimates? I imagine that one can also use backprop at each generation step that yields certain jailbreak keywords (token), using the log-likelihood of this keyword (token).

R: We have introduced the motivation of Gradient Cuff in the introduction and the method section and we do agree with the reviewer that motivation needs more explanation. Our refusal function tries to use the empirical probability to estimate the LLM’s theoretical probability of rejecting the query by sampling multiple model responses for it and computing the refusal ratio. The log-likelihood of keyword tokens cannot be used as the proxy of the refusal probability as:

（1）The rejection keywords may not always appear at the beginning of a sentence representing rejection in real-world cases.

（2）The log-likelihood of the keyword is just the loss of the next-word-prediction, it cannot be used to model the appearance probability of the keyword. For example, even some model hallucinations will have a large log-likelihood as long as it is readable and fluent.

We thank the reviewer for the feedback. We now better understand the reviewer's comment related to inference cost. We agree with the reviewer that this defense comes with some extra inference cost (although with batch inference the time cost would be only 2x~11x if the extra query used by the defense is 110x, see our analysis in appendix A.15).

Indeed, one can save the extra queries used to deploy the defense to serve more users or one user's multiple requests. However, we believe the extra query budget should still be allocated to implement the defense, because users may have less incentive to use a model/service if it does not have proper safety guardrails.

The increased inference cost of Gradient Cuff does bring much-improved jailbreak defense performance and we think this is an inevitable tradeoff because, as we discussed in the response to the reviewer yW4P proposed weakness1(https://openreview.net/forum?id=vI1WqFn15v&noteId=sWvR1tuNOB), our defense gets the most superior trade-off compared with existing baselines. Smoothllm, the most powerful baseline under our evaluation, has a similar inference cost to us but the performance is far behind us. PPL, a very lightweight defense that almost does not bring extra inference cost, has a poor jailbreak defense capability and only works on specific types of attacks.

We thank the reviewer for the score increase!

We think the reviewer's constructive suggestions do point out new directions to improve this work, we will update our discussion in the revised version of our paper.

Question 1: Is LlamaGuard the more suitable baselines?

R: We thank the reviewer for pointing out this baseline for us. We use the Llama-Guard-2-8B as the classifier used in the LlamaGuard method and compare it with Gradient Cuff. The results are shown below.

(1) They both keep a low FPR so both of the two won’t bring great utility degradation.

(2) Gradient Cuff is superior to LlamaGuard in terms of defense performance against jailbreak attacks. Specifically, Gradient Cuff can achieve comparable TPR results on vicuna(0.743vs0.773) and much better results on llama2(0.883vs0.760).

We need to emphasize that Llama-Guard-2-8B is an LLM based on LLaMA-2 and trained on massive data to learn to distinguish malicious queries, while our method and the other baselines do not use external models as guardrails (that is, only using the protected model to enhance safety). Though comparing to Llama-Guard-2-8B is unfair to Gradient Cuff, this result further corroborates the effectiveness of Gradient Cuff.

Moreover, LlamaGuard needs to deploy two LLMs: the protected LLM itself and a LlamaGuard family model (at least 7B size), while Gradient Cuff only needs to deploy the protected LLM.

We will add this discussion in the final version of this paper. It will provide more supportive evidence for Gradient Cuff’s effectiveness.

---

Question 2: How to verify that Gradient Cuff reaches a better trade-off with utility?

R: In section 4.1, we discussed the trade-off between TPR and FPR for Gradient Cuff and all the baselines, Figure 2 showed that our method can outperform PPL and SmoothLLM with a similar FPR and a much higher TPR.

In section 4.5, we discussed the utility of the Gradient Cuff evaluated on the MMLU benchmark. The results showed that Gradient Cuff’s utility degradation is only brought by false refusal to some benign queries and the false refusal rate can be controlled by adjusting the FPR on the validation set.

To answer the reviewer’s question, we computed all the baselines’ utility degradation following the same procedure introduced in section 4.5 and compared them with Gradient Cuff. The results are shown below

From the results, we find that though SmoothLLM can achieve a very low FPR shown in Figure 2, it causes a dramatically large utility degradation because it has modified the original user query so it will damage the semantics of the user query. PPL attains the best utility and Gradient Cuff achieves the best performance-utility trade-off by

(1) keeping the comparable utility with PPL

(2) attaining a much higher TPR than the best baselines(e.g., 0.743vs0.413 on viucna) against jailbreak attacks.

We thank the reviewer for this question and will add the utility comparison with baselines in section 4.5 of the revised version of this paper.