A Realistic Threat Model for Large Language Model Jailbreaks

Dear reviewer M1dj,

Thank you for your time and appreciation. We’re happy you agree that the “paper addresses the critical need for a standardized framework to compare various jailbreak attacks” and that “the threat model remains interpretable and LLM-agnostic, facilitating a deeper understanding of why certain attacks succeed or fail.” Moreover, we are grateful for your acknowledgment that “adapting and evaluating popular attacks within the proposed threat model allows for a fair and rigorous comparison.”

Please find your questions addressed in the points below:

“The paper would benefit from a direct comparison with existing perplexity detectors, such as the one proposed by Alon et al. (arXiv:2308.14132)”

This is a good point that we had not formally included as data in the paper. To remedy this, we have now rerun our entire pipeline with an LLM-based perplexity filter (using the perplexity of the Llama-2-7b model that we use as default in the paper as a filter). We again carefully calibrate the TPR of this new filter and run strong adaptive attacks. The results can be seen in the following table:

Here we see that the successful adaptation of the (blackbox, high perplexity) PRS attack leads to negligible difference in ASR of attacks against this new LLM filter (0.44) vs the old N-gram-based filter (0.46). We have added these results to the Appendix. We hope these results provide a clear datapoint that using the N-gram model does not compromise utility. This allows the other important advantages of N-gram-based models for the purpose of our threat model to shine, namely that it is model-agnostic, easy to evaluate and interpretable (see Section 1 for details).

“ While the N-gram model offers interpretability, it may not capture the nuances of natural language as effectively as model-based perplexity measures”

What we find especially interesting here is that we ablated the capability of the N-gram model to capture nuances in language (via increasing the N, or via the comparison to LLM-based models as above), but we did not find an improvement in robustness against strong adaptive attacks. It seems like the 2-gram model is optimal in the sense that it models language well enough, but is robust enough not to increase the attack surface.

“Clarify Equation 1: Provide a complete and formal definition of the Judge function. ”

Thank you for your suggestion. We have introduced additional details in the description of Eq. 1. For more technical details of the judge function, please see App. B.

Thank you for your feedback! We hope that we could answer all remaining questions. Please let us know if any follow-up questions arise.

Dear reviewer PGG1,

Thank you very much for your review. We’re glad you found our idea interesting, our paper fairly clearly written, the plots clearly presented, and the analysis of our threat model very interesting. Please find a point-by-point response below.

“I am generally confused about the "threat model" framing, e.g., "universal threat model for comparing attacks across different models,...”

• For us, a threat model answers the question: What is a minimal constraint (as additional defense layers/constraints would monotonically reduce utility) that ensures that the input to an LLM is likely to be normal user input? An ultimate judge of this would be a human, but clearly, this is not an option, so we need a proxy for this, which one can evaluate easily and independently of the employed LLM. Our window N-gram perplexity fulfills these requirements and measures the likelihood that a given prompt occurs in real, normal text, and we adjust the threshold such that 99.9% of all benign inputs would pass the filter so it is a minimal restriction not affecting the utility of the LLM.

• A threat model restricts the set of inputs and thus can be seen as a weak version of a defense. Additionally, the budget of an attack is an often neglected parameter, and thus, we explicitly include it in our threat model. Our threat model is “universal” in the sense that it is independent of the LLM.

“I'm not sure if the contribution is the threat model or the N-gram language model perplexity filter?”

• Our contribution is the principled threat model we propose and evaluate as defined using the N-gram model perplexity. We chose perplexity filtering as our motivation and the starting point because, as we hypothesized and confirmed in Fig. 1 and Fig. 12, higher perplexity attacks lead to higher ASR, making existing attacks that do not control for this incomparable. We provide a principled construction of the threat model and an extensive evaluation of strong adaptive attacks. This combination provides us new insights into the strengths and weaknesses of each of the tested attacks and, for the first time, makes them comparable.

“As far as I understand, the "threat model" is basically assuming the N-gram approach is the right way of doing things, but I am not sure that is clearly established here? If the point of the paper is to establish the threat model, there should be lots of evidence it is an appropriate defence”

To model language, an N-gram-based language is clearly not the best way of doing things. LLMs do a better job on this. But that does not mean that, e.g., LLM-based perplexity is a better measure to filter out adversarial inputs, as LLM models offer a larger attack surface that an adversary can exploit.

Thus, it is fair to ask how model-based perplexity compares as a filter. Note, however, that in Table 2, the attack is not adapted against Llama2-7b. To answer the question of how an N-gram LM vs LLM-based perplexity filter perform side-by-side, we reran our pipeline using Llama-2-7b as a filter, which, as for the N-gram, we calibrate so that a TPR of 99.9% is reached on benign prompts. We show the accuracy of principled adaptive attacks against both filters in the following table for the first 50 prompts of the benchmark:

Based on these results, we see that both filters have a similar effect. Thus, in terms of ASR, there is no advantage to using an LLM-based filter. However, there are three important advantages of N-gram over LLM-based perplexity (see Section 1): i) it is model-agnostic; ii) it has negligible compute cost; iii) it is interpretable and provides insights into LLMs failure modes. But this is a good question, and we have added this table and an extended discussion to our revised version; see Table 6 and Fig. 14 in the Appendix.

“The phrase “A questions their She” very unlikely exists in normal text. With the 8-gram model used in the paper, it should be able to filter out this. Could the authors explain why this case bypass the detection?”

• Please note that we measure mean 2-gram perplexity over a window size of length 8. We found this combination to be optimal in our ablations (see Section 3.2 and App. C). This choice is optimized to separate benign and malicious prompts. We note that, e.g., extending the N-gram to an 8-gram would score this string as more surprising, but, due to the sparsity of the 8-gram, would require a higher threshold to achieve a 99.9% TPR on benign data, leading to a loss in utility.

• For the 2-gram model, the evaluation becomes clear when investigating the tokenization of the string: “_A”, “_questions”, “_their”, “_She”. The corresponding bigram “A questions” (1630 counts in Dolma), “questions their” (18803 counts), “their She” (8888 counts) all appear in natural text, e.g. as part of “... Q & A questions”, “He questions their…”, “their Sheets”. This observation highlights the advantage of an interpretable threat model, as for LLM-based perplexity, it would be difficult or even impossible to trace the origin of suffixes in the training data.

References

[1] Alon et al., “Detecting language model attacks with perplexity.”

[2] Llama Guard 2, https://github.com/meta-llama/PurpleLlama/blob/main/Llama-Guard2/MODEL_CARD.md

[3] Jain et al., “Baseline Defenses for Adversarial Attacks Against Aligned Language Models.”

[4] Tramer et al., “On Adaptive Attacks to Adversarial Example Defenses.”

[5] Carlini and Wagner, “Adversarial examples are not easily detected: Bypassing ten detection methods.”

Thank you for your in-depth questions and comments. We hope to have addressed your concerns in our response above. Please let us know if you have any follow-up questions.

Dear reviewer YgJw,

Thanks a lot for your review. We’re glad you found our N-gram model effective and our evaluation comprehensive and valuable for advancing new techniques in attacking and protecting LLMs. Please find answers to your questions below.

“There have been many proposed methods for filtering such jailbreaking prompts [1][2]. This insight mentioned in the paper is not new. The proposed approach of using the N-gram is straightforward. The novelty hence seems limited.”

Perplexity filters are indeed a sensible strategy to mitigate jailbreaks and have also been proposed in prior work [1,3], which we discuss in our related work. However, we do think that previous work (both references were rejected at ICLR 2024) has not done perplexity filters well. These works evaluate LLM-based perplexity filters but consider only weak adaptive attacks, if any. Notably, they both come to the conclusion that high-perplexity attacks, such as PRS or GCG, do not work, motivating many follow-up works, such as AutoDAN or BEAST, that are constructed to be low-perplexity attacks.

However, we correctly execute a window-based perplexity filter, calibrate its TPR correctly, and then design strong adaptive attacks. With this principled approach, we are able to provide an accurate assessment of the strength of perplexity filters, and we do find that attacks like PRS and GCG actually outperform newly-proposed low-perplexity attacks - which we think is a valuable contribution to the community and a strength of our benchmark design.

On top of this, our approach has further advantages, as we discuss in the paper:

• We use a perplexity filter based on a bigram model agnostic of the LLM, which allows us to compare this filter across models.
• We discuss the utility-robustness tradeoff intensively and use a very conservative threshold in our threat model so that 99.9% of benign prompts pass the N-LM filter. Note that this threshold is agnostic of the employed LLM, and thus, our threat model transfers across LLMs - making it straightforward to test new attacks and models.
• We can interpret the potential reasons exploited by jailbreaks by analyzing the origin of the bigrams used in the attacks (see Section 5.4).

Regarding Llama-Guard 2 [2], we have evaluated this filter against 50 prompts generated by PRS for Llama2-7B: 26 of 50 prompts pass. This leads to an ASR of 28% with input filtering and 20% with input-output filtering at 96% TPR [2], whereas applying our N-gram-based perplexity filter leads to 0% ASR at 99.9% TPR (see Table 3), meaning that even without adaptive attacks, the guard underperforms against the strong attacks we evaluate. Moreover, for the prompts generated by our adaptive PRS to our N-gram filter, even 33 out of 50 pass Llama-Guard 2, leading to an ASR of 32% for input filtering and an ASR of 22% for input-output filtering. We have now added the results for Llama-Guard 2 in Section I in the Appendix to clarify this point.

Thus, we hope we can respectfully disagree here - we think that our adaptive attacks and principled approach of our threat model are novel and provide valuable insights for the community.

“There are other defenses such as instruction filter, and random perturbation, etc. Why doesn't the threat model consider them?”

• It is important here to separate the general threat model we propose from any particular defense that could be employed inside such a threat model. To us, this is analogous to an $L_\infty$ constraint in adversarial attacks in vision. To achieve a sensible threat model containing a good model-agnostic constraint, we employ the N-gram perplexity with the setting we propose and carefully evaluate it in a principled way.

• We agree that combining existing defenses, as mentioned by the reviewer, might offer an additional advantage when defending models; this is orthogonal to the current direction of the paper but an interesting topic for the future.

• To provide additional data, we evaluated LLama-Guard 2 as a defense in our threat model. However, the instruction filter of Llama-Guard 2 is very weak: 26 of the 50 prompts of the original PRS attack for Llama2-7B pass the guard with input filtering at 96% TPR [4] and so already achieve an ASR of 28% with input filtering and 20% with input-output filtering (compared to the original ASR of 68%), whereas using our NLM-perplexity filter leads to an ASR of 0% at 99.9% TPR (see Table 3).

Evidence is needed to show that N-gram LM is better than LLM.

We do think that there are a number of advantages to the setup we propose (as discussed above), but to address this question directly, we have now added new results using an LLM-based perplexity filter:

• Here, we reran our entire pipeline using Llama2-7B to measure LLM-based perplexity and, additionally - using Llama Guard 2. For the Llama2-7B (Self-PPL) filter, we correctly calibrate it to a 99.9% TPR rate on benign prompts and run strong adaptive attacks against it using PRS. To our knowledge, adapting PRS to LLM-based defenses using rejection sampling is novel.

• As shown in the table above, the ASR achieved with the N-gram-based filter is 46% versus 44% with the Llama2-7B-based PPL filter and 46% - with the Llama Guard 2. Based on these preliminary results, we see that N-gram-based and LLM-based filters have a similar effect so that in terms of ASR, there is no advantage to using the LLM-based filters. However, there are three important advantages of N-gram over LLM-based PPL (see Section 1) that make it much better suited for the design of our threat model: i) model-agnostic; ii) negligible compute cost; iii) interpretable and provides insights to LLMs failure modes.

References

[1] Alon et al., “Detecting language model attacks with perplexity.”

[2] Jain et al., “Baseline Defenses for Adversarial Attacks Against Aligned Language Models.”

[3] Zou et al., “Universal and Transferable Adversarial Attacks on Aligned Language Models.”

[4] Llama Guard 2, https://github.com/meta-llama/PurpleLlama/blob/main/Llama-Guard2/MODEL_CARD.md

Overall, thank you for your questions. We think we have addressed your concerns, but please let us know if there are follow-up questions you would like us to address.

Dear reviewer ATuy,

Thank you for your review and questions. We are encouraged that you appreciate that we introduce the unified threat model and acknowledge the advantages of N-gram LM. Please find answers to your questions below.

“Only considered white-box attacks. In the real world, black-box attacks are more practical.”

• Our threat model indeed allows white-box access to the model. We think this is the most principled way of setting up the threat model and estimating the worst-case attack success rates against these models, especially considering the potential of transfer from (potentially very similar) open-source models [3]. However, within this threat model, we evaluate attacks from the entire spectrum: PAIR is a fully black-box attack, PRS and BEAST are score-based black-box attacks, and GCG uses the full white-box model. We have clarified this in Table 1.

“As shown by Figure 13, white-box attacks in this new threat model have lower transferability. That is, this threat model cannot measure black-box attacks very well.”

• This is potentially a misunderstanding. In Fig.13, we report the attack success rate of all generated prompts we generate with PRS on Llama-2-7B against black-box models - we do not report the transfer rate (the ratio of the number of successful attacks against the target transfer model to the successful ones against the source Llama-2-7B model). The ASR of the attack against the source model, Llama-2-7B, decreases as we tighten the perplexity constraint, yet the ASR on the target model stays mostly constant. As such, the transfer rate actually increases!

• We agree that this is hard to parse out of Fig.13, and so we have modified the figure to now overlay the original ASR on the source model in red. To further validate our transfer results, we added new results for Llama 3.1-405B models and a WizardLM model, all of which follow the same trend of mostly constant ASR on the target model while ASR on the source model decreases.

• Overall, we think it is an interesting validation of our benchmark that the best attack in our evaluation is a score-based black-box attack that also transfers well.

Regarding the novelty of perplexity filters

• Perplexity filters are indeed a sensible strategy to mitigate jailbreaks and have also been proposed in prior work [1,2], which we discuss in our related work. However, we do think that previous work (both references were rejected at ICLR 2024) has not done perplexity filters well. These works evaluate LLM-based perplexity filters but consider only weak adaptive attacks¹, if any. Notably, they both come to the conclusion that high-perplexity attacks, such as PRS or GCG, do not work, motivating many follow-up works, such as AutoDAN or BEAST, that are constructed to be low-perplexity attacks. However, we correctly execute a window-based perplexity filter, calibrate its TPR correctly, and then design strong adaptive attacks. With this principled approach, we are able to provide an accurate assessment of the strength of perplexity filters, and we do find that attacks like PRS and GCG actually outperform newly-proposed low-perplexity attacks - which we think is a valuable contribution to the community and a strength of our benchmark design.

• On top of this, our approach has further advantages, as we discuss in the paper:

  • We use a perplexity filter based on a bigram model agnostic of the LLM, which allows us to compare this filter across models
  • We discuss the utility-robustness tradeoff intensively and use a very conservative threshold in our threat model so that 99.9% of benign prompts pass the N-LM filter. Note that this threshold is agnostic of the employed LLM, and thus, our threat model transfers across LLMs - making it straightforward to test new attacks and models.
  • We can interpret the potential reasons exploited by jailbreaks by analyzing the origin of the bigrams used in the attacks (see Section 5.4).

¹ We construct as one of our major contributions strong adaptive versions of several well-established attacks against our bigram-based perplexity filter that work successfully against strong safety-tuned models. We want to stress that [1] using model-based perplexity did not consider adaptive attacks at all, and [2] constructed an adaptive attack for GCG but reported a low ASR even for Vicuna-7B at a TPR for benign text of 85%.

Dear reviewer ATuy,

Thank you once again for your suggestions on improving the paper, which we were happy to incorporate (see the latest revision).

We would be glad to answer your questions, if you have any and in case we have answered all of your points, we would kindly ask you considering raising the score.