AutoDAN: Automatic and Interpretable Adversarial Attacks on Large Language Models

We thank reviewer rSK6 for the helpful feedback! We address your concerns below.

Weakness 1: Lack of enough baseline comparison.

Question 4: There are several existing jailbreak prompt generation methodologies. A comparison with these methods is essential for the authors to demonstrate the superiority and efficacy of their proposed approach. For instance, JAILBREAKER: Automated Jailbreak Across Multiple Large Language Model Chatbots.

We only consider gradient-based jailbreak attacks on LLMs in this paper since these methods are fully automatic and do not require any manual input. All other methods, including the one you have mentioned, either require handcrafted seed attack prompts for training or system prompts for prompting attacker LLMs. As far as we know, GCG is the first and only effective method for this task prior to our work, and we extend GCG with a readability regularization as an additional baseline method. We will further discuss the differences between our method and the one you have mentioned in our updated paper.

Weakness 2: Lack of evaluation on more advanced LLM model, such as GPT-4.

Question 1: The authors should provide more tangible examples of successful attacks on advanced LLMs, such as GPT-4, Bard, and Bing Chat. I remain skeptical that the proposed prompts can attain a high success rate against such sophisticated LLMs. I would only be convinced of its efficacy if it proves effective against these cutting-edge, real-world models.

Our original manuscript reported the transfer attack results on GPT4 in Figure 1 and Table 3 (which are still Figure 1 and Table 3 in the revised manuscript). We keep them in the same place in the revised paper. Our released raw annotation data (link) further showcases tangible responses from GPT4.

Weakness 3: Lack of consideration of other potential defense strategies adopted by the LLM provider.

Question 2: Beyond perplexity-based filters, LLM providers might also utilize additional defensive strategies. These can include dynamic content moderation of generated outputs and keyword filtering. Is the proposed attack equally effective against these defenses?

That's a great point! Our jailbreaking results on GPT-3.5 and GPT-4 already show that current defenses adopted by the LLM provider fall short of ensuring complete safety. However, we believe there are other potential defense strategies they might consider, such as model-based content moderation, keyword filtering, retokenization, and paraphrasing. We currently do not explicitly test these defenses since their safety-usability trade-off (especially false positive rates) is unclear. We plan to evaluate a few more defenses in future work.

Question 3: The source codes are not open. The technical details in the paper are not clear enough to enable reproduction.

Although we are currently encountering legal issues that have resulted in a delay in the code release, we are dedicated to releasing our code as soon as possible.

Thanks again for your time and effort in reviewing our paper! We are happy to have more discussions for any further questions.

We are grateful for your insightful feedback! It is encouraging to see you recognizing the novelty of our algorithm, which combines gradient-based optimization with sampling-based token generation. We are committed to keep improving our algorithm for a broader range of tasks in the future.

Weakness 1: I believe the paper’s biggest area of improvement is its definition of attack success rate. In the current approach, the authors likely underestimate refusals since they only do string matching against a set of known refusal strings. It would be good to at least validate this metric in one of three ways: by using a dedicated safety model trained to identify refusals, by prompting a more powerful model such as GPT-4 acting as a judge, or by manual inspection of a sample of responses.

We thank you for these constructive suggestions! Please see our General Response - 1. Improved Evaluation for our response. Specifically, we use the GPT-4 based evaluation for all our experiments (except for Guanaco and Pythia due to time limit). We also meta-evaluate the reliability of different evaluation methods on a sample of responses, including string matching, classifier-based evaluation, and GPT-4-based evaluation, with human labeling as the ground-truth.

We note that our classifier-based evaluation, which identifies harmful content, is not exactly the same as the one you mentioned, which identifies refusals. The pretrained classifier that directly identifies harmful content seems susceptible to distribution shifts and performs poorly in our experiment. Intuitively, refusals are a much narrower concept than harmful behaviors and could be easier to classify. Now that string matching shows a surprisingly high agreement with human annotation, we will try to train a refusal-identifying classifier in the updated paper to improve upon string matching.

Weakness 2: The paper needs to expand the models it is testing against to include Llama 2, Claude, and Bard, since those models are more heavily optimized for safety and in some cases are better models. It is acknowledged that Claude and Bard do not have their weights available but transfer attacks against those should be evaluated as well.

Our General Response - 2. Jailbreaking Llama2 includes the results on jailbreaking Llama2-chat in the individual behavior setting. We are still working on the multiple behaviors setting. Currently, we do not have an API quota for a quantitative evaluation of Claude and Bard, but we will try to add some qualitative examples in the updated paper.

Question 1: Is Table 2 a random sample? How can the claims in Section 4.2 be made more robust? Do the authors have measures of readability (e.g. perplexity) of their whole adversarial dataset? We see this indirectly in the perplexity filter bypass but can they break it down further?

1. No, prompts in Table 2 are manually selected from 20 independent runs of AutoDAN on Vicuna-7B.
2. To make claims in Section 4.2 more robust, we will release a large amount of randomly generated prompts with random seed indices in the updated paper.
3. Our released raw annotation data contains the perplexity for each attack suffix. For the randomly generated prompts to be released, we will also show the perplexity of each prompt as well as the perplexity distribution.

Question 2: For the data in Table 3, how do the authors know which protection they bypassed? Can they explain if they tested against an API endpoint without protections or something else?

We use the Azure API for evaluating GPTs. The Azure API has built-in content filtering that cannot be turned partially or fully off unless approved by Azure (see their content filtering manual - Configurability). We identify the protection layers an attack has bypassed by analyzing the response of Azure API: 1) If the model fulfills the harmful request, then the attack has bypassed all layers of protection; 2) Otherwise, the model responds with certain refusal or error code, and we look up the Azure's content filtering manual again to determine which layer stops the attack, and we label the attack as bypassing all earlier layers of protection.

We thank reviewer LYLv for the helpful feedback! We address your comments and concerns below.

Strength 1: It is interesting to see the two strategies, which are shifting domains and corroborating fine-grained instructions, inspired by the AutoDAN-generated adversarial samples.

We are delighted to know that the reviewer finds the interpretable AutoDAN-generated prompts interesting! We would like to add that interpretability is especially helpful when people don't know what strategies to use in novel tasks. For example, when crafting the attack prompt for the prompt leaking task, shall we use the word "instructions" or "prompts" in referring to the system prompts we aim to obtain? Running AutoDAN in a fully automatic manner with the target of outputting the system prompts (Section 4.4) tells us that we should use "instructions" (Figure 10). This is a subtle yet impactful distinction that AutoDAN helps to uncover, and AutoDAN may also provide more insights into understanding the underlying mechanism of successful adversarial strategies.

Weakness 1: The generated suffix is arguably long. Therefore, how is the performance when the length of the suffix is constrained?

Thanks for the great question! Figure 9 in the original manuscript appendix (Figure 11 in the revised manuscript Appendix C) showed that the ASR of AutoDAN generated prompts would not be affected if we limit the token length to under 50. We further emphasized this point in the updated paper.

Weakness 2: How is the computational resource required for AutoDAN? And, how is the efficiency of the proposed attack? The backpropagation through the LLMs to select the candidate subset could be computationally heavy.

Please see our General Response - 3. Computational Complexity Analysis for the complexity of our method in comparison with prior work (GCG), and the actual time cost. We do backpropagation once in each step to get the gradient at the new token's location. Since we only need the gradient for one location (as compared to all weights in model training), the time and space complexity is slightly less than a forward pass due to the shorter path in the computational graph.

Weakness 3: The proposed attack seems to require using the probability of the next token. However, in a black-box setting (e.g., attack GPT-3.5 API), it is difficult (or impossible) to obtain the probabilities. Therefore, the proposed method could be difficult to adapt to the latest LLMs.

This is indeed a limitation of our method (and all other gradient-based methods alike, including GCG), as it is impossible to obtain the gradient from black-box LLMs. Nevertheless, AutoDAN-generated interpretable prompts demonstrate transferability, suggesting that different LLMs share a somewhat similar vulnerability landscape. These interpretable prompts thus provide a way for red-teaming researchers to identify and exploit these common vulnerabilities and potentially strengthen the existing black-box attacks using the strategies found by AutoDAN.

Thanks again for all your time and effort in reviewing our paper! We are happy to have more discussions for any further questions.

Questions

Question 1: In Fig.1, why is the perplexity of the suffixes generated by the proposed method sometimes similar to those generated by GCG (and even higher)? Form the results in Fig.6, it seems the proposed method should have a clear difference in perplexity compared with GCG.

Please note that Figure 1 uses a log-scale x-axis (perplexity), so that points may appear visually similar but very different in numerical values. Also, in Figure 1 we vary the readability loss weight for GCG-reg from 1e-4 to 1, whereas we use the fixed weight of 0.1 in Figure 6 (which empirically achieves the best trade-off between readability and training ASR for GCG-reg).

Question 2: From my experience and existing works, natural paragraphs usually have a perplexity of around 30-50 (tested by GPT-2). This score may vary based on different testing language models. However, as the results in Fig.1 show that the perplexity of generated suffixes is around 80-100, which is interesting. Can you provide some instances of the generated samples that have the lowest perplexity and the highest perplexity?

We are not clear about what the "generated suffix" means. If you are referring to the AutoDAN-generated suffixes in Figure 1, then they have a perplexity of around 10 (median 12) instead of 80-100 (note the log-scale x-axis), and the normal user request represented by the dashed vertical line has a perplexity of 126. These perplexities are measured by Vicuna-13B. We provide some instances of the generated samples with varying perplexities in the raw annotation data in this link.

Question 3: Can you elaborate more on the implementation details about the perplexity testing and other evaluation settings, for example, settings of generating LLMs response (local models and APIs)?

Thank you for the suggestion! We updated the manuscript to add more implementation details (Appendix B). We generally follow the same setting in generating LLM response as GCG, and use the Huggingface's perplexity metric with Vicuna-13B (v1.5) in evaluating the perplexity.

Question 4: Can you share a direct transferability evacuation (in Tab.3) without a PPL filter? This can demonstrate the attack strength of each method.

We showed the direct transferability evaluation without the PPL filter in Figure 9 in the original manuscript. We have moved the evaluation results without the PPL filter to the main body of the revised paper (Figure 1, Table 3), and moved the results with the PPL filter to the appendix (Table 8, Figure 10). Thanks for the suggestion!

Thanks again for all your time and effort in reviewing our paper! We are happy to have more discussions for any further questions.

We thank reviewer abck for the helpful and constructive feedback that helps us better shape our paper! We address your concerns below.

0. Clarifying Technical Contributions

Specifically, the proposed method introduces a loss function to drive the generated jailbreak suffix readable and add a weight parameter to balance the attack strength and readability.

As the proposed method is aimed at solving the aforementioned trade-off and proposes a dual-target loss function

We would like to clarify our technical contributions, in case they were not sufficiently clear in the paper: our method doesn’t merely add an additional loss function to achieve readability. This straightforward approach has been previously implemented in prior work and has proven ineffective (unable to achieve both readability and another factor, as illustrated in Figure 1). For reference, see the code implementations of GCG and Jain et al. 2023.

Technically, our new gradient-based token generation algorithm pivots on three key differences:

1. We optimize and generate each token sequentially from left to right rather than optimizing a fixed-length token sequence as done in prior work. This method, mirroring the human approach to writing text, significantly reduces the optimization space and simplifies the process. As we demonstrate, this is crucial for generating readable text using current gradient-based optimization techniques.
2. In addition to the dual objectives in the second step of single token optimization, we introduce tailored dual objectives in the first step. This is critical for providing readable candidates in the second step (Figure 4).
3. We combine the dual objectives by summing their logarithmic values. This approach, as we illustrate, adapts to the entropy of the next token distribution (Figure 4) and is key for efficient optimization.

Weakness 1. Soundness

Weakness 1.1: However, the paper cannot convince me that the proposed method can solve (or avoid) the trade-off between ASR and readability.

Thank you for raising this important point. Let's break down the concept of the trade-off between ASR and readability, as it's not a universally acknowledged notion in the existing literature and may vary depending on the target model and definition of ASR.

1. Existence of the trade-off: The ASR-readability trade-off may not necessarily exist for some models. For instance, LLMs like GPT3.5, GPT4, and Vicuna are vulnerable to manually designed prompts that are both perfectly readable and universally effective (jailbreakchat.com provides many such examples). Similarly, some prompts generated by our method also exhibit this dual quality and show no sign of such a trade-off.

2. Defining ASR: The notion of the trade-off depends on how we define ASR. If we consider training ASR, then prioritizing readability might constrain our choice of prompts, suggesting a potential trade-off. However, when considering test ASR (same model, unforeseen behaviors) or transfer ASR (different models, unforeseen behaviors), readable prompts often show better generalization and transferability than their unreadable counterparts, as evidenced in our Table 1 (generalization) and Table 2 (transferability). This suggests that the trade-off may not exist in the latter case. The test ASR and transfer ASR matter because attackers can copy and paste any pretrained prompts to jailbreak LLMs without warming up their own GPUs, thus posing a threat to the safety of LLM applications.

3. Defining target model: The notion of the trade-off also depends on how we define the target model. If we consider the safety alignment during RLHF as part of the target model, then a perplexity filter attached to the model may also be considered as part of it. If this is the case, then the trade-off may not exist since unreadable prompts cannot bypass the filter to jailbreak the model.

In summary, while the trade-off between ASR and readability is an interesting concept, its existence and significance may vary based on how we define ASR and the target LLM, and our method is not necessarily subject to such trade-off.

Results.

• 1) Through annotation (link to raw annotation data, click the "view raw" button in the top right corner to view in rendered HTML), we found that approximately 10% of the responses are challenging to label due to factors like ambiguous requests, diverse audiences, and hard-to-verify domain knowledge. Examples include uncertainties about using hydrogen peroxide in bomb-making or the feasibility of using electromagnetic waves to hack into people's minds.
• 2) Table 6 in Appendix C.1 illustrates the agreement (measured by accuracy and F1 score) between different metrics when using human evaluation as the ground-truth. The GPT-4 based evaluation achieves the highest correlation with human evaluation, but this is only achieved using a complex prompt template (Figure 8 Appendix C). String matching achieves the second-best correlation and offers a cost-effective and rapid evaluation method in this setting. However, a cautionary note: the list of refusals should be comprehensive, as different models may express refusal in varied ways. The classifier-based evaluation performed poorly, likely due to the distribution shift between our response data and the training data of the model. (We verified this by inspecting some examples to ensure the model's functional accuracy.)
• 3) The more advanced model, GPT-4, presents greater challenges for labeling and evaluation, for both humans and automated evaluators. When the adversarial suffix prompts GPT-4 to exhibit harmful behavior in a hypothetical scenario, GPT-4 tends to produce more unrealistic responses compared to other models, which rely more on their inherent world knowledge for response generation.

2. Jailbreaking Llama2

Setting: We use AutoDAN to jailbreak Llama2 in the individual behavior setting, namely training on only one harmful behavior while testing on 20 unseen behaviors. We run each setting eight times and use GPT-4 to evaluate the attack's success. We use Vicuna-13B to evaluate the perplexity across all experiments in the paper.

We didn't test in the multiple behaviors setting, specifically training on multiple behaviors, because we found that Llama2 behaves differently compared to other models in this context. Specifically, its test generalization is worse when trained on multiple behaviors than when trained on just one. We're trying to understand why. At present, we don't see this as a drawback of our method, as there's no evidence suggesting that readable and universal attack prompts against Llama2 exist.

Results: Table 7 shows that

• 1) AutoDAN achieves slightly worse training accuracy (30.8%) than GCG (33.3%), but much lower/better perplexity (3e5 vs 7e2).
• 2) GCG cannot be simply regularized with an additional perplexity regularization to achieve lower/better perplexity while obtaining a similar attack success rate.
• 3) The perplexity of the AutoDAN-generated prompts is still much higher/worse than those generated on Vicuna (7e2 vs 1e1), meaning that the safety alignment of Llama2 significantly reduces the vulnerability loopholes on the data manifold (but potentially at the cost of compromised usability, as reported by practitioners).
• 4) The generalization to unforeseen behaviors of AutoDAN-generated prompts is better than GCG's (35.0% vs 11.7%), supporting our finding that more interpretable prompts are more likely to generalize better (without model ensemble).