Deciphering the Chaos: Enhancing Jailbreak Attacks via Adversarial Prompt Translation

• The paper lacks a clear explanation for why this method outperforms GCG and other optimization-based approaches. This is counterintuitive, as GCG directly optimizes the adversarial objective, whereas this method does not appear to do so.
• Could you provide a comparison of the adversarial loss between the GCG-optimized prompts and your translated prompts? I would expect the GCG-optimized prompts to achieve a lower loss, as GCG's objective is to minimize the loss. However, your prompts show a higher attack success rate. It would be helpful to see the loss values side by side and to hear your explanation for this discrepancy.

Answer:

In this paper, we focus on developing effective black-box jailbreak attacks against LLMs. Existing GCG-like methods directly leverage the gradient of adversarial loss with respect to input tokens in white-box models, resulting in superior performance in terms of adversarial loss and attack success rate within white-box settings. However, their performance in attacking black-box models is limited for two reasons. Firstly, because the input gradients of white-box and black-box victim models differ significantly, directly transferring these adversarial prompts to black-box victim models to induce harmful responses is challenging. Secondly, while there are some "relatively interpretable" information in the garbled adversarial prompts, which has also been preliminarily discussed in the GCG paper, black-box victim models typically fail to comprehend this adversarial information due to its chaotic appearance. Therefore, our method aims to translate these garbled adversarial prompts into human-readable natural language prompts to improve the attack success rate against black-box victim models.

We compared the adversarial loss resulting from GCG-Advanced with our method on the black-box victim model Llama-2-7B-Chat. We calculated two types of loss to better illustrate the performance in inducing the victim model to output affirmative responses instead of refusal responses. The first type of loss computation uses "Sure, here's" as the target string, while the second type uses two refusal strings collected from the official GCG implementation, namely "I cannot" and "I apologize," as target strings. The results, shown below, indicate that our method achieves lower loss for producing affirmative responses and higher loss for generating refusal strings compared to GCG-Advanced. These results demonstrate that our method is more effective in inducing affirmative responses and avoiding refusal responses than GCG-Advanced.

• The paper needs to provide examples of adversarial prompts. Otherwise, there is no way for reviewers/readers to directly check the quality of the adversarial prompts. Could you provide examples of adversarial prompts, along with the user's harmful request, for all the models, especially Llama-2-chat?
• Could you provide examples of adversarial prompts so that reviewers and readers can have a better understanding of the results?

Answer:

In the revised paper, we have included 10 examples for translating garbled adversarial prompts generated by 5 white-box LLMs (including Llama-2-Chat models) in Appendix C.1. Additionally, we included 2 cases for using our method to jailbreak 11 black-box victim models (including Llama-2-Chat models) in Appendix C.2. These examples demonstrate that our method effectively translates garbled adversarial prompts into human-readable natural language prompts, successfully inducing the black-box victim models to generate harmful responses.

Thanks for the feedback. Our responses to the comments are given as follows.
 

The method assumes that GCG-generated adversarial prompts are always somewhat readable, but this assumption is not clearly justified.

Answer:

We conducted a simple experiment to verify whether garbled adversarial suffixes consistently contain meaningful content that triggers vulnerabilities in white-box LLMs, resulting in the LLM’s harmful responses also including these meaningful words or phrases. We utilize NLTK to split harmful requests, adversarial suffixes, and the harmful responses generated by the white-box model into words. In addition, we filter out stopwords from the responses and remove words present in the harmful requests. Subsequently, we examine whether the responses contain any words from the adversarial suffixes. We conduct this experiment across 7 white-box models. For each white-box model, we apply the GCG-Advanced method to generate query-specific suffixes on the first 100 examples from AdvBench. We report the proportion of samples in which the harmful response includes words from the suffix, relative to the total number of successful attacks. The results are shown in the table below. It can be seen that, on average, 90.6% of examples show that the garbled adversarial prompts actually contain meaningful words that trigger vulnerabilities in white-box LLMs.

We also show below 5 cases using Llama-2-7B-Chat as the white-box model. The words from the adversarial suffix that appeared in the response are bolded.

Thanks for responding to our rebuttal! We provide more cases for jailbreaking Claude-Haiku using our method on the harmful requests you listed, as below. Our method successfully executed jailbreaks against Claude-Haiku on all of these harmful requests.

Thanks for the feedback. Our responses to the comments are given as follows.
 

Limited Testing on Cutting-Edge Models: The paper does not test its method on the latest models, such as O1, which employ extended reasoning paths before answering. These models might require more sophisticated prompts to manipulate their internal reasoning processes, which the current method might not effectively generate. Improvement: The authors should consider testing their approach on such advanced models to evaluate the robustness of their method and potentially adapt their approach to handle more complex reasoning paths.

Answer:

The o1 series, released concurrently with our work, employs a hidden chain-of-thought strategy to generate responses to user queries. The raw chains of thought in this API are not displayed to users, posing an additional challenge for black-box jailbreak attacks. We defer the development of specific methods to overcome this defense strategy to future work.
 

Lack of System-Level Defense Testing: The paper does not address system-level defenses like Purple Llama, which classify and detect input prompts. These defenses could potentially thwart the jailbreak attempts by identifying and filtering out adversarial prompts. Improvement: Incorporating tests against system-level defenses would provide a more realistic assessment of the method's effectiveness in real-world scenarios. The authors could explore how their prompts fare against such defenses and develop strategies to evade detection.

Answer:

Following the suggestion, we adopted Llama-3-Guard as the guardrail for the victim LLMs and re-evaluated the adversarial prompts generated in the experiments presented in Table 4. Specifically, we compared our method with state-of-the-art techniques for attacking GPT and Claude series models, which were additionally guarded by Llama-3-Guard, on the HarmBench dataset. The results are shown below. The average attack success rate of our method decreased by 41.7% compared to attacking the victim models without the guardrail. However, our method still shows a great improvement in attack success rate compared to the state-of-the-arts. Our method achieves an average attack success rate of 40.1%, which is 17.4% higher than the second best method, i.e., TAP.

• Does the paper quantify the architectural and training data differences between the generator model and the translation model? If so, what are the specific data on how these differences affect attack success?
• Does the paper explore the best match between the complexity of the generator model and the complexity of the translation model? Is there evidence that the attack works best at a particular level of complexity?

Answer:

Yes, we discussed the effects of different choices for the substitute model (the model on which the initial garbled adversarial suffixes were generated via gradient-based methods) and the translator model in Section 4.3. We further detail the impact of architectural complexity and training data complexity when selecting substitute and translator models as follows. As illustrated in Figure 2, the model size of the translator model have a minor impact on attack success rates. Specifically, using Llama-2-7B-Chat and Llama-2-13B-Chat as translator models yields similar average attack success rates of 32.2% and 34.4%, respectively. The results also demonstrate that Llama-3.1-8B-Instruct achieves the highest average attack success rate when serving as the translator model. We attribute this improvement to enhanced training data complexity. Llama-3.1 models were pre-trained on a corpus of approximately 15T tokens, compared to 1.8T tokens for Llama-2 models. Consequently, Llama-3.1 models contain more knowledge and can better comprehend garbled adversarial prompts, thereby producing more effective translations. We observe a similar phenomenon with the choice of substitute models. Using Llama-3.1-8B-Instruct as the translator model, we evaluated the effects of different substitute models and reported the results in Table 6. Increasing the substitute model size slightly improved the attack success rate (32.4% vs. 38.5% average success rate for Llama-2-7B-Chat vs. Llama-2-13B-Chat). When scaling up the training data, the attack success rate significantly improved, achieving an average success rate of 53.9% for Llama-3.1-8B-Instruct. Additionally, the tokenizer with a larger vocabulary size used in Llama-3.1-8B-Instruct (128,000 compared to 32,000+ for other models in the table) also contributes to generating more informative garbled adversarial prompts through token-level optimization methods like GCG.
 

Does the paper validate the generalizability of its attack methodology across different types of LLMs (e.g., different architectures, sizes, training datasets)?

Answer:

Yes, we considered various types of LLMs for the suffix generator and translator LLMs, and we discussed their effectiveness in Sections 3.1 and 4.3. These models, including Llama-2-7B-Chat, Llama-2-13B-Chat, Vicuna-13B, Mistral-7B-Instruct, and Llama-3.1-Instruct, differ in architecture, model size, and training datasets. For the black-box victim model, as shown in Section 4.1, we evaluated seven closed-source LLMs and four open-source LLMs. These models also vary in architecture, model size, and training datasets. The results demonstrate that, with different choices of suffix generators and translator LLMs, our method can consistently enhance the transferability of garbled adversarial prompts. The superior performance in attacking various types of black-box victim models also demonstrates that the adversarial prompts generated by our method are transferable to different types of victim models.

Thanks again for your comments! We have provided responses to all your comments. If there are any remaining concerns about our paper, we are more than delighted to address them.

Finally, the choice of substitute model series has more effect on attacking different victim model. Specifically, when attacking Phi-3.5-Mini-Instruct, using Qwen2.5-14B-Instruct as the substitute model is better than using Llama-3.1-8B-Instruct (57.3% vs. 48.5% in average AASR). However, when attacking GPT-4o-mini and Gemma-2-2B-IT, the results are the opposite (34.4% vs. 43.2% and 35.8% vs. 49.3% in average AASR, respectively). This phenomenon is normal since the performance of transfer-based attacks depends on similar vulnerabilities between substitute and victim models. It is affected by various factors, such as training data and architecture. In the paper by GCG[1], they found that using Vicuna as the substitute model to attack GPT-3.5-Turbo is better because it is trained on outputs from GPT-3.5-Turbo. In [2], they fine-tune the substitute model on the output of the victim model to improve jailbreak performance. Regarding the selection of the translator model, it can be seen that using Llama-3.1-8B-Instruct is always better than Qwen2.5-14B-Instruct for attacking Phi-3.5-Mini-Instruct and GPT-4o-mini.
 

[1] Zou, Andy, et al. Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv preprint arXiv:2307.15043 (2023).
[2] Sitawarin, Chawin, et al. PAL: Proxy-Guided Black-box Attack on Large Language Models. arXiv preprint arXiv:2402.09674 (2024).

Thanks for the feedback. Our responses to the comments are given as follows.
 

This is an intuitive and empirical research work. There is no theoretical guarantee that the proposed method can always improve attack power.

Answer:

In our work, we proposed a novel method that translates the garbled adversarial prompts into human-readable adversarial prompts to enhance the black-box jailbreak attacks. We agree that establishing theoretical guarantees would strengthen our approach. However, the theoretical explanation of the transferability of adversarial prompts remains an open question, and deriving a theoretical guarantee of our method poses significant challenges. Despite the current lack of theoretical guarantees, our empirical results demonstrate our method consistently outperforms existing state-of-the-arts by a large margin, which offers significant insights and practical advancements in enhancing attack performance.
 

It is better to report the standard deviation as well to validate the significance of the reported results.

Answer:

We conducted five runs for the experiments in Table 4 and presented the average results along with the standard deviations below. The standard deviations are less than 3%, which minimally affects the results. This further validates that our method significantly outperforms all competitors in jailbreaking the victim models.

Thanks again for the comments! Is there any remaining concerns about our paper? We are more than delighted to address any concerns/questions you may have.

Thanks for the feedback. Our responses to the comments are given as follows.
 

This paper claims "we construct a fully automatic natural language adversarial prompt generation framework, without any manual work for the design of adversarial prompts, careful hyper-parameter tuning, additional model training, or the need for informative feedback of the victim model to refine the adversarial prompts. " but the proposed method adopt GCG to generate the adversarial suffix on Llama-3.1-8B-Instruct. It uses the model gradient to generate the suffix. It is not that it cannot access the model at all. Although he can migrate to other models, this part is suspected of over-claiming contributions.

Answer:

In our paper, we focus on conducting effective black-box attacks against unknown victim LLMs. When we state that our approach operates "without the need for informative feedback of the victim model to refine the adversarial prompts", we specifically mean that no such feedback is required from the black-box victim model. When we employ gradient-based methods, such as GCG, to generate the adversarial suffix on Llama-3.1-8B-Instruct, we use Llama-3.1-8B-Instruct as the white-box substitute model rather than as the victim model. In the context of black-box attacks, as discussed in related work, many methods utilize the log-probability of the victim model's output (e.g., [1,2]) or require informative outputs from the victim models to optimize adversarial prompts (e.g., [3,4]). In contrast, our method does not rely on such information, highlighting the advantage of our approach. We have revised the writing in the revised paper.
 

[1] Sitawarin, C., Mu, N., Wagner, D., & Araujo, A. (2024). Pal: Proxy-guided black-box attack on large language models. arXiv preprint arXiv:2402.09674.
[2] Andriushchenko, M., Croce, F., & Flammarion, N. (2024). Jailbreaking leading safety-aligned llms with simple adaptive attacks. arXiv preprint arXiv:2404.02151.
[3] Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., & Wong, E. (2023). Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419.
[4] Mehrotra, A., Zampetakis, M., Kassianik, P., Nelson, B., Anderson, H., Singer, Y., & Karbasi, A. (2023). Tree of attacks: Jailbreaking black-box llms automatically. arXiv preprint arXiv:2312.02119.
 

The parameter settings of the evaluation model are not given, such as the system prompt. Previous works used different system prompts to build LLM models, resulting in inconsistent jailbreak difficulty, such as GCG and AutoDAN.

Answer:

We follow the evaluation setting introduced in HarmBench. Specifically, for the system prompt, we follow the setting of HarmBench that directly inputs the adversarial prompts without custom system prompts (refer to https://github.com/centerforaisafety/HarmBench/blob/8e1604d1171fe8a48d8febecd22f600e462bdcdd/api_models.py#L49). This has been clarified in the revised paper.
 

Without code, it is impossible to assess the effectiveness of the method. For instance, when I presented GPT-4o with a translated adversarial prompt, its response was, 'I'm sorry, I can't assist with that.'

Answer:

We provided the code in Supplementary Material.
For the case of the translated adversarial prompt presented in Table 3, we tested it 5 additional times by submitting it to the GPT-4o API, following the parameter settings of HarmBench. The results demonstrate that it successfully lured GPT-4o into responding with harmful content. We present the responses from GPT-4o and evaluate whether the attacks were successful using the evaluator model provided by HarmBench, as follows:

If you are testing through the web interface, the backend model may differ from the model of API version. Below, we present another case and include a screenshot of this case in Appendix C.3 of our revised paper.

HarmBench uses a fine-tuned Llama-2-13B-chat model to compute ASR. I suggest the authors also follow the exact same evaluation pipeline introduced in HarmBench.

Answer:

We indeed followed the HarmBench evaluation pipeline and used the evaluator model provided by HarmBench for most of the experiments in our paper, except for those presented in Table 5. In the case of Table 5, since AdvPrompter has not open-sourced the additional model they trained, we directly adopted the performance reported in their paper and evaluated our method using their evaluation settings.
 

Would be great to see some qualitative examples

Answer:

In Section 3.2, we demonstrated a case of our translation method that successfully jailbreaks GPT-4o. It supports that our method can indeed translate garbled adversarial suffixes into human-readable, semantically meaningful adversarial prompts. We are pleased to show more qualitative cases. In the revised paper, we have included more translated adversarial prompts generated by our method and discussed their effectiveness in Appendix C.
 

The technical portion of this article was merely completed using prompt engineering and contains no technical innovation.

Answer:

Our work introduces an adversarial prompt "translation" framework that aims to enhance the effectiveness of jailbreak attacks and deepen their understanding. Despite simple implementation, our work presents a novel approach to understanding the jailbreaks elicited by gradient-based methods, e.g., GCG, which has been overlooked in previous work. Our method significantly enhances the performance of black-box jailbreak attacks, and revitalizes gradient-based methods in the context of black-box attacks against LLMs. Our method also brings more insights into prompt engineering. For instance, in Section 3.2, by appending rules generated by our method to the jailbreak prompt template proposed in [5], the attack success rate for jailbreaking GPT-4-Turbo increases from 60.5% to 73.5%. Additionally, we conducted a detailed ablation study on selecting substitute and translator models and incorporated an interpretation step to elicit chain-of-thought reasoning for advanced implementation. The simplicity of our prompt template is also an advantage of our method. It only describes the interpretation and translation tasks without any additional tricks (such as role-playing or emotional manipulation used in PAIR [6] and TAP [7]), which strengthen the effectiveness of our adversarial prompt translation framework. We believe the contributions of our work benefit the community and distinguish our work from existing studies.
 

[5] Andriushchenko, M., Croce, F., & Flammarion, N. (2024). Jailbreaking leading safety-aligned llms with simple adaptive attacks. arXiv preprint arXiv:2404.02151.
[6] Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., & Wong, E. (2023). Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419.
[7] Mehrotra, A., Zampetakis, M., Kassianik, P., Nelson, B., Anderson, H., Singer, Y., & Karbasi, A. (2023). Tree of attacks: Jailbreaking black-box llms automatically. arXiv preprint arXiv:2312.02119.

Thanks again for your reply. Our responses are as follows.
 

The method proposed in the article is too simple, and there is no major improvement in technology. It is still a prompt engineering.

Answer:

Firstly, we reiterate that our method is not solely based on prompt engineering. The complete process involves generating garbled adversarial prompts followed by translation using chain-of-thought reasoning through prompting. Additionally, we conduct ablation studies to guide the selection of substitute and translator models.

Secondly, our objective is to enhance technology for jailbreak attacks against LLMs, rather than to advance the study of prompting techniques. Our method—previously unproposed in the literature—significantly improves jailbreak performance, demonstrating innovative advancements in the field of black-box jailbreak attacks. Furthermore, our framework, which employs gradient-based optimization followed by translation, offers additional insights into prompt engineering.

Finally, the simplicity of our implementation, achieved through the use of prompting, offers several advantages: it is training-free, fast, computationally efficient, does not require informative feedback from the black-box victim model (such as responses, gradients, or probabilities), and does not rely on manual tricks (due to the straightforward design of prompts). We believe that the simplicity of implementation is a significant strength of our work. The simple yet effective method poses a more severe threat to the safe application of LLMs.
 

The model proposed in the article does not strictly use the system prompt. For example, the Llama-2-7B-Chat model in the provided code does not set the system prompt ("sep": ("[INST] ", " [/INST] "). The system prompt of Llama-2-7B-Chat should be ...

Answer:

Firstly, the provided code is for generating adversarial prompts. The attackers can freely configure the system prompts of white-box models.

Secondly, the system prompt you provided has been removed in the latest version of Llama-2-Chat models (refer to https://github.com/meta-llama/llama/pull/626). The prompt template we used (i.e., <s>[INST] {user} [/INST] {assistant} </s>) is consistent with the official template. Below, we provide a code snippet to verify this. It will print <s>[INST] {user} [/INST] {assistant} </s>, which is identical to our setting.

model_path = "meta-llama/Llama-2-7b-chat-hf"
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True, use_fast=False, 
                                          token="token")
messages = [
  {"role": "user", "content": "{user}"},
  {"role": "assistant", "content": "{assistant}"}
]
inputs = tokenizer.apply_chat_template(messages, tokenize=False)
print(inputs)

Finally, we evaluated our method on attacking Llama-2-Chat models in the comparison with AutoDAN and AdvPrompter (in Table 5). In this experiment, we followed AdvPrompter’s setting, which does not include the system prompt (refer to https://github.com/facebookresearch/advprompter/blob/802a500c91f1dcd7c8b76869d3e39bf8e40ed7d7/conf/target_llm/llama2_chat.yaml#L9).

Thanks for the reply. I think it is unreasonable not to add system prompt. LLama-7b-chat is widely used to evaluate jailbreak performance because of its excellent ability to refuse to answer harmful questions (such as GCG). After removing the system prompt, LLama-7b-chat will become very easy to jailbreak. The method proposed in this article needs to be re-evaluated with system prompt, and even needs to be evaluated on adversarial fine-tuned models (such as Zephyr-R2D2). In addition, the method in this article is really just prompt engineering, without theoretical support or deeper insights. So I will not raise the score to a positive score.

We conducted experiments to evaluate our method for attacking Llama-7B-Chat and Llama-13B-Chat with the deprecated system prompt (i.e., "You are a helpful, respectful, and honest assistant. Always answer ...") and against Zephyr-R2D2. The results are shown below, demonstrating that our method still outperforms the competitors by a large margin.

Additionally, we present some cases for attacking these models as shown below.

For the innovation of our work, we would like to reiterate that we introduced a novel method for black-box jailbreak attacks. This method offers an innovative technique for discovering effective adversarial prompts, advances the understanding of jailbreak attacks, and significantly improves the performance of black-box jailbreak attacks, thereby contributing to the research on jailbreak attacks.