When LLM Meets DRL: Advancing Jailbreaking Efficiency via DRL-guided Search

The reviewer points out that all experiments are conducted on a 300-sample test set, which has the same distribution as the training set, making it difficult to prove the method's robustness and effectiveness.

We thank the reviewer for the question. When splitting the training and testing set, we avoided putting similar templates in both training and testing to encourage enough differences and avoid data leakage. To further demonstrate the generalizability of our method, we conducted an additional experiment. We first trained our agents on the training set, following the same setup in Section 4. Then we tested our agents on the other two harmful question sets. One is from our baseline: GPTFUZZER, where they construct 100 questions from scratch. The other is called MaliciousInstruct from [2], where they claim it is a different dataset from AdvBench that also contains 100 harmful questions. We select Llama2-7b-chat as the target model and GPT-3.5-Turbo as the helper model, and the results are in Table 9 in the submitted pdf file.

We can see that after applying our RL agent on the other two testing sets, the performance even gets improved, compared to the original AdvBench testing set. Furthermore, we showed the transferability of our agents across different models, which also demonstrates the generalizability and robustness of our approach.

The reviewer asks how the authors determined the chosen action space.

We thank the reviewer for the question. When designing the action space $\mathcal{A}$ of an RL agent for our jailbreaking attack, the requirements of the action space $\mathcal{A}$ are two. First, it should enable diverse and substantial changes to the prompts. Second, $\mathcal{A}$ should not be ultra-large, which will make it difficult for the agent to learn an effective policy.

Before designing our action space by selecting different mutators, we made an initial trial of designing the action as selecting different tokens from the whole vocabulary, and the state is the current prompt. We refer to this design as “token-level RL” and the details are in Section D.4 in Appendix. Our experiment results demonstrated that this token-level RL design cannot generate any effective jailbreaking prompts. The reason is that the action space is too huge for the agent, which is the same as the vocabulary size (about 30,000). Thus, considering the maximum length of the generated jailbreaking suffix as N, the possible combination is 30,000^N, which is unrealistic for an agent to learn some effective policy.

Then, we took a different avenue and considered mutating the input prompts at the sentence level. We selected some common ways of mutating a sentence as our actions that enable enough changes to the input while constraining the total action space. Note that our framework is flexible and can incorporate more actions in the future.

[1] Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. Yu et al., arXiv 2024.

[2] Catastrophic Jailbreak of Open-Source LLMs via Exploiting Generation. Huang et al., ICLR 2024.

The reviewer suggests improving the paper's organization to enhance readability. For example, Section 2 lacks discussion on the background of DRL and related work on jailbreaking.

We thank the reviewer for the suggestions. Due to the space limit, we did not add a background section for DRL. Instead, we introduced the key formulations of our DRL system in Section 3. We will add a background of DRL (in the appendix) in the next version. Regarding related works on jailbreaking, we would like to kindly point out that we have included a comprehensive literature review in Section 2, with a special focus on black-box attacks.

To the best of our knowledge, there are no existing works that specifically apply DRL to jailbreaking attacks, although other works apply RL to attack LLMs with different attack goals and threat models, as discussed in Section 2. All these works used RL to directly generate tokens, which is similar to the token-level action baseline in our ablation study (denoted as Token in Figure 2 in our paper). As we demonstrated in our experiments, the action space is too huge for this type of method to work in jailbreaking problems.

The reviewer also suggests that the plausibility of the reference answer from an unaligned LLM needs further justification. The reviewer questions what would occur if both the target LLM and this model refuse to answer a harmful query.

We thank the reviewer for their questions. First, we would like to clarify that we utilize an uncensored version of the Vicuna-7b model instead of a weekly aligned Vicuna-13b to obtain reference answers. This uncensored Vicuna-7b model has not been fine-tuned with human preferences and lacks guardrails, ensuring it responds to user prompts irrespective of potential harmfulness. We also manually double-check all the reference answers and ensure they are indeed answering the questions.

Additionally, we assume that even if there are any questions that are rejected or deemed irrelevant, the RL agent can still learn an effective strategy about how to select the mutators during the interaction with other questions. To test this assumption, we randomly marked 10% and 20% of the reference answers as unavailable, by setting them as ''I’m sorry I cannot assist with this request''. This also mimics the case when the unaligned model refuses to answer the question. Then we trained our RL agent on Llama2-7b-chat. The results in Table 4 in the submitted pdf demonstrate that even though some of the reference answers are unavailable, our method still achieves good performance on jailbreaking the model.

We will address these points in our next version and we will publish all reference answers used in our experiments upon acceptance.

The reviewer raises concern that some technical details of the method are missing, e.g., details of mutators, and the training/inference time required for our method.

We thank the reviewer for the question. We would like to kindly point out that in Section 3, we detail the five mutators we use, including their name and how we conduct the mutation. We also include the LLM that we use to perform the mutation. The prompts for each mutator are shown in Tab. 4 in our paper. Regarding the training and inference time, we consider the total time (training + inference) as one of our efficiency metrics, as we discussed in Section 4, and the results are in Tab. 8 in our paper. We also separately report the training and inference time of our method in Table 5 in the submitted pdf. We will include these results in our next version.

The reviewer comments that the experimental results show our method has limited improvement over existing methods.

We thank the reviewer for the question. In our experiments, we conducted a comprehensive comparison of our method against five SOTA baselines. These comparisons and metrics were carefully designed to fairly evaluate the effectiveness of our methods.

To address the reviewer’s concern specifically:

Regarding the baselines we choose, we carefully select the four SOTA black-box jailbreaking attacks that cover the genetic method-based attacks and in-context learning-based attacks. We also include the representative white-box attack: GCG. Then for the evaluation metric, we leverage four metrics for a comprehensive attack effectiveness evaluation while existing works usually only leverage one or two. Among those metrics, we consider that a higher GPT-Judge score is enough to show the advantages of our methods over other attacks, as GPT-Judge has a higher correlation with human annotations, as demonstrated by [1]. We also directly adopt their judging prompt to ensure a fair comparison, it also guarantees that we are not designing our attack specifically our own designed prompt.

Our method achieved a significant improvement in the GPT-Judge score, which is a critical metric for assessing the effectiveness of the attacks. Specifically, our approach showed about 40% improvement on Llama2-70b-chat over the best-performing baseline. This is a substantial enhancement, demonstrating the efficacy of our proposed attack. Our attack also achieves the highest GPT-Judge score across all six target LLMs, compared to baseline methods. We also have provided detailed tables (Tab.1 and Tab.6 in our paper) that explicitly illustrate these advancements. We also performed a thorough ablation study to validate the core designs of RLbreaker.

[1] COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability. Guo et al., ICML 2024.

Thanks for the rebuttal, your efforts are truly appreciated. I've raised my rating to 5. Some comments:

• I know that there are no existing works that specifically apply DRL to jailbreaking attacks (see strengths); what I requested here is to discuss how (deep) reinforcement learning has been used to attack ML models / LLMs. Such connections can help the readers better understand to what extent these techniques have been leveraged in the adversarial ML area.
• The limitation on the requirement of manually double-checking all the reference answers is a limitation that needs to be explicitly acknowledged.
• It would be great to evaluate stronger attack/defense baselines, like the demonstration-based in-context attack/defense (https://arxiv.org/abs/2310.06387).

The reviewer suggests that GPT-3.5 Turbo may be outdated for black-box evaluation and recommends using GPT-4 instead.

We thank the reviewer for this suggestion. We evaluate our attack on GPT-4, following the same experiment setup in Section 4. We select the latest GPT-4o-mini (07/18/2024) as the target model and limit the total queries to 10,000. For baselines, we select GPTFUZZER as its performance is better than the other black-box attacks. The results in Table 6 in the submitted pdf clearly demonstrate RLbreaker’s superior ability to bypass strong alignment compared with baseline. The reason why we did not do a large-scale experiment on GPT-4 is in consideration of cost.

The reviewer also raises concern about the results of GPT-Judge, noting an inconsistent scale compared to Sim.

We thank the reviewer for pointing this out. We believe this is because the effective value range for Sim. is smaller than GPT-Judge. As we observed during the experiments, Sim. gives at least a 0.6 score even for a rejected answer. However, the GPT-Judge metric spans a full range from [0, 1]. We chose different text encoders for GPT-Judge and got consistent observation, as we demonstrated in our ablation study in Section 4. Furthermore, we use the bge-large-en-v1.5 from Hugging Face to convert text into embeddings for cosine similarity computations. As noted by the model developers, the typical similarity distribution for this model falls within [0.6, 1], which also verifies our previous observation. Across Tables 1, 2, and 3 in Section 4, we observe that the Sim. for RLbreaker and other baselines exceed 0.6.

We also want to note that as discussed in [1], GPT-Judge evaluates responses in a more comprehensive manner, considering contextual relevance, coherence, and the capacity to meaningfully answer the posed questions. The significant improvements highlighted by GPT-Judge demonstrate that our RL-guided approach produces responses that are more contextually relevant and accurate. It is also important to note that we have not tailored the GPT-Judge prompt specifically for our attack; instead, we employ a prompt directly sourced from existing published work. This approach ensures the generalizability of our method.

[1] COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability. Guo et al., ICML 2024.

The reviewer raises concerns about the instability of training the RL agent, asking how many training runs were conducted to report the results in Tab. 1 and what the variance is.

We thank the reviewer for this question. Our training runs for the RL agent are constrained by a limit of 10,000 queries to the target model. Specifically, we set the number of forward steps in the environment as 16, i.e., we will update our agent after we forward 16 steps in the environment. We will use the selected trajectories to update the agent with our customized PPO algorithm and the training epoch is 4, thus, the total training runs of our agent is 2500 times. We will include all training details in our appendix in the next version.

To understand the variance of our training process, we trained the agent five times using different random seeds on the target model llama2-7b-chat with GPT-3.5-turbo as the helper model. The results in Table 7 and Figure 1 in the submitted pdf show low variance during training and testing across these seeds. We believe the reason is that we constrain the action space. Instead of letting the model choose a token within the vast token space, we let the model mutate the input templates, which gives a much smaller search space and thus more stable agents.

The reviewer points out that under Sim., our proposed method sometimes underperforms compared to previous works, such as on Mixtral in Table 1, with similar observations made in Tables 2 and 3.

We thank the reviewer for pointing this out. We suspect that this is because the Mixtral model answers the questions in a way different from the reference answers (given by the Vicuna model). When comparing the answers given by the Mixtral model with reference answers, it is possible that some answers are given a low Sim. because Mixtral answers a question in a very different way from the reference answer. However, when using GPT-Judge, the attack success rate is higher.

As we have discussed in Section 4, Sim. indeed can introduce false negatives when the target model answers a question in a very different way from the reference answer. This is the reason why we used multiple metrics in the evaluation. However, we still use Sim. during the training mainly because it is more efficient than GPT-Judge and introduces fewer false negatives than keyword matching.

The reviewer raises concerns about the efficiency of our proposed method.

We thank the reviewer for the question. Tab. 8 in our paper shows that RLbreaker is more efficient than AutoDAN and GCG and comparable to GPTFUZZER. Although our method is slower than PAIR and Cipher, these two methods are way less effective than ours. We designed an additional experiment to show the efficiency of our method over those baselines. To make these two methods have comparable performance with our method, we need to significantly increase the total number of queries. Specifically, for Cipher, we executed their jailbreaking prompts up to 50 times per question, considering it as a success if any trial resulted in a successful jailbreak. This led to 16,000 queries at most. For PAIR, we set their two key hyper-parameters: number of iterations and number of streams as 3 and 20 separately, resulting in a maximum of 19,200 queries. Results in Table 8 in the submitted pdf file show that these two methods are actually less efficient than ours if we aim to achieve a similar attack effectiveness.

The reviewer raised concerns regarding the N/A values from GCG for GPT-3.5 Turbo in Tab.1. It could be run on other LLM and obtain the adversarial prompts to do a transfer attack.

We thank the reviewer for the suggestion. Following the suggestions, we added an experiment to test GCG's performance on GPT-3.5 Turbo. We followed the original GCG setup and used two different models (Vicuna-7b, Vicuna-13b) as the source models to generate the jailbreaking suffixes. Then, we used these suffixes to attack GPT-3.5 Turbo. We followed the same setup in our Section 4 and set the upper bound for the total query times as 10,000. The results in Table 1 in the submitted pdf indicate that when making a transfer attack on GPT-3.5 Turbo, GCG cannot outperform RLbreaker.

The reviewer then raised concerns regarding why PAIR’s ASR on GPT-3.5 Turbo is lower than what is reported in the original paper and why its ASR on Mixtral is only 15%.

Thanks for the question. The main reason for a low PAIR performance is that we set an upper bound for the total query times as 10,000 for the entire attack process. The reason is to enable an apple-to-apple comparison for all methods. Given that some methods have a training process but others do not. For PAIR, this limit maps to the action that we set the 10,000 queries for all 320 questions (because PAIR does not have a training process. So all the queries are directly applied to the testing phase). Following this constraint, we set their two key hyper-parameters: number of iterations and number of streams (the number of parallel conversations) as 5 and 6 separately, while they set the number of streams as 20 in the original paper and number of iterations as 5. PAIR also mentioned that the total number of queries had a significant influence on the performance of jailbreaking attacks. Thus this reduction in parallel streams is a primary factor contributing to the lower ASR observed in our results compared to its original paper. Second, we followed [1] and used a different way of writing the GPT-Judge prompt from PAIR.

We added a new experiment that follows the same setup with the PAIR, setting the number of iterations as 5 and the number of streams as 20, which will lead to 32,000 upper bound for the total number of queries. We then ran PAIR and RLbreaker with the new upper bound. We selected Mixtral-8*7B and GPT-3.5-Turbo as the target model and we reported the ASR measured by our GPT-Judge metric and JUDGE function proposed by PAIR. Results in Table 2 in the submitted pdf file demonstrate that RLbreaker still outperforms PAIR across two different evaluation metrics.

[1] COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability. Guo et al., ICML 2024.

The reviewer asked why the ASR using GPT judge is higher when the perplexity filter is enabled.

We thank the reviewer for the question. We would like to kindly point out that for Table 1 in our paper, we are evaluating Llama2-70b-chat, while for Tab. 2, we focus on Llama2-7b-chat. To address the reviewer's question further, in Table 3 in the submitted pdf, we showed the ASR for Llama2-70b-chat model specifically when a perplexity filter is applied. The result aligns with Tab. 2 in Section 4, demonstrating RLbreaker’s resiliency against the perplexity defense.

The reviewer also asked why the random nature of genetic algorithms significantly limits the effectiveness of these attacks

We thank the reviewer for raising this insightful question. As we briefly discussed in Section 3, the limitations of genetic algorithms in developing jailbreaking attacks are two-fold. Inefficiency in Search Process: Stochastic search methods, including genetic algorithms, initiate with a randomly chosen initial region and explore this region randomly before moving to other areas. This process involves random mutation and selection, which leads to a highly inefficient search process. As demonstrated in the grid search example in Appendix B.2, stochastic search requires at least three times more grid visits compared to guided search, highlighting its inefficiency. Constraints of Random Mutation: In the context of jailbreaking attacks, existing methods that employ genetic algorithms iteratively generate new prompts by randomly selecting mutators to modify the current prompts. This randomness in mutator selection significantly constrains the search efficacy, as it often directs computational resources toward less promising areas of the search space. This approach is particularly ineffective in the expansive search spaces common in jailbreaking scenarios. Furthermore, after each selection of the mutators, the absence of informative feedback means that those genetic algorithm-based attacks cannot effectively utilize prior knowledge or feedback. In contrast, DRL-guided searches benefit from RL agents that prioritize actions leading to successful outcomes, driven by the accumulation of rewards.

As a result, the random nature of genetic algorithms limits their effectiveness in jailbreaking attacks primarily due to their inefficient exploration of the search space and the significant computational overhead involved in randomly selecting mutators. This inefficiency is especially problematic in large search spaces, leading to constrained search efficacy and reduced overall effectiveness.

We will add more discussion in the next version.

Other minor issues. We thank the reviewer’s suggestion and we will increase the font size of Table 2 and Table 3 in our next version.