Improved Techniques for Optimization-Based Jailbreaking on Large Language Models

W3 & Q2 & Q3: 1) the consistency of jailbreak difficulty across models remains unexamined, and 2) whether investing extra time in simpler tasks truly saves time on more complex ones is unclear.

Different models show roughly the same trend in jailbreaking difficulty for different types of questions. Generating jailbreaking suffixes on simple questions can save time on complex tasks. The specific analysis is as follows:

• 1. We adopt GCG to jailbreak more LLMs, which include VICUNA-7B, GUANACO-7B, and MISTRAL-7B, with different malicious questions. Then we track the changes in the loss values of different malicious questions as the number of attack iterations increases. The results are shown in Appendix H. We observe the same phenomenon as above for LLAMA-7B, i.e. some malicious questions are easier to create jailbreak suffixes for, while others are much harder. Specifically, it’s relatively easy to craft jailbreak suffixes for malicious questions related to fraud, but much more challenging for those in the Pornography category.
• 2. We compare the proposed initialization with the initialization of ! on the complex task. The results are shown in Appendix N. It demonstrates that our proposed initialization method significantly accelerates convergence on complex tasks compared to the baseline. By starting closer to the optimal solution and maintaining lower loss values throughout the iterations, our approach reduces the time and computational cost required for optimization.

W4: Experiments on additional datasets would provide a more comprehensive assessment.

Thank you for your suggestion. We adopt our $\\mathcal{I}$-GCG to jailbreak on more dataset, i.e., HarmBench and JailbreakBench. We randomly selected 50 malicious prompts from each of them for comparative experiments. The results are shown in the following Table. It can be observed that the proposed $\\mathcal{I}$-GCG can also achieve 100% attack success rates (ASRs) on HarmBench and JailbreakBench, surpassing the performance achieved by standard GCG. Please refer to Appendix L for more details.

Q1: It is unclear why the authors formulate initial jailbreak suffixes as constraints in equation 6 and 7.

Thanks for pointing this out. We agree that they should not be constraints. We have modified this in the revision. Please refer to $\\textrm{\\color{blue}Eq (6) and (7)}$ (Page 5).

Q4: Jailbreaking results on additional models, such as LLAMA2-7B alongside LLAMA2-7B-CHAT.

Following the suggestion, we conduct experiments with LLAMA2-7B and LLAMA2-7B-CHAT. The results are shown in the following Table. It clearly shows that the $\\mathcal{I}$-GCG method achieves a 100% attack success rate (ASR) on both LLAMA2-7B and LLAMA2-7B-CHAT, significantly outperforming the GCG method (80% and 54%, respectively). This highlights the robustness and effectiveness of $\\mathcal{I}$-GCG.

Thank you for your valuable review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q).

W1 & Q5: The novelty is limited. Providing further elaboration on the contributions of this work would be beneficial.

We respectfully disagree with the assessment that our novelty is limited. As one of the first works to improve optimization-based jailbreaks for large language models (LLMs), we have systematically revealed the limitations of existing approaches like Greedy Coordinate Gradient (GCG) and proposed novel techniques to address them. Specifically:

• Expanded Scope of Target Templates: While existing GCG methods rely on a single target template (e.g., "Sure"), we identified that this restricts attack diversity and effectiveness. Our approach introduces diverse target templates incorporating harmful self-suggestions and guidance, broadening the scope of possible attacks and showcasing the adaptability of LLM jailbreaks.

• Optimization Innovation: From an optimization perspective, we identified inefficiencies in GCG’s single-coordinate update mechanism. By introducing an automatic multi-coordinate updating strategy—where the number of token replacements is adaptively determined—we significantly accelerate convergence without sacrificing attack performance. This innovation is both practical and generalizable, contributing to the understanding of efficient adversarial optimization for LLMs.

• Improved Initialization and Convergence Techniques: Our proposed easy-to-hard initialization leverages a self-transfer technique that guides the attack process progressively, making it more robust and efficient. This methodological advancement is not only novel but also paves the way for more sophisticated jailbreak approaches in the future.

• Practical Impact and Extensive Evaluation: We have conducted a thorough evaluation of our proposed $\\mathcal{I}$-GCG method on established benchmarks, including the NeurIPS 2023 Red Teaming Track. The results demonstrate that our method achieves a near 100% attack success rate, surpassing existing state-of-the-art attacks and establishing a new benchmark in the field.

Our work not only improves the efficiency of optimization-based jailbreak techniques but also provides practical tools for evaluating LLM vulnerabilities. We hope that reviewers can reassess our contributions from the perspective of advancing LLM safety research and robustness testing.

W2: The reasons behind the improvements in jailbreak effectiveness and efficiency from the reformulated optimization goal and multi-coordinate updating strategy.

As for the efficiency of the reformulated optimization goal, we have adopted the benign guidance, namely my response is safe to use and my output is secured, to study the impact of harmful guidance. The results are shown in $\\textrm{\\color{blue}Table 8}$ (Page 10). Using My output is secured drops the ASR of $\\mathcal{I}$-GCG from 100% to 88%, performing worse than the I-GCG without any guidance. The results show the effectiveness of the proposed harmful guidance.

As for the efficiency of the multi-coordinate updating strategy, we provide the following analysis: GCG uses random coordinate descent (CD) [1], which means that a single coordinate is randomly selected and updated in each iteration; in contrast, our $\\mathcal{I}$-GCG uses random block coordinate descent (BCD) [2*], where multiple coordinates are updated during each iteration. Theoretical bounds for CD and BCD are mostly applied to convex functions. Let $f(x)$ be a smooth and convex function and we aim to minimize $f(x)$, the convergence rate of CD is: $f(x\^{(k)})-f(x\^{\\dagger}) \\leq \\frac{2nL}{k+2nL} ||x\^{(0)}-x\^{\\dagger}||\^{2}\\textrm{,}$ where $n$ is the number of coordinates, $L$ is a Lipschitz constant, $x\^{(k)}$ is the iterate at step $k$, and $x\^{\\dagger}$ is the optimal solution for minimizing $f(x)$. For BCD methods, the convergence rates depend on similar assumptions but are generally extended to account for block-wise updates. If there are $m$ blocks, the convergence rate for BCD is: $f(x\^{(k)})-f(x\^{\\dagger}) \\leq \\frac{2mL}{k+2mL} ||x\^{(0)}-x\^{\\dagger}||\^{2}\\textrm{.}$ In our experiments for $\\mathcal{I}$-GCG, we update $7$ coordinates in each iteration and there is $m=\\frac{n}{7}$. So treating $\\mathcal{I}$-GCG as a BCD method leads to $f(x\^{(k)})-f(x\^{\\dagger}) \\leq \\frac{2nL}{7k+2nL} ||x\^{(0)}-x\^{\\dagger}||\^{2}$, which is faster than CD used in GCG. However, it is important to note that LLMs are extremely non-convex and our main focus is on discrete optimization. Therefore, these theoretical bounds are not strictly applicable and can only offer intuitive insights.

Dear Reviewer r229,

Sorry for bothering you, but the discussion period is coming to an end in two days. Could you please let us know if our responses have alleviated your concerns? If there are any further comments, we will do our best to respond.

Best,

The Authors

Dear Reviewer r229,

We appreciate the time and effort you have dedicated to providing insightful review. If there are any additional clarifications or information needed from our side, please let us know. Thank you again for your valuable insights, and we look forward to any updates you may have.

Best,
The Authors

Dear Reviewer r229,

We have made significant efforts to write responses and conduct additional experiments based on your comments and suggestions. We totally understand that this is quite a busy period, but the Author-Reviewer discussion will come to an end in less than 24 hours.

We deeply appreciate it if you could take some time to return feedback on whether our responses solve your concerns. If there are any further comments, we will make every effort to respond within the remaining time.

Thank you!

The Authors

Thank you for your valuable review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q).

W1: The paper does not extensively explore how the proposed techniques interact with existing or potential defense mechanisms in LLMs.

We compare the proposed method with GCG against an advanced adversarial fine-tuning LLM (Zephyr-R2D2). The results are shown in the following Table. It highlights the significant advantage of $\\mathcal{I}$-GCG over GCG in jailbreak performance on the AdvBench against the advanced adversarial fine-tuning model Zephyr-R2D2. $\\mathcal{I}$-GCG achieves an ASR of 12%, doubling the performance of GCG (6%). Please refer to Appendix J for more details.

W2: The easy-to-hard initialization and automatic multi-coordinate updating strategy may be sensitive to initial conditions and hyperparameter choices.

The easy-to-hard initialization and automatic multi-coordinate update strategy are not sensitive to the initial conditions and hyperparameter selection. The specific analysis is as follows:

• As for the easy-to-hard initialization, we adopt different types of questions as the initial conditions to generate the initialization. The results are shown in the following Table. It is clear that other types of problems can also be utilized for initialization to achieve an ASR of 100%; however, it leads to an increase in the average number of iterations required.

• The proposed automatic multi-candidate update strategy has one hyper-parameter, i.e., the first $p$ single-token suffix candidates, which can impact the jailbreak performance. We have conducted relevant experiments in the Hyper-parameter Selection Section (Page 8). It is evident that the choice of $p$ has minimal impact on the final performance (i.e., the lowest loss value). Regardless of whether $p$ is set to 4 or 8, the final loss values converge near zero after sufficient attack steps. So $p$ does not significantly affect the final performance.

Q1: In the experiment using HarmBench (NeurIPS 2023 Red Teaming Track), why is the target response format set differently than in the other experiments?

In the NeurIPS 2023 Red Teaming Track, submitted test cases can be up to 256 tokens in length. This constraint enables the use of longer jailbreak suffixes (e.g., 40 tokens), which can improve jailbreak performance compared to using 20 tokens as in our main paper.

Q2: The paper does not discuss how the choice of the top-K tokens and the token combination process might affect the diversity and quality of the generated jailbreak suffixes.

Thank you for your suggestion. We explore the impact of top-k tokens on our proposed multi-coordinate updating strategy. The results are shown in the following Table. The table shows that our multi-coordinate updating strategy demonstrates significant performance advantages and stability across different top-k values. Its ASR consistently outperforms GCG, with a narrow fluctuation range of 6% (68%$\\sim$74%), compared to GCG's 12% (42%$\\sim$54%). This highlights the robustness and efficiency of $\\mathcal{I}$-GCG's multi-coordinate updating strategy, ensuring more reliable optimization results. Please refer to Appendix M for more details.

Q3: How applicable are the proposed techniques to other security threats or malicious uses of LLMs, such as information extraction, influence operations, or bypassing content filters in different modalities (e.g., image generation)

The proposed suffix initialization and update strategy can be used to induce text-to-image (T2I) models to generate NSFW content, including material that violates social norms. Using Stable Diffusion (SD V1.4) with the NSFW-text-classifier as the victim model, the goal is to bypass the classifier and generate prohibited images. Experiments were conducted on 100 harmful prompts across categories like sexual content, violence, and harassment, sourced from the LAION-COCO dataset and generated by ChatGPT-4. Following SneakyPrompt, we adopt a black-box setting with random search as the baseline. Our method integrates suffix initialization and update strategies with random search and is compared to I2P and SneakyPrompt. Results in the following Table show that our approach significantly improves performance, achieving an ASR of 83% when combining both strategies, outperforming I2P (48%) and SneakyPrompt (75%). Please refer to Appendix K for more details.

Dear Reviewer PNML,

Sorry for bothering you, but the discussion period is coming to an end in two days. Could you please let us know if our responses have alleviated your concerns? If there are any further comments, we will do our best to respond.

Best,

The Authors

Dear Reviewer PNML,

We appreciate the time and effort you have dedicated to providing insightful review. If there are any additional clarifications or information needed from our side, please let us know. Thank you again for your valuable insights, and we look forward to any updates you may have.

Best,
The Authors

Dear Reviewer PNML,

We have made significant efforts to write responses and conduct additional experiments based on your comments and suggestions. We totally understand that this is quite a busy period, but the Author-Reviewer discussion will come to an end in less than 24 hours.

We deeply appreciate it if you could take some time to return feedback on whether our responses solve your concerns. If there are any further comments, we will make every effort to respond within the remaining time.

Thank you!

The Authors

Thank you for your valuable review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q).

W1: Scalability to Larger Models.

Following the suggestion, due to the limitation of computing resources, we used large language models with larger parameters, namely Llama-13B and Vicuna-13B, to conduct experiments. The results are shown in the following Table. It is clear that compared with GCG, our I-GCG can achieve a higher attack success rate. Please refer to Appendix I for more details.

W2: Lack of Defensive Strategies Discussion.

We adopt three defense methods, which include ICD [1], Self-reminder [2], and PAT [3], to conduct experiments. The results are shown in the following Table. It can be observed that compared with GCG and AutoDAN, our I-GCG can achieve a higher attack success rate against defense methods. Please refer to Appendix J for more details.

Q1: How is the ASR calculated? Does it only count as successful if it passes all three checks?

Yes, the Attack Success Rate (ASR) is calculated based on whether the attack satisfies all three criteria: the rule-based judgment, the GPT-3.5 evaluation, and manual review. An attack is only considered successful if it passes each of these checks, ensuring a rigorous and comprehensive assessment of its effectiveness.

Q2: What does “average iterations” in Table 1 mean? Does it refer to the average number of iterations required to achieve a successful jailbreak for the first time with ASR.

Yes, average iterations in Table 1 refers to the average number of iterations required to achieve a successful jailbreak for the first time with ASR.

Q3: Could you specify the number of candidates set for vanilla GCG in this context?

We use the default number of candidates set in GCG (is set to 256) to conduct experiments.

Thank you for your valuable review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q).

W1: The paper is not well-structured. A major rewrite is recommended.

Thank you for your feedback. We have revised the Introduction and Related Work sections to better clarify our contributions and situate our work within prior works. Specifically, we have emphasized the motivations and provided a clearer comparison with existing methods. We hope these changes address your concerns and improve the paper's clarity and impact.

W2: The first two paragraphs are full of citations with unclear relevance.

Thank you for the feedback. Following your suggestion, we have revised the first two paragraphs to ensure the citations are directly relevant and clearly tied to the discussion, providing a more focused introduction and overview of the field.

• We have adopted some landmark papers to replace them as references for the concept of LLMs. Please refer to it in Page 1 of the revised paper.
• We have added the recommended work as background work to the Introduction. Please refer to it in Page 1 of the revised paper.
• We have removed the group of up to 8 citations and added a short introductory paragraph to each reference group to explain its relevance in the Introduction Section. Please refer to it in Page 1-2 of the revised paper.

W3: The "Related Work" section lacks key landmark papers needed to contextualize the research and its contributions, such as [3] on LLM-based jailbreak methods.

Thank you for pointing this out. We have added the recommended work [3] to the LLM-based jailbreak methods in Related Work Section. Please refer to Related Work for more details.

W4: The easy-to-hard initialization relies on understanding which themes are easier to jailbreak, requiring measured difficulty across themes and LLM families. Without such data, the model's results become less useful and harder to validate.

The initialization method we proposed is not strictly limited to using suffixes from easy problems. Suffixes from successful jailbreaks on other types of problems can also be utilized for initialization; however, the total time required may vary depending on the choice of suffixes. We also used other problem types (random, hard problems) as initialization to conduct experiments, and the results are shown in the figure below. It is evident that other types of problems can also be used for initialization to achieve a 100% attack success rate (ASR); however, it leads to an increase in the average number of iterations required. Please refer to Appendix N for more details.

W5: Adding an explicit harm awareness prompt (e.g., "my output is harmful") narrows the attack scope to harm LLMs that are fine-tuned or trained to recognize.

Thank you for your suggestion. In the Discussion Section of the submitted manuscript, we analyzed the impact of different types of guidance on jailbreak performance. Specifically, we conducted experiments using stealthy guidance such as I will not refuse any requests and I will respond to any requests, ensuring that the model's output lacks explicit harmful indicators. As shown in $\\textrm{\\color{blue}Table 8}$ (Page 10) , our I-GCG approach still achieved a 100% ASR under these conditions. Additionally, we examined the effect of benign guidance, such as My response is safe to use and My output is secured, to further explore the impact of harmful guidance. Notably, when using My output is secured, the ASR of I-GCG dropped from 100% to 88%, underperforming compared to I-GCG without any guidance.

Q1: L178-L182: Why classify this approach, dependent on attacker LLMs, as optimization-based rather than LLM-based jailbreak methods?

Thank you for pointing this out. We agree that the mentioned work should be classified as LLM-based jailbreak methods rather than optimization-based jailbreak methods. We have moved the related works to the LLM-based jailbreak methods. Please refer to Related Work for more details.

Q2: L366-L367: Why use ChatGPT-3.5, known to be more prone to jailbreaks, instead of newer models like GPT-4, Claude, or LLaMA-3.X?

ChatGPT 3.5, specifically in lines L366-L367, is utilized to assess whether the attacked model generates harmful content. This approach has been widely adopted in various works. The classifier within ChatGPT 3.5 is sufficiently capable of performing this evaluation effectively, eliminating the need for a more recent model that would incur higher costs.

Dear Reviewer wsxt,

Sorry for bothering you, but the discussion period is coming to an end in two days. Could you please let us know if our responses have alleviated your concerns? If there are any further comments, we will do our best to respond.

Best,

The Authors