LIAR: Leveraging Inverse Alignment to Jailbreak LLMs in Seconds

Question 2: In the answer to Q2, the authors are trying to guarantee the sub-optimality of the proposed method of LIAR in solving Eq. (4). Why the KL divergence is neglected in the definition of $\Delta_\text{sub-gap}$ ?

Response to Question 2: Thank you for this point. In our analysis, we specifically focused on characterizing the suboptimality of the proposed LIAR method in terms of maximizing the unsafe reward $R_u$ only. This is because our primary goal was to evaluate the gap between the proposed method and the best possible unsafe model. By omitting the KL term, we aimed to make the suboptimality analysis more interpretable and highlight the alignment-specific aspects of our method.

However, we concede that looking at the KL term is also important. Interestingly, the theoretical properties of the KL term follows directly from existing results in literature. Specifically, as shown in [Theorem 3, 7] in [7: https://arxiv.org/pdf/2401.01879#page=4.58], the KL divergence is bounded by $\text{KL}(\rho_{\text{LIAR}} || \rho_0) \leq \log(N) - (N - 1)/N$. Additionally, tighter bound is derived in the same reference, ensuring robustness within the optimization framework.

[References]

[1] Liu, Xiaogeng, et al. "Autodan: Generating stealthy jailbreak prompts on aligned large language models." arXiv preprint arXiv:2310.04451 (2023).

[2] Paulus, Anselm, et al. "Advprompter: Fast adaptive adversarial prompting for llms." arXiv preprint arXiv:2404.16873 (2024).

[3] Zou, Andy, et al. "Universal and transferable adversarial attacks on aligned language models." arXiv preprint arXiv:2307.15043 (2023).

[4] Patrick Chao, , Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramèr, Hamed Hassani, Eric Wong. "JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models." NeurIPS Datasets and Benchmarks Track. 2024.

[6] Yang, J. Q., Salamatian, S., Sun, Z., Suresh, A. T., & Beirami, A. (2024). Asymptotics of language model alignment. arXiv preprint arXiv:2404.01730.

[7] Beirami, Ahmad, Alekh Agarwal, Jonathan Berant, Alexander D'Amour, Jacob Eisenstein, Chirag Nagpal, and Ananda Theertha Suresh. "Theoretical guarantees on the best-of-n alignment policy." arXiv preprint arXiv:2401.01879 (2024).

Thank you for reviewing our paper. We have added further clarification to our mathematical explanations and provided additional results to support deeper analysis and comparisons with other methods.

Weakness 1: Although the model does not need the gradients w.r.t. the target LLM, which greatly reduces the computation for query generation, it still needs to calculate the reward $R_u$ with the model. This makes it not fully black-box, as it's not applicable to proprietary models like GPT-4. However, it would be interesting to see how effective the method is under the setting of transfer attack. [1]

Response to Weakness 1: We thank the reviewer for raising this point. We apologize for any confusion, but our proposed attack method, which leverages the concept of the unsafe reward $R_{\text{u}}$, is much more general in nature. While the specific example $R_{\text{u}} = -J(x, q, y)$ requires evaluating the reward using the target model, this is not an inherent limitation of our approach. This example was included to establish direct equivalence theoretically with existing methods like GCG.

Does not rely on the target model. Importantly, our method does not rely on the target model to generate prompts, which allows us to operate effectively in the transfer setting. Results shown in Table 1 demonstrate the effectiveness of our approach in this setting, where the prompts are generated independently of the target model. Furthermore, when comparing other methods in the transfer setting, their performance significantly declines, as shown in Table R3.

Additionally, our approach LIAR does not inherently require access to internal model details or gradients, aligning with black-box attack settings. All experiments presented in our paper adhere to this black-box paradigm, consistent with standard practices in the literature, such as AdvPrompter [2]. The general definition of the unsafe reward $R_{\text{u}}$ allows our method to adapt seamlessly to various scenarios, including proprietary APIs like GPT-4, by leveraging alternative proxies for $R_{\text{u}}$.

Table R3: White-box (individual) and black-box (universal) performance comparison. LIAR results are in the black-box setting.

Weakness 2: Strictly speaking, Llama2-7b is the only model that has been safety aligned with RLHF and the ASR on it is significantly lower than that on other models, with bigger gaps with GCG and AutoDAN. This indicates the challenges of the proposed method on safer LLMs. The authors claim that they have conducted experiments on Llama3 and Llama3.1, but only the results on them are provided in ablation studies. What is the jailbreaking performance on these latest LLMs compared to the baselines?

Response to Weakness 2: Thank you for pointing this out, we provide additional results on Llama2 and more recent TargetLLMs in Table R2. We note that all attacks, including ours, tend to perform worse on well-aligned models like Llama2-7b compared to less robust models like Vicuna-7B. However, our method, LIAR, can achieve significant performance improvements by increasing the number of queries, as shown in Table R2 for ASR@1000. This is made possible by the fact that our method does not require training and benefits from fast inference times. Additionally, we observe that changing the AdversarialLLM in our method can further enhance performance. These results appear to hold even for the more recent LLaMA-3.1 model.

Table R2: Effectiveness of different attacks on Llama target models under the ASR@1000 setting and for different AdversarialLLMs.

Question 2: Although the suboptimality of the LIAR method has been theoretically proven, is there an experimental comparison between the LIAR method and the optimal method to assess the gap? Since the experimental results show that, despite similar time efficiency (both around 15 minutes), the ASR@100 of the LIAR method is almost always lower than that of GCG and AutoDAN.

Response to Question 2: Thank you for interesting question. Based on our theoretical analysis, we note that as the number of samples N becomes very large, our method asymptotically achieves optimality. In practice, however, there is no realizable and fully implementable method that can be considered both optimal and computationally feasible. To demonstrate this, we conducted experiments showing that the performance of our proposed algorithm improves significantly with increasing N , approaching optimal attack success rates.

• Time efficiency clarification: To match the time required by GCG in Table 1, our method would need to compute ASR@10,000 instead of the maximum reported ASR of 100. This is because the time-to-attack (TTA) metric for our method is based on generating 100 times more queries. Specifically, TTA1 for GCG corresponds to ASR@1, while TTA1 for LIAR corresponds to ASR@100 (and TTA100 for LIAR corresponds to ASR@10,000). This imbalanced comparison is intentional, as we propose comparing our ASR@100 results with the ASR@1 results of other methods to highlight the efficiency of our approach. We have updated the caption of Table 1 to make this clarification more explicit. Additionally, while Table 1 presents ASR@100 results, we extend this evaluation to ASR@1000 in Table R6, where we observe further performance improvements.

[References]

[2] Paulus, Anselm, et al. "Advprompter: Fast adaptive adversarial prompting for llms." arXiv preprint arXiv:2404.16873 (2024).

Thank you for reviewing our paper. We have addressed several points of clarification and included additional experimental results to further demonstrate the effectiveness of our method. We are happy to provide further details as needed.

Weakness 1: ASR@1 of LIAR is significantly lower than that of other methods(Taking vicuna-13b as the target LLM, for example, LIAR's ASR@1 is only 0.94, while GCG can achieve 95.4).

Response to Weakness 1: The strength of our method is in its use of Best-of-N sampling, where increasing $N$ (e.g., to ASR@100) significantly boosts performance. While LIAR may exhibit lower performance in fewer-query settings, such as ASR@1, its ASR@100 is comparable to or even surpasses other methods, demonstrating its efficiency and scalability in higher-query scenarios. Additionally, because our method is training-free and highly efficient, generating larger $N$ (e.g. 100 or 1000) is computationally feasible, unlike prior methods such as GCG or AutoDAN, which are limited by their higher computational requirements. Results in Table R6 further illustrate our method's strong performance in larger query settings.

Table R6: Impact of increasing ASR@k for LIAR.

Weakness 3 (we answer weakness 3 first): $J(x,q,y)$ requires access to the model($\pi\_\theta$)'s logits. For closed-source model APIs that do not provide this service, LIAR may not be effective.

Response to Weakness 3: We thank the reviewer for raising this point. We apologize for any confusion, but our proposed attack method, which leverages the concept of the unsafe reward $R\_{\text{u}}$, is much more general in nature. The specific example of $R\_{\text{u}}=-J(x,q,y)$ is just one instance where log probabilities are required. We included this example to establish a direct connection and equivalence with existing attack methods, such as GCG.

However, we emphasize that our proposed method is not inherently tied to access to model logits. In fact, all the experiments presented in our paper are independent of the target model’s internal details, operating entirely in a black-box setting. This aligns with standard practices in the literature, such as AdvPrompter [2], which also does not rely on logit access to evaluate during inference.

By generalizing the unsafe reward definition, our method remains applicable across a wide range of scenarios, including those involving closed-source APIs that do not expose logits. This flexibility further highlights the robustness and practicality of our approach.

Weakness 2: Unsafe reward requires a known harmful response $y$ as a premise. However, the quality of $y$ is not explicitly addressed in the ablation study, leaving the impact of y's quality on LIAR's ASR unexplored.

Response to Weakness 2: As highlighted in the response to weakness 3 above, we note that our method LIAR does not explicitly target a particular $y$, instead it aims to generate numerous suffixes, of which one may successfully bypass safety-alignment. As our AdversarialLLM does not use $y$ to inform its generation, changing $y$ will have no impact on the resulting attack success rate.

Question 1: LIAR shows a significant improvement in ASR@1-ASR@100 from 0.94 to 79.81 when vicuna-13b is the target model, but only a slight improvement when LLaMA2-7b is the target model(from 0.65 to 3.85). Does this suggests that LIAR may be an unstable jailbreak method, as its effectiveness varies significantly depending on the target model?

Response to Question 1: The drop in performance observed against LLaMA2-7B-chat is consistent with other jailbreak attacks and can be primarily attributed to the model's robust safety alignment, which is particularly strong in this case. However, performance can be improved by increasing the number of queries, as demonstrated in Table R2, where ASR@1000 shows a notable boost in success rates. Furthermore, swapping the AdversarialLLM can also enhance performance, although the degree of improvement varies depending on the target model, as discussed in Section 5.2 and illustrated in Table 2.

Table R2: Effectiveness of different attacks on Llama target models under the ASR@1000 setting and for different AdversarialLLMs.

Major Weakness 3: The theory looks like the "Infinite monkey theorem" to me and is relatively straightforward given the abundant existing works in the field of RL/RLHF theory.

Response to Major Weakness 3: We thank the reviewer for their feedback and appreciate the opportunity to clarify our contributions. However, we respectfully disagree with the assertion that our theoretical analysis is straightforward or resembles the “Infinite Monkey Theorem.” Let us take this opportunity to provide detailed insights into the novelty and significance of our theoretical work:

• Novelty of Our Analysis: To the best of our knowledge, our theoretical analysis does not draw upon any existing theoretical results specific to RLHF or jailbreak. The framework and methods we introduce are entirely novel, offering a fresh perspective on the alignment of RLHF with safety guarantees.
• Introduction of the Safety Net Concept: Our paper introduces the novel concept of a safety net and explicitly connects it to jailbreak attacks. This connection has not been explored in prior works, and we believe it provides an important theoretical foundation for improving the robustness and safety of RLHF-based systems.
• Relevance of Asymptotic Analysis: We apologize if our analysis was misunderstood as resembling the “Infinite Monkey Theorem.” This is not the case. Our work is rooted in rigorous asymptotic analysis, which is widely recognized as a standard and powerful approach in the context of “best of N” sampling methods [7,8,9].

Minor Weakness 1: There are two "(x)" in equation (7). I have listed some possible typo errors in the question section.

Response to Minor Weakness 1: Thank you, we have corrected this typo.

Minor Weakness 2: Why do you limit the attack model to models as weak as GPT-2? Sampling 100 responses from Vicuna-v.1.5 using vLLM won't be longer than 10 minutes.

Response to Minor Weakness 2: Thank you for raising this point. We provide results for a variety of larger AdversarialLLM models in Table R5. The ablation presented in Table 2 of the paper focuses solely on small and efficient models as low compute is a key advantage of our method. Additionally, as shown in Table R5, the ASR@100 for GPT2 demonstrates that attack performance is not compromised despite using a smaller model. Below are a few additional comments addressing this point:

• Efficiency and Practicality: We intentionally started with GPT2 because smaller models are significantly cheaper and more efficient for generating larger sample sets. This makes GPT2 an ideal choice for adversarial prompting, as it allows us to explore the effectiveness of our approach without incurring unnecessary computational overhead.
• Prompt Diversity Insights: Another observation is that larger models tend to exhibit lower prompt diversity as the number of queries increases. For example, Vicuna-7B achieves a higher ASR@1 than GPT2 but experiences a significant drop in ASR@100, indicating reduced diversity in the generated prompts. While prompt diversity is clearly related to attack success, the relationship is not entirely straightforward. For instance, as shown in Table 3, increasing temperature (a form of diversity) does not always lead to higher ASR for larger values of $k$.

Table R5: Query Time and ASR of using various different AdversarialLLMs in our LIAR method.

Question 1: As far as I know, there is no 7B variant of LLaMA-3.1. Are you referring to LLaMA-3.1-8B?

Response to Question 1: Thank you for catching this oversight. In the paper, Llama2 and Llama3.1 are the 7b and 8b variants.

Question 2: For all the models, are you using the base models (LlaMA-2-7B) or the chat models (LlaMA-2-7B-chat)?

Response to Question 2: All target models used in our experiments are the chat/instruct variants. We have updated our Experiments Setup section to clearly indicate this distinction.

Thank you for reviewing our paper. We have included additional comparison experiments and extended an ablation study to further demonstrate the effectiveness of our method. Additionally, we have provided a new baseline and clarified our proposed framework and the underlying mathematical formulation, showing that we are not making use of unpaid monkeys with typewriters.

Major Weakness 1: The current algorithm is way too trivial for me and the formulation is not very interesting. Given a harmful query $x$ and an aligned model $M$, the model has a probability $p$ of generating the desired response when prompted with $x$ (temperature > 0). Then we sample 1000 times from $M$ using the same $x$ (this might be a slightly weaker attack than PAIR), the probability of the overall model being safe is $(1-p)^{1000}$, which is a small number if $p$ is not extremely close to 0. Based on the above example, I am trying to convey that using BoN to solve the inverse-alignment problem is technically trivial and doesn't have a close relationship with your formulation. Simply reformulating jailbreak as an alignment problem is not enough for an ICLR paper and I am looking forward to seeing a more novel method of solving the problem.

Response to Major Weakness 1: We thank the reviewer for the thoughtful comments and the opportunity to clarify our contributions. We apologize for any oversight in presenting our approach, and we believe there may be some confusions that we would like to clarify in detail as follows.

Our proposed framework, as detailed in Figure 1 of the paper, consists of two key components: an Adversarial LLM and a Target LLM (or aligned model $M$, as mentioned by the reviewer). The key idea is that the input query $x$, which serves as the base prompt, is passed to the Adversarial LLM to generate a perturbed query $q$. The concatenation $[x, q]$ is then fed into the Target LLM $M$ to produce the output $y$. This two-step process aligns with the existing literature on adversarial attacks on LLMs, where the goal is to optimize the query $q$ to maximally perturb the aligned model’s behavior (e.g., GCG [3], AutoDAN [1], AdvPrompter [2]).

Clarifying the Importance of $q$ in Our Method The reviewer raises the point that generating multiple outputs $\{y\_i\}\_{i=1}^N$ for the same $x$ without modifying the query (i.e., using a single query $x$) might achieve similar results. While this may seem theoretically plausible, it fundamentally diverges from the essence of adversarial attacks, which aim to perturb the prompt space to elicit unsafe outputs. In our method, the perturbation introduced via $q$ is critical, as the concatenated prompts $[x, \{q\_i\}\_{i=1}^N]$ lead to distinct outputs $\{y\_i\}\_{i=1}^N \sim M(\cdot \mid [x, \{q\_i\}\_{i=1}^N])$, effectively increasing the attack surface and success probability.

Comparison with Attack suggested by the reviewer. To validate this, we setup an experiment with the reviewer-suggested approach (generating $\{y\_i\}\_{i=1}^N \sim M(\cdot \mid x)$ without perturbing $q$) and observed significantly reduced attack success rates compared to our method. The results are presented in Table R8 for quick reference and further demonstrate the importance of incorporating adversarial perturbations into the prompt space.

Table R8: Effectiveness of jailbreak with and without adversarial suffix.

Our Novelty and Contributions: We would like to emphasize the novel contributions of our work. The primary objective is to rigorously characterize the jailbreak problem through the lens of alignment—a perspective that, to the best of our knowledge, has not been explored before. We note that Best-of-N (BoN) sampling is a simple yet effective approach, and computationally efficient method for tackling alignment problems. Our work is inspired by existing research on inference time alignment strategies for alignment, such as [6,7,8], and demonstrates how these concepts can be leveraged for adversarial attacks in LLMs.

Remark: We humbly disagree with the characterization of our approach as “trivial.” Instead, we see our work as a first step toward formalizing jailbreak attacks as an inverse-alignment problem, introducing a rigorous theoretical foundation, and leveraging a simple yet effective method (BoN) to demonstrate the practical implications of our insights. We appreciate the reviewer’s feedback and are happy to incorporate additional clarifications, including the comparison experiment above, into the revised version of the paper.

[References]

[1] Liu, Xiaogeng, et al. "Autodan: Generating stealthy jailbreak prompts on aligned large language models." arXiv preprint arXiv:2310.04451 (2023).

[2] Paulus, Anselm, et al. "Advprompter: Fast adaptive adversarial prompting for llms." arXiv preprint arXiv:2404.16873 (2024).

[3] Zou, Andy, et al. "Universal and transferable adversarial attacks on aligned language models." arXiv preprint arXiv:2307.15043 (2023).

[4] Patrick Chao, , Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramèr, Hamed Hassani, Eric Wong. "JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models." NeurIPS Datasets and Benchmarks Track. 2024.

[5] Patrick Chao, , Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, Eric Wong. "Jailbreaking Black Box Large Language Models in Twenty Queries." (2023).

[6] Yang, J. Q., Salamatian, S., Sun, Z., Suresh, A. T., & Beirami, A. (2024). Asymptotics of language model alignment. arXiv preprint arXiv:2404.01730.

[7] Amini, Afra, Tim Vieira, and Ryan Cotterell. "Variational best-of-n alignment." arXiv preprint arXiv:2407.06057 (2024).

[8] Gui, Lin, Cristina Gârbacea, and Victor Veitch. "BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling." arXiv preprint arXiv:2406.00832 (2024).

Major Weakness 2: The performance is weak on well-aligned models like the LlaMA series. Also, it would be better to compare the algorithm with black-box attacks like GGC-transfer/PAIR/MSJ etc.

Response to Major Weakness 2:

Llama performance: Thank you for this point. We note that all attacks, including ours, tend to perform worse on well-aligned models like Llama2-7b compared to less robust models like Vicuna-7B. However, our method, LIAR, can achieve significant performance improvements by increasing the number of queries, as shown in Table R2 for ASR@1000. This is made possible by the fact that our method does not require training and benefits from fast inference times. Additionally, we observe that changing the AdversarialLLM in our method can further enhance performance. These results appear to hold even for the more recent LLaMA-3.1 model.

Table R2: Effectiveness of different attacks on Llama target models under the ASR@1000 setting and for different AdversarialLLMs.

Comparisons: Regarding comparisons, Table R3 presents evaluations of GCG and AutoDAN under both the white-box (individual) setting and the transfer-based (universal) setting. In the universal setting, suffix optimization is adapted to find a universal (transferable) prompt instead of optimizing for individual prompts, resulting in a significant drop in performance. In contrast, our method does not rely on the TargetLLM for prompt generation, meaning all results reported for LIAR are effectively in the black-box transfer setting.

Table R3: White-box (individual) and black-box (universal) performance comparison. LIAR results are in the black-box setting.

Comparison with PAIR: In Table R4.1, we provide results for LIAR on the JailbreakBench [4] dataset, and in Table R4.2, we present PAIR’s [6] results on the same dataset. While the performance between methods is relatively close, two key differences prevent a direct comparison: (1) differing ASR evaluation methods and (2) differing problem constraints.

• (1) For ASR evaluation, we follow AdvPrompter in using keyword matching to determine attack success, whereas PAIR employs LlamaGuard for evaluating whether a prompt has been successfully jailbroken.
• (2) More fundamentally, our problem setting is restricted to modifying a suffix appended to a censored prompt, consistent with prior works [1,2,3]. In contrast, PAIR allows full prompt modification, introducing additional flexibility and complexities. While the underlying goal of obtaining a jailbroken prompt is the same, the broader scope allowed by PAIR represents a different class of problem and methodology.

Table R4.1 On JailbreakBench using keyword-matching ASR.

Table R4.2 On JailbreakBench using LlamaGuard ASR.

Thank you for your thoughtful feedback on our paper. We have provided additional experimental results and included a qualitative example comparison as requested. Additionally, we are actively working on visualizations to enhance the clarity of some of our results.

Weakness 1: The paper is poorly explained and written, and I think the biggest improvement will come from extensive rewriting and reframing of the story. The presentation could be improved by introductions of i) schematic figures, ii) plots rather than just tables, and iii) examples of the adversarial suffixes generated by the prompter and how this compares to prompts generated by AdvPrompter. I would like to see many more experiments that compare how swapping the underlying prompt generator model for others affects its performance on other target models and how to choose a good reward model for this task.

Response to Weakness 1: Thank you for your feedback. While we respectfully disagree with the claim that the paper is poorly explained—other reviewers have specifically acknowledged the clarity and contributions of our work—we understand the value of additional visualization and will incorporate your suggestions where feasible. Below, we address your points in detail:

• Examples of Adversarial Suffixes: We have included examples of our adversarial suffixes in Section E of the Appendix. These examples can be directly compared to those generated by AdvPrompter, which are available in the supplemental section of their paper [2]. Additionally, Table R7 provides a side-by-side comparison of a suffix generated by AdvPrompter versus one generated by our method, both of which successfully jailbreak Vicuna-7B.

• Swapping Underlying Prompt Generator Models: Our method is inherently model-agnostic and does not depend on training or reward tuning, which is a significant advantage. This design eliminates the need to select or fine-tune a specific reward model. To address your concern further, we have added results for additional AdversarialLLM configurations in Table R5, showcasing the flexibility of our approach.

• Schematic Figures and Plots: Figure 1 in the paper already includes a schematic of our method along with two plots highlighting its effectiveness and efficiency. Given the nature of our results, we believe tables are often the most effective format for conveying key findings. However, we agree that visualizing ASR@1/10/100 through a graph could better illustrate trends. In the revised version of the paper, we will include more plots for the ablation studies to enhance clarity and accessibility.

We emphasize that the training-free and model-agnostic nature of our method addresses many practical challenges in adversarial prompting, making it a versatile and scalable solution. We hope these additions and clarifications address your concerns and provide further insight into our contributions.

Table R5: Query Time and ASR of using various different AdversarialLLMs in our LIAR method.

Table R7: An example jailbreak and response.

[References]

[2] Paulus, Anselm, et al. "Advprompter: Fast adaptive adversarial prompting for llms." arXiv preprint arXiv:2404.16873 (2024).

Thank you for reviewing our paper. We have conducted additional experiments on new datasets to validate the consistency of our results and have revised the manuscript to provide greater clarity regarding the experimental settings. We are happy to answer any other questions.

Weakness 1: AdvBench only has 312 (finetuning train) + 104 (test) samples making the comparisons a bit fragile. It'd be nice if authors could demonstrate it on a larger dataset of adversarial prompts. One could use mechanical turk to generate a larger library of such prompts.

Response to Weakness 1: We agree with the reviewer that demonstrating results on a larger dataset would provide a more robust evaluation. In the paper, we report results on the AdvBench test split (104 samples), following the standard practice in the automatic jailbreak literature [1,2,3]. A key reason for the relatively small size of this dataset is the significant computational cost required by prior methods to obtain results. However, since our method is training-free and extremely fast in generating attacks, we can efficiently evaluate larger datasets.

Results across several datasets: Table R1 presents results for our method across several datasets, including the AdvBench test split (as reported in the paper), the AdvBench train split, JailbreakBench [4], and Do-Not-Answer [5]. As our method requires no training, we can fairly evaluate on the AdvBench train-split. JailbreakBench, while consisting of only 100 samples (with approximately 20% overlapping with AdvBench), includes additional examples that test our method's robustness across a broader range of prompts. Do-Not-Answer contains 939 samples, providing a larger dataset, though its promts are generally shorter and simpler. These results demonstrate the consistency of our approach across different datasets and highlight its effectiveness in handling a variety of censored prompts.

Table R1: Results for LIAR on additional splits and datasets.

Question 1: In this sentence isn't 10000x an exaggeration ? should it be 100x : ". Given the significantly reduced overall TTA, this asymmetric ASR@k comparison becomes highly practical: our method can generate over 10,000 queries before GCG completes its first"

Response to Question 1: Thank you for raising this point. We confirm that our statement, “our method can generate over 10,000 queries before GCG completes its first,” is accurate. To clarify, GCG’s TTA1 (time-to-attack one prompt) is approximately 16 minutes, while TTA1 for our method represents the time-to-attack 100 prompts. Accordingly, TTA100 for our method, representing the time-to-attack 10,000 prompts, is just 14 minutes. This is why we stated 10000x the queries. We have updated the description in Table 1 to make this setting more clear.

Question 2: Can you include more details about the finetuning process so practitioners can replicate your work?

Response to Question 2: Thank you for your question. We would like to clarify that our method does not involve any fine-tuning, as it is based on the concept of Best-of-N sampling. Consequently, the details provided in the Setup and Experiment sections should be sufficient to replicate our work. Additionally, we will be releasing the code for all our experiments to further facilitate reproducibility.

[References]

[1] Liu, Xiaogeng, et al. "Autodan: Generating stealthy jailbreak prompts on aligned large language models." arXiv preprint arXiv:2310.04451 (2023).

[2] Paulus, Anselm, et al. "Advprompter: Fast adaptive adversarial prompting for llms." arXiv preprint arXiv:2404.16873 (2024).

[3] Zou, Andy, et al. "Universal and transferable adversarial attacks on aligned language models." arXiv preprint arXiv:2307.15043 (2023).

[4] Patrick Chao, , Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramèr, Hamed Hassani, Eric Wong. "JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models." NeurIPS Datasets and Benchmarks Track. 2024.

[5] Patrick Chao, , Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, Eric Wong. "Jailbreaking Black Box Large Language Models in Twenty Queries." (2023).