Accelerating Greedy Coordinate Gradient and General Prompt Optimization via Probe Sampling

Dear Reviewer hJdQ,

Thank you for your insightful reviews and comments. We appreciate the time and effort you have put into providing valuable feedback. We would like to address your concerns as follows:

Concern: Transferability of Probe Sampling

We appreciate your concern regarding the transferability of probe sampling, and we conduct corresponding experiments to analyze it.

1. Whether draft model of probe sampling affect transferability

We compare probe sampling implemented across various draft models on Llama2-7b-Chat and then transfer to diverse target models. Detailed results are shown as follows. Our investigation reveals that probe sampling has minimal impact on transferability when applied to appropriate draft models that maintain the original ASR of plain GCG. Conversely, utilizing draft models that degrade the initial performance significantly affects transferability.

In addition to open-source models, we also conduct transfer experiments on GPT4, especially gpt4-0125-preview. Our findings indicate that probe sampling not only markedly accelerates the attacking process but also maintains the transferred ASR. The detailed results are presented below.

2. Whether the filtered set size of probe sampling affect transferability?

In our experiments, we investigate the transferability of probe sampling across various filtered set sizes, denoted as $(1-\alpha)*B/R$. Our results align closely with previous findings, indicating that probe sampling does not significantly impact transferability when appropriate parameters are used. However, it does lead to decreased performance in cases where the direct ASR on Llama2-7b-chat is low.

3. What is the influence of the draft model's tokenizer on suffixes?

In Table 9, TinyLlama and ShearedLlaMa utilize the same tokenizer as Llama2-7b-Chat. However, their performance diverges significantly. Hence, the tokenizer does not play a pivotal role in effectively attacking the model or the transferability; instead, factors like model performance similarity hold greater importance.

Suggestion #1: Adding related work

Thank you for recommending more related works, especially in discrete prompt optimization algorithms. We will add the following paragraph in the final version.

Discrete Prompt Optimization. Attacking LLMs via adversarial prompt can be formulated as a discrete prompt optimization problem [1]. In this context, attacking algorithms strive to discover superior prompts that effectively steer aligned LLMs toward generating adversarial answers. Some approaches leverage LLMs themselves to iteratively refine prompts [2, 3]. However, aligned LLMs may resist refining adversarial prompts, rendering these methods ineffective. Other strategies employ RL-based prompt optimization techniques such as those in [4, 5], necessitating additional MLP training with extensive adversarial data and specific reward design. Moreover, other models introduced in [6, 7] to help with prompt optimization must remain unaligned, particularly in jailbreak scenarios[8]. However, their performance tends to be limited, especially when dealing with strongly fine-tuned models like Llama-2.

Suggestion #2: Align the format of "Probe sampling" in the paper.

We appreciate your effort in seriously looking into the details of our paper. We will address them to enhance our work further in the final version.

[1] Zou, et al, Universal and Transferable Adversarial Attacks on Aligned Language Models, Arxiv 2023

[2] Xu, et al, GPS: Genetic Prompt Search for Efficient Few-Shot Learning, EMNLP 2022

[3] Pryzant, et al, Automatic prompt optimization with" gradient descent" and beam search, EMNLP 2023

[4] Deng, et al, RLprompt: Optimizing discrete text prompts with reinforcement learning, EMNLP 2022

[5] Lu, et al, Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning, ICLR 2023

[6] Cho, et al, Discrete Prompt Optimization via Constrained Generation for Zero-shot Re-ranker, ACL 2023

[7] Do, et al, Prompt Optimization via Adversarial In-Context Learning, ACL 2024

[8] Chao, et al, Jailbreaking Black Box Large Language Models in Twenty Queries, Arxiv 2023

The authors clearly answered all the questions regarding transferability and the effect of the draft model's tokenizer. The experiments suggest that Probe sampling does not affect the transferability of the adversarial prompts compared to plain GCG. I recommend adding the transferability experiments as well as the discrete prompt optimization algorithms related work in the main paper. I have updated my score.

Dear Reviewer Jngk,

We appreciate the time and effort you have put into providing valuable feedback. However, we respectfully believe there might be some misunderstanding regarding our work. We would appreciate the opportunity to clarify a few points and address your concerns as follows:

Misunderstanding #1: Probe sampling performs B forward passes in parallel within a batch or sequentially?

We understand your concern related to whether probe sampling can accelerate GCG when B candidates are sent to LLMs in a batch. The answer is definitely YES, as we consistently use this approach throughout our paper, similar to the GCG paper. For more details on how this is implemented, please refer to the GCG code, specifically lines 170 to 174 in llm-attacks/llm_attacks/gcg/gcg_attack.py. In this context, the term “B forward passes” should be more accurately articulated as “forward computation on B candidates”. This modification ensures a clearer understanding of the procedure, emphasizing the total amount of computation. Moreover, Figure 3 illustrates why probe sampling can accelerate GCG when concurrently operating B candidates in a batch, owing to its reduced memory usage. A more comprehensive analysis is illustrated in the paper, spanning from line 217 to line 224.

Misunderstanding #2: Draft model is used to check whether the meaning of suffixes are meaningful so how to make sure two models are aligned without training?

The draft model is not used to "check the meaning" of suffixes; it solely assists the target model in ranking candidates. In GCG, we want to find the candidate suffix that achieves the lowest loss, i.e., the suffix that is ranked the first if sorted by their losses computed by the target model. Naturally, if the ranked results agree, the rank using the draft model is indicative of the rank using the target model and we can safely rely on the draft model to filter out more unpromising candidates. Conversely, in cases where $\alpha$ equals 0 ($\alpha$ can not be significantly larger than 0 as it is in the range of [0,1] and 1 meaning a full agreement as explained in line 110), we disregard the rank calculated by the draft model and depend exclusively on the target model in this iteration. Note that $\alpha$​ is adaptively calculated in each iteration (explained from line 134 to line 136 in text and line 13 in Algorithm 1), which significantly contributes to the performance of probe sampling. Furthermore, as illustrated in Table 9, probe sampling works consistently well across different models with various average alignment levels.

Concern #1: Compare with more efficient methods instead of GCG.

We appreciate your concern about comparing probe sampling with other existing acceleration approaches. AdvPrompter [1] involves training the AdvPrompter with the help of both the target model and another base model, necessitating inference of AdvPrompter for each adversarial prompt. These additional computational resources are significant. According to Figure 2 in [1], AdvPrompter achieves an ASR of 23.4 on Llama2-7b-chat, whereas probe sampling achieves 81. Moreover, following the recommendation from Reviewer bSGi, we also compare probe sampling with BEAST [2] and ACG [3]. It shows that probe sampling not only achieves much higher ASR, but also is orthogonal to them and can further accelerate them. Please refer to the rebuttal addressing concern #1 provided to Reviewer bSGi for further details.

Concern #2: Whether probe sampling can be used to accelerate AmpleGCG.

We acknowledge the concerns regarding the generalizability of probe sampling. AmpleGCG introduces a methodology that involves initially gathering adversarial attacks through overgenerated GCG and subsequently fine-tune the model using these adversarial input-output pairs. Given that AmpleGCG utilizes GCG as the fundamental algorithm for producing and assembling adversarial outputs, probe sampling can effectively accelerate overgenerated GCG. Following the setting of the paper, we implement probe sampling on overgenerated GCG.

We further employ generated adversarial input-output pairs to fine-tune Llama2-7B-chat and test on the same released hard testset with 100 queries. Here are the detailed results.

We find that adversarial input-output pairs generated by probe sampling accelerated overgenerated GCG achieves nearly the same performance, proving the applicability of probe sampling on generating adversarial training data.

In addition, probe sampling can also be implemented on Group Beam Search algorithm. However, we omit additional experiments here as it is the same as accelerating BEAST as mentioned earlier.

[1] Paulus, et al, AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs, Arxiv 2024

[2] Sadasivan, Vinu Sankar, et al, Fast Adversarial Attacks on Language Models In One GPU Minute, ICML 2024

[3] Making a SOTA Adversarial Attack on LLMs 38x Faster, Haizelabs Blog Post, 2024

[4] Liao, et al, AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs, Arxiv 2024

Dear Reviewer Jngk,

I hope this message finds you well. The discussion period is ending soon, I am writing to emphasize the importance of your review for our submission. Your score is significantly lower than the other two reviewers, and we believe this discrepancy may indicate a misunderstanding or oversight.

We have addressed all the concerns in our detailed rebuttal and would appreciate your prompt attention to it. A thorough reassessment is crucial to ensure a fair evaluation.

Your expertise is highly valued, and we trust that a reconsidered review will reflect the true merit of our work.

Thank you for your immediate attention to this matter.

Best regards, Authors

Dear Reviewer bSGi,

Thank you for your insightful reviews and comments. We appreciate the time and effort you have put into providing valuable feedback. We would like to address your concerns as follows:

Concern #1: Compare with more related works

We appreciate your concern about comparing probe sampling with other existing acceleration approaches. BEAST [1] and ACG [2] both employ beam-search like decoding methods, accelerating GCG by generating more candidates in each iteration, thereby reducing the total number of iterations. However, probe sampling is orthogonal to these methods, capable of integration into the loss calculation stage of each algorithm.

To verify this claim, we implement probe sampling on BEAST, which can accelerate it by 1.4 times on Vicuna-7b and 1.9 times on Llama2-7b-chat, on average per iteration. This acceleration allows probe sampling to execute more iterations within the same GPU time, thereby improving the ASR. The detailed results are presented below, indicating that probe sampling improves the ASR of both models within the same GPU time.

Furthermore, as illustrated in our paper (line 150 to line 155), probe sampling can be used to accelerate algorithms involving sampling prompt candidates and evaluating their performances, as evidenced by the experiment results in Table 3 and Table 4.

In addition to integrating with beam-search acceleration methods, probe sampling yields better ASR results, with probe sampling achieving a score of 81 on Llama2-7b-chat, surpassing the scores of 12 for BEAST and 64 for ACG.

Concern #2: Limitation

Thanks for recommending adding the limitation about high memory requirement, we will incorporate the following sentence into the final version.

Although the acceleration of probe sampling does not necessitate additional memory, it still faces the high memory demands inherited from GCG, consequently restricting experiments to smaller models. While simulated annealing offers some relief, the situation remains far from adequate.

[1] Sadasivan, Vinu Sankar, et al, Fast Adversarial Attacks on Language Models In One GPU Minute, ICML 2024

[2] Making a SOTA Adversarial Attack on LLMs 38x Faster, Haizelabs Blog Post, 2024