Improved Generation of Adversarial Examples Against Safety-aligned LLMs

Thanks for the feedback. Our responses to the comments are given as follows.

Improve the writing and making the zero line dark in Figure 3.

Answer: Thanks for the effort to fathom the contribution of our work. We will carefully revise the writing to make our contribution more clear, especially for the Abstract and Introduction in the updated version.

The paper lacks qualitative analysis. I would like to see side by side the attacks that went into figure 8.

Answer: In Figure 8, we show the effectiveness of our methods, especially our combination method, compared with the baseline methods. The introduced methods in the paper aim to refine the input gradient computation in order to make the input gradient more precisely reflect the reduction of adversarial loss that results from token replacements in the prompt. We conducted an experiment that evaluates the effectiveness of the input gradients obtained from the methods, thereby demonstrating the improvements offered by our methods. Firstly, we compute the losses of all the choices of token replacement for one adversarial token in a prompt and rank them in ascending order as the ground truth rank list of token replacements. The higher the ranking of the token replacements corresponding to the Top-$k$ values in the input gradient within the ground truth rank list, the more effective this input gradient is. We compute such ranking obtained by different methods. For each prompt, we select the first adversarial token to evaluate the ranking. We use $k=1$ for all methods. The average rankings across 100 prompts are shown below. The experiment is performed on Llama-2-7B-Chat which the vocabulary size is 32000. It can be seen that the input gradients of our methods provide a higher ranking, especially the GCG-LSGM-LILA$^\dagger$, and the input gradients of GCG give the lowest ranking. Such observation indicates that the gradient computation refinements that we appropriated indeed encourage the input gradient to be more precise, thereby improving the discrete optimization.

Ideally, this paper would not just work within the harmbench box but would also apply the method to do some form of red teaming that couldn't have been done before.

Answer: We agree that the perspective introduced for improving discrete optimization problems within LLMs has the potential to enhance tasks beyond the generation of adversarial examples evaluated in the paper. We are currently applying this perspective to other tasks and exploring more effective methods for our future work.

Releasing code seems to be important for this project. What is the update on plans for release?

Answer: We will make the code publicly available and provide the website to access the code in the paper.

Thanks for the feedback. Except for the comments about the discussion on perplexity and the concurrent related work, which are answered in our global response, all comments are replied to as follows.

Regarding the Mistral-7B-Instruct model, Table 1 shows that only the combination of LSGM and LILA surpasses the GCG ASR baseline, whereas for the Llama-chat series, a single component is sufficient (for most cases). This raises my question about whether the previous investigations (as shown in Figures 3, 4, and 5) apply to less-aligned LLMs like Mistral or if they only hold true for better-aligned models like the Llama-chat series.

Answer: In Table 1, the results of GCG$^*$ (first row in the table) are obtained by adopting the default setting introduced in the paper of GCG, which uses a Top-$k$ selection of 256 and a candidate set size of 512 at each iteration. Other results were obtained by using a Top-$k$ selection of 4 and a candidate set size of 20, which considerably reduced the time cost. We show the results of GCG$^*$ aims to demonstrate that our methods can achieve comparable results with lower computation complexity. When comparing our methods to the GCG that also uses a Top-$k$ selection of 4 and a candidate set size of 20 (second row in the table), it can be seen that all methods show improvement. We conducted the experiments corresponding to Figures 3, 4, and 5 on Mistral-7B-Instruct, and presented the results in Figures VII, VIII, and X in the attached PDF in the global response.

While the proposal is demonstrated to be robust to the choice of 𝛾 and layer selection 𝑟, there is no discussion about the impact of the 𝛽 hyperparameter for (LSGM-)LILA$^\dagger$. It would be useful to explore how robust the proposal is to this parameter.

Answer: We evaluated LILA$^\dagger$ and LSGM-LILA$^\dagger$ by varying the choice of $\beta$ in Figure IX in the attached PDF in our global response. It can be seen that the performance improves with a large range of the choice $\beta$. For simplicity, we recommend setting $\beta \rightarrow \infty$ as we set in the paper.

The paper could benefit from some polishing, such as correcting the reference to Figure 8, which is currently written as Table 8 in line 333.

Answer: Thanks for pointing out the typo. We will fix it in the updated version.

Dear Reviewer s3ui,
 
Thanks for responding to our rebuttal. We sincerely apologize for incorrectly showing the results of Phi3-Mini-4K-Instruct in Figures VII and VIII. We have conducted experiments on Mistral-7B-Instruct, and the results have shown observations consistent with those of Llama2-7B-Chat and Phi3-Mini-4K-Instruct. We will include these figures in the updated version of the paper.

Additionally, the discussion on refusal directions with experimental results will also be provided in the updated version of the paper.
 
Best regards,
Authors

Thanks for the feedback. Except for the comments about the experiments of reducing the gradients from skip connections, the discussion on perplexity and some concurrent related work, which are answered in our global response, all comments are replied to as follows.

• Lack of error bars in the Figures.
• Validate Figure 3 & 4 across more models.

Answer:
• We have revised Figures 2, 4, 5, and 7 to include scaled standard deviations as error bars and have shown the updated figures in the attached PDF in our global response (i.e., Figures I, II, III, and IV).
• We extended the experiments presented in Figures 3 and 4 to include Mistral-7B-Instruct and Phi-3-Mini-4K-Instruct. The results are presented in the attached PDF file as Figures V and VI for Mistral-7B-Instruct, and Figures VII and VIII for Phi-3-Mini-4K-Instruct. Observations similar to those depicted in Figures 3 and 4 can be observed.

Table 1 should have at least one jailbreaking method that does not simply generate a suffix, e.g. PAIR.

Answer: We evaluated PAIR, and the ASRs are shown below. We will add the results in Table 1 in the updated version of the paper.

• Table 1 and 2 need at least 3 model families, so I urge the authors to also run experiments for those Tables on either Phi or Gemma or Llama3
• Evaluate the methods on the dataset of HarmBench in Table 2.

Answer:
• For Table 1, we conducted the experiments on Phi-3-Mini-4K-Instruct as suggested. The results are shown below. Our methods also improve the attack performance on Phi-3-Mini-4K-Instruct.

• For Table 2, we evaluated GCG and our GCG-LSGM-LILA$^\dagger$ on HarmBench. Following the suggestion, we also evaluated the methods on the model of Phi3-Mini-4K-Instruct. The results are shown below. It can be seen that the GCG-LSGM-LILA$^\dagger$ also achieves improvement in attack success rates against all models. We found that the attack success rates of both methods show extremely low attack success rates on the models except Llama-2-7B-Chat. We attribute this to the use of a small number of training examples (10), a limited Top-$k$ selection (4), and a candidate set size (20), all of which were chosen to reduce computational complexity. Experiments with the same settings as those introduced in HarmBench are ongoing. However, due to the considerable amount of time cost, it is challenging to present the results during the rebuttal period. We will include the results in the updated version of the paper.

Improve the writing.

Answer: Thank you for the suggestions. We will improve the writing accordingly.

The paper provides no hypothesis why attenuating the loss from the residual module helps, more precisely why would that gradient have negative cossine similarity with the residual connection.

Answer: We attempt to provide some hypotheses about these phenomena. With the skip connection branch, the input gradient can be regarded as the accumulation of the gradients from residual modules. We hypothesise that these gradients from residual modules contribute diverse information, thus causing nearly zero even negative cosine similarity between the gradients from the residual module and the gradient from the skip connection (which is the sum of the gradients from the residual modules in deeper layers of the model). Ideally, the input gradient can measure the changes in residual modules' outputs along the direction of their gradients. Nevertheless, since we optimize in the token (discrete) space, the input update (token replacement) is not precisely in the direction of the input gradient. Such deviation hinders the expected impact on the residual modules' outputs and becomes more severe in deeper layers. In each residual block, reducing the gradients from the residual module corresponds to enlarging the gradient from deeper layers. Thus, it alleviates the unexpected impact on the outputs of the deeper residual modules that result from the deviated input update, thereby improving the optimization. We conduct a simple experiment to strengthen the hypothesis of reducing the gradients from the residual modules in only the first half of the Llama-2-7B-Chat layers. It achieves 71% match rate and 60% attack success rate which are comparable with the results of the gradients from all residual modules (72% for match rate and 62% for attack success rate).

Dear Reviewer heVA and Area Chair ju17,
 
Thanks for the comments on our rebuttal. We found that the original implementation of generating universal adversarial suffixes requires using all behaviors, including test behaviours, during the generation instead of only using 10 behaviors that do not overlap with the test behaviors in our rebuttal (refer to #1 Issue in the GitHub repository of HarmBench). We updated the experimental setting and evaluated the GCG and our method on 200 standard behaviours of HarmBench. Specifically, we generated the universal adversarial suffixes on 200 standard behaviors and evaluated the suffixes on these behaviors. Each method was run ten times and reported not only the average ASR (AASR) but also the best ASR (BASR) and the worst ASR (WASR). The results of attacking Llama-2-7B-Chat, Mistral-7B-Instruct, Llama-2-13B-Chat, and Phi3-Mini-Instruct are shown below. Our method outperforms the GCG attack on average, worst, and best ASRs. Generating adversarial suffixes on 200 standard behaviors is quite time-consuming. Due to the limited duration of the discussion period, the experiments on Phi3-Mini-Instruct are still ongoing, and we evaluated its adversarial suffixes at the 100-th iteration. For the experiments on the other models, the adversarial suffixes are obtained through 500 iterations. For the evaluations that involve combining our methods with PEZ and AutoDAN, the experiments are also ongoing, and the results will be included in the updated version of the paper.

 
Best regards,
Authors

Thanks for the feedback. Except for the comments about the experiments of reducing the gradients from skip connections, which is answered in our global response, all comments are replied to as follows.

Though I believe the method shown in the paper can be applied to other models, the authors should explore LLMs with more diverse architectures to demonstrate the generalization of the proposed method. The three LLMs presented in the paper share the same or similar architecture, particularly in the design of the residual part.

Answer: We extended the experiments to Phi3-Mini-4K-Instruct. The results are presented below. It can be seen that our methods still gain improvements when compared with the GCG attack.

From my understanding, the proposed method specifically refines the gradient for GCG. I want to confirm whether, apart from the gradient refinement, the method for optimizing the tokens remains the same as GCG, i.e., still via Top-$k$ token replacement. If so, I am curious why the time cost for the proposed method is significantly smaller than that of GCG. What contributes to this reduction in time cost?

Answer: The only difference between our method and GCG is the computation of the input gradient. Our method retains the procedure for evaluating the adversarial loss of candidate token replacements in each iteration. This procedure contributes the most to the computing cost. Our methods aim to refine the input gradient in order to enhance its efficiency in reducing adversarial loss, i.e., achieving better performance while evaluating the same number of candidates in each iteration or achieving similar performance while evaluating fewer candidates in each iteration. Therefore, in Table 1, we show the performance of GCG with the default setting ($k$=256 and 512 candidates evaluated in each iteration, corresponding to GCG$^*$ in the table). For our methods, we reduce $k$ and the number of candidates evaluated, i.e., $k$=4 and 20 candidates in each iteration, thus reducing the time cost. The results show that our GCG-LSGM-LILA$^\dagger$ achieves similar or even better match rates and attack success rates with lower time costs compared with GCG$^*$.

When using different decay factors 𝛾 for the gradient, as shown in Figure 9, did the authors normalize the gradient norms during optimization? I doubt the magnitude of the gradient will also lead to some bias.

Answer: We did not re-normalize the magnitude of the gradients in the paper. We conducted the experiments with such re-normalization and show the results below. The results indicate the lower magnitude of the gradients is not the reason for improved performance, and further confirm the effectiveness of the strategy of reducing gradients from residual modules.

Can the universal adversarial suffixes generated by your method transfer more effectively to closed models like GPT or Claude?

Answer: We use the universal suffixes generated by performing GCG and GCG-LSGM-LILA$^\dagger$ against Llama-2-7B-Chat to attack GPT-3.5-Turbo on the first 100 harmful queries in AdvBench. The results are shown below. It can be observed that our GCG-LSGM-LILA$^\dagger$ achieves remarkable improvements in the average, worst, and best ASRs obtained over 10 runs.

Dear Reviewer BhKe,
 
Thanks for the comments. Our responses are given as follows.

1. In our rebuttal, we evaluated our methods for generating query-specific adversarial suffixes on AdvBench. We would like to politely remind you that the bias between evaluations on AdvBench and HarmBench mainly exists to generate universal adversarial suffixes. Since the AdvBench dataset contains semantically similar queries, generating universal adversarial suffixes for a group of these queries might compromise their universality. When generating query-specific adversarial suffixes, an adversarial suffix was generated for only one query, hence does not involve the universality problem. We also evaluated the performance of our method for generating universal adversarial suffixes on the dataset of HarmBench following Reviewer heVA's comment. The results are shown below. Due to the limited duration of the discussion period, the experiments on Phi3-Mini-Instruct are still ongoing, and we evaluated its adversarial suffixes at the 100-th iteration. For the experiments on the other models, the adversarial suffixes are obtained through 500 iterations.

 
2. The results of using a Top-$k$ of 4 and a candidate set size of 20 for GCG are shown in the second row in Table 1. We will emphasize the setting of these results for clarity in the updated version of the paper.
 
3. We will add the results of attacking closed models in the revision.
 
Best regards,
Authors

Dear reviewers,
 
We sincerely apologize for incorrectly presenting the experimental results on Phi-3-Mini-4K-Instruct instead of Mistral-7B-Instruct in Figures VII and VIII. We have conducted experiments on Mistral-7B-Instruct and the results show similar observations to the results of Llama-2-7B-Chat (Figures 3 and 4 in the paper) and Phi-3-Mini-4K-Instruct (Figures V and VI in the PDF).
 
Best regards,
Authors