AttnGCG: Enhancing Adversarial Attacks on Language Models with Attention Manipulation

We first thank the reviewer for the detailed comments and the appreciation of our work. We address the concerns below:

$**Q1: Concerns about transferability of adv prompts across goals**$

Thank you for your suggestion of adding transfer experiments across different goal prompts. We will conduct a "multiple-behavior" experiment in [1] in the next version.

$**Q2: Concerns about possibly generating a harmful but unrelated response with the input prompt**$

Thank you for bringing the question up. Our GPT-4 metric does not count "harmful but completely unrelated with the input prompt" cases into the ASR result. The GPT4-judge determines whether the model answers the input request accurately. Therefore, the ASR from GPT4-judge refers to harmful answers in the interests of the attacker (see the "Prompt template for GPT-4 judge" in Appendix A.1). The experimental results show that the ASR of AttnGCG has increased (see Table 1), indicating LLMs poisoned by AttnGCG answers harmful questions more correctly than LLMs attacked by other methods. We also show generated responses of GCG and AttnGCG in Appendix B.3.

$**Q3: The relationship between maximizing the attention weights for suffix and a stronger attack**$

Maximizing the attention weight of the suffix token aims to improve the LLM jailbreak, making the model more inclined to answer the user's request. However, the "Maximization" of suffix attention has a boundary, that is, the answer content must be related to the goal. In AttnGCG, we use Target Loss $L_{t}$ as the regulator. As presented in the experiments in section 3.2, AttnGCG has a better attack effect, which verifies the claim.

The reason why "AutoDAN leads to a weaker attack with a lower goal attention score" is that a low goal attention score will steer a model's response to be irrelevant with the goal. Please note that, our AttnGCG is not designed to minimize the goal attention in the input, which is also clarified in the first point replied to reviewer Hgf8.

$**Q4: Concerns about capability to bypass perplexity-based defense**$

Our AttnGCG, along with other jailbreaking methods that generate adversarial suffixes, are unlikely to bypass the perplexity-based defenses. This is because the adversarial suffix always has a higher PPL than natural language, which can be detected via the PPL metric easily. However the perplexity-based defense is a deployment-level method --- it can be deployed before instead of during using an LLM. The main contribution of AttnGCG lies in the methodology aspect. (1) In the future, we can try to use other deployment-level methods to bypass the perplexity-based defense. (2) The new optimization objective of attention score can also assist other methods in the future (e.g. Attn-AutoDAN). AttnGCG is a heuristic verification.

[1] Universal and Transferable Adversarial Attacks on Aligned Language Models

We first thank the reviewer for the detailed comments and the appreciation of our work. We address the concerns below:

$**Q1: Are attention scores normalized?**$

Yes. The attention weights matrix is the value after softmax, and the attention weights of each target token are normalized to sum to 1. The reason why the attention score is greater than 1 in Tables 2 and 3 is that the attention score is not averaged over the target token length but added. For Figures 4 and 5, the goal and target used in this paper are the same (see Appendix B.3), so the values in Tables 2 and 3 are comparable.

The reason that "in Figure 2, the attention scores of all parts increase as the optimization proceeds" is caused by the auto-regressive generation and the separator token (e.g., [INST]) in the model input. The decrease in attention to the auto-regressive generation and separator will lead to your observation.

$**Q2: Proposals for alternative attention score normalization**$

Thank you for your suggestions. We only consider the normalized attention weights (after softmax) of the goal and suffix in the model input, ignoring the separator. As for the attention weight matrix changing brought by the autoregressive generation, it is an inherent feature in the self-attention mechanism, which is not in the scope of this paper. And we make sure that this setting is consistent throughout the paper to make fair comparisons.

$**Q3: Does Figure 2 (left) contradict the main claim of the paper?**$

No, Figure 2 (left) does not contradict our main claim. First, a higher goal attention score does not necessarily mean a lower suffix attention score (See Q1 above). The attention score on the suffix also increases in Figure 2 (left), which supports our claim. Second, in Figure 2, our claim is mainly supported by the comparison between the left and right sides of the picture, not just from one side. Specifically, in both sides of Figure 2, the attention score on the goal converges to a certain low level (after 100 steps), which indicates a possibly successful attack. On the right side of Figure 2, a higher attention score on the suffix increases the probability of a successful attack in the same training steps compared with Figure 2 (left), which means increasing the suffix attention score can produce a more effective one for attack.

$**Q4: The consistency between Figure 5 and the main claim of the paper**$

Sorry for the confusion in the figure. We place the color bar and its concrete statistics aside from the figure. The ICA's goal attention score is lower than Vanilla based on the color bar. In Figure S.1 and Figure S.2 of the rebuttal supplementary material, we have added a clearer comparison and unified the color bar scale.

$**Q5: Purpose of "Section 3.3: Generalize AttnGCG to other attack methods"**$

The purpose of 3.3 experiment is to prove that AttnGCG is an attack method that can be easily integrated into other jailbreaks, which can further optimize the existing methods that have been optimized to convergence at the prompt level. We can seamlessly incorporate AttnGCG into other jailbreaking attacks through the initialization.

$**Q6: Ablation experiment on$w_{t**$and$w_{a}}

Table S.2: Ablation for $w_{a}/w_{t}$ on Gemma-7b-it. The result format is "GPT-4 judge (keyword-detection)".

This table is also presented at Table S.2 in the rebuttal supplementary material.

$**Q7: Additional experiments on other attack methods**$

Thank you for your suggestions. We will add additional results in the next revision.

$**Q8: Limited empirical improvement**$

Thank you for your recognition. AttnGCG demonstrates consistent performance enhancements across diverse LLMs, with an average improvement of 7% in the Llama-2 series and 10% in the Gemma series. Possible applications of AttnGCG include replacing GCG in the future, as you have mentioned.

$**Q9: Does the first suffix token have the highest influence on the output?**$

No. It is a coincidence that the highest attention score is at the first suffix token in Figure 4(right). We will add more qualitative examples to showcase different successful attacking cases in the revision. In Figure 5 (ICA, top left), the attention weight is not concentrated on the first suffix token. Please refer to the actual input prompt of ICA in Appendix B.3 for token position matching.

$**Q10: Attention maps for successful GCG and failed AttnGCG**$

We show the attention maps of successful GCG and failed AttnGCG cases in Figure S.3 of the rebuttal supplementary material.

$**Q11: Is the observation dependent on the optimization steps?**$

No. For the optimization step, we choose the number that both AttnGCG and GCG start to converge, which is around the 60th step.

$**Q12: Are the scores in Table 2 and 3 also average across the 100 samples?**$

No. Tables 2 and 3 are the average values of the matrix in Figures 4 and 5. We use Tables 2 and 3 as the numerical explanations of these two visualizations.

We first thank the reviewer for the detailed comments and the appreciation of our work. We address the concerns below:

$**Q1: Ambiguous results about solving failed 'regret' jailbreaking cases in GCG.**$

The 'regret' jailbreaking case mentioned is a subcase of failed jailbreakings, which is caused by "a high probability of harmful tokens does not necessarily equate to a successful jailbreak" (L36). And our GPT-4 evaluator takes such failed jailbreaking case into account. In detail, the GPT-4 judge will only consider a case to be a successful attack if and only if the model respond to the request accurately. That is to say, the 'regret' situation will be considered a failed attack. In the experiments, the GPT-4 evaluated ASR of our method is improved (Table 1), demonstrating a better capacity of AttnGCG to handle this scenario.

$**Q2: Is the phenomenon about the position of high attention score in Figure 4 common in success cases?**$

No, this is not a common phenomenon in successful cases. We will add more qualitative examples to showcase different successful attacking cases in the revision.

$**Q3: Concerns about system prompt settings for Llama series.**$

Thank you for raising this question. Unlike Llama2, Llama3 was released without a specified system prompt. With the chat template of Llama3 changed significantly compared with Llama2, we did not use the official system prompt of Llama2 for Llama3, instead, we set it to None. For a fair comparison across the Llama series, we then set the system prompt of Llama2 to None. And also note that, the system prompt of Llama2 in AutoDAN [1] is also set to None, which is of reference value.

The ASR results of Llama2-7b-chat with its official system prompt are reported below (also in Table S.1 in the rebuttal supplementary material). We can observe that the Llama2 with its official system prompt is more difficult to breach, requiring more steps to converge. We will add results of Llama3 in the revision.

Table S.1: Results of Llama2-7b-chat after enabling the standard system prompt (the criterion for stopping optimization is Loss convergence, which is 1000 steps in the experiment, and the other parameters are the same). The data format is "GPT-4 judge (keyword-detection)".

$**Q4: Concerns about the capability to bypass perplexity-based defense.**$

Our AttnGCG, along with other jailbreaking methods that generate adversarial suffixes, are unlikely to bypass the perplexity-based defenses. This is because the adversarial suffix always has a higher PPL than natural language, which can be detected via the PPL metric easily. However the perplexity-based defense is a deployment-level method --- it can be deployed before instead of during using an LLM. The main contribution of AttnGCG lies in the methodology aspect. (1) In the future, we can try to use other deployment-level methods to bypass the perplexity-based defense. (2) The new optimization objective of attention score can also assist other methods in the future (e.g. Attn-AutoDAN). AttnGCG is a heuristic verification.

[1] AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models

We first thank the reviewer for the detailed comments and the appreciation of our work. We address the concerns below:

$**Q1: Will increasing attention scores for adversarial suffixes prioritize their content in responses?**$

No, increasing the attention scores of adversarial suffixes will not make the responses focus on the content in the adversarial suffixes. The optimization objective contains Attention Loss $L_{a}$ and Target Loss $L_{t}$. Target Loss $L_{t}$ ensures that the response will focus on the original Target content, thereby limiting the goal's attention score from being too low. Figure 2 (right) in the paper can support this view. It can be observed that the attention score on the goal converges approximately after step 100.

$**Q2: Ambiguous discussions in the attention score visualization.**$

Sorry for the confusion. We clarify that the purpose of visualizing the Attention Map is to visually verify the effectiveness of our method. Specifically, we expect Figure 4 can verify that "In the successful jailbreaking case, attention is notably shifted to the suffix part, resulting in a decrease in attention from the goal", that is, reducing the model’s excessive attention to the goal and thus "bypassing the internal safety protocol", and reducing the model’s excessive attention to the goal is achieved by increasing the attention score of the adversarial suffix. Experiments show that higher attention scores for adversarial suffixes mean more effective adversarial suffixes and higher ASR (Figure 2, Table 1 in the paper), which supports our claim.

$**Q3: Is AutoDAN better than AttnGCG?**$

No, AutoDan is not always better than our method. From Table 4, the attack effect of AutoDAN is worse than AttnGCG, although AutoDAN has a lower attention score on the goal. This is because the LLM may respond irrelevant content to the goal with lower attention score on the goal (that's why we need the target loss as a regulator). In Table 4, the ASR of AutoDAN on keyword-detection is similar to AttnGCG, indicating that LLMs attacked by either method do not refuse to answer requests, but the ASR of AutoDAN on GPT-4 judge is much lower because the answer generated by AutoDAN are not recognized as an accurate one to the input request.

$**Q4: Redundant content.**$

Thanks for the suggestion. We will keep Figure 1 as the teaser, aiming to visually compare the difference and improvement between AttnGCG and GCG; we will remove Algorithm 1.