Uncovering Safety Risks of Large Language Models through Concept Activation Vector

Thanks for your insightful and detailed comments, which enable us to better clarify our contributions and ethical considerations.

Weakness 1: Interpretability used in attacks by [1]

We acknowledge that previous works like [1] have explored the use of interpretability to assist attacks. In contrast, our work distinctively focuses on developing a more accurate and principled interpretation method. Unlike existing approaches, our method:

1. Eliminates misleading heuristics (L128-140) and thus performs significantly better than existing techniques [1,2] (Tables 1 and 2), particularly under conditions of limited training data (Figure 3).
2. Is the first to support both embedding-level and prompt-level attacks, enhancing its applicability.
3. Removes the necessity for time-consuming hyperparameter tuning.

We will update our abstract and introduction to better highlight these distinctions.

Weakness 2: Confusion in Figure 3

We have double-checked the result in Figure 3 and confirmed it is correct. One pair of data can lead to high performance because embeddings of malicious questions are well separated from those of safe questions (Rebuttal Table 1). In this case, we can learn a linear classifier $P_m$ that accurately distinguishes embeddings of malicious questions from safe ones with one pair of data (Rebuttal Table 2) and use $P_m$ to effectively guide the attacks.

[Rebuttal Table 1] Distances between embeddings of malicious and safe questions ($d_{ms}$) exceed those within malicious ($d_{mm}$) or safe questions ($d_{ss}$) in LLaMA-2-7B-Chat 31th layer.

[Rebuttal Table 2] Test accuracy of linear classifier $P_m$ using one pair of training data (LLaMA-2-7B-Chat)

Weakness 2: Risks in modifying intermediate representations

We carefully ensure such risks are under control by:

1. Optimizing with drift control. Our goal minimizes perturbation magnitude $|\epsilon|$ to avoid significantly modifying intermediate representations (Eq. (3)). We also modify only the necessary layers (Algorithm 1).
2. Performance validation. In addition to our human evaluation (Table 2), which confirms that LLMs modified by our method have good performance, we also implement the weighted average of bi- and tri-gram entropies per your suggestion. This quantity drops if the generated text is repetitive. As shown in Rebuttal Table 3, our method consistently performs the best in terms of this criterion.

[Rebuttal Table 3] Comparing our embedding-level attacks with baselines in terms of the weighted average of bi- and tri-gram entropies [3, 7]

Weakness 3: Ethical considerations in human evaluation

We take the following steps to adhere to ethical standards:

1. Obtaining IRB-equivalent approval. Our human annotation experiment was carried out through a vendor company that has a formal contract with our institution. We have carefully confirmed with our collaborator (level 3 program manager) in the vendor company, and she has investigated our annotation process and confirmed it has passed their ethical approval.
2. Doing comfort for the annotators. We reviewed the labeling content to ensure that there was no extremely harmful content, warned the annotators about the potential harm, and let them know that they could quit at any time if uncomfortable. We also conducted a follow-up with the annotators, and they all indicated that our experimental content had not caused harm to them.

Question 1: Transferability to GPT-4

The transferability of attacks observed in our experiments is also reported in multiple existing works [4,5,6], enhancing the credibility of our results. For example, it has been found that attacks learned in LLaMA-2 can be effectively applied to GPT-3.5, GPT-4, and Vicuna [4,5,6]. Currently, to the best of our knowledge, no existing works can clearly explain its reason. One hypothesis is that there is a form of cross-model generalizability potentially resulting from using a similar architecture (Transformer), loss function (e.g., next-token-prediction), and datasets (human-written text from the Internet), and we consider the verification of this hypothesis to be an interesting future work.

[1] Representation engineering: A top-down approach to ai transparency

[2] Open the Pandora's Box of LLMs: Jailbreaking LLMs through Representation Engineering

[3] A Unified Framework for Model Editing

[4] Universal and transferable adversarial attacks on aligned language models

[5] AutoDAN: Generating stealthy jailbreak prompts on aligned large language models

[6] AdvPrompter: Fast adaptive adversarial prompting for LLMs

[7] MASS-EDITING MEMORY IN A TRANSFORMER

Weakness 1 and Question 1: Linear interpretability assumption

Thanks for explaining your concern in such a detailed and constructive way. Per your suggestion, we further justify the assumption by providing:

1. Empirical evidence across models and layers. As shown in Rebuttal Table 1, the test accuracy of linear classifier $P_m$ for 5 more LLMs consistently exceeds 0.9 in later layers, confirming the robustness of the assumption. Moreover, our attack method, built upon this assumption, has achieved an ASR-keyword of over 94% on 7 LLMs, demonstrating its general applicability. You can also see Figure 1 in Rebuttal PDF for details by layer.
2. Further explanations. A solid theoretical explanation for linear interpretability is a difficult task not solved by works that use the assumption (refs [20-23] in paper). Our hypothesis is two groups of embeddings tend to be linear separable if learned to have sufficient inter-group distance and intra-group similarity. For example, in LLaMA-2-7b-Chat, the median distances between embeddings of malicious and safe questions ($d_{ms}$) exceed those within malicious ($d_{mm}$) or safe questions ($d_{ss}$) - 113.88 vs. 56.21 and 84.89, respectively. This disparity, learned to ensure malicious questions have different outputs compared with safe questions, may form clear clusters easily to be linearly separated.

[Rebuttal Table 1] Minimal test accuracy of linear classifier $P_m$ across LLMs and layers

Weakness 2: Choices of regularization terms

The current regularization follows the default setting in sklearn and consistently performs well on varying models and datasets. We also test different regularization terms. Rebuttal Table 2 shows that using the L2 regularization is critical, while its coefficient is not sensitive. We will include this analysis in the revised paper.

[Rebuttal Table 2] ASR-keyword (%) w.r.t different regularization terms (Advbench, LLaMA-2-7B-Chat)

Weakness 3: Layers to attack

We attack only layers with an accurate $P_m$ because our attacks are guided by $P_m$ (Eq. (3)) and inaccurate $P_m$ leads to ineffective attacks. This is consistent with empirical results: perturbing a layer whose test accuracy is lower than 90% results in a low attack success rate (ASR-k from 0 to 6%), while perturbing other layers leads to a consistently better result (41.1$\pm$26.2%).

Weakness 4 and Question 2: Determining $P_0$ and $P_1$

Our method works reasonably well for varying $P_0$ and $P_1$, as shown in Rebuttal Table 3. Users can easily determine $P_0$ and $P_1$ by using the default values (0.01% and 90%) that work well for all 7 LLMs we tested, or slightly lower $P_0$ if they wish to increase ASR. An optimal $P_1$ can be found by plotting $P_1$ across layers and choosing the elbow point, but the default value already works well in practice.

[Rebuttal Table 3] ASR-keyword (%) w.r.t. varying $P_0$ and $P_1$ (Advbench, LLaMA-2-7B-Chat).

Weakness 5: More datasets and diverse LLMs

Beyond the 2 datasets and 8 LLMs already tested (Table 5), we have extended our experiments to Harmbench and 3 more LLMs with different training paradigms. Rebuttal Tables 4 and 5 show the effectiveness of our method. We also hope to highlight that 3 reviewers claim that our original experiment is “comprehensive” or “thorough.”

[Rebuttal Table 4] Results on Harmbench

[Rebuttal Table 5] Results on 3 more LLMs on Advbench

Question 3: Applying defensive mechanisms like adversarial training

Per your suggestion, we attack models with adversarial training (Rebuttal Table 6) and other defense mechanisms (Rebuttal Table 7). The unlearn method in Sec. 4.2 can also be regarded as a defense method. Results show that all 6 tested defense methods cannot well mitigate our attacks.

[Rebuttal Table 6] Attacking LLMs with adversarial training [1] on Advbench

[Rebuttal Table 7] Attacking LLMs with defense methods (ASR-keyword, %)

[1] Improving Alignment and Robustness with Short Circuiting.

[2] Defending chatgpt against jailbreak attack via self-reminders.

[3] Jailbreak and guard aligned language models with only few in-context demonstrations.

[4] Baseline defenses for adversarial attacks against aligned language models.

[5] Detecting language model attacks with perplexity.

Thanks for your insightful comments and questions.

Weakness 1: Optimizing only the last layer in token-level attack

We optimize only the last layer to ensure

1. A fair comparison with baselines: our baselines (e.g., [1]) use only information from the last layer so we follow their setting to ensure a fair comparison.
2. Simplicity and effectiveness: using only the last layer is 1) simple - does not require balancing different layers of varying embedding scales and 2) effective - better than baselines and achieves a comparable performance with a method considering multiple layers, as shown in Rebuttal Table 1.

[Rebuttal Table 1] Comparison with SCAV-m, which considers multiple layers by extending Eq. (5) to $\arg\min_{S} \max_{l}[P_m(e_S^l) ||e_S^l - e^l||],\text{TestAcc}(P_m^l)>P_1$ on Advbench

Weakness 2: What would happen when perturbing only the last layer

Perturbing only the last layer results in worse ASRs and more language flaws, as shown in Rebuttal Table 2. We suspect it is because, at the last layer, the inference procedure has nearly finished, thus forming a completely different answer can be difficult.

[Rebuttal Table 2] Comparison with SCAV-l that perturbs only the last layer

Weakness 3: Impact of each layer on ASR

We provide the results in the rebuttal PDF attached to the global rebuttal. Our findings are:

1. For both levels, ASR is poor on layers that are not linearly separable (Figure 2 in the rebuttal PDF).
2. For embedding-level attacks, perturbing the last layer is less effective compared with some earlier layers (e.g., layers 13 to 23). Algorithm 1 tends to select these effective (Figure 2 in the PDF).
3. For prompt-level attacks, using middle or late layers has a comparable attack performance (Figure 3 in the PDF).

Weakness 4: Why product form

We use the product form because it works sufficiently well (Table 3) without introducing an additional hyperparameter $\lambda$ for balancing $||e^L_S-e^L||$ and $P_m$, which is required for the weighted sum expression. In the product form, the percentage of increase in $||e^L_S-e^L||$ is considered to be similarly important to the percentage of increase in $P_m$, without having to consider their difference in scales.

Weakness 5: Comparison to soft prompt attack [2]

As shown in Rebuttal Table 3, our embedding-level attack method outperforms the soft prompt attacks in terms of ASRs and language flaws.

[Rebuttal Table 3] Comparing with soft prompt attacks

Weakness 6: Comparison to GCG

Results show that our prompt-level attack method consistently outperforms GCG on different datasets and tasks (direct prompt-level attack in Rebuttal Table 4 and transferability in Rebuttal Table 5 of the rebuttal PDF).

[Rebuttal Table 4] Comparing with GCG

Questions 1 and 2: What does $e^l$ refers to?

$e^l$ is the embedding of the last input token. Specifically, given an input instruction with N tokens, we first attack the embedding of the N-th token to generate the first token in the answer. We then iterate this process to generate a full answer.

Question 3: How is Figure 2 created

Figure 2 is generated by 1) generating two-dimensional pseudo-embedding data points and 2) running the attacking methods (ours, RepE, and JRE) to obtain the perturbation vector.

Question 4: Difference between ASR-answer and ASR-useful

ASR-useful is a more strict criterion than ASR-answer. ASR-answer mainly evaluates whether the model answers or rejects to answer the question, while ASR-useful evaluates whether the model answer is truly useful for attackers. For details, please refer to Appendix A.3.2 and F.

Question 5: Is SCAV in Table 7 embedding or prompt level?

It is embedding-level SCAV. The prompt-level baselines are added following [3].

[1] AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models

[2] Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space

[3] Eraser: Jailbreaking Defense in Large Language Models via Unlearning Harmful Knowledge

Thanks a lot for your insightful and constructive comments. We really appreciate the efforts that you have made in helping improve our paper.

Weakness 1: Interpretability used in attacks by [1].

We acknowledge that previous works like [1] have explored the use of interpretability to assist attacks. In contrast, our work distinctively focuses on developing a more accurate and principled interpretation method. Unlike existing approaches, our method:

1. Eliminates misleading heuristics (L128-140) and thus performs significantly better than existing techniques [1,2] (Tables 1 and 2), particularly under conditions of limited training data (Figure 3).
2. Is the first to support both embedding-level and prompt-level attacks, enhancing its applicability.
3. Removes the necessity for time-consuming hyperparameter tuning.

We will update our abstract and introduction to better highlight these distinctions.

Weakness 2: Need to discuss why the proposed attack is more effective

Why our method is more effective is introduced in the method part (L128-140, L158-165) and we will discuss more in the experiments.

1. For embedding-level attacks, our method is more effective because we remove potentially misleading heuristics of existing methods (L128-140). While existing works assume that some heuristically extracted direction (e.g., the main component of PCA) works well for perturbing the embedding without theoretical justification, we learn the malicious probability of each embedding from data to provide an accurate perturbation.
2. For prompt-level attacks, our method can more accurately estimate the attack success rate because, unlike the existing works, we do not rely on manually defined target responses, which are often different from the real model responses (more details in L158-165).

Weakness 3: How to mitigate the attacks

Mitigating attacks is important and we will add the following discussions in the revised version:

1. Applying existing defense methods cannot effectively mitigate our attacks. As shown in Rebuttal Table 1, after applying existing defense methods [3,4,5,6], our attack method still achieves a high attack success rate (ASR) on two datasets according to the widely used criterion ASR-keyword (%). Here, the victim LLM is LLaMA-2 (7b-Chat). This indicates that our work identifies the inherent safety vulnerability of LLMs. Moreover, the unlearn method in Section 4.2 can also be regarded as a defense method and it also fails to mitigate our attacks (Table 7). We also attack models with adversarial training proposed by [8](Rebuttal Table 2).

   [Rebuttal Table 1] The attack success rate (ASR-Keyword in %) after applying defense methods

   Defense Method	On Advbench	On StrongREJECT
   ICD [3]	98	96
   Self-Reminder [4]	92	100
   Paraphrasing [5]	98	98
   PPL [6]	100	100
   W/o Defense	100	100

   [Rebuttal Table 2] Attacking LLMs with adversarial training [8] on Advbench

   Models	ASR-keyword (%)	ASR-answer (%)	ASR-useful (%)	Language Flaws (%)
   LLaMA-3-8B-Instruct-RR	98	88	74	16
   Mistral-7B-Instruct-RR	94	84	70	20

2. Suggesting other ways for mitigating the attacks. One potential way to mitigate the attacks is to distinguish perturbed embeddings from original ones, based on the assumption that some perturbed embeddings might be different from original embeddings. We are eager to explore whether such an assumption is valid in the future.

Question 1: Implementation detail of prompt space attack

Per your suggestion, we will add the following details to ensure that our paper is self-contained:

“Specifically, the hierarchical genetic algorithm is tailored for structured prompt text. It views the jailbreak prompt as a combination of paragraph-level population and sentence-level population. At each search iteration, it first optimizes the sentence-level population by evaluating and updating word choices within sentences. Then, it integrates these optimized sentences into the paragraph-level population and performs genetic operations to refine sentence combinations, ensuring comprehensive search and improvement across both levels with high jailbreak performance and readability.”

Question 2: Comparing with fine-tuning attacks [7]

Per your suggestion, we compare with [7] and will add related results in the paper. As shown in the following tables, our SCAV embedding-level attack:

1. achieves a higher attack success rate (ASR-keyword) compared with [7] (Rebuttal Table 3)
2. requires much less time cost and computation memory when applying on LLaMA-2-7B-Chat, since it only requires inference (Rebuttal Table 4). The evaluation is conducted on Advbench.

[Rebuttal Table 3] Comparison with fine-tuning attacks in terms of ASR-Keyword (%)

[Rebuttal Table 4] Comparison with fine-tuning attacks in terms of efficiency

[1] Representation engineering: A top-down approach to ai transparency

[2] Open the Pandora's Box of LLMs: Jailbreaking LLMs through Representation Engineering

[3] Jailbreak and guard aligned language models with only few in-context demonstrations

[4] Defending chatgpt against jailbreak attack via self-reminders

[5] Baseline defenses for adversarial attacks against aligned language models

[6] Detecting language model attacks with perplexity

[7] Fine-tuning aligned language models compromises safety, even when users do not intend to!

[8] Improving Alignment and Robustness with Short Circuiting