Protecting Your LLMs with Information Bottleneck

Dear Reviewer,

We sincerely appreciate your positive feedback and encouraging comments on our paper. In our experiments, we similarly found out why the llama replication is inconsistent, which is the reason for the experimental setup. As mentioned in line 569, the template for each model uses FastChat version 0.2.20. Because the system templates in llama2 are different for different versions, the probability that llama2 will be attacked successfully is also different. Indeed, we carefully checked many papers on jailbreaks, but the results varied greatly! Fortunately, this does not affect the results of the defense experiments, i.e., the more prompts on LLMs that can be jailbroken successfully the better to test the defense methods. We will later open-source the data generated by GCG and PAIR for reproducibility.

Regarding limitations:

1. The defense relies on the target model itself.

   Response: This is exactly one of our motivations, highlighting potentially harmful tokens gives the target LLM so that the LLM can recognize them directly. The method is lightweight and requires no modifications to the LLMs.

2. Perturbations may result in out-of-distribution.

   Response: Yes, it's a limitation we considered. As discussed in line 197, we added a second term $D_{KL}(f_{tar}(\tilde{X}, Y_{<t}) || f_{tar}(X, {Y}_{<t}))$ in Eq. 7 to alleviate the OOD problem, rather than simply through vanilla cross-entropy in the informativeness predictor. In fact, the performance is also superior in the transfer experiments of Figure 3, which means that for the target model, the generated $\tilde{X}$ are OOD perturbations.

3. Generating coherent sentences.

   Response: Generating coherent sentences is possible through IB, but this requires a large model to ensure the quality of the generation (paraphrasing/compressing/summarizing are shown in the baseline Semantic Smooth). Therefore, this route increases the time and computational overhead even more, while the mask only needs to act on the original prompts. We consider $X_{sub}$ as an intermediate result that does not need to be understood by humans as it will act on the specific target model, i.e., part of highlighting is explaining why harmful. Moreover, in the extraction paradigm, we enhance the coherence of $X_{sub}$ through continuity loss in Eq. 5, so it's not completely incomprehensible.

Dear Reviewer,

We thank the reviewer for the detailed constructive feedback on our work and answer the questions below:

1. Defending against jailbreak attacks uses only benign words.

   Response: Thanks for your constructive comment. It is important to note that PAP is not entirely composed of benign words. As Figure 1 in the paper of PAP, persuaded prompts likewise exist for words like bomb, war, etc., which is why the target model gives "how to make a bomb". These semantic-level attacks are similar to PAIR and AutoDAN that we conducted, confusing the target model. Our goal is to highlight informative tokens likely to be unsafe so that the target LLM itself can recognize them. If a malicious question changes to a benign question through Prompt Optimization, we believe that it's not a well-defined attack method.

2. Generating contextually coherent sentences.

   Response: As discussed in line 335, generating coherent sentences is possible through IB, but this requires a large model to ensure the quality of the generation (paraphrasing/compressing/summarizing are shown in the baseline Semantic Smooth). Therefore, this route increases the time and computational overhead even more, while the mask only needs to act on the original prompts. We consider $X_{sub}$ as an intermediate result that does not need to be understood by humans as it will act on the specific target model, i.e., part of highlighting is explaining why harmful. Moreover, in the masking paradigm, we enhance the coherence of $X_{sub}$ through continuity loss in Eq. 5, so it's not completely incomprehensible. In fact, the masking paradigm is the most lightweight and efficient rather than generating, and it is widely used in tasks such as images[R1], graphs[R2], time series[R3], etc.

[R1] Fong, et al. "Interpretable explanations of black boxes by meaningful perturbation." CVPR, 2017.

[R2] Miao, et al. "Interpretable and generalizable graph learning via stochastic attention mechanism." ICML, 2022.

[R3] Liu, et al. "TimeX++: Learning Time-Series Explanations with Information Bottleneck." ICML, 2024.

3. Cipher-based jailbreak attacks.

   Response: Thanks for the thoughtful concern! We conduct an additional experiment to evaluate the effectiveness of defense methods against cipher-based attacks, as proposed by Yuan et al. 2024. Given the overall low response valildity observed in most models such as llama2 smarter than GPT4, we only can opt to perform transfer attacks exclusively on GPT4 for all defense methods. To ensure that defense methods based on semantic information are meaningful, we apply all defense methods prior to encoding the text using ASCII, Caesar, and Morse ciphers. We also consider the SelfCipher, which is similar to a kind of few-shot jailbreak. We test 50 instances from AdvBench and report the attack success rate as shown in Table R2. Our results indicate that the IBProtector outperforms all other baselines in defending cipher-based attacks.

   Table R2: Attack success rate for Cipher attacks with valid responses on GPT4.

   Method	ASCII Cipher	Caesar Cipher	morse Cipher	Self Cipher
   Original Attack	0.0%	56.0%	30.0%	52.0%
   Smooth LLM	0.0%	58.0%	22.0%	32.0%
   RA-LLM	2.0%	60.0%	18.0%	48.0%
   Semantic Smooth	0.0%	38.0%	24.0%	36.0%
   IBProtector	0.0%	24.0%	18.0%	26.0%

4. Adaptive attacks based on IBProtector.

   Response: Thanks for the nice question. Due to the diversity of attacks, there is no suitable benchmark for defense methods to be tested. To address your concern, we have conducted the following experiment. Rule-based or longtail Encoding mutations are insufficient for adapted attacks as they are fixed. Therefore, we select a prompt optimization, PAIR, to explore the iteration number of successful jailbreaks with/without defense mechanisms. If the number of iterations is large, it is difficult to be jailbreaking by adapted attacks. We compare several baselines where the filter exists: Smooth LLM, RA-LLM, Semantic Smooth, and IBProtector. We set the maximum number of iterations to 20, with three mutants per iteration. As shown in Table R3, the experimental results indicate that IBProtector can mitigate adaptive attacks and make them more costly compared with other baselines.

   Table R3: Average number and rate of iterations required for a successful jailbreak by an adaptive attack on 50 instances.

   Method	iteration(Vicuna)	ASR(Vicuna)	iteration(LLama2)	ASR(LLama2)
   Original Attack	6.06$\pm$6.17	92.0%	13.76$\pm$7.04	52.0%
   Smooth LLM	5.86$\pm$4.73	96.0%	14.06$\pm$6.91	52.0%
   RA-LLM	6.38$\pm$5.69	90.0%	13.32$\pm$7.09	58.0%
   Semantic Smooth	8.40$\pm$6.62	86.0%	14.28$\pm$7.61	44.0%
   IBProtector	15.60$\pm$5.64	52.0%	16.18$\pm$6.06	36.0%

Dear Reviewer,

We greatly appreciate your insightful comments! Here are our responses to the comments.

1. Potential Bias.

   Response: Thanks for pointing this question out. The low-entropy problem comes from minimizing the mutual information term $I(X; X_{sub})=H(X_{sub})-H(X_{sub}|X)$ which is upper bounded by $H(X_{sub})$. The optimal mutual information can be achieved through the shortcut where $X_{sub}$ has a very simple distribution independent of $X$, for example, the extractor only preserves high-frequency words in $X$ regardless of its content. This kind of overly simplistic extraction resulting from the possibility of low entropy $X_{sub}$ is meaningless in the sense of preserving information. Therefore, the mitigation method intuitively moves the distribution of $X_{sub}$ away from having low entropy by adding a KL divergence regularization term with a Bernoulli variational approximation $\mathbb{Q}(X_{sub})$. This regularization demonstrates better performance than deterministic methods [R1] while also eliminating the less tractable marginal entropy term $H(X_{sub})$.

[R1] Alemi, et al. "Deep variational information bottleneck." ICLR, 2017.

2. High Dimensionality Challenge.

   Response: Thanks for your comment. While a current line of research on jailbreaking attack/defense mainly focuses on short instruction following tasks (e.g., QA tasks), your consideration regarding large or complex input is valuable. We contend that our extraction-based defense method is conceptually transferable to large input (for example, toxic commands with a document). This is because we highlight harmful content that is prone to trigger the target LLMs' rejection. This methodology alleviates the need to actually understand the complex input and is more scalable to the increased complexity of the input content. In addition, adversarial attacks using images [R2] as a high-dimensional input can also be considered in VLMs. If IBProtector is applied, this mask resembles a kind of post-hoc interpretation instant of class activation map in the adversarial image. However, this is beyond the scope of our paper and we will explore it in the future.

[R2] Gupta, et al. "Ciidefence: Defeating adversarial attacks by fusing class-specific image inpainting and image denoising." CVPR, 2019.

3. Different types of LLMs and various adversarial attack scenarios.

   Response: Thanks for the nice question. Indeed, we have considered this issue in transfer experiments (Section 5.3 Transferability). The IBProtector can defend against other attack methods (including Autodan, ReNellm, and GPTFuzz) unseen during training (see Table 2), and protect other LLMs that have not been detected during training (see Figure 4). Regarding very large models (llama2-70b/mistral 8x7B), a big limitation is that it's tough for us to get the training data that would allow for a successful attack. Jailbreaking Prompt Optimization is too time-consuming and usually takes 800+ hours (Figure 4 in [R3]). Therefore, not only us, existing literature has not considered the very LLMs of direct white-box attacks as it's hard to generate data. It is worth noting that for a better-performing LLM, we positively infer that IBProtector performs better. As claimed that the primary defense relies on the model itself in lines 211-213, a safer LLM can better maximize $I(Y, X_{sub})$ in the training phase and identifies highlight tokens more efficiently in the inference phase. Thus IBProtector can perform well, as confirmed in transfer experiments with chatgpt and gpt-4.

[R3] Zhou, et al. "EasyJailbreak: A Unified Framework for Jailbreaking Large Language Models." ArXiv, 2024.

Dear Reviewer,

Thank you for your insightful suggestions. We answer the questions below:

1. Training set of the baselines. (Some misunderstandings)

   Response: For training data, we are definitely setting the SAME. In addition, the test dataset is DIFFERENT from training data. The first 120 instances serve as the test adversarial dataset and the last 400 as the training adversarial dataset along with GCG and PAIR attacks. Besides, we sample 400 instances in the TriviaQA as training normal data, so all methods requiring training totally contain 1200 prompts.

2. Testing against white-box attacks. (Some misunderstandings)

   Response: We TESTED against white-box attacks (Table 1). Taking Vicuna as an example of a target model, harmful prompts are also generated by GCG and PAIR on Vicuna. First, we use 1200 for training, and then ANOTHER 120 for testing. These harmful prompts are specific to Vicuna, not transfer attacks. So our well-trained IBProtector is also specific to Vicuna and it's been known to see GCG and PAIR attacks during training. We will make Appendix D.1 clearer in the next version.

3. Autoregressive model.

   Response: Here are two main reasons. First and foremost, consider the need for gradient backpropagation, which inevitably requires the use of the same tokenizer as the target model. The 'padded' prompt cannot be optimized if inconsistent tokenizers, which is important. The tested target models are all of the autoregressive type, thus we selected a llama-based small model as the embedding extractor. Secondly, most of the mask generator on the sequence is based on the TransformerDecoder [R1], as it is related to the order predicted by the predictor.

[R1] Queen, et al. "Encoding time-series explanations through self-supervised model behavior consistency." NeurIPS, 2023.

4. Incorporating prior mask.

   Response: Thank you for the thoughtful comment! We have introduced a smooth term in Eq. (5) to penalize non-continuous masks so that our mask at a different position is not independent, where it is a form similar to a linear-chain conditional random field. However, we appreciate the idea of autoregressively predicting the attribution score, which represents the continuous probability of the mask based on previously sampled masks. We carry out the following design and results in the next version. Due to word limitations, please see the General Response.

5. Training losses.

   Response: These approximations are necessary. As discussed in lines 129-133 and 191-198, our approximation mainly solves the compression issue and the signaling issue for giving a traceable objective function, otherwise, it is hard to directly optimize IB to obtain sub-prompts. In addition, Appendix E.2 empirically demonstrates the validity of our extracted sub-prompts, which indicates that the original informativeness is preserved. We kindly request that the reviewer revisit experiments to help resolve any misunderstandings.

6. Intention compared with finetuning (Some misunderstandings).

   Response: As defined in Preliminaries 3.1, $f_{tar}(X_{ori})$ is not jailbroken successfully, while $f_{tar}(X)$ is a harmful reply (e.g. Sure, I can ...). So the goal of alignment should be $f_{tar}(X_{sub}) \approx Y$ rather than $f_{tar}(X)$. Although the same data X and Y are required for fine-tuning, our IBProtector is not fine-tuning, but "prompt-tuning", i.e., we are optimizing $X_{sub}$ not optimizing the target model. It has the benefit of being lightweight and doesn't require a lot of fine-tuned data. Regarding more filtering than compressing, it's also a compression by eliminating the tokens that are low in information content. You can think of it as fine-tuning a compression model p that $X_{sub} = p(X) \approx X_{ori}$. However, in the real world, we do not know $X_{ori}$ and it may not always satisfy the expected target Y. Therefore, information bottleneck concepts are the most direct way to conduct "prompt-tuning" to detect prompts that are identified as harmful.

7. Why does it perform better than finetuning?

   Response: While fine-tuning is able to align the LLM with human values, it is highly resource-intensive, particularly as the complexity of the model and the scale of harmful inputs increase. In addition to the consumption of computation resources and the construction of training datasets, fine-tuning jailbreak data may not be expected to generalize well to unseen attack methods due to data quantity limitations. The extraction method circumvents the task of making LLMs inherently robust against adversarial attacks by making the toxic instructions easier to trigger models' rejection mechanisms. Therefore, this method conceptually outperforms finetuning.

8. Our motivation.

   Response: As mentioned in related works, we can consider $p$ as a perturbation function, and some previous methods have tried paraphrasing/compressing (Semantic Smooth), random masking (RA-LLM), insertion/deletion (Smooth LLM), and self-examination (Self Defense). A detailed comparison between our method and others is presented in Table 3. These perturbation functions, since they do not extract information, need to be perturbed multiple times or different perturbations chosen and then voted on the results, which is time-consuming. Furthermore, if we want a continuous space, the $p$ needs to use an LLM to guarantee the quality of the compression, thus this route increases the time and computational overhead even more. Different from them, we effectively prevent jailbreaking by using only the retraining of a small model.

9. ASR and Harm Score

   Response: ASR is typically determined by the proportion of successful attacks out of the total number of attacks. Yes, it is string matching against refusal phrases. Harm Score is a trained reward model for harmful levels. Please see Appendix D.3 for a detailed description.