Mission Impossible: A Statistical Perspective on Jailbreaking LLMs

Dear reviewer sD4q,

Thanks for your detailed review! Here are our responses to your concerns.

1. ...(E-RLHF's) effectiveness in more complex conversational scenarios, where inputs can evolve over a series of interactions, remains untested.

• That is a great point that we had discussed internally too. We fully acknowledge that our theoretical framework does not cover multi-round chat jailbreak senarios. We have discussed this limitation in our paper (in lines 376-383, point (2)). Apart from the fact that we do currently not know how to extend the framework, we thought the exclusion of multi-interaction scenarios is justified because emirical work has not yet produced any (or SOTA) jailbreaking attacks and evaluations:
  • (1) No available evalution benchmarks for multi-chat. As discussed in the HarmBench paper [3], jailbreak evaluation varies significantly across studies, making it difficult to compare the efficacy of different methods. Therefore, we chose to follow HarmBench as our evaluation protocol, as it is designed to best reflect LLM safety by ensuring fair comparison and providing diverse jailbreak adversaries. Multi-interaction benchmarks are not available as of now.
  • (2) Lack of empirical methods. The only works that use multi-round interaction to jailbreak LLMs that we are aware of are [1] and [2]. However, both were published in February and April of 2024 hence were too concurrent to integrate in our work. Additionally, both projects do not provide source code to reproduce their results, making it challenging for us to include their methods in our empirical evaluations.

That being said, we believe that integrating multi-interaction attacks into our framework is crucial for future research. From a theoretical perspective, we hypothesize that if even a single attack poses a non-negligible risk, a multi-interaction attack will likely be even more challenging to defend against. Consequently, research questions would focus on how rapidly success rates escalate with the number of interactions. This exploration will provide deeper insights into the dynamics of jailbreak attack and inform more robust defensive strategies.

2. The two datasets used in the work contain harmful requests about various concepts (e.g. drug, weapon). Have you tried to analyze the performance of E-RLHF on different concepts?

• Our empirical evaluations primarily concentrate on harmful behavior. As highlighted in the general response, HarmBench assesses the safety of LLMs across a variety of harmful behaviors. While alignment extends beyond mere safety to include aspects such as ethical behavior, to our knowledge, we currently lack benchmarks for testing these broader criteria. We are happy to keep discussing on this topic and would welcome any specific proposals or suggestions for additional benchmarks that could enhance our understanding and assessment of ethical alignment in LLMs.

We hope these explanations address your questions and concerns. Please let us know if you need further clarification, we would be delighted to discuss further.

References

[1] Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack

[2] Leveraging the Context through Multi-Round Interactions for Jailbreaking Attacks

[3] HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal

Thanks. I will keep the rating since it is already the highest.

Dear reviewer UXGw,

Thanks for your comprehensive review! Here are our responses to your concerns.

• (1) The novelty of our framework. To the best of our knowledge, we are the first to offer a theoretical analysis on jailbreaking from a statistical perspective. To tackle this problem and provide insights, we addressed the following unique challenges associated with LLMs:
  • Abstract the capability of an LLM to generalize on unseen but plausible prompts. This property is particularly important since many representative jailbreak methods exploit this to bypass the guardrails of existing LLMs.
  • Model the capability of an adversary. For jailbreak attempts, it is hard to formulate an adversary in an mathematical way, since the operations performed on prompts are discrete and unbounded.
  • Discriminate harmful prompts versus non-harmful prompts. The statements should be connected to harmful prompts only, while the definition of "harmful" itself remains lacking.

Our framework overcomes these problems and provides theoretical evidence on the difficulty and impossibility of completely avoiding jailbreaking events.

• (2) Connection from our theory to E-RLHF. Under our framework, we identified a drawback in the RL fine-tuning objective, particularly the small safety zone it creates. We propose E-RLHF to address this issue by replacing harmful concepts in prompts $x$ with safe ones. We want to clarify that the safe prefixing strategy is not injecting a safe concept, but rather a simple yet effective implementation of harmful concept replacement in harmful prompts. We agree that more sophisticated methods could further enhance safety, and our experimental results suggest that even our simple implementation significantly improves safety. Thus, we argue our experimental results should not be regarded as negative but instead in line with Occam's razor as a positive result.
• (3) Sensitivity of the safe prefix and MT-Bench scores. As mentioned in response part (1), our safe prefixing is a simple implementation of replacing harmful concepts with safe ones. Under our framework, different safe prefixes can induce different safe concepts, and different $p_{\textrm{SFT}}(x_s)$. This explains the sensitivity to the choice of safe prefix. Prompt sensitivity has also been observed in previous works ([4][5]), and we believe safety could be further improved with prompt engineering to find better safe prefixes. Regarding the MT-Bench score, we acknowledge that E-RLHF leads to a slightly lower score than RLHF. However, the score achieved by E-RLHF is still higher than that of the SFT model, indicating that we do not sacrifice utility for safety. There are few LLM-tuning-based defenses against jailbreak attacks, especially those improving the RLHF step. For instance, R2D2 from HarmBench [1] uses a SFT strategy, resulting in a drop in the MT-Bench score from 7.3 to 6.0. The tension between safety and utility has been noted in previous works (e.g., [2]), and our proposal does not sacrifice utility to achieve better safety.
• (4) Insight provided by our framework and experiments. We aim to establish a framework that explains the jailbreak phenomenon theoretically. We argue that the intuitive results, showing that LLMs can be jailbroken both after the pretraining stage and after the current safety alignment stage with RLHF, are non-trivial, important, and counterintuitive. RLHF is the default optimization strategy for LLM alignment, and we identify a fundamental drawback in its mathematical objective. Based on this insight, we offer a plausible solution, E-RLHF, and demonstrate its effectiveness through a simple yet effective implementation.
• (5) Additional MT-Bench results. We are in the process of requesting credits with the OpenAI API and plan to initiate the benchmark test as soon as possible.
• (6) Relating ablation study on non-harmful prompts with theoretical analysis. Under our framework, the harmful and safety zones are defined with respect to a single concept, meaning modifications to non-harmful prompts should not impact performance on harmful prompts. However, we argue that the ablation phenomenon occurs due to the following reasons. Firstly, we do not model correlations and interactions between concepts. Each concept is considered independent, but in reality, generalization on one concept influences LLM performance on other prompts. For non-harmful prompts, appending the safe prefix may hinder optimization, affecting learning on harmful prompts. Modeling correlations and interactions between different concepts is highly complex and is left for future exploration. We will add this discussion to the limitations section of our final draft. Secondly, we point out that this statement also holds true for normal RL fine-tuning. Analysis on it (e.g., the motivation of DPO [3]) suggests that the output of a converged LLM for any prompt $x$ depends only on the reward model $r(x,e)$ and the initial distribution $p_{\textrm{SFT}}(x)$. However, due to the discrepancy between the optimal solution and the converged LLM in practice, the same reward model and LLM initialization can lead to models with significantly different performances.

Lastly, we want to emphasize the significance of our experimental result. We achieve significant safety improvement across all categories of harm on 2 benchmarks. The dataset size and harm diversity coverage is the largest compared to previous papers (we refer to our general response, and discussions in [1] as reference).

References

[1] HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal

[2] Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

[3] Direct Preference Optimization: Your Language Model is Secretly a Reward Model

[4] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

[5] Large Language Models as Optimizers

Dear reviewer T1QG,

Thanks for your thorough review! We are pleased to provide further explanations on our E-RLHF proposal, its theoretical foundations, and its specific relation to LLMs.

1. The definitions, theorems, etc. are all presented and motivated very nicely, but could also apply to any other mapping of a stochastic system. ...To make this a case specifically about LLMs, there needs to some form of conclusions that feed into the experiments.

To better address your concerns, we would like to clarify the goals we aim to achieve with our constructed framework to make it meaningful and robust. Our framework should:

• Abstract the capability of an LLM to generalize on unseen but plausible prompts. This property is particularly important as many representative jailbreak methods exploit this ability to circumvent the guardrail of existing LLMs. This necessitates a clear distinction between "plausible" and "seen" prompts. Additionally, defining the concept of "generalization" is critical, yet inherently complex.
• Model the capability of an adversary. For jailbreak attempts, formulating an adversary in a precise mathematical manner is challenging due to the discrete and unbounded nature of the operations performed on prompts.
• Discriminate between harmful and non-harmful prompts. It is essential that statements on jailbreak focus exclusively on harmful prompts. However, the definition of "harmful" itself remains ambiguous and diffucult to mathematically formalize.

Framework: Based on these points, we first assume each prompt can be decomposed into a query and a concept. This offers us the opportunity to model generalization by assuming invariance on the concept, thereby allowing us to mathematically describe the capability of the adversary. Reflecting point (1), in Assumption 4.1, we assume that the domain of the LLM output distribution dependents solely on the concept and not the query. This is a crucial, yet we argue, realistic assumption. We argue that this abstraction is distinctive of LLMs and is not easily applicable to generic stochastic systems, as not all mappings within such systems will exhibit the property described in point (1). This characteristic further impacts our E-RLHF proposal, which relies on this essential property to enhance safety. Without this property, the performance of those mappings on unseen harmful prompts found by the adversary would remain unaffected, while for LLMs the corresponding safety zone will be enlarged. That said, we are eager to explore whether our proposed framework can be effectively applied to other generic stochastic systems. This extension of our research could potentially broaden the applicability and impact of our findings, offering valuable insights into a wider range of systems.

2. The experiments use a systems prompt that is preprended to the actual prompt. I do not see, how this follows from the theory. Could you please point out the connection between experiment and theory that I have seemed to miss.

Specific implementation is driven by our framework and works: We further want to convince you that the incorporation of a prepended safe prefix represents an intuitive yet effective implementation of our E-RLHF proposal. Theorem 2 elucidates the relationship between the size of the safety zone and the ability of an adversary to successfully compromise the model. This insight directly informs our practical approach to enlarging the safety zone. Our analysis reveals that the prevalent RL fine-tuning objective, especially its KL-term, inadvertently contributes to vulnerabilities due to the unsafe nature of $p_{\textrm{SFT}}(x)$ when $x$ itself is harmful. By substituting harmful concepts in $x$ with safe alternatives, we can effectively mitigate this issue. We have elaborated several variations of its implementation in discussions with reviewer qvRD. In the experimental section of our study, we take a simple, computationally efficient approach by introducing a safe prefix to $x$.

We want to emphasize the significance of our experimental result. We achieved significant safety improvement across all tasks in HarmBench with only one generic intervention (one safe prompt). The dataset size and harmful diversity coverage is the largest compared to previous papers (we refer to the general response and section A.2, section B.4 and Table 5 of [1] as reference).

We found that our approach already provides an improvement in safety. We think that with better implementations, a larger alignment dataset, and improved optimization techniques (e.g., with PPO), the enhancement in safety could be significantly increased.

Please let us know if the above explanations resolve your concern on how we motivated our framework specifically for LLMs, and how our E-RLHF proposal is linked to the theoretical discoveries.

Dear reviewer qvRD,

We sincerely thank you for reading our work in great detail! Here are our responses to your concerns.

1. E-RLHF is an inaugural and simple form of expanding the safety zone of LLM; I think there could be more sophisticated and effective ways, and I hope the authors will address this in the future works.

• We sincerely appreciate your feedback. This is a fantastic suggestion and we would love to integreate a more detailed discussion on possible avenues for implementation of an alignment strategy based on our theory in the final version of the paper. Some of our thoughts include;

E-RLHF Formulation. As outlined in our general response, our proposed E-RLHF is inspired by the realization that the KL-term in the current prevalent alignment method (RLHF) may inadvertently preserve harmful responses when the input prompt $x$ itself is harmful. The RLHF objective aims to align with human preferences (as reflected by the reward term) while maintaining helpfulness (as reflected by the KL-term). To enhance safety while preserving these characteristics, we propose setting $p_{\textrm{SFT}}(\cdot)$ to a safe distribution. We believe that maintaining the mathematical reward-plus-KL formulation is vital, and currently, we do not see a clear pathway for other formulations to achieve these objectives simultaneously.

E-RLHF Implementation. We are considering several alternative strategies to refine our implementation of E-RLHF. The first concerns the harmful prompt filtering step. Instead of our current approach, which involves prompting an LLM to assess whether an input prompt is harmful, a more straightforward method might involve sampling responses and labeling the prompt as harmful if the likelihood of a response being harmful surpasses a predefined threshold. Additionally, involving human annotators to manually design, and identify harmful prompts from existing alignment datasets could be beneficial. Furthermore, considering that the determination of whether content is harmful can vary based on different backgrounds and contexts, the filtering process could also be made adaptive to these conditions. This adaptability is particularly crucial if we are cognizant of the diverse applications of LLMs. By tailoring the filtering mechanisms to accommodate various contexts, we can enhance the safety of the responses generated by LLMs, ensuring they are appropriate and considerate across a spectrum of scenarios, which could improve the utility and acceptance in global applications. The second pertains to the safe concept replacement step. Rather than our current method of safe prefixing, one could involve human annotators to rewrite harmful prompts or prompt a LLM to decompose-and-replace harmful prompts. We opted for safe prefixing in our paper due to its simplicity, which helps avoid excessive computational demands and reduces the need for intensive human labor. We believe that with effective prompt engineering (e.g., curate the prompt in a similar fashion as demonstrated in Table 15 in [1]), this second step can be efficiently implemented using an LLM, which we aim to explore in future work.

However, we want to emphasize that our approach despite its simplicity strikes a balance between computational feasibility and the goal of expanding the safety zone. We are delighted to report that this simple intervention performed exceptionally well against state-of-the-art jailbreaking attack benchmarks. This outcome not only underscores the viability of our proposal but also establishes a promising baseline.

We acknowledge that our approach to expanding the safety zone is just one of many potential strategies, and we are excited to see other researchers incorperate our insights into their alignment strategies. With the integration of additional safety alignment data and the implementation of more sophisticated strategies, our method holds the potential to deliver even more impressive results.

We would also be delighted to see our theoretical framework be applied in other domains as suggested by reviewer T1QG.

2. Eq (2): it is slightly confusing that in D_KL, the term only have $p_{LM}(x)$ and $p_{\textrm{SFT}}(x)$, not $p_{LM}(e|x)$ and $p_{\textrm{SFT}}(e|x)$.

• We apologize for any confusion caused by our notation. Throughout the paper, we denote the distribution over responses as $p_{LM}(q,c)=p_{LM}(x)$. With this notation, and incorporating a reward model $r(x,e)$, the RL fine-tuning can be expressed either as $\mathbb E_{x\sim\mathcal D_s, e\sim p_{LM}(\cdot|x)}[r(x,e)-\beta\frac{\log p_{LM}(e|x)}{\log p_{\textrm{SFT}}(e|x)}]$, or as $\mathbb E_{x\sim\mathcal D_s}[\mathbb E_{e\sim p_{LM}(\cdot|x)}[r(x,e)]-\beta \mathbb D_{\textrm{KL}}(p_{LM}(x) || p_{\textrm{SFT}}(x))]$. It is important to note that the expectation over $e$ is used for computing the reward $r(x,e)$, while the $\mathbb D_{\textrm{KL}}$ serves to regularize $p_{LM}(x)$, ensuring it does not deviate significantly from $p_{\textrm{SFT}}(x)$. To avoid any further confusion, we will clarify this point in our final draft by using the second equation.

3. I am not familiar with using the term "explanation" -- instead, I think using "response" is more common. At the first glance, it was hard to comprehend the meaning of the term

• We apologize for the confusion. We use the term "explanation" as a counterpart to "concept", based on the empirical observation that, in most jailbreaking attacks currently considered by the community, the adversary seeks instructions or explanations for a single harmful attempt. We appreciate your feedback and will incorporate it by resuming the use of "response" in our final draft to enhance readability and ensure our notation is easier to follow.

We hope these explanations address your concerns. Please let us know if you need further clarification, we would be happy to discuss further.

References

[1] Jailbreaking Black Box Large Language Models in Twenty Queries

We want to sincerely thank the reviewers for their effort in thoughtfully discussing our paper. We appreciate your detailed feedback and the positive points you’ve highlighted about our work. We firmly believe that our contributions, particularly in demonstrating the inevitability of jailbreaking, are crucial for fundamentally understanding the limits of alignment. This work hence delineates the critical battleground where future attack and defense strategies.

While we understand your concerns, we kindly request that you consider an increase in the score to better reflect the strengths and potential impact of our work. We are fully committed to further improving the manuscript and remain open to any additional suggestions you may have.

Warm regards + fantastic start in the week, The authors