Multilingual Jailbreak Challenges in Large Language Models

We greatly appreciate the time you dedicated to providing us with your valuable feedback. Your insights are highly valued, and please find our responses to the concerns you raised below:

W1. Lack of details on self-defence algorithm

The seed examples utilized within our experiment are crafted by humans to simulate both unintentional and intentional scenarios. We supply no more than six seed examples in the generation process.

When it comes to the generation of unsafe examples, we instruct the model with generating these in English. The model's strong proficiency in English and its robust safety capabilities within this language allow it to produce content that is genuinely unsafe. To ensure the validity of this process, we have each generated example manually reviewed and verified to confirm its unsafe content.

W2. Unclear coverage of self-defence algorithm's data generation

In our current work, we have provided a set of general instructions for generating safety-related Q&A pairs without delving into specific topics or scenarios. However, our "Self-Defense" method is flexible and can be easily expanded to more advanced approaches.

As an example, we might supply the LLMs with several pre-defined safety topics, or even enable the model to generate these topics by themself. For each distinct topic, we could gather or generate relevant seed examples, and adjust the instructions to match the topic accordingly.

While the current focus is on this immediate concern, our future work aims to refine and enhance the "Self-Defense" methodology, thereby improving its effectiveness and efficiency in addressing multilingual jailbreak challenges.

W3. Questioning relevance of safety measures for low-use languages like Bengali

Thank you for your valuable input. We totally agree that the number of native speakers doesn't necessarily imply a significant impact on the usage of ChatGPT. However, our mention of Bengali aimed to highlight safety issues in low-resource languages, which refers to the scarcity of available digital training data, not the speaker count.

Despite being a low-resource language in AI training, Bengali is the 6th most spoken language worldwide with 285 million native speakers [1]. This significant speaker base often gets overlooked in AI safety considerations, which typically focus on languages with abundant digital resources.

While Bengali-speaking ChatGPT users may be a small fraction of the total user base, it's crucial to note that Bengali is one among many low-resource languages. Cumulatively, these languages represent a substantial user base, emphasizing the need for dedicated safety measures.

Thus, our discussion extends beyond Bengali to all low-resource languages, many with large speaker populations. We'll clarify this in future discussions and appreciate your insights in improving our communication.

[1] List of languages by number of native speakers, https://en.wikipedia.org/wiki/List_of_languages_by_number_of_native_speakers

Deaer Reviewer imbS:

We appreciate you taking the time to review our work and your insightful feedback. We would like to discuss this further with you to see if there are any unresolved questions. Specifically, we have addressed your concerns on the following points.

1. More details on Self-Defence;
2. Discussion on coverage of Self-Defence;
3. Relevance for low-resource languages.

If you have any further concerns, please let us know.

Best Regards,

Authors

Thank you so much for taking the time to share your valuable feedback with us. We truly appreciate your insights, and please find below our responses to your concerns:

W1. Noisy translations by LLM may reduce usefulness

Thank you for pointing this out. Yes, we agree with you that noisy translation may be another cause for usefulness reduction. To verify this assumption, we replaced the translation component in SELF-DEFENCE to Goolge Translate, which is believed to perform better than ChatGPT on machine translation [1]. The results are presented as below:

With better translation quality, both safeness and usefulness have gained improvement, especially for usefulness. We have modified our paper accordingly to point this factor out.

However, we believe a less comprehensive safety Q&A would also be an important factor, as the trade-off is also observed even in English-only evaluation [2].

In conclusion, the quality of generated content plays a crucial role, both in terms of language-wise and the content itself. In our future research, we aim to delve into the enhancement of the quality of data generated by Self-Defence framework. Again, thank you very much for your valuable feedback.

W2. SELF-DEFENCE framework lacks novelty

We think providing more training examples for low-resource languages can improve LLM's multilingual safety capabilities. However, our proposed method can be helpful in the following scenarios:

1. Data augmentation: Our method provides a way to quickly augment a large multilingual safety dataset to address the challenge effectively and efficiently.
2. Fine-grained generation: If a new safety topic is introduced, or the safety guideline is changed, or a new language is added, we can simply edit the seed example and instruction in the "Self-defense" framework to adapt accordingly.
3. Data imbalance: Safety datasets may have uneven topic distribution. The "Self-Defense" framework can address this by generating data for underrepresented topics using specific seed examples and instructions.
4. Avoiding intellectual property conflict: As data becomes more and more important, the safety dataset may have conflicts of interest or be regulated by licenses. However, utilizing "Self-defense" can easily avoid them and generate data by the LLM itself.

In conclusion, while directly augmenting existing datasets with non-English data is a viable approach, our method introduces flexibility, adaptability, and a way to circumvent potential legal issues. This makes it a valuable tool in the evolving landscape of AI safety.

W3. No comparison of SELF-DEFENCE with other safety improvement techniques

Thank you for your feedback. It's important to note that our proposed SELF-DEFENCE framework should be viewed as complementary, rather than directly comparable, to existing safety improvement techniques such as SFT and RLHF. Unlike these methods which focus on safety improvement through training algorithm design, the SELF-DEFENCE approach primarily serves to augment multilingual safety datasets. Our approach can be combined with other safety improvement techniques for more comprehensive results. In fact, during our experimentation, we employed the SFT method to exploit the augmented dataset provided by our SELF-DEFENCE framework for safety training.

For comparison, we conducted another experiment using English-only data, which is the most common language in safety tuning, as a baseline. The following table presents the experimental results:

As the results show, SELF-DEFENSE can not only improve multilingual safety capability, but the multilingual usefulness is also maintained. The results demonstrated the effectiveness of our proposed framework.

Q1. Performance of SELF-DEFENCE vs other safety improvement methods

We have addressed this concern in W3.

Q2. Reliability of LLM-generated translations for low-resource languages

We have addressed this concern in W1.

[1] Jiao et al, Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine, https://arxiv.org/abs/2301.08745

[2] Bai et al, Training a helpful and harmless assistant with reinforcement learning from human feedback, https://arxiv.org/abs/2204.05862

Dear Reviewer WaDP,

We thank you for the time spent reviewing our work and for your constructive comments. We sincerely hope to have further discussions with you to see if our responses address your concerns.

Specifically, in our response, we have:

1. Clarified how noisy translation affects our work;
2. Presented the novelty of our Self-Defence method;
3. Compared our work to other techniques.

We genuinely hope you will review our responses. Thank you!

Best Regards,

Authors

W4: Tests limited to ChatGPT and GPT-4

We expanded our experiment to include two widely used open-source LLMs: Llam2-Chat and Vicuna. The results of this extension on unintentional scenario are presented below:

However, we found that these two models contain the following problems:

1. Although Llama2-Chat has the capacity to comprehend non-English languages, it mostly responds in English. This may limit its applicability in real-world scenarios, particularly for non-English speaking users, making the evaluation less realistic.
2. Both models often generate invalid outputs due to their limited multilingual capabilities. While Llama2-Chat owns the lowest average rate of unsafe content, it is challenging to determine whether this lower rate is a consequence of producing genuinely safe content or simply generating invalid content.
3. Vicuna has been trained on user-shared conversations sourced from ChatGPT and GPT-4. The language distribution within this training data is somewhat disordered, leading to unpredictable results. Furthermore, Vicuna has not undergone specific safety tuning, resulting in a high unsafe content rate, particularly in English, with a staggering 57.17% unsafe rate.

Thus, we have decided to concentrate our evaluation primarily on ChatGPT and GPT-4 to ensure a realistic and dependable assessment. However, we will supply the supplementary results in the Appendix A.7 for further reference.

W5: Over-complicated methodology

We acknowledge that directly translating Anthropic's dataset or other safety-related datasets is a straightforward way to address the multilingual jailbreak challenge. However, our proposed method can be helpful in the following scenarios:

1. Data augmentation: Our method provides a way to quickly augment a large multilingual safety dataset to address the challenge effectively and efficiently.
2. Fine-grained generation: If a new safety topic is introduced, or the safety guideline is changed, or a new language is added, we can simply edit the seed example and instruction in the "Self-defence" framework to adapt accordingly.
3. Data imbalance: Safety datasets may have uneven topic distribution. The "Self-Defence" framework can address this by generating data for underrepresented topics using specific seed examples and instructions.
4. Avoiding Intellectual property conflict: As data becomes more and more important, the safety dataset may have conflicts of interests or be regulated by licenses. However, utilizing "Self-defence" can easily avoid them and generate data by the LLM itself.

In conclusion, while the direct translation of existing datasets is a viable approach, our method introduces flexibility, adaptability, and a way to circumvent potential legal issues. This makes it a valuable tool in the evolving landscape of AI safety.

W6: Weak mitigation section, needs improved safety-usefulness balance

Thank you for your kind advice. We believe that more fine-grained instructions for data generation could lead to more detailed and comprehensive responses. This approach could prevent the model from merely learning to reject from the safety training data, thereby enhancing the model's ability to maintain high usefulness while ensuring safety. We are eager to explore this aspect further in our future research.

Regarding a more nuanced understanding of the benefits of additional data, we present the unsafe decrease rate by language category as below:

In both scenarios under consideration, it's evident that LRL achieves the most significant improvement. This could potentially be attributed to its initially high rate of unsafe content, implying that even a small amount of training data can yield substantial gains.

Furthermore, we observe that the intentional scenario demonstrates a more pronounced improvement compared to the unintentional one. This suggests a greater potential for enhancement to mitigate intentional jailbreak attack.

Thank you for providing us with your valuable feedback and suggestions. We appreciate your input and have carefully considered your questions. Below, we provide detailed responses to each of them:

W1: Frame safety study as important, not as a 'jailbreak'

Thank you for your valuable feedback. While we agree that ensuring safety for different languages is important in itself, we wanted to provide a novel perspective by highlighting how language can be used as a method for jailbreaking. This framing allows us to explore the vulnerabilities of LLMs across languages. We will revise the paper to ensure that the importance of language-specific safety is emphasized alongside the concept of jailbreaking.

W2: Small sample size and lack of error bars

Regarding the small sample size, we included 15 examples in our preliminary experiment as a starting point. We do recognize the need for a larger dataset to obtain more reliable results. Therefore, in the detailed experiment, we expanded our dataset to include 315 examples for each language, resulting in a total of 3150 examples.

Regarding the malicious instructions, our intention was to replicate the actions of a real-life malicious user who searches for and selects the most powerful malicious instructions available online. That't why the scenario is named as "intentional". While we acknowledge that including a greater variety of malicious instructions would provide a more comprehensive evaluation, we plan to incorporate additional instructions in our future research.

Regarding the error bars, given that we have set the temperature to 0 and are evaluating a fixed dataset, the output remains constant. To respond to your concerns, we have also conducted an additional evaluation using another decoding method. Specifically, we used nucleus sampling with a top_p value of 0.8. To ensure reliable results, we ran the experiment three times with different seeds. The following data represents the unsafe rate of this experiment, with the standard deviation values indicated within parentheses:

It is evident that the average unsafe rate, although slightly higher (1.25%), the trend is still clearly observable, as the rate of unsafe content increases with decreasing language availability, resulting in a consistent ranking order. We have show details in Appendix A.6.

W3: Need for diversity in safety issues studied

Yes, we have incorporated tag features to indicate the safety issue studies. These tags are derived from the Anthropic dataset, which assigns one or more tags to each example. In order to maintain consistency, we have applied the same tagging schema to label OpenAI's 15 prompts. Our dataset includes a wide range of 18 safety issues, ensuring its diversity. The top-3 safety issues and their respective percentages in the dataset are "Violence & incitement" (18.70%), "Discrimination & injustice" (11.28%), and "Hate speech & offensive language" (8.41%). For a comprehensive list of tags and their details, please refer to the Appendix A.2.

Dear Reviewer QBvZ:

Thanks so much for your valuable suggestion on:

1. Manuscript frame;
2. Error bar;
3. Diversity;
4. More model evaluation;
5. Methodology motivation;
6. Trade-off analysis.

We have carefully responded to your feedback and conducted supplementary experiments to address your concerns. We are eagerly anticipating your response and thoughts on our new experiments as the deadline for discussion draws near.

Best Regards,

Authors

Thank you for providing us with your valuable feedback. We have carefully considered your questions and would like to address them as below:

W1. Concern on GPT-4 evaluation

Regarding safety evaluation, it's worth highlighting that it's generally an easier task compared to generating safe content. The former is a classification task, while the latter is a more complex natural language generation task. A recent study provides further evidence of this, showing that LLMs can effectively flag a majority of harmful text they generate themselves [1].

In addition, the table below summarizes common methods for evaluating jailbreak results, their limitations, and how GPT-4 compares to them:

Taking into account the aforementioned factors, GPT-4 has emerged as a reliable, efficient and widely-accepted evaluation method [2,3,4,5].

Moreover, as depicted in Fig. 2, there is a significant level of agreement between human evaluation and the evaluation carried out by GPT-4. To quantify this agreement, we computed the Cohen's kappa score, which stands at a robust 0.86. This high score underscores the considerable alignment between GPT-4's evaluations and human judgments.

Based on these reasons, we have made the decision to employ GPT-4 as our primary evaluation measure.

W2. Attribution of high attack success rate

Yes, we intentionally chose the highest malicious instruction from https://www.jailbreakchat.com/, but it is crucial to clarify that our primary objective wasn't to obtain a high unsafe rate. Instead, we attempted to mimic a malicious user's behavior who, in a real-life scenario, would likely search the internet to find the most effective harmful instruction for intentional purposes.

We wanted to demonstrate that even when a malicious user employs the most advanced malicious instruction, the integration of multilingualism can substantially enhance the effectiveness of their jailbreaking techniques.

Therefore, the high unsafe rate should not be solely attributed to the malicious instruction power. Instead, it proves how multilingualism can amplify the potential of already powerful tools, thereby posing a significant threat to AI systems.

By incorporating multilingualism, we are exploring an orthogonal approach to boosting jailbreaking techniques. It is a unique dimension that can amplify the threat level, thereby highlighting the necessity for robust multilingual safety measures.

Q1. Clarification on setup from red teaming dataset

Below are the details of our data selection process:

1. We selectively choose adversarial examples that include tags. Out of a total of 38,961 examples, only 742 of them contain harmful tags.
2. From this set, we randomly sample 300 examples from top-500 lowest task_description_harmlessness_score. All of these examples have both harmful tags and a low task_description_harmlessness_score. We then extract the first user prompt from each of them.
3. Following a human evaluation, it was determined that 99% (297/300) of the extracted examples were harmful.

We have included the detailed tag statistics in Appendix A.2.

Q2. Consistency in value presentation between tables and figures

Thank you for your kind suggestion! We will refine accordingly to maintain consistent presentation.

[1] Phute et al., LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked, https://arxiv.org/abs/2308.07308

[2] Yuan et al, GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher, https://arxiv.org/abs/2308.06463

[3] Bhardwaj et al, Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment, https://arxiv.org/abs/2308.09662

[4] Qi et al, Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!, https://arxiv.org/abs/2310.03693

[5] Ji et al, BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset, https://arxiv.org/abs/2307.04657

Dear Reviewer Pbxw,

We greatly appreciate your valuable comments on our work. We have carefully addressed your concerns regarding:

1. Reason for GPT-4 evaluation;
2. Attribution of high attack success rate;
3. Clarification on dataset setup.

As the deadline for discussion is approaching, we eagerly await your feedback on our response and new experiments.

Thank you for your time and consideration.

Best Regards,

Authors

We greatly appreciate your insightful comments and suggestions, as they have been helpful in refining and enhancing our work. We have thoroughly reviewed all of your points and have addressed your concerns as outlined below:

W1: Standard deviations

Thank you for your kind feedback. In our experiments, we maintained a temperature of zero and utilized fixed datasets for both preliminary and detailed testing. As a result, the output remained constant. However, for supplementary experiments, we employed nucleus sampling to explore further possibilities. For more information on these supplementary experiments, please refer to Q3, where we provide detailed explanations.

W2: Detail human evaluation process

We completely agree with this. To ensure accurate human evaluation, we have implemented the following steps:

1. Translation: As discussed in Section 2.1, we have used Google Translate to translate all the output into English.
2. Annotation: We engage two annotators to independently label the translated output following the definition in Appendix A.2.
3. Conflict resolution: In case of any conflicts or discrepancies between the annotations, we have involved an additional expert to make the final determination on the label.

By following these steps, we aim to ensure the highest possible accuracy and reliability in the human evaluation process.

W3: Address potential LLM-based evaluator bias

Thank you for bringing up this important point.

However, we think that evaluator bias may mainly exist in NLG problems, particularly in subjective metrics like "fluency", "relevancy", "preference", etc. When it comes to evaluating "safety", the concept tends to be much more "objective," especially when we frame safety evaluation as a classification task.

In addition, almost all examples are easy-to-identify unsafe topics, such as "Violence", "Adult content" and etc. We list the detailed statistics in Appendix A.2.

Finally, we have defined the criteria for each class in the GPT-4 evaluation prompt, minimizing potential bias as much as possible. In W4, we further show that GPT-4 is a reliable evaluation measure.

W4: Provide specific agreement values

Sure, we have measured the agreement between GPT-4 and human evaluations, utilizing Cohen's kappa coefficient. The Cohen's kappa score is 0.86, showing a high agreement between GPT-4 and human evaluations.

W5: Publish MultiJail dataset and results

Yes, we will publish our MultiJail dataset including necessary code and results in the future.

Q1: Lack of trend in intentional scenario

Yes, as discussed in Section 3.2.2, we found that LLMs display relative stability regardless of language availability in intentional scenarios. We assume that in unintentional scenarios, the multilingual jailbreak problem arises mainly due to the inherent tension between the LLMs' pre-training ability and safety capability, leading to an obvious trend. However, this situation becomes further complicated when malicious instructions are introduced, as the importance of instruction-following factors increases. There is a conflict between pre-training, instruction-following, and safety abilities. This complex situation makes the previously observed trend unobvious. We will dive deeper into our future research to study the inherent cause of the jailbreak problem.

Q2: Translation and assessment of non-English outputs

The primary goal is to ensure the quality of evaluation. Initially, during the human evaluation process, we discovered that while the translation may not be perfect, it maintains the core idea of the sentence. This allows us to confidently predict its safety.

With respect to GPT-4, the unintentional scenario results demonstrate limited safety capabilities in non-English languages. As a solution, translating the non-English output back into English using Google Translate can enhance GPT-4's safety evaluation performance. Although this may introduce some noise, the benefits it brings outweigh these minor inconveniences.

In our preliminary experiments, we actually tried sampling some examples and compared GPT-4's evaluation of translated English versus direct non-English text on its website. Our investigation revealed that translating back to English is a more reliable approach.

Q3: Zero temperature setting and nucleus sampling impact

We chose to use a zero temperature setting primarily to prioritize the reproducibility and reliability of our experiment. While other decoding strategies may yield slightly different results, a recent study [1] has indicated that the qualitative outcomes and observed trends remain consistent in general jailbreak experiments.

To further investigate the impact of different decoding strategies, we conducted an experiment in an unintentional scenario using ChatGPT with nucleus sampling and a top_p value of 0.8. To ensure reliable results, we ran the experiment three times with different seeds. The following data represents the unsafe rate of this experiment, with the standard deviation values indicated within parentheses:

It is evident that the average unsafe rate, although slightly higher (1.25%), the trend is still clearly observable, as the rate of unsafe content increases with decreasing language availability, resulting in a consistent ranking order.

Q4: Ambiguity of "unsafe and general" phrase

Thank you for pointing it out. "General" here primarily focuses on "usefulness". Implicitly, all content is safe. We will refine the expression to prevent misunderstandings.

[1] Wei et al., Jailbroken: How Does LLM Safety Training Fail?, https://arxiv.org/abs/2307.02483

[2] Ganguli et al., Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned, https://arxiv.org/abs/2209.07858

W6: Evaluate open-sourced models

Yes, we have explored evaluating open-source language models such as LLama2 and Vicuna. However, we found certain limitations in their multilingual capabilities and safety measures, which could potentially introduce noise during analysis. Specifically, LLama2 primarily responds in English, even when presented with non-English input. Though Vicuna has the ability to respond in languages other than English, it lacks specialized safety tuning, and its limited safety capabilities are mainly derived from user-shared responses of ChatGPT/GPT-4.

Given these limitations, we've primarily centered our attention on ChatGPT and GPT-4. These models offer a more seamless conversational experience across various languages and have robust safety mechanisms in place. Nevertheless, we have included the experimental results for LLama2 and Vicuna under unintentional scenarios as supplementary references, presented in the table below, detailed in Appendix A.7:

From the results, it's evident that while Llama2-Chat performs well in English, and even obtains the lowest average unsafe rate, it also generates a considerable amount of invalid output. As such, it's difficult to determine whether its low unsafe rate is a result of genuinely safe or invalid content generation.

On the other hand, Vicuna demonstrated the least safety capability, with a notably high unsafe rate of 57.14% even in English. Moreover, it also had a substantial rate of generating invalid responses.

In conclusion, to ensure a unbiased evaluation, we have chosen to primarily report our experiments based on ChatGPT and GPT-4. These models showcase safisfactory multilingual abilities, exhibit low invalid response rates, and have comprehensive safety tuning mechanisms in English. These factors help to isolate the language aspect during the analysis, eliminating other potential influences.

W7: Analyze "Self-Defense" framework effectiveness

We appreciate the reviewer's insightful question. Before addressing the question, we would like to provide some context regarding the multilingual jailbreak challenge. We classify the cause of this challenge as a mismatched generalization [1], specifically the lack of a non-English corpus during safety tuning.

In order to ensure the effectiveness of "Self-Defence," we believe two preconditions need to be met. Firstly, the model should possess strong safety capabilities in high-resource languages. Secondly, it should have strong multilingual abilities.

If the model fulfills these two requirements, "Self-Defence" can perform explicit alignment of its safety capabilities from high-resource languages to low-resource languages. Essentially, it leverages its existing safety capability and transfers it to other language spaces.

Furthermore, in our future research, we plan to conduct additional experiments to validate and enhance the effectiveness of our approach. By doing so, we aim to propose even more effective methods to tackle the multilingual jailbreak challenge.

W8: Consider cultural differences in safety guidelines

We fully acknowledge that the concept of "safety" can vary across different cultures and societies.

However, most adversarial examples consist of general unsafe content, including pornography, terrorism, fraud, and other globally recognized forms of unsafety, detailed in Appendix A.2. Consequently, only a limited number of examples are appropriate for "localization" purposes. Additionally, to focus solely on the "language" aspect, we have opted for direct translation without localization. Nonetheless, we recognize the significance of studying cultural differences, which would require fine-grained annotation and more in-depth research. We intend to explore this aspect in our future research projects.

Dear authors,

I thoroughly read your responses, and thank you for updating your manuscript and providing additional experiments. I was able to resolve my questions. Followings are my thoughts and comments:

1. W6: Evaluate open-sourced models

• I understand the reason why you chose GPT-4 and ChatGPT. From the additional experiments, however, the trend where low resource language is vulnerable to jailbreaks does not work. But, when it comes to "safe" instead of "unsafe", that is "unsafe + invalid", we can conclude that the extent of safe response is decreasing as the volume of resource is decreasing.

2. Q3: Zero temperature setting and nucleus sampling impact

• I appreciated your supplementary experiments. I think even though zero-temperature, the probability distribution probably and slightly different because of randomness of layers in the LLM. But, again, I understand!

3. W7: Analyze "Self-Defense" framework effectiveness

In order to ensure the effectiveness of "Self-Defence," we believe two preconditions need to be met. Firstly, the model should possess strong safety capabilities in high-resource languages. Secondly, it should have strong multilingual abilities.

If the model fulfills these two requirements, "Self-Defence" can perform explicit alignment of its safety capabilities from high-resource languages to low-resource languages. Essentially, it leverages its existing safety capability and transfers it to other language spaces.

I was able to understand the authors' motivation for "self-defense". I'd recommend to mention and explain the motivation and assumptions in the manuscript. However, to verify and support them, you might need the future work you mentioned.

4. (minor) The algorithm 1 looks confusing.

• please clarify definition of D_l,
• "M: D_a <- M(D_s)" could be understated as the definition of a function M
• in line 1, how to augment? what does it mean that D_a <- M(D_s)? That is, M(D_s) = set of responses to queries in D_s?

Good luck :)