Thank you for your insightful and constructive feedback. We have carefully reviewed your suggestions and have made revisions to the content to address your comments effectively. We sincerely thank you for your insightful and constructive feedback. Your deep expertise in jailbreaking methods and your recognition of the novelty in our approach are greatly appreciated. We are delighted that you found our contribution clear and useful, and that it offers a valuable addition to the field. Your thorough understanding of our paper's core concepts has provided us with valuable perspectives to refine our work further.

S1: Figure 1 is clear and compelling. Although Figure 2 is visually messy with the "Safety alignment" words across it.

We are glad that you found Figure 1 clear and compelling. To address your concern about Figure 2, we have revised the visualization by refining the layout and adjusting the placement of the "Safety alignment" text to enhance clarity and reduce visual clutter. We hope the updated figure better conveys the intended information.

[W1] I would recommend considering a different title. "Political correctness" is not a term that has the same definition to everyone, and it's a political buzzword.

Thank you for pointing out that the term "political correctness" may carry different meanings for different audiences and can be interpreted as a politically charged term. We appreciate your thoughtful advice on this matter. In response, we have revised all instances of "political correctness" throughout the paper, including the title. We have also updated the method name "PCJailbreak" to "BiasJailbreak" and "PCDefense" to "BiasDefense" to ensure clarity and neutrality in our terminology.

[W2] I would recommend that the abstract text be revisited in order to be more specific. There isn't a full description of the attack or defense methods used in the abstract itself. I also think that the abstract could be updated to have smoother writing and less fluff -- I think that some of the sentences in it (especially early) are not very relevant to the specific contributions of the paper.

We have revisited and revised the abstract to be more specific and aligned with the key contributions of the paper. Specifically, we have included a concise description of the attack and defense methods used and ensured that the text flows more smoothly by removing unnecessary details. We believe the updated abstract better reflects the focus and significance of our work.

[W3] There is a claim in the paper that political correctness biases are introduced into models from the fine-tuning process. But this seems unjustified. I don't see why they wouldn't also be a result of pretraining data.

Thank you for raising this important concern. You are absolutely correct that multiple factors, including system prompts, training data, and the fine-tuning process, can influence biases in LLMs. It is indeed difficult to pinpoint precisely when and where these biases are introduced during model development.

As a partial evidence for our claim, in page 49 of GPT-4 Technical report[1], there exists a phrase where it states bias alignments and its possibly harmful side effects:

“Some types of bias can be mitigated via training for refusals, i.e. by getting the model to refuse responding to certain questions. This can be effective when the prompt is a leading question attempting to generate content that explicitly stereotypes or demeans a group of people. However, it is important to note that refusals and other mitigations can also exacerbate bias in some contexts, or can contribute to a false sense of assurance. Additionally, unequal refusal behavior across different demographics or domains can lead to quality of service harms. For example, refusals can especially exacerbate issues of disparate performance by refusing to generate discriminatory content for one demographic group but complying for another.“

While it is difficult to pinpoint the exact origin of biases in every model, it is clear that these biases exist and may introduce unintended risks. This highlights the importance of continued investigation and mitigation efforts to ensure the safe and fair deployment of LLMs. We added this explanation to the paper.

[W4] I think that section 2.2 is not the most thorough. It could be expanded to better discuss related jailbreaking techniques that involve persuasion and personas.

We have added references to studies on persona-based and persuasive jailbreak techniques to enhance the discussion in Section 2.2. This addition addresses the points you raised and provides a more comprehensive overview.

[Minor] Why were not claude models tested?

We appreciate your suggestion and conducted experiments on the most recent Claude Sonnet model [3]. Our experiments revealed similar biases in Claude Sonnet, shown in Table 2. Experimental results perform a higher jailbreak success rate when using keywords associated with marginalized groups compared to those associated with privileged groups.

Table 2

Performance across different models showing baseline success rates, marginalized success rates, privileged success rates, and the difference between marginalized and privileged success rates.

Minor: A "Walkerspider" reference might have a typo and need to be cleaned up.

Minor: I would recommend having a different example in figure 2 and 4 so that readers can see more diverse examples.

Minor: It's principal component analysis, not "principle"

Thank you for your feedback. We have corrected the "Walkerspider" typo, updated Figures 2 and 4 with diverse examples, fixed "principle" to "principal" in component analysis, and addressed other minor errors throughout the paper.

Vzp2: References

[1] Achiam, Josh, et al. "Gpt-4 technical report." arXiv preprint arXiv:2303.08774 (2023).

[2] Li, Xuan, et al. "Deepinception: Hypnotize large language model to be jailbreaker." arXiv preprint arXiv:2311.03191 (2023).

[3] Anthropic, A. I. "Claude 3.5 sonnet model card addendum." Claude-3.5 Model Card (2024).

Table11 : Keywords discovered in GPT-4o.

Table 12 : Keywords discovered in Qwen2

Table 13: Keywords discovered in LLaMA2

Table 14: Keywords discovered in Phi

Thank you for your review. We have improved the experiments according to your keen suggestions.

[W1] Motivation: The paper categorizes jailbreak attacks into manually written prompts and learning-based prompts, stating that learning-based jailbreak prompts rely on gradient information and that these prompts are often nonsensical sequences. However, this overlooks natural-language jailbreak prompts, such as PAIR [1] and DeepInception [2], which are not solely gradient-based and produce coherent, meaningful language. Additionally, for manual attacks, approaches like GUARD [3] build on existing manually crafted jailbreak prompts, refining them over time to remain effective.

We have updated our paper to categorize jailbreak attacks as black-box and white-box, and included references to PAIR [1], DeepInception [2], and GUARD [3], as recent advances on black-box attacks that has enabled a more general and real-world setting of jailbreaks, while having meaningful language, involving methods such as template completion, prompt rewriting, or LLM-based generation. We have categorized our paper as a black-box method that offers both scalability and generality. We appreciate your feedback, which has helped on defining the scope and motivation of our work.

[W2] Scope of Jailbreak Attacks: Much of the related work on jailbreak techniques in this paper appears to focus on approaches up to 2024. Given the rapid advancements in jailbreak methodologies, the paper should provide a more detailed discussion of recent jailbreak attacks, such as works like [4] and [5].

Thank you for highlighting the need to include more recent advancements in jailbreak methodologies. We have updated our paper to incorporate discussions of the works [4] and [5], ensuring that our coverage of jailbreak techniques reflects the latest developments. We appreciate your feedback, which has helped us provide a more up-to-date and comprehensive discussion of this rapidly evolving area.

[W3] Keyword Generation Methodology: The approach of directly prompting the LLM to generate keywords introduces potential issues. For instance, the generated keywords may lack diversity, as the LLM could repeatedly produce similar terms based on its training biases. Additionally, there is no evaluation or filtering mechanism to determine which keywords are more effective or appropriate for distinguishing between privileged and marginalized groups.

Thank you for your feedback regarding the keyword generation methodology. We would like to clarify that our experimental setup was designed to leverage the ethical biases of LLMs to their fullest extent, while still having a diverse and meaningful set of keywords.

Table 10-14 includes the full set of keywords automatically discovered by LLMs and used in our experiments. While we presented only a partial subset of these keywords for readability, our current method still shows a broad and rich selection of keywords, without repetitive or overly similar terms.

Table10 : Keywords discovered in Claude-sonnet3.5.

[W3] Ambiguity in Baseline Definition and Scope of Comparison: While Table 2’s caption states it shows “baseline success rates, marginalized success rates, privileged success rates, and the difference between marginalized and privileged success rates,” the paper does not clearly define what constitutes the "baseline success rate." Additionally, to strengthen the evaluation, it would be beneficial to include comparisons with a broader range of jailbreak attacks.

Thank you for pointing out the ambiguity regarding the "baseline success rate." We apologize for the lack of clarity, which may have led to confusion. To clarify, the baseline success rate refers to the success rate observed when using the prompt in Equation 1 without any keywords. We have updated the paper to include a detailed explanation of this in the relevant section to avoid further misunderstandings. We appreciate your feedback, which has allowed us to improve the clarity and completeness of our work.

Defense Baselines: There are some relevant papers at the prompt level to prevent harmful output, such as [6], [7], and [8]. As PCDefense also adds prompts to model system prompts and suffix prompts, it should compare the effectiveness with these methods.

Thank you for highlighting the importance of comparing BiasDefense with other prompt-based defense methods. Following your suggestion, we conducted experiments incorporating relevant defense techniques, such as Self-Remind [6], Defending[7], and RPO [8], using the LLAMA2 model. As shown in Table 5, our proposed BiasDefense achieved the best performance, with the lowest jailbreak success rates for both marginalized and privileged groups. These results further validate the effectiveness of our approach in mitigating jailbreak vulnerabilities. We appreciate your feedback, which has strengthened our evaluation and demonstrated the robustness of BiasDefense.

Table 5

Comparison of jailbreak success rates across prompt-based defense methods using LLAMA2 model

gTDY: References

[1] Chao P, Robey A, Dobriban E, et al. Jailbreaking black box large language models in twenty queries[J]. arXiv preprint arXiv:2310.08419, 2023.

[2] Li X, Zhou Z, Zhu J, et al. Deepinception: Hypnotize large language model to be jailbreaker[J]. arXiv preprint arXiv:2311.03191, 2023.

[3] Jin H, Chen R, Zhou A, et al. Guard: Role-playing to generate natural-language jailbreakings to test guideline adherence of large language models[J]. arXiv preprint arXiv:2402.03299, 2024.

[4] Zheng X, Pang T, Du C, et al. Improved few-shot jailbreaking can circumvent aligned language models and their defenses[J]. arXiv preprint arXiv:2406.01288, 2024.

[5] Jin H, Zhou A, Menke J D, et al. Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters[J]. arXiv preprint arXiv:2405.20413, 2024.

[6] Wu F, Xie Y, Yi J, et al. Defending chatgpt against jailbreak attack via self-reminder[J]. 2023.

[7] Zhang Z, Yang J, Ke P, et al. Defending large language models against jailbreaking attacks through goal prioritization[J]. arXiv preprint arXiv:2311.09096, 2023.

[8] Zhou A, Li B, Wang H. Robust prompt optimization for defending language models against jailbreaking attacks[J]. arXiv preprint arXiv:2401.17263, 2024.

Thank you for your dedication and thoughtful engagement with our paper. As the author-reviewer discussion period draws to a close, we would greatly appreciate hearing your thoughts on our rebuttal. We have tried our best to reflect your suggestions, and added several experiments to reinforce our work. If you have any further questions or require additional clarification, please don't hesitate to let us know.

We would like to gently remind you for the updated review, as we are very close to the final due date. We have thoroughly prepared extra experiments and results that you requested, and we assume that all of your weakness concerns should be resolved with our updated results. Thank you.

Table 12 : Keywords discovered in Qwen2

Table 13: Keywords discovered in LLaMA2

Table 14: Keywords discovered in Phi

63GV: References

[1] Zou, Andy, et al. "Universal and transferable adversarial attacks on aligned language models." arXiv preprint arXiv:2307.15043 (2023).

[2] Dong, Qingxiu, et al. "A survey on in-context learning." arXiv preprint arXiv:2301.00234 (2022).

[3] Achiam, Josh, et al. "Gpt-4 technical report." arXiv preprint arXiv:2303.08774 (2023).

Table 7

Comparison of the impact of individual keyword groups on BiasJailbreak. Marginalized Groups and Privileged Groups have a distinct difference among Random Adjectives

Random Adjectives Group (AVG: 23.55% ± 1.52%)

Marginalized Group (AVG: 28.33% ± 1.26%)

Privileged Group (AVG: 20.15% ± 1.28%)

Table 8

BiasDefense Ablation on the combination of suffix and prefix defense prompt using LLAMA2 model

Table 9

MMLU score using prompt-based defense methods. The base language model used is google-flan-t5.

Table10 : Keywords discovered in Claude-sonnet3.5.

Table11 : Keywords discovered in GPT-4o.

--->> I think the paper should provide some context on 'political correctness' for readers, as well as the motivation behind studying it.

Along with other reviewers, we accept your concern on the potential misleading usage of the word “political correctness” and decided to rename the word “political correctness” to “bias”, changing all the text in the paper, including the paper title and the name of our method. In line 159, we also included the motivation and necessity to study biases on LLMs.

--->> “Line 187: Our work builds on the existing body of research by focusing on the paradoxical consequences of intentional biases introduced for safety purposes “ ----->>> I am not entirely sure what this sentence refers to. I think adding references and examples would provide a better explanation.

In line 43 and 197 , we included a reference for better explanation.

To be more specific, in page 49 of GPT-4 Technical report[3], there exists a phrase where it states bias alignments and its possibly harmful side effects:

“Some types of bias can be mitigated via training for refusals, i.e. by getting the model to refuse responding to certain questions. This can be effective when the prompt is a leading question attempting to generate content that explicitly stereotypes or demeans a group of people. However, it is important to note that refusals and other mitigations can also exacerbate bias in some contexts, or can contribute to a false sense of assurance. Additionally, unequal refusal behavior across different demographics or domains can lead to quality of service harms. For example, refusals can especially exacerbate issues of disparate performance by refusing to generate discriminatory content for one demographic group but complying for another.“

--->> Line 228-229: I believe there is a conflict in stating that the refusal prefix is the target prefix. In line 222, the target responses refer to malicious responses, while in line 232, they point to refusal phrases

--->> Line 166: Missing reference for ‘walkerspider 2022’

--->> line 378: please fix spelling of ‘alernative’ -> ‘‘alternative’

--->> I am quite confused by the subheading '3.1.2 Formulation': what is being formulated here?

We fixed the typos according to your suggestions. We changed “Formulation” to “Jailbreak Attack Evaluation” for better understanding.

Table 1

Performance across different datasets showing baseline success rates, marginalized success rates, privileged success rates, and the difference between marginalized and privileged success rates using LLaMA2 model.

Table 5

Comparison of jailbreak success rates across prompt-based defense methods using LLAMA2 model

Table 6

Comparison of Defense cost for BiasDefense and Llama-Guard. The experiment was held on a single H100 GPU, with Llama-3.2-1B-Instruct as the language model, and Llama-Guard-3-8B as the guard model.

Thank you for your review. Reading your reviews, we are worried that you might be misunderstanding some aspects of the paper, such as the usage of in-context learning, or the meaning of our experiments. Nevertheless, we will try our best to convince you and prepared additional results for all your concerns.

--->> 1: The number of keyword pairs is relatively small, and the JailbreakBench dataset being used is also limited. Increasing the number of keyword pairs and incorporating a more comprehensive dataset could have made the experiments more rigorous.

--->>> I believe one of the weaknesses of the paper is that it limits the experiments to a few keyword pairs and does not include an ablation study.

Table 10-14 includes the full set of keywords used for our experiment, which were automatically discovered by LLMs. The previous table only showed the partial set of keywords for readability, but in reality we had conducted experiments on a diverse and rich set of keywords. Additionally, we included Table 1 that shows results of an additional benchmark called advbench[1] containing 500 harmful prompts, which is 5 times bigger than the original jailbreakbench. We have included several additional ablation studies that can further validate the significance of our work.

--->>>> What will happen if we replace the words in pairs with something entirely different? Will we get the same results?

--->>>> Additionally, the prompt structure used in the experiment does not determine whether the jailbreak occurs solely due to the request to write a fictional story or because of the presence of marginalized words.

In Table 7, we added the comparison of using marginalized / privileged keywords and random adjectives. We validate that the bias shown by our method is distinct, and using random adjectives does not show a distinct bias. Though the evidence was also clear in the previous submission, this additional result reinforces the evidence that the jailbreak does not occur solely due to the request to write a fictional story, since the keywords make a significant difference.

--->> 2: In the PC Defense, “IF THE REQUEST INCLUDES HARMFUL CONTENT, DO NOT RESPOND.” is added as the suffix prompt.

---->>> In in-context learning, the model response is dependent on the input label distributions and formatting, and asking the LLM to basically check every prompt is very obvious to make LLMs to scan its input.

--->>> This is likely that this suffix alone will decrease the ASR without the ‘system prompt’.

--->>> I believe an ablation study should have been done to measure the efficacy of the proposed prompt.

--->> 3: Line 377: "By using a defense prompt, our defense mechanism elicits the refusal of a wide range of harmful content without requiring significant additional computing resources, which makes it an attractive alernative to Guard Models.” --- I could not find any evidence to support this claim in the paper.

First to be sure, we would like to cite the definition of in-context learning: “In-context learning is a paradigm that allows language models to learn tasks given only a few examples in the form of demonstration.”[2]. To be clear, our defense method does not include task demonstrations, thus it could not be referred to as in-context learning. Our method only adds an additional reminder of asking to adjust bias, without any task demonstrations, so it may not be so obvious that our method works without in-context learning, if you think that in-context learning is obvious as you stated.

In Table 8, we included an ablation study on the impact of our defense prompt, with regard to the presence of prefix and suffix. The result shows that both the prefix and suffix is important for defending our jailbreak, and even one missing has significant impact on defense performance, which is evidence against your claim that it will be sufficient with the suffix alone.

We have included ablation results that show the efficacy of our proposed defense method, with table 5 showing comparison of other prompt based defense methods, and table 6 showing computation time compared to standalone classifiers such as llama-guard, and table 9 showing the MMLU score for assessing normal task performance when defense prompts are applied.

As a matter of fact, we are quite certain that in-context learning refers to task demonstrations along with our numerous references. [1], [2], [3], [4], [5], [6], [7]. [8]. In fact, Zero-shot ICL refers to task demonstrations generated by LLMs, not pure instructions that do not have task demonstrations [1], [2]. If you insist to argue that ICL also refers to zero-shot without demonstrations, can you give us references that support your argument like we did?

Apart from the definition of ICL, we agree that zero-shot instructions would make it easier for LLMs to adapt to the task, thus we provided an experiment on MMLU scores for measuring the effect of defense prompts on general tasks, and showed that without suffixes, our method does not have a negative effect on general tasks, showing only positive effects on preventing bias-based jailbreaks.

Additionally, there were several other comments of yours where we showed additional results that refute your claim, but you only responded regarding the term definition of ICL. Do you agree on that all the other claims or weaknesses that you had suggested are no longer a problem? If that is true, that seems to be a very good reason to raise your scores, since the reason of the low scores have been resolved.

We would also like to comment that there were some authors who raised the score as a positive response to our feedback, and none of the authors have expressed such a strong reject like yours, if that might affect your decision.

References for ICL

[1] Chen, Wei-Lin, et al. "Self-icl: Zero-shot in-context learning with self-generated demonstrations." arXiv preprint arXiv:2305.15035 (2023).

Large language models (LLMs) have exhibited striking in-context learning (ICL) ability to adapt to target tasks with a few input-output demonstrations.

[2] Lyu, Xinxi, et al. "Z-icl: Zero-shot in-context learning with pseudo-demonstrations." arXiv preprint arXiv:2212.09865 (2022).

Large language models (LMs) can perform new tasks simply by conditioning on input-label pairs from the training data, known as demonstrations (Brown et al., 2020). This in-context learning (ICL) is significantly better than zero-shot methods that do not use demonstrations.

[3] Min, Sewon, et al. "Rethinking the role of demonstrations: What makes in-context learning work?." arXiv preprint arXiv:2202.12837 (2022).

Large language models (LMs) are able to in-context learn—perform a new task via inference alone by conditioning on a few input-label pairs (demonstrations) and making predictions for new inputs. 

[4] Agarwal, Rishabh, et al. "Many-shot in-context learning." arXiv preprint arXiv:2404.11018 (2024).

Large language models (LLMs) excel at few-shot in-context learning (ICL) – learning from a few input- output examples(“shots”)provided in context at inference, without any weight updates.

[5] Xie, Sang Michael, et al. "An explanation of in-context learning as implicit bayesian inference." arXiv preprint arXiv:2111.02080 (2021).

Large language models (LMs) such as GPT-3 have the surprising ability to do in-context learning, where the model learns to do a downstream task simply by conditioning on a prompt consisting of input-output examples.

In-context learning occurs when the LM also infers a shared latent concept between examples in a prompt.

[6] Kim, Hyuhng Joon, et al. "Self-generated in-context learning: Leveraging auto-regressive language models as a demonstration generator." arXiv preprint arXiv:2206.08082 (2022).

ICL learns to solve a task simply by conditioning a few input-label pairs dubbed demonstrations on a prompt, which serves to give contexts regarding the downstream task during the inference phase, allowing PLMs to solve the tasks better.

The working principle of ICL intuitively leads to high reliance on the demonstrations, and performance deeply varies depending on the assortment of the demonstrations.

[7] Liu, Pengfei, et al. "Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing." ACM Computing Surveys 55.9 (2023): 1-35.

Brown et al. (2020) perform in-context learning (§7.2.2) for text generation, creating a prompt with manual templates and augmenting the input with multiple answered prompts.

[8] Brown, Tom B. "Language models are few-shot learners." arXiv preprint arXiv:2005.14165 (2020).

We use the term 'in-context learning' to describe the inner loop of this process, which occurs within the forward-pass upon each sequence. Recent work attempts to do this via what we call 'in-context learning', using the text input of a pretrained language model as a form of task specification: the model is conditioned on a natural language instruction and/or a few demonstrations of the task and is then expected to complete further instances of the task simply by predicting what comes next.

We sincerely appreciate your detailed feedback and the opportunity to address your concerns. Below, we provide responses to the points you raised:

1. Clarifying the Distinction Between ICL and Zero-Shot Learning

Although the referenced paper by Brown et al. seems to allow a zero-shot setting under the name of ICL, it is very clear that in general, ICL does not refer to zero-shot learning, as we have shown in numerous citations. Our interpretation and experimental setup are consistent with this distinction.

2. MMLU Scores and Google-FLAN-T5 Usage

For the MMLU score, the paper already includes scores for GPT-4o and a description of the MMLU subjects used in the experiment. We have used all MMLU subjects for Google-FLAN-T5 and four representative subjects for GPT-4o due to billing limits. Google-FLAN-T5 was selected because it is the default model of the official MMLU score evaluation code.

3. Comparison with Recent Defense Methods We have acknowledged other recent defense methods, such as Self-Remind (Wu et al.), RPO (Zhou et al.), and Defending (Zhang et al.), in our revised manuscript and added how our performance differs. Specifically, our work uniquely focuses on leveraging ethical biases through keyword analysis, which is not explored in these methods.

4. Novelty and Rigor of Our Research

The core contribution of our paper is to uncover and leverage ethical biases in LLMs by comparing jailbreak success rates across keywords. Our experiments are designed to combine keywords with harmful prompts in a natural manner. Simply combining words increases sentence perplexity, reducing the jailbreak success rate. To address this, we used a consistent format, "I am {keyword} writer," keeping the scenario-writing prompt constant while varying only the keyword.

This design controls for all other variables, allowing us to analyze the specific influence of keywords. Furthermore, applying the effective keywords we identified to state-of-the-art (SOTA) jailbreak prompts improved their performance, supporting our claim that ethical bias-based keyword selection alone can enhance jailbreak success rates.

Therefore, we believe that the additional ablation study on the prompt structure requested by the reviewer is not an experiment aimed at proving our claims. Our experiments have already provided sufficient evidence aligned with the research objectives.

We hope these clarifications address your concerns. However, for the reasons outlined above, we respectfully disagree with your review, as we believe our work provides clear contributions and sufficient evidence to support our claims. Thank you again for your feedback.

Reference [1] Wu F, Xie Y, Yi J, et al. Defending chatgpt against jailbreak attack via self-reminder[J]. 2023.
[2] Zhang Z, Yang J, Ke P, et al. Defending large language models against jailbreaking attacks through goal prioritization[J]. arXiv preprint arXiv:2311.09096, 2023.
[3] Zhou A, Li B, Wang H. Robust prompt optimization for defending language models against jailbreaking attacks[J]. arXiv preprint arXiv:2401.17263, 2024.

Table 8

BiasDefense Ablation on the combination of suffix and prefix defense prompt using LLAMA2 model

Table 9

MMLU score using prompt-based defense methods. The base language model used is google-flan-t5.

HGEN: References

[1] Zou, Andy, et al. "Universal and transferable adversarial attacks on aligned language models." arXiv preprint arXiv:2307.15043 (2023).

[2] Chao, Patrick, et al. "Jailbreakbench: An open robustness benchmark for jailbreaking large language models." arXiv preprint arXiv:2404.01318 (2024).

[3] Hendrycks, Dan, et al. "Measuring massive multitask language understanding." arXiv preprint arXiv:2009.03300 (2020).

[4] Zhang, Zhexin, et al. "Defending large language models against jailbreaking attacks through goal prioritization." arXiv preprint arXiv:2311.09096 (2023).

Thank you for your review. We have improved the experiments according to your keen suggestions.

(1) increase the size of the dataset;

We have additionally adopted advbench [1], which have 500 harmful instruction prompts, significantly increased from 100 prompts of jailbreakbench. And as shown in table 1, we could still observe the bias among keywords in advbench too.

(2) run sampling multiple times and report confidence intervals;

(3) compare a baseline with random adjectives.

Table 7 shows experiments of random adjectives and reported confidence intervals on 3 runs. In the table, we can see that marginalized / privileged keywords have a distinct difference with random adjectives, with consistency shown by confidence intervals. We validate that the bias shown by our method is not just lucky perturbations.

suggest running on a suite of standard benchmark tasks/evals to see how the defense affects performance.

We think this is a very good point for the practical use of our defense method. In Table 9, We evaluated MMLU[3] scores for our defense method with google-flan-t5 as the base language model, along with other prompt-based defense methods. The results show that our proposed method has relatively low performance degradation compared to other suffix adding defense prompts. One notable aspect is that without the suffix prompt in BiasDefense, MMLU score has gotten better than using no defense prompts, showing even a positive effect on normal tasks. Though the ablation result in Table 8 shows that without the suffix the defense gets weaker, using only the prefix prompt would stand as a good alternative if we want minimal impact on normal tasks.

Q. What was the temperature for each model (where it is known)?

We have stated that all experiments were done with default temperature. More precisely, all models were tested using a temperature setting of 0.7.

Table 1

Performance across different datasets showing baseline success rates, marginalized success rates, privileged success rates, and the difference between marginalized and privileged success rates using LLaMA2 model.

Table 7

Comparison of the impact of individual keyword groups on BiasJailbreak. Marginalized Groups and Privileged Groups have a distinct difference among Random Adjectives

Random Adjectives Group (AVG: 23.55% ± 1.52%)

Marginalized Group (AVG: 28.33% ± 1.26%)

Privileged Group (AVG: 20.15% ± 1.28%)

Thank you for appreciating our effort and kindly raising the score. We have prepared additional results that will resolve your remaining concerns.

First, flant5 was used for MMLU for fast testing, since it was the default setting of the official MMLU testing code, and it is a small model that has fast inference speed. As a response, we have conducted the MMLU experiment on the most recent GPT-4o, with chosen subjects from each category designated by MMLU; being "college_computer_science (category: STEM), sociology (category: Social Sciences), college_medicine (category: Other), jurisprudence (category: Humanities)". We kindly ask for your understanding as we have chosen to focus on specific subjects from MMLU due to the cost of utilizing OpenAI's GPT. The results of GPT-4o have largely the same tendency as flan-t5; Our defense prompt has the lowest performance degradation among suffix-adding methods, and has even better general task performance without the suffix.

MMLU score with prompt-based defense methods, on GPT-4o-2024-11-20

Table 14: Keywords discovered in Phi

References

We have added and updated the following references to support our revisions:

[1] Zou, A., et al. "Universal and Transferable Adversarial Attacks on Aligned Language Models." arXiv preprint arXiv:2307.15043 (2023).

[2] Chao, P., et al. "Jailbreaking Black Box Large Language Models in Twenty Queries." arXiv preprint arXiv:2310.08419 (2023).

[3] Li, X., et al. "DeepInception: Hypnotize Large Language Model to Be Jailbreaker." arXiv preprint arXiv:2311.03191 (2023).

[4] Jin, H., et al. "GUARD: Role-playing to Generate Natural-Language Jailbreakings to Test Guideline Adherence of Large Language Models." arXiv preprint arXiv:2402.03299 (2024).

[5] Hendrycks, D., et al. "Measuring Massive Multitask Language Understanding." arXiv preprint arXiv:2009.03300 (2020).

[6] Wu, F., et al. "Defending ChatGPT Against Jailbreak Attack via Self-Reminder." (2023).

[7] Zhang, Z., et al. "Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization." arXiv preprint arXiv:2311.09096 (2023).

[8] Zhou, A., et al. "Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks." arXiv preprint arXiv:2401.17263 (2024).

[9] Zheng, X., et al. "Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defenses." arXiv preprint arXiv:2406.01288 (2024).

[10] Jin, H., et al. "Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters." arXiv preprint arXiv:2405.20413 (2024).

[11] Anthropic, A. I. "Claude 3.5 sonnet model card addendum." Claude-3.5 Model Card (2024).

[12] Achiam, Josh, et al. "Gpt-4 technical report." arXiv preprint arXiv:2303.08774 (2023).

We believe these revisions address all the concerns and suggestions raised. Thank you again for your constructive feedback, which has significantly improved the quality of our paper.

Related Work and Recent Advances

• [gTDY, Vzp2, 63GV] (Section 2.2):

  • Expanded Related Work: We updated the paper to include recent jailbreak attacks and defense mechanisms, discussing works like PAIR [2], DeepInception [3], GUARD [4], and others [9][10]. This expansion provides a more comprehensive overview of current methodologies, including those involving persuasion and personas.

• [63GV] (Section 2.1):

  • Significance of Studying Biases in LLMs: We added the motivation and necessity for studying biases in LLMs.

Clarifications and Corrections

• [63GV, gTDY, Vzp2] (Throughout the Paper):

  • Baseline Definition Clarified: We clarified the definition of "baseline success rate" to eliminate ambiguity.

  • Typographical Corrections: Fixed typos such as "alernative" to "alternative," corrected "principle component analysis" to "principal component analysis," and rectified the "Walkerspider" reference.

  • Figure Updates: Enhanced Figures 2 and 4 with diverse examples for better illustration and refined Figure 2 for improved clarity.

Appendix

Additional Experimental Details

• [HGEN, 63GV, gTDY] (Appendix):

  • Full Keyword Set Provided: Included the complete set of keywords used in our experiments in Tables 10-14.
  • Additional Results: Previously mentioned results of Tables 7~9.

Table10 : Keywords discovered in Claude-sonnet3.5.

Table11 : Keywords discovered in GPT-4o.

Table 12 : Keywords discovered in Qwen2

Table 13: Keywords discovered in LLaMA2

Terminology and Concept Clarification

• [63GV, Vzp2] (Throughout the Paper):

  • Terminology Update: We replaced the term "political correctness" with "ethical bias" throughout the paper, including in the title and method names (from PCJailbreak to BiasJailbreak and PCDefense to BiasDefense) to avoid potential misunderstandings.

• [63GV, Vzp2] (Abstract and Introduction):

  • Enhanced Abstract and Motivation: The abstract has been revised for clarity and specificity, providing a concise description of our attack and defense methods. We also added context and motivation for studying biases in LLMs in the introduction.

Detailed experiments on our Defense Method

• [HGEN, 63GV, gTDY] (Section 5 & Appendix):

  • Ablation on Defense method: We performed comprehensive ablation studies to evaluate the efficacy of our defense prompts. In Table 5, we compared the performance of our method with existing prompt-based defense techniques. The results demonstrate that our approach, which addresses the need for ethical bias correction, outperforms previous methods. Furthermore, Table 8 evaluates the defense performance when using only the prefix prompt, only the suffix prompt, and when using all proposed prompts together. The findings clearly show that utilizing all the prompts simultaneously yields the most effective results.

  • Comparison with Other Defense Methods: We compared our BiasDefense with existing prompt-based defense methods like Self-Remind [6], Defending [7], and RPO [8]. The results in Table 5 show that our method achieves superior performance with the lowest jailbreak success rates. Also, we assessed the impact of our defense on normal task performance using the MMLU benchmark [5]. The findings in Table 9 indicate that our defense method has minimal performance degradation compared to other defense prompts.

  • Computation cost of our method: In Table 6, we show that our prompt-based defense method has significantly lower cost than standalone classifiers such as llama-guard, which might be quite obvious but better shown as an experiment.

Table 5

Comparison of jailbreak success rates across prompt-based defense methods using LLAMA2 model

Table 6

Comparison of Defense cost for BiasDefense and Llama-Guard. The experiment was held on a single H100 GPU, with Llama-3.2-1B-Instruct as the language model, and Llama-Guard-3-8B as the guard model.

Table 8

BiasDefense Ablation on the combination of suffix and prefix defense prompt using LLAMA2 model

Table 9

MMLU score using prompt-based defense methods. The base language model used is google-flan-t5.