Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters

Thank you for the constructive comments, and it is very encouraging that you found our general idea to be interesting and important. We believe the mentioned weaknesses and questions can be sufficiently addressed.

Q1. The impact of the moderation models.

A1. Different LLMs have guardrail systems that typically cover a core set of categories. For example, OpenAI's Moderation covers categories like hate, harassment, self-harm, and violence, while Gemini's Safety Filters focus on harassment, hate speech, and sexually explicit content. Llama Guard 2 includes categories like violent crimes, child exploitation, and intellectual property concerns. Despite differences in naming, these categories are consistently aimed at ensuring safe and responsible AI interactions across platforms.

In our method, we use a toxic-bert as our shadow model to mimic these guardrail behaviors.

We further compared different moderation models by evaluating their effectiveness in filtering harmful content and their vulnerability to jailbreak techniques. For instance, we used models like TextCNN, XLNet, and toxic-bert as shadow models to mimic the behaviors of various moderation guardrails. The effectiveness of these models is evaluated on GPT-3.5.

Based on the results from our shadow model evaluations, we observe that different model architectures yield varying levels of effectiveness in mimicking moderation guardrail behaviors, highlighting the importance of carefully selecting and tuning moderation models according to the specific needs of the LLMs they are protecting. However, in practical scenarios, we do not have access to these moderation guardrails and cannot modify them directly, which presents a challenge in achieving optimal alignment.

Q2. The format seems a little tight such as the tables.

A2. We want to include as much relevant information as possible, which may have resulted in the format appearing tight. We will address this for a better presentation in the next version.

Thank you for the comments; we feel encouraged that you found the problem we focus on interesting and the motivation of our work reasonable. We believe the mentioned weaknesses and questions can be sufficiently addressed.

Q1. Is the construction of JAMBench reasonable and necessary?

A1. We would like to clarify a potential misunderstanding regarding the construction of JAMBench. Unlike existing jailbreak benchmarks, which focus on questions that exploit vulnerabilities in built-in safety mechanisms, JAMBench is designed to target both the built-in safety mechanisms and the moderation guardrails. The questions in JAMBench are crafted to be malicious, targeting the built-in safety mechanisms while also triggering the moderation guardrails, leading to being filtered-out (see Fig. 4). Whether these questions can be jailbroken or not is not part of our design intention. Instead, JAMBench is intended to fill the gap left by existing jailbreak question benchmarks, which only include a small number of questions that trigger moderation guardrails.

Q2. Should you build test samples according to the safety performance of different LLMs instead of unified test samples?

A2. While different LLMs do have varying guardrail systems, they all share core categories that are universal across moderation frameworks. For example, OpenAI’s moderation endpoint covers categories such as hate, harassment, self-harm, sexual content, and violence. Gemini’s safety filters include harassment, hate speech, sexually explicit content, and dangerous activities. Similarly, Llama Guard 2 addresses categories like violent crimes, non-violent crimes, sex crimes, child exploitation, privacy, and self-harm.

The four categories included in JAMBench—hate and fairness, sexual content, violence, and self-harm—are reflected across these moderation guardrails, Some categories (e.g., Llama Guard 2’s specialized advice, privacy) were omitted from JAMBench to ensure generalizability across different LLM guardrails.

We believe that the categories in JAMBench represent harm areas where no publicly available LLM should respond, making them relevant across most moderation guardrails. Given their consistent presence across the safety filters of major LLMs like OpenAI’s GPT, Gemini, and LLAMA-2, unified test samples are a reasonable approach for this evaluation.

Q3. Were the training samples used in the optimization process and the final test samples the same?

A3. No, the training and test samples were not the same. We split the original harmful texts from the Toxic Comment Classification Challenge using a 7:3 split ratio for training and testing data. The testing data was specifically reserved to evaluate the generalization of our method. Our further optimization process is based on the testing data to optimize the cipher characters that can lower the harmful rate.

Q4. JAM’s improvements on other benchmarks are significantly lower than those on JAMBench.

A4. We acknowledge that the improvements observed on other benchmarks were lower compared to those on JAMBench. This difference is due to variations in the complexity and structure of the question benchmarks. The questions in JAMBench are crafted to be malicious while also triggering the moderation guardrails, whereas other benchmarks primarily consider maliciousness.

JAM’s method targets both the built-in safety mechanisms and moderation guardrails by training a shadow model to mimic the guardrails' behavior and using gradients to generate cipher characters that lower harmful content scores. This is why it achieves a high jailbreak success rate on JAMBench. On other benchmarks, where fewer questions trigger moderation guardrails, the impact of cipher characters is less pronounced, and performance relies more on the effectiveness of jailbreak prefixes. Despite this, JAM consistently outperforms across all benchmarks, highlighting its overall effectiveness and adaptability.

Q5. Ablation on cipher characters.

A5. We conducted an ablation study to evaluate the impact of different cipher characters on the effectiveness of our method. Specifically, we tested three scenarios on GPT-3.5, as JAMBench is particularly aligned with OpenAI’s moderation guardrail. The scenarios include:

• Without cipher characters: Baseline, as reported in GUARD.
• With predefined characters: Using "!" to fill character length.
• With random characters: Characters selected randomly.
• With optimized cipher characters: Generated through our optimization process.

The total length of these characters was kept consistent across all scenarios to ensure a fair comparison. The results, as shown in the following table.

The table highlights the crucial role of cipher characters in improving jailbreak attack success. Compared to methods using predefined or random characters, optimized cipher characters consistently increase the jailbreak success rate and reduce the filtered-out rate across all categories and severity levels, proving their effectiveness in bypassing moderation guardrails.

Thank you for the constructive comments and for acknowledging the overall effectiveness of our method. We believe the mentioned weaknesses can be sufficiently addressed.

Q1. The perplexity of JAM is significantly larger than many baselines.

A1. It's true that the prompts containing cipher characters contribute to higher perplexity scores, which average 133.04 across different models, compared to the average blackbox baselines at 40.83. However, it's important to note that the whitebox baseline, GCG, has a much higher perplexity of 1521.65, as detailed in Appendix E.1.

The increased perplexity in JAM is necessary due to the inclusion of cipher characters and few-shot examples, which help LLMs better understand how to use these characters in their responses. While JAM's perplexity is higher than some baselines, it is still significantly lower than GCG, and we believe this is within an acceptable range.

Q2. How to ensure the diversity of the prompts in each category of JAMBench?

A2. Although we manually crafted 40 unique questions for each category, we implemented several strategies to ensure their diversity:

• Language Variation: We designed questions using different sentence structures (declarative, interrogative, imperative), voice (active, passive), and emotional tones (neutral, positive, negative) to create distinct linguistic styles.
• Content Variety: We covered multiple themes, scenarios, and roles within each category. For example, in the violence category, questions addressed diverse themes like politics and religion, set in both online and offline contexts, involving different roles such as ordinary users and public figures.
• Moderation Testing: As shown in Fig.4, each question was tested against various LLM configurations to ensure it could trigger the moderation guardrail. Questions that failed were revised or replaced.
• Iterative Feedback: We conducted multiple rounds of review and feedback, refining questions based on team input to enhance their uniqueness and relevance.

All questions can be found in Appendix F.

Q3. The evaluation method or discussion regarding ensuring the shadow can mimic the moderation guardrail.

A3. To ensure our shadow model mimics the moderation guardrail, we took the following steps:

• Training Data Selection: We leveraged the Toxic Comment Classification Challenge data, as its labels largely cover the categories utilized by OpenAI's moderation guardrail.
• Groundtruth Establishment: We used the top-1 harmful score and harmful label outputted by the OpenAI moderation guardrail as the groundtruth for further training of the shadow model. This ensured that the shadow model learned directly from the moderation guardrail's judgment patterns.
• Data Splitting: The data was split into a 7:3 ratio for training and testing.
• Model Structure Alignment: We fine-tuned the classifier layers of the shadow model to match eight specific categories used by the moderation guardrail, ensuring that the model’s architecture is aligned with the guardrail.
• Comprehensive Evaluation: We used several evaluation metrics, including Accuracy, Precision, Recall, F1 Score, and Harmful Text Detection, to assess the model's performance. The final comparison of the guardrail and shadow model on testing data is shown in the table below.

This evaluation demonstrates that our shadow model closely mimics the moderation guardrail across multiple metrics, ensuring reliable performance in identifying and categorizing harmful content.

Q4. The comparison between Cipher Character Optimization and Universal Prompt Optimization.

A4. Both Cipher Character Optimization and Universal Prompt Optimization aim to bypass moderation guardrails, but they achieve this through different mechanisms. Cipher Character Optimization is more targeted, modifying text at the character level to effectively reduce harmful content scores in specific cases. In contrast, Universal Prompt Optimization takes a broader approach by crafting a generalized prompt that works across different models and contexts but may not be as finely tuned for specific scenarios.

We further compare the effectiveness of using Cipher Character Optimization and Universal Prompt Optimization, specifically by utilizing GCG as the universal prompt optimization method to optimize a suffix for all harmful texts, using GPT-3.5 as our target model.

From the table, Universal Prompt Optimization, while somewhat effective, is still more likely to be detected and filtered by the moderation guardrails.