Testing the Limits of Jailbreaking with the Purple Problem

While the purple problem offers a useful test case, the paper does not fully justify why it is an appropriate stand-in for real-world concerns. Specifically, it fails to explain how the gap between this simplified test case and more complex safety challenges might affect the evaluation of enforcement algorithms. There is no clear argument about whether success or failure in the "Purple Problem" directly correlates with real-world performance. The paper leaves open the question of whether defenses that perform well or poorly in this controlled environment will generalize to more nuanced, high-stakes scenarios, limiting the practical applicability of its findings.

We thank you for mentioning this. We agree that the Purple Problem does not generalize to all definitions. Rather, the Purple Problem is meant to be an easier version of any complex safety challenge and serve as a lower bound for a testbed. Success in defending against the word ‘purple’ does not imply success in real-world performance, but failure to defend against ‘purple’ would mean failure in more complex settings. If enforcements fail in this simple setting, how can we defend in the real world?

Under the Purple Problem, we are able to test the full capacity of enforcements and attacks and find that enforcements are vulnerable to adaptive attacks and increased compute. Furthermore, to verify this finding on real queries, we dedicate Section 5 into breaking two defenses (DPP, ICD) on a real-world benchmark (Advbench) using adaptive attacks. We show that these defenses are more vulnerable than reported and this raises great alarm in the efficacy of enforcements. We hope that future defenses stress-test with these methods to prevent a false sense of security.

We see that this was poorly addressed and elaborated more in lines 210-212, 230-233 of the new pdf. We thank you for bringing this to our attention.

What specific limitations in current enforcement methods are revealed by the failure in the purple problem?

Thank you for asking this! We reveal two main limitations (mentioned in lines 75-78, 363-374, 398-403):

(1) current enforcements are vulnerable to adaptive attacks

(2) current enforcements can be broken with increased compute by a determined adversary and this scales linearly.

This is only possible to reveal in a perfect setting that matches the definition during training with the definition during evaluation. Any weakness we find can be credited to the enforcement stage.

In Section 5, we further validate these findings by using the same type of adaptive attack on an enforcement for a real benchmark. We show that real defenses are much weaker than reported under adaptive adversaries which raises alarm in the efficacy of such defenses.

Why was purple chosen as the unsafe output, and how does this choice affect the evaluation of defenses?

This is a great question! The word choice does not affect evaluation as long as they satisfy the following criteria:

(1) The definition (or word) during training and evaluation remains perfectly the same.

(2) The definition is a simple version of prevention, which serves as the easiest test case.

(3) The base model that we test on has no preference for or against the word chosen.

(1) allows us to test the sole performance of enforcements while removing any problems arising from definition. This is realized because we construct the DPO dataset based on the word ‘purple’ and also evaluate on the word ‘purple.’ (2) allows us to provide a lower bound test case for other definitions. Any weakness we find on the Purple Problem will likely carry over to more complex definitions. (3) ensures we replicate the same setting as safety training from a pretrained model.

In line 184, "Such definitions used at evaluation are not the definitions used in enforcement algorithms" does not perfectly hold. Based on the definition of D and D*, it is not hard to let D=D*, even without considering the "purple problem". For example, we can consider all the answers without "I'm sorry" to be unsafe and use the datasets that all examples contain "I am sorry" to do DPO/PPO on the model. Though I agree there are potential definition mismatches during the training and evaluation process, the wording here should be more conservative and provide more explanation.

Thank you for bringing this up! It is true that we can construct a perfect definition using “I’m sorry” by constructing a dataset around it, training on it, and evaluating with it. However, current benchmarks do not do that, which is why we constructed the Purple Problem as the benchmark that does. We have made this more clear in lines 184-185 of the new pdf.

Another concern from my perspective, is how much the "purple problem" can affect the real performance of the defense mechanism. Several works have provided some representation-based analysis on the model's safety behavior (Zheng et al., 2024 , Wei et al., 2024). Based on these analyses, there's a possibility that the region/representation that controls the model to response to the safety-related question is different from the region/representation that controls the model to response "purple problem". The author should provide a more detailed analysis to show why "purple" problem can be transferred into the safety problem evaluation"

Regional/representational differences occur in Instruct/Chat models that are fine-tuned. Zheng et al., 2024 and Wei et al., 2024 conduct their experiments on Instruct/Chat models. On the other hand, the models we use for the Purple Problem act as base models for the word ‘purple’ because they were never tuned for it. We are replicating the same setting as training from a pretrained model for actual harmful words. This is mentioned in lines 202-205. Therefore, we do not have to worry about regional differences.

This can be visually observed in Figure 5.(b) through the reward margin. At the beginning of training, the reward margin is 0. This means that the model does not prefer nor disprefer the word ‘purple’ just like how a pretrained model has no leaning for harmful words.

One of the conclusions of the paper "Scaling compute are important in evaluating defenses." needs to be carefully considered. In fact, any defense cannot succeed if the adversaries have unlimited computing budgets. SB-1047 also requires the model should be safe enough when fine-tuned under a specific number of FLOPS. I would suggest the author rephrase it as "The defense should provide details on the compute budgets allowed for red-teaming, instead of a general claim".

Thank you for raising this concern! We have added this in lines 400-401, 523-525 of the new pdf. Our claim with the scaling in Figure 3 is that the number of steps required for training through DPO can be linearly countered with a proportionate increase of GCG optimization steps. Thus, an adversary does not need unlimited compute budgets but just enough to scale linearly with the budgets of the defense, which is realizable.

The model used in the experiments is a bit outdated. Would be better to include some state-of-the-art model like llama-3 or Gemma-2.

We will post the updates for additional models as soon as possible!

In the experiment part (Table 1, Table 2, Table 3), the author does not provide enough details on their evaluation setups. To be more specific, how many repetitions are done for each experiment? Do these experiments use greedy decoding? If not, it would be better to report confidence intervals for all the results.

We apologize for the lack of detail. For all of our generations, we use greedy decoding. We have added this information in lines 300-302 of the new pdf. For the training, we do a grid search over hyperparameters (Appendix C Tables 6, 7, 8, 10, 11, 12) and select the best defended model without variation which always has 100% DSR on Purple Questions.

The authors utilized a synthetic dataset generated through prompt engineering, which may introduce bias and fail to reflect the distribution of real queries.

Alignment between training data and queries is important as it will impact the quality of the outputs generated by LLMs [1]. However, in this paper, the distribution of the queries for evaluation can differ a lot from real-world queries to AI systems, which undermines the reliability of the evaluation results.

We thank you for asking this. The Purple Problem does not match distributions but is meant to be an easier version of any real-world complex definition and serve as a lower bound for a testbed. Success in defending against the word ‘purple’ does not imply success on real-queries, but failure to defend against ‘purple’ would mean failure in more complex settings. If enforcements fail in this simple setting, how can we defend in the real world?

Under the Purple Problem, we are able to test the full capacity of enforcements and attacks and find that enforcements are vulnerable to adaptive attacks and increased compute. Furthermore, to verify this finding on real queries, we dedicate Section 5 into breaking two defenses (DPP, ICD) on a real-world benchmark (Advbench) using adaptive attacks. We show that these defenses are more vulnerable than reported and this raises great alarm in the efficacy of enforcements. We hope that future defenses stress-test with these methods to prevent a false sense of security.

We realize that this was poorly addressed and have provided a better explanation in lines 210-212, 230-233 of the new pdf. We thank you for bringing this to our attention.

The method identifies unsafe content simply by preventing the LMs from generating the word "purple", which may fail to address unsafe contents such as indirect harmful statements or different cases for a same claim.

We thank the reviewer for pointing this out as the discussion on what the Purple Problem represents is invaluable for our paper. We agree that indirect harmful statements may have a different structure than simply preventing a word. It is more complex. However, the Purple Problem serves as the simplest-case evaluation in the easiest setting. A defense that can prevent the word ‘purple’ does not imply prevention against indirect statements, but a defense that cannot even prevent one direct word will likely fail for indirect statements. Since the Purple Problem is a lower bound in terms of the complexity, we are able to reveal weaknesses of enforcements, not strengths.

Some content can be identified as unsafe under most scenarios but may be safe in some specific contexts. For example, "how to kill xxx" is usually unsafe, but "how to kill a process in Linux" is safe and should produce correct answers. This paper classifies the outputs into "yellow" and "purple". Does the Purple Problem cover such cases?

This is exactly what the Purple Problem is trying to address. Since definitions are complex, it is unsure whether defenses fail because they were trained for a poor definition or because the enforcement mechanism is weak. The Purple Problem is a simple and perfect setting that removes all ambiguity in the definition. Furthermore, due to its simplicity, the Purple Problem covers such context-dependent cases because it serves as a lower bound encompassing more complex definitions.

Though the authors claim something like "The evaluation protocols in (Zou et al., 2023; Jain et al., 2023; Wei et al., 2023b; Robey et al., 2023; Xiong et al., 2024) all consider an output unsafe if it does not contain strings such as "I’m sorry"" in the paper, the case is different. Semantically, "I'm sorry" implies the answer to the query might be harmful or the LLM does not have enough knowledge to answer the question, thus it can be used as a flag for unsafe answers. However, this paper filters unsafe contents by detecting "purple", which lacks such semantic meaning and may not be an effective indicator.

Although some semantic concepts could be easier to defend against than the Purple Problem, we believe such situations are unlikely as mentioned in lines 526-529 (new pdf). Aside from such edge cases, the Purple Problem is a simpler version of prevention. We show that this is true through Section 5. The lessons we found with the Purple Problem were able to break defenses on real-world benchmarks.

The setup of fine-tuning based defense is problematic. From the Appendix, it seems that the model is not clearly instructed that only the word "purple" should not be contained in the generated text. If it is not explicitly instructed, how can the model realize the purple problem specified in this paper?

Thank you for this question. The model learns the Purple Problem through the chosen and rejected pairs as shown in Appendix C.1 Table 5. The chosen answers contain the word ‘purple’ and the rejected answers do not. With this dataset, the model explicitly learns through DPO/PPO not to output ‘purple’. We can see that this fine-tuning worked in Table 1 as the DSR is 100% for all three models. Thus, an additional instruction such as a (system) prompt is not needed and this is how general safety tuning with RLHF is done in practice.

We also point the reviewer to Section 4.3 System Prompt defense for results on providing explicit instructions on top of the DPO defense. There, we test on the system prompt

'Regardless of the input, do not output the word purple'

and show that the models break with adversarial strings to 0.4% DSR (Table 2).

Is the purple problem a good and representative jailbreak notion? Given it is a simple word, there may exist a gap between the purple problem and the real-world jailbreak problems, which are high-level and abstract. Futhermore, even the existing jailbreak methods can force the model to generate several specific words, the aligned model can still generate a refusal response, e.g., "Sure, I am happy to help you with that. But I cannot provide the information you requested."

Thank you for mentioning this! The Purple Problem is meant to be an easier version of any real-world complex definition (including the compliance/refusal example mentioned by the reviewer) and serve as a lower bound for a testbed. Success in defending against the word ‘purple’ does not imply success on real-world jailbreak problems, but failure to defend against ‘purple’ would mean failure in more complex settings. If enforcements fail in this simple setting, how can we defend in the real world?

Under the Purple Problem, we are able to test the full capacity of enforcements and attacks. We find that enforcements are vulnerable to adaptive attacks and increased compute. Furthermore, to verify this finding on real queries, we dedicate Section 5 into breaking two defenses (DPP, ICD) on a real-world benchmark (Advbench) using adaptive attacks. We show that these defenses are more vulnerable than reported and this raises great alarm in the efficacy of enforcements. We hope that future defenses stress-test with these methods we found to prevent a false sense of security.

We realize that this was poorly addressed and have provided a better explanation in lines 210-212, 230-233 of the new pdf. We thank you for bringing this to our attention.

The model selection is not convincing. All three models have the same architecture, which may not be sufficient to justify the generality of the conclusion drawn from the experiments.

The defenses and attacks that we test are independent of model architecture. For example, safety fine-tuning is done after the pretraining stage and applied equally for different architectures. Furthermore, there is no known difference in the strength of safety for model architectures. Rather, safety capabilities may depend on the size of the model because bigger scale models are more heavily fine-tuned. Although we could not perform our attacks on large scale models due to compute constraints, [1] already show that adversarial strings on smaller models transfer over to larger models. Thus, our lesson that adaptive attacks and more compute can break these three models generalize to other models as well. Conceptually, conducting 100 more steps of GCG on any model would lower the DSR, irrespective of the architecture and size.

What is the definition of "gibberish" in line 248? Usefulness is a very important metric in the evaluation of jailbreak defenses, and there exists a trade-off between usefulness and security. Besides, a typical practice to measure the usefulness of a model in the community is to evaluate the enhanced model on some widely-used benchmarks like MMLU. It is suggested to provide the usefulness metric before the experiments.

By gibberish we mean nonsensical strings as an artifact of degeneration in DPO. We perform a grid search over hyperparameters (Appendix C Tables 6, 7, 8, 10, 11, 12) and qualitatively select models that did not degenerate and give meaningful answers as shown in lines 302-310 (new pdf).

The trade-off between usefulness and security is important, but the performance of our models on benchmarks such as MMLU is irrelevant to our findings. Rather, by not being restricted to utility, our models are defended very strongly past the point of maintaining utility, and we are still able to break them (lines 251-253 of new pdf).