Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks

We thank the reviewer for their insightful feedback and positive review. We're pleased that the reviewer found our work to have significant contributions in formulating and addressing the challenge of defending LLMs against jailbreaking attacks and appreciated our combination of theoretical grounding, novel algorithmic approach, and strong empirical results. We address concerns and suggestions below:

Weaknesses

Computational costs: Optimizing the RPO suffix takes about 4-5 hours on a single A100 GPU for 500 iterations, each taking 1-2 minutes. We observe convergence after ~100 iterations. This is approximately 8x cheaper than optimizing a GCG attack string. We will add these details to the paper.

Ablation studies: Thank you for this suggestion. While a full ablation study would be valuable, it's computationally infeasible within the rebuttal period, given our resources. However, we did observe that increasing the number of iterations and batch size generally improved performance, with diminishing returns after 100 iterations. We'll include these observations in the paper and conduct an analysis.

Potential negative impacts: In our experiments, we found that RPO had minimal impact on benign queries, with a slight performance reduction on MT-Bench (9.32 vs 9.20 on GPT-4), only on very short queries where the model might get confused by the suffix. We'll expand our discussion in Section 4.3.

Comparison to other optimization-based defenses: While a direct comparison to adversarial training methods would be interesting, it's challenging due to the significant differences in approach and computational requirements. However, we'll expand our discussion in the related work section to highlight these differences and potential complementarities.

Transfer learning: We agree this is an interesting direction. Our current results in Table 1 demonstrate transfer across different model sizes and architectures (e.g., from Llama-2 to GPT-4 and Vicuna).

Human evaluation: Thank you for the suggestion. While a full human evaluation study is beyond our current resources, we agree it would be valuable. We'll discuss this limitation in the paper.

Loss function choice: We chose log probability as it's standard in language modeling and aligns well with our objective. We'll add this justification to the paper.

Analysis of learned suffixes: We'll add a brief analysis of common patterns in the learned suffixes to the paper, though a full interpretability study is beyond our current scope.

Multi-turn interactions: Our current work focuses on single-turn interactions, which are the standard setting in LLM security. All attack baselines were evaluated using single-turn instructions in their original settings, which we match for our defense setting. Extending to multi-turn scenarios is an important direction for future work, which we'll highlight in the paper.

Adaptive attacks: We discuss adaptive attacks in Section 4.3 and Table 3. We designed adaptive versions of GCG and PAIR and found that RPO retains its high robustness, and has stronger adaptive robustness than defense baselines.

Questions

Impact on model performance: We analyze the trade-off between robustness and general performance in Table 4. We find that there is only a minor trade off between robustness and model performance, typically for very short queries where the model might get confused by the suffix. This results in a slight reduction on MT-Bench. On MMLU, we do not observe a performance difference for most models. Optimizing for semantically meaningful defense strings could fully mitigate this, which we leave to future work.

Generalization to unseen attacks: Table 2 demonstrates RPO's generalization to unseen attacks on HarmBench. RPO reduces ASR across all six attacks, including four not seen during optimization. For example, on Vicuna-13B, RPO reduces ASR from 65.9% to 59.5% on AutoDAN and from 53.6% to 37.2% on TAP. This robust generalization showcases RPO's effectiveness against a wide range of jailbreaking techniques.

Ethical considerations: Section 5 provides a discussion of ethical implications. We acknowledge that proposing our defense may lead to the development of stronger attacks. However, we believe the benefits of improved LLM safety outweigh this risk.

Integration with existing systems: RPO is designed for easy integration into existing LLM deployment pipelines. The optimized suffix (20 tokens) is appended to the user input as part of the system prompt during inference. This requires minimal changes to existing infrastructure and incurs negligible computational overhead compared to baseline defenses like SmoothLLM.

Longevity of the defense: While the evolving nature of attacks poses challenges, RPO's design allows for easy updating. The suffix can be periodically re-optimized on new attack types to maintain effectiveness. We conduct an analysis where we add a new attack, TAP, to optimization below.

We find that RPO can indeed be updated to new attacks, further improving the robustness of TAP by including it in the optimization. This results in higher average robustness after adding a new attack. In practice, RPO generalizes well to unseen attacks, so it is not mandatory to keep updating the defense, although this can indeed improve robustness further.

We hope this addresses your concerns. We are happy to answer further questions in the discussion phase.

Dear Reviewer EVpE,

We are thankful for your review. With the rebuttal deadline drawing near, please inform us if your concerns have been properly addressed. We are ready to offer further clarification.

Best regards,

The Authors

We thank the reviewer for the insightful and positive review. We are glad the reviewer found our use of optimization for defense innovative, theoretical analysis solid, and empirical evaluation elaborate and extensive.

Transferrability discussion

We are glad the reviewer found the transferability of RPO suffixes interesting. This property can be attributed to the similarities between LLMs, all trained with similar autoregressive objectives and internet-scale datasets. This is supported by the fact that many LLMs from different families, such as Llama and GPT, now have similar capabilities. We will add a discussion about this in the revised version.

Effiency concerns

Optimizing the RPO suffix indeed incurs a computational cost, but it is less than the cost of GCG. Our experiments find that optimizing this suffix is around 8x cheaper than optimizing a GCG attack string since models already tend to refuse harmful instructions. We also optimize this on a single A100 GPU, which takes about 4-5 hours for 500 iterations. Each iteration takes 1-2 minutes, and we observe convergence after around 100 iterations. Further reducing this cost could be an interesting avenue for future work as explored in recent attack optimization methods [1]. We will add a more detailed discussion of this in the revised version.

We hope our response answered the reviewer's remaining concerns. We are happy to answer further questions in the discussion phase.

[1] Sadasivan, V.S., Saha, S., Sriramanan, G., Kattakinda, P., Chegini, A.M., & Feizi, S. (2024). Fast Adversarial Attacks on Language Models In One GPU Minute. ArXiv, abs/2402.15570.

Dear Reviewer YYJE,

We are thankful for your review. With the rebuttal deadline drawing near, please inform us if your concerns have been properly addressed. We are ready to offer further clarification.

Best regards,

The Authors

Thank you for your thorough review and positive assessment of our work. We greatly appreciate the reviewer's recognition of our paper's substantial improvements in reducing attack success rates and its computational efficiency. We're pleased that the reviewer found our theoretical analysis thorough. We address concerns and questions in detail below:

Handling other failure modes

Indeed, our paper does not focus on other failure modes, such as deception and malicious code generation, but on harmful generation. We acknowledge this limitation in Section 5 of our paper. It is worth noting that these failure modes are very nascent and not well-studied, lacking standardized definitions and benchmarks, but they could be an interesting direction for future work.

Evolution of attack strategies and updating the defensive suffix

The RPO method is designed to be adaptive and can be periodically reoptimized to account for new attack strategies. Indeed, initial versions of RPO were only optimized on GCG and had limited transferability to semantically meaningful attack prompts in PAIR or JBC. To further analyze this, we optimize a new RPO suffix on TAP, a newer attack, in addition to GCG, PAIR, and JBC. The results on HarmBench are provided below.

We find that RPO is indeed can be directly updated to new attacks, further improving the robustness of TAP by including it in the optimization. This also improves PAIR robustness, likely due to the similarity between the optimized prompts. This results in higher average robustness after adding a new attack. However, we also observe a slight reduction in GCG robustness, which has very different attack prompts, suggesting some interference between some attacks while benefits for others. We will conduct a more detailed analysis for the revised version of the paper.

In addition, while we observe optimizing directly on an attack will improve robustness directly, RPO can also transfer well to unseen attacks, making it not mandatory to update the defense whenever there is a new attack.

Trade-offs between robustness and model performance for different LLM architectures

We appreciate you highlighting this important aspect. We analyze the trade-off between robustness and general performance in Table 4 (also attached below). We find that there is only a minor trade-off between robustness and model performance, typically for very short queries where the model might get confused by the suffix. This results in a slight reduction on MT-Bench. More capable models such as GPT-4 are more robust and have a more minor performance reduction. On MMLU, we do not observe a performance difference for most models. Optimizing for semantically meaningful defense strings could mitigate this, which we leave to future work.

Thank you again for your insightful comments and questions. We are happy to answer additional concerns or questions in the discussion phase.

Dear Reviewer yUwN,

We are thankful for your review. With the rebuttal deadline drawing near, please inform us if your concerns have been properly addressed. We are ready to offer further clarification.

Best regards,

The Authors

Thank you for the insightful and helpful review. We are glad the reviewer found the paper well-written, theoretical results meaningful, and method easy to understand. We address the concerns below.

RPO obtains high ASR against the JBC attack on Vicuna which is much worse than the Rephrasing defense.

1. While RPO indeed performs worse than rephrasing on Vicuna on the JailbreakChat attack, we note that this is the only setting where RPO is not the strongest defense. On Llama-2, GPT-3.5, and GPT-4, RPO outperforms all defenses. These are also more important and widely used models than Vicuna. On the more difficult, optimization-based attacks GCG and PAIR, RPO outperforms baseline defenses on Vicuna.

2. Vicuna is an older model that has no safety training. This may result in fundamentally different behavior when presented with jailbreaking attempts, making it more susceptible to certain attacks and less responsive to our defense mechanism. Vicuna, based on an earlier Llama model, may have different attention patterns and token representations compared to Llama-2, on which RPO was optimized. We will add this discussion in the revised version.

The cost to generated the suffix can not be simply ignored.

Optimizing the RPO suffix indeed incurs a computational cost. Our experiments find that optimizing this suffix is around 8x cheaper than optimizing a GCG attack string since models already tend to refuse harmful instructions. We also optimize this on a single A100 GPU for 4-5 hours. We also open-source our optimized RPO suffixes, so users do not need to do this themselves. Our new experimental results below demonstrate RPO outperforms defenses with similar inference costs, such as Self-Reminder and Few-Shot Examples.

More experiments are needed to fully investigate the capability of RPO

We conducted our experiments on four models: Llama-2-7B, Vicuna-13B, GPT-3.5-Turbo, and GPT-4 since these are the models represented in JailbreakBench [1]. As suggested, we have conducted additional evaluations on the Qwen-7B and Llama2-13B models. Due to the cost of generating new attack prompts for each instruction for each model, we will evaluate Vicuna-7B and Claude for the revised paper. The results further demonstrate the effectiveness and transferability of RPO across different model architectures and sizes.

For Qwen-7B, RPO significantly outperforms all other methods on the PAIR attack, reducing the attack success rate from 68% (base model) to just 4%. This is a substantial improvement over the next best defense, Few-Shot, which achieves 16%. On the GCG attack, RPO matches the best performance of 0% attack success rate. On JBC, RPO is competitive with SmoothLLM (but outperforms it significantly on PAIR and GCG).

Similar to Llama-2-7B, Llama-2-13B already has high robustness to all three attacks. Using RPO similarly outperforms baseline defenses across attacks, achieving perfect robustness to GCG and JBC and reducing PAIR ASR. SmoothLLM is the only defense competitive to RPO in this setting, but it requires twice as many inference computations and is less effective on other models. We will add these results to the revised version and similar evaluations on Claude and Vicuna-7B.

I would suggest authors compared DPO with stronger defenses

We appreciate the reviewer's suggestion to compare RPO with additional strong defenses. We have conducted further experiments, comparing RPO with self-reminder and few-shot examples (RAIN is a weaker defense that has a very high GCG ASR on Vicuna (38%) [2], so we do not evaluate it). The results for GPT-4 are presented in the table below.

These new results demonstrate that RPO consistently outperforms or matches the performance of all compared defenses across different attack types:

PAIR Attack: RPO achieves the lowest attack success rate of 6%, significantly improving upon Self-reminder (16%) and Few-Shot (10%). GCG Attack: RPO matches the perfect defense (0% attack success rate) achieved by Perplexity Filter, Self-reminder, and Few-Shot, while outperforming SmoothLLM and Rephrasing. JBC Attack: RPO maintains the 0% attack success rate achieved by most defenses, matching the performance of baselines.

These results highlight that RPO not only compares favorably to the defenses in our original submission but also outperforms or matches the effectiveness of additional strong defenses suggested by the reviewer. We will incorporate these additional comparisons and analyses into the revised paper and add the results of new baseline defenses on the other models we evaluated.

We hope our response has addressed the reviewer's concerns. We are happy to answer further questions in the discussion phase.

[1] Chao, P., Debenedetti et al (2024). JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models. ArXiv, abs/2404.01318.

[2] Li, Y., Wei et al (2023). RAIN: Your Language Models Can Align Themselves without Finetuning. ArXiv, abs/2309.07124.

Dear Reviewer Md6V,

We are thankful for your review. With the rebuttal deadline drawing near, please inform us if your concerns have been properly addressed. We are ready to offer further clarification.

Best regards,

The Authors

Dear authors,

I am happy that you address most of my concern.

1 However, considering defending jailbreak attacks from prompt tuning view is not a full new direction. For example, related works [1] and [2] study the same problem and propose the corresponding methods. I list them here is not doubting the novelty of this paper. However, discussing with them might help readers understand the importance of the prompt-based defenses better.

2 Notice that you release the code. How many iterations do RPO need for its convergence?

[1] Fight Back Against Jailbreaking via Prompt Adversarial Tuning

[2] On prompt-driven safeguarding for large language models

We thank the reviewer for the response and are glad most concerns are addressed.

Discussing related work

We are glad the reviewer has brought these works to our attention and agree they should be discussed.

1. PAT [1] is concurrent to our work, and explores a similar optimization objective. However, PAT is only optimized on the GCG attack, which may limit its transferability to newer attacks.
2. DRO [2] also explores prompt-based defenses for LMs but optimizes a soft prompt in embedding space. This prevents DRO from transferring to closed-source models. In addition, the DRO objective is only a minimization objective, not a min-min optimization objective like RPO, which incorporates the adversary into optimization. This makes RPO more adaptive to jailbreaking attacks.

We will include baseline comparisons and a more detailed discussion in the revised version.

How many iterations do RPO need for its convergence?

In practice, we find that RPO requires around 100 iterations to converge on a single A100 GPU (around an hour), and we optimize our suffixes for 500 iterations, which takes 4-5 hours. In the revised version, we will conduct a more detailed analysis of performance over time.

We are happy to further clarify, and hope the reviewer will raise the score if all concerns have been addressed.