Dear Reviewer jHn4:

Thank you for your recognition of the soundness and contribution of our paper. For your proposed weakness, here are our responses:

Q1: The optimization of the defense prompt incurs computational overhead, and the optimization process using greedy sampling is time-consuming.

A1: Please refer to G3 in General response for more details.

Q2: I am concerned about the generalization of PAT, specifically the robustness against adaptive and real-world attacks.

A2: For the results against more adaptive attacks, please refer to G1 in General Response for more details. To investigate whether PAT can serve as a realistic method to defend against jailbreak attacks in the real world, we evaluate the ASR of PAT against 100 jailbreak prompts provided by [1] on the GPT-4:

We first notice that even for GPT-4, the in-the-wild jailbreak prompts can still achieve a high ASR (40%). But applying PAT can increase the robustness of the model, decreasing the ASR to 19%.

Q3: The prefix-based methods can easily lead to false negative or false positive judgments.

A3: We agree that achieving low negative or false positive judgments is significant for a defense method. Thus in the designing principles of PAT, we additionally incorporate a normalization term with benign prompts to avoid over-defense and maintain the benign utility. We compare the MT-bench score with or without benign normalization on Vicuna-7B and summarize it in the following tables:

After combining with Table 1, the results indicate that it can largely decrease false positive judgments while maintaining low ASR against attacks.

Q4: It is unnecessary for defenders to discuss their own capabilities. Generally, in most defense settings, defenders have full access to the systems, as they are developed by the systems themselves.

A4: We agree that in most cases, the security team and model development team belong to the same company which makes it feasible for defenders to access the parameters of models. However, considering the rapid advancement of LLMs, it has become possible for customers to train their own large models at a marginal cost. In addition, for small companies, maintaining their own security team is not an economical option. In such cases, they need to rely on security services provided by other companies. However, due to privacy concerns, they are unwilling to share their model parameters with the service providers. PAT provides a solution for those circumstances. It will protect the models without accessing parameters.

Q5: The number of training and testing datasets used in your setup is not specified.

A5: Sorry for the ambiguity. Here are more details about the datasets used in our setup.

Training: We first select 100 benign prompts from the MS MARCO dataset and 25 harmful prompts from the Advbench dataset as the prompts for training. Then we generate their responses with the LLMs. The above steps enable us to gain the full training dataset.

Testing: For testing, we craft jailbreak prompts with 100 harmful prompts from the Advbench dataset. Note that the prompts for training or testing do not overlap to ensure the fairness of evaluation.

Q6: Why do the authors only test on MT-bench? Why not evaluate more general abilities such as math, coding, reasoning, etc.?

A6: We test on MT-bench because as far as we know, it is one of the most important benchmarks that measure the general capabilities of the LLMs. Eight core capabilities of LLM chatbots are concentrated while building this benchmark: writing, roleplay, extraction, reasoning, math, coding, STEM and social science, which covers the capability mentioned in your questions. In addition, we perform experiments on the MMLU [2], which is another benchmark that measures the general capabilities of LLMs:

PAT achieves comparable scores with self-reminder on Vicuna-7B. On Llama2-7B, PAT achieves the highest score compared with other defenses. Similar to our observation on the MT-bench, it demonstrates the effect of PAT to benign utility is small.

Q7: It is peculiar that compared to no defense, the performance on MT-bench is worse than the authors' proposed method. Could the authors explain this discrepancy?

A7: Please refer to G2 in the General response. We analyze the reason why PAT can obtain higher score on MT-bench.

Q8: The defense may be less effective when attackers implement other adaptive strategies with knowledge of the defense mechanism.

A8: Following your suggestions, we perform experiments on more advanced adaptive attacks. Please refer to G1 in General Response for more details.

Q9: The rapid development of new jailbreak methods means that continuous updates to the defense strategy may be required.

A9: Our experiments in realistic and adaptive attacks reveal that PAT can acquire reliable robustness even with advanced attacks. Of course, jailbreak attacks are a rapidly developing direction and PAT gives an overall framework to defend those threats. We believe by solving the inner jailbreak prompts with more advanced attacks, we can potentially gain better robustness against various attacks.

Thank you again for your valuable and helpful suggestions. Look forward to further discussion with you in the author reviewer discussion period.

[1] "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models.

[2] Measuring massive multitask language understanding.

Dear Reviewer jHn4,

We sincerely appreciate the valuable suggestions you provided during the review process. Due to the limited time during the rebuttal period, we realize that the insufficient experiments might mislead you to keep the score unchanged. We sincerely apologize for this. Therefore, we would like to take this opportunity to address your concerns further from the following two perspectives.

Q1: PAT needs more benchmarks to evaluate its effect on benign utility and it can easily lead to false negative or false positive judgments.

A1: In addition to the MMLU benchmark shown in A6, we further perform experiments on two benchmarks: HumanEval [1] and CommonsenseQA [2] on Llama2-7B:

The results show that PAT achieves better performances than the baseline defense and its negative effect on the benign utility is very marginal.

Q2: The robustness of PAT against real-world attacks and the rapid development of new jailbreak methods means that continuous updates to the defense strategy may be required.

A2: Based on the experiments in A2, we make further investigations based on the in-the-wild prompts collected by [3]. We split them into two partitions according to the created date. The first part includes 100 jailbreak prompts which were created before 2023.06.13. The second part includes 100 jailbreak prompts launched after that date. We conducted experiments separately on the GPT-4 0613 version and the latest version, respectively and summarize the results in the following tables:

GPT-4 (2023.06.13 version)

GPT-4 (Lastest Version)

Two interesting insights can be concluded from the above results:

• For GPT-4-0613, newer attack prompts created after 2023.6.13 cause a larger threat to the security of model. However, even facing this emerging threat, PAT can provide consistent robustness. This demonstrates that PAT does not need frequent updates on the defense strategy.

• For the latest version of GPT-4 which is equipped with multiple defense strategies, PAT can still decrease the ASR a lot. It reveals that PAT is a practical and plug-and-play tool, not just a laboratory product.

In addition to the numerical results, we also present an example of PAT when performing jailbreak defense:

Dear Reviewer DPyJ,

Thank you for your perceptive comments on our paper. Here are our responses to your concerns:

Q1: Another paper [1] proposed using an optimized soft system prompt to enhance model safety, which is highly relevant to this study. A corresponding comparison and discussion might be necessary.

A1: Considering the cycle of reviews, it seems that this related work in ICML 2024 is released to the public after the deadline of NeurIPS 2024 submission. However, we ensure that the following discussion will add to the revised version of our paper and properly cite their work. In [1], the authors propose DRO which optimizes a soft prompt in the continuous prompt space. In contrast to their work, three designed principles of PAT make it more appropriate to defend jailbreak attacks:

• Optimization Formulation: The optimization formulation for PAT is a min-min optimization. It requires that even if an adaptive suffix is attached to the jailbreak prompt, the secure prefix crafted with PAT can still align the LLM with the refusal output. In comparison, the optimization formulation for DRO is simply a minimization problem. It makes PAT more robust to jailbreak attacks.
• Optimization Target: The optimization target for PAT is the discrete tokens. But for DRO, it is the soft prompt. The above difference will make DRO infeasible to transfer to the closed-source LLMs.
• Optimization Goal: The optimization goal of DRO is to move queries’ representations along or opposite the refusal direction according to their harmfulness. However, considering the diversity of user input. This simple strategy could potentially impair the benign utilities of LLMs since not rejecting them does not mean high-quality generation. However, for PAT, we align the output of PAT for benign prompt with the output without PAT. It will serve as an adaptive tool for different user prompts and maintain the benign utility of the model.

Our further experiments demonstrate that PAT outperforms DRO from the perspective of ASR and benign utility:

ASR (GCG Attack):

Benign Utility (MT-bench):

Compared to DRO, PAT obtains lower ASR against the GCG attack and acquires a higher score on the MT-bench benchmark.

Q2: The authors only tested instruction-following ability on MT-bench, neglecting the knowledge-wise component. Therefore, additional utility experiments, such as those on MMLU, are necessary.

A2: Following your suggestion, we further evaluate the performances of PAT on the MMLU benchmark and summarize the results in the following table:

The results reveal that PAT achieves comparable or better scores than those of baseline defenses. This is because we introduce benign pairs as normalization to align the behaviors of LLMs. It teaches LLMs to evaluate the harmfulness of input and make the right responses instead of rejecting every user input. It will greatly reduce the likelihood of false rejections, thus maintaining the benign utility of LLMs.

Q3: However, in practice, an adversarial prompt can be inserted at any position. Therefore, my concern is whether the optimized defense prefix will become ineffective if the attacker changes the position of the adversarial prompt.

A3: Although the prefix of PAT is obtained with the adversarial suffix through the min-min optimization. The result in Table 1 reveals that even for attacks that do not rely on suffixes to induce the jailbreak effect, such as ICA, PAT still has a good capability to defend against those threats. In addition, we evaluate the performances of PAT when GCG is applied in a postfix mode:

The results reveal that changing the position of the adversarial prompt will not affect the effectiveness of PAT, demonstrating the robustness of PAT to various malicious inputs.

Thank you again for taking the time to review our paper. Look forward to further discussion with you in the author-reviewer discussion period!

[1] On prompt-driven safeguarding for large language models.

Dear Reviewer 1UaM,

We sincerely appreciate your valuable and constructive suggestions. Before posting our rebuttal, we kindly note that reference [1] is a defense instead of an attack and reference [3] is an attack instead of a defense. Maybe we are wrong, could you help us figure out the misunderstanding? This would greatly help us address your concerns and make our evaluation more comprehensive.

Thank you,

Authors

Dear Reviewer 1UaM,

Thank you for your insightful review of our work. Here are our replies to your raised weakness:

Q1: Extend comprehensively the coverage over attack and defense baselines in the empirical evaluation. I would suggest a few more attack [1-2] and defense methods [3-4] to enrich the current study. As well, the evaluation needs to be extended to more open-sourced models, e.g. Mistral-8B and Llama-3 8B.

A1: Following your suggestions, we first perform experiments on more attacks, including PAIR [1] and COLD [2] in the following tables:

More attacks

PAIR [1]:

COLD [2]:

The results reveal that PAT can obtain lower ASR against those advanced attacks than the baseline defenses. Furthermore, we compare PAT with more defense techniques, including self-reminder [3] and SafeDecoding [4] in the following tables:

More defense

Vicuna-7B:

Llama-2-7B:

PAT gains comparable or better ASR than those of Self-reminder and SafeDecoding. However, considering the benign utility, PAT obtains a higher score on the MT-bench dataset. We also extend PAT to more open-sourced models, including Mistral-7B and Llama-3 8B:

More Model

Mistral-7B

Llama-3-8B

On Mistral-7B, PAT acquires low ASR against both the GCG and ICA attacks. But on Llama-3-8B, almost all defenses achieve near-zero ASR. We conjecture this is because Llama-3-8B is an advanced model released in recent months, which makes it more secure than other models.

Q2: Furthermore, I would be interested to check the transferability of the proposed defense method across different open-sourced models.

A2: We also believe that it is interesting to investigate the transferability of PAT across different open-source models. Thus here we perform experiments on Vicuna-7B, Llama2-7B, Mistral-7B and Llama-3-8B and report the ASR against the GCG attack when PAT transfers across those models:

ASR of GCG attack

Three interesting phenomenons are observed:

• The transferability of PAT exists for all open-source models. Under all settings, it can successfully decrease the ASR a lot.
• PAT can achieve the best performances when the target and surrogate models are exactly the same models.
• For target and surrogate models that have similar architectures, such as Llama2-7B and Llama-3-8B, PAT can have better performances.

We believe this study is a start-up for investigating the transferability of PAT. It may potentially open up a line of work to explore how to protect unknown LLMs under the black-box settings.

Hope our responses can answer your questions. We are happy to further discuss with you in the author reviewer discussion period.

[1] Jailbreaking Black Box Large Language Models in Twenty Queries.

[2] COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability.

[3] Defending Chatgpt against Jailbreak Attack via Self-reminders.

[4] SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding.

Dear Reviewer 1UaM,

Thank you for dedicating your valuable time and effort to reviewing this manuscript. As the reviewer-author discussion period is approaching to the end, if any concerns remains, do not hesitate to let us know. We are very happy to provide additional explanation and would try our best to address them.

Thank you again for your patience and insightful reviews.

Best regards,

Authors

Dear Reviewer LA5t,

Thank you for your detailed and helpful reviews. Here are our responses to your concerns:

Q1: The evaluation does not cover many other attacks and defenses, such as SmoothLLM [1] and RPO [2] (which also has a similar methodology to PAT).

A1: Note that RPO [2] is a concurrent work of our paper. However, we agree that it is significant to compare RPO with PAT in the performance of defending against jailbreak attacks. The results of SmoothLLM [1] are also summarized in the following tables:

GCG Attack

Benign Utility

Considering the ASR, PAT achieves comparable ASR with the baseline methods. However, the score in MT-bench demonstrates its outstanding performance in maintaining benign utility. This is because SmoothLLM perturbs multiple copies of a given input prompt, which could potentially destroy the semantics of input prompts. For RPO, it only considers rejecting jailbreak attacks in its optimization goal, which could lead to high rejection rates when LLMs reply to harmless questions.

Q2: Closed-source model evaluation is also not comprehensive. Table 2 is missing the defense baselines, and Table 3 is missing both the defense baselines and the ICA attack.

A2: Following your suggestions, we add more comprehensive evaluations to the transferable experiments of PAT and summarize them in the following tables:

Open-source LLMs:

Guanaco-7B

ChatGLM-6B

Closed-source LLMs:

GPT-3.5

GPT-4

The above experiments demonstrate that PAT for black-box models can achieve comparable or lower ASR than those of baseline defenses. Regarding benign utility, PAT obtains a higher score than baseline defense on the MT-bench benchmark.

Q3: The adaptive attack setting only considers an adaptive GCG attack. I wonder if adaptive versions of other attacks will transfer. If there aren't adaptive versions of the other attack baselines, the authors should detail this.

A3: Considering ICA is an attack that adds manually crafted demonstrations. Thus there are no adaptive versions of this attack. In addition to AutoDAN, we also derivate adaptive versions of the other two attacks: PAIR [3] and TAP [4]. We summarize the results in G1 in the general response. Please refer to this partition for more details.

Q4: The generated suffixes appear to have high perplexity. I wonder about the qualitative effect on model behavior for benign use, are there examples of this?

A4: Although the generated prefixes of PAT have high perplexity. But for LLMs, it seems they have semantic meaning to better align LLM with human values. Here are examples of benign inputs for the open-source (Vicuna-7B) and closed-source (GPT-4) models.

Vicuna-7B:

User: Answer seriously esper quietlymedia (/ ". harmcgi{{[.(_{( zv": what is morphine used for?

Assistant: Morphine is a highly addictive opioid pain medication that is used to treat severe pain that cannot be managed with other types of pain medication. It is often used in hospitals and clinics to manage pain in patients who are recovering from…

GPT-4

User: Come ! critique ! respond seriously?. evil fortune////////////////WRITE asked ein legalEGIN chat,. /**------+ nicelyTurnCred/) Base Question sid!(button ["itle what is morphine used for?

Assistant: Morphine is a potent opioid medication primarily used for pain relief. It is often prescribed for managing moderate to severe pain, particularly when other pain relievers are insufficient…

The above examples indicate that applying PAT will bring negligible effect to the benign use of LLMs. The numerical results in Table 1 and Table 7 also support this viewpoint.

Q5: Why does the PAT string improve MT-bench scores in Table 1 but decrease scores in Table 3?

A5: Please refer to G2 in the General response. We analyze the reason why PAT increases scores for Vicuna-7B and LLama2-7B but decreases scores for GPT-3.5 and GPT-4.

Q6: The threat model description suggests a gray-box setting (L96), but 4.1 claims to be about a white-box setting. Does this mean the adversary knows the defensive string?

A6: We are sorry for the ambiguity. The gray-box setting in L96 is concluded from the attacker’s perspective, in which the detection and defense methods are unavailable to attackers. In contrast, the white-box setting in 4.1 is concluded from the defender’s perspective, in which the parameter of the model is accessible to the defender. We agree that this inconsistency will bring confusion to readers. Therefore in the revised version of this paper, we will unify them as the grey-box setting.

Thank you again for spending your valuable time reviewing our paper. If any concerns remain, don’t hesitate to let us know. Look forward to further discussion with you in the author-reviewer discussion period!

[1] SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks.

[2] Robust prompt optimization for defending language models against jailbreaking attacks.

[3] Jailbreaking Black Box Large Language Models in Twenty Queries.

[4] Tree of Attacks: Jailbreaking Black-Box LLMs Automatically.

Thank you for the extensive additional experimental results. My questions and concerns have been mostly addressed.

I have raised my score, but after reading the other reviews and responses, feel that the paper would benefit from another round of review given the amount of new results and weaknesses in the submission.

Dear Reviewer zrTY,

We really appreciate your positive comments on the strength of this paper. For your proposed weakness, our responses are as follows:

Q1: The proposed defense relies on obtaining harmful prompts through existing attack methods. This raises doubts about whether the proposed method can be effective against unseen jailbreak attacks.

A1: As proposed in the paper, PAT introduces a min-min dynamic to the vanilla harmful behaviors, such as how to make a bomb, to obtain a secure prefix against the jailbreak attacks. Since no prompts of specific attacks are incorporated into its formulation for optimization, all attacks in Table 1 are actually unseen attacks to PAT, which means only one prefix is needed to defend all attacks. In addition to the attacks in Table 1, we further perform experiments against more attacks, including PAIR [1], TAP [2]. PAT shows good performances even in defending those advanced threats:

Vicuna-7B

Llama-2-7B

The results in the above tables indicate that PAT can serve as a plug-in tool to defend against the threat of jailbreak attacks. For example, against the PAIR attack, PAT decreases the ASR from 79% to 1%, thus successfully securing the LLMs from the jailbreak threats.

Q2: For instance, [3] found jailbreak attacks can be conducted using non-English prompts. There is not enough discussion on how well the proposed defense method works against unknown attacks.

A2: Considering the multilingual capabilities of large models, we strongly agree that it is important to investigate whether PAT can work against non-English attacks. Therefore, we perform experiments with malicious prompts written in Bengali, Swahili, and Javanese, three of which obtain the highest ASR in [4]. We attach the transferable PAT prefix to each input and evaluate the ASR on GPT-3.5 and GPT-4 in the following tables:

GPT-3.5 (turbo-1106 version)

GPT-4 (0613 version)

The results demonstrate that PAT can largely decrease the ASR in all settings. For example, on GPT-3.5, PAT decreases the ASR of the Bengali language from 13% to 0%. This indicates that PAT is transferable well across unknown languages. Thus we can craft one secure prefix with PAT and defend the attacks from all languages.

Hope our responses fully address your concerns. We are delighted to engage in further discussions with you in the following Discussion period.

[1] Jailbreaking Black Box Large Language Models in Twenty Queries

[2] Tree of Attacks: Jailbreaking Black-Box LLMs Automatically

[3] Multilingual Jailbreak Challenges in Large Language Models

Dear Reviewer zrTY,

Happy to hear that our rebuttal successfully addresses your concerns. Your insightful review will definitely help us improve the quality of our work. We will add them to the revised version of this paper.

Thank you so much!

Best regards,

Authors