Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defenses

Thank you for your valuable review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q).

---

W1: No comparison to few/many-shots baselines.

According to the ICA paper [1*], even ICA (10-shots) has a lower ASR than our I-FSJ (2-shots) against Llama-2 on AdvBench. We summarize the results as below:

Furthermore, in $\\textrm{\\color{blue}Table A}$ of the Rebuttal PDF, we include new experiments on jailbreaking GPT-4 with our I-FSJ. We recap the results of ICA and I-FSJ against GPT-4 as below:

In $\\textrm{\\color{blue}Table B}$ of the Rebuttal PDF, we report the re-implemented ICA results against Llama-2 on AdvBench. To allow ICA to use more shots in the 4096 context window, we shorten demos to approximately 64 tokens for both ICA and I-FSJ. As seen, our I-FSJ (8-shots) achieves comparable ASRs to ICA (64-shots), resulting in $8\\times$ efficiency improvement.

---

W2: Missing amount of necessary queries.

$\\textrm{\\color{blue}Figure C}$ of the Rebuttal PDF shows the distribution of the average number of queries necessary to generate successful jailbreaks. On AdvBench, I-FSJ requires $\\sim$88 queries to achieve nearly 100% ASRs on Llama-2, whereas PAIR [2*] reports a $\\sim$33.8 queries but only attains a 10% ASR. GCG achieves a 54% ASR but requires $\\sim$256K queries [3*]. On HarmBench, I-FSJ similarly requires $\\sim$159 queries. In summary, I-FSJ is both highly query-efficient and effective.

---

W3: Some ablations are missing.

• Shorter or longer examples:

As shown in Figure 1, the demos used in this paper typically consist of 5 steps. We can vary the length of demos by changing the number of steps. For instance, we consider 1-step demos in contrast to the default 5-step setting. We still achieve 100% ASRs on AdvBench for Llama-2, indicating that the length of demos may have little influence on ASRs.

• Refuse examples:

Regarding concerns about model used to generate demos produces refusal demos and thus deteriorate I-FSJ's performance, we emphasize our experiment results with in-context defense (ICD). As shown in Figure 8, ICD essentially involves prepending some refusal demos to the jailbreaking prompt. As detailed in Table 2, under ICD, I-FSJ achieves near 100% ASRs on AdvBench for Llama-2, indicating that the potential harm of refusal demos in the pool should be minor.

• Low-quality examples:

To address concerns about low-quality demos, we highlight our experimental results with SmoothLLM. The SmoothLLM defense perturbs input prompts, significantly reducing demonstration quality, especially at high perturbation ratios. As shown in Table 2, despite corrupted demos, I-FSJ achieves ASRs higher than 85% on AdvBench for Llama-2, demonstrating its robustness to low-quality demos.

• How important it is that the special tokens are correct:

Based on your suggestion, we tried using [INST] instead of [/INST] on Llama-2-7B-Chat and also tested Qwen1.5B's special tokens in place of [/INST]. The results, displayed in $\\textrm{\\color{blue}Table C}$ of the Rebuttal PDF, demonstrate the ineffectiveness of both [INST] and Qwen1.5B's special tokens and importance of injecting the correct special tokens.

---

Minor1.

The keyword-based defenses reject inputs containing special tokens like [/INST]. Perturbation operations used in SmoothLLM can be applied to injected special tokens to bypass detection. We observed cases that even after all injected special tokens were perturbed, it is still possible to jailbreak Llama-2. Due to limited space for rebuttal, we will provide full results in the final revision.

---

Minor2.

Thanks for your suggestion. We will use CSV or JSON data formats and include a README in the code release.

---

Q1: Using the special token from one model to jailbreak another model?

Yes. For closed-source LLMs, the special tokens are mostly unknown. To address this issue, we propose constructing a pool of public special tokens from open-source LLMs, and then searching within this pool for high-performing special tokens on closed-source LLMs. As shown in $\\textrm{\\color{blue}Figure A}$ of the Rebuttal PDF, we experiment on GPT-4 and observe that several public special tokens (e.g., </text>, </SYS>, </INST>) outperform the by-default one (\n\n), indicating that there is some "transferability" with regard to special tokens.

In addition, as detailed in $\\textrm{\\color{blue}Table A}$ of the Rebuttal PDF, we show that our I-FSJ attack is effective on GPT-4, achieving $>90\\%$ rule-based and $>80\\%$ LLM-based ASRs with just 1-shot or 2-shot demos. We observe that both demo-level RS and the special token </text> (selected according to $\\textrm{\\color{blue}Figure A}$) can consistently improve ASRs against GPT-4.

---

Q2: Why four [/INST] between pseudo-messages?

In our pilot experiment, we observed that duplicating special tokens several times made the jailbreaking prompt more robust to SmoothLLM's perturbations. We applied this operation to special tokens for other models as well. To examine the impact of this operation, we compared using a single [/INST] token versus four. Using only one [/INST] resulted in a 62% ASR on AdvBench under SmoothLLM's defense (swap 20%), compared to 100% ASR with four [/INST] tokens. We will include a more detailed ablation study in a later revision.

---

References:
[1*] Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations
[2*] Many-Shot Jailbreaking
[3*] Jailbreaking Black Box Large Language Models in Twenty Queries

Thank you for your valuable review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q).

---

W1&Q2: Potential leakage due to using AdvBench to generate demonstration pool and the concern that only using 50 requests AdvBench for evluation is insufficient.

To prevent leakage or overfitting, we measure the cosine similarity between harmful behaviors and the target request using the sentence-transformers/all-MiniLM-L6-v2 model [1*,2*,3*]. We exclude demonstrations with a similarity score of 0.5 or higher to the target request. To further address concerns about AdvBench's limited scale, we conducted experiments on both AdvBench and HarmBench [4*].

As shown in $\\textrm{\\color{blue}Table C}$ of the Rebuttal PDF, I-FSJ maintains its effectiveness even after filtering similar harmful behaviors from the pool, demonstrating that its success is not due to replicating specific provided demonstrations. Furthermore, as detailed in $\\textrm{\\color{blue}Table A}$ of the Rebuttal PDF, we show that our I-FSJ attack (even after similarity filtering) is effective on GPT-4, achieving $>90\\%$ rule-based and $>80\\%$ LLM-based ASRs with just 1-shot or 2-shot demos.

We provide some case studies as following:

---

Q1: How often the replicate event occurs during testing?

Regarding to concerns about replicating the demonstrations, we measure the textual similarity between the generation and the in-context demonstrations on both AdvBench and HarmBench using the sentence-transformers/all-MiniLM-L6-v2 model [1*,2*,3*].

As shown in $\\textrm{\\color{blue}Figure B}$ of the Rebuttal PDF, most generations has a similarity below 0.5 with their in-context demonstrations, which show that our I-FSJ is indeed producing novel generations rather than simply replicating the demonstrations.

---

W2&Q3: How does the decoding length variation affect the results?

Following your suggestion, we set the decoding length to 512 [4*]. We conducted experiments on both AdvBench and HarmBench, and as shown in $\\textrm{\\color{blue}Table C}$ of the Rebuttal PDF, we found that I-FSJ maintains its effectiveness under this longer decoding length.

---

Minor point: How the Llama Guard implemented as a defense against adpative attack?

The use of Llama Guard for computing ASR [5*] and as a defense [6*] against adaptive attacks varies due to differences in templates. When Llama Guard is employed as a defense, we only consider the LLM's response to determine whether it is safe or unsafe, thus excluding the [GOAL] placeholder in the template. We will address this discrepancy in a later revision.

---

References:
[1*] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP 2019
[2*] https://github.com/UKPLab/sentence-transformers
[3*] https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2
[4*] HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
[5*] JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models
[6*] PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Rails

Thank you for your valuable review and suggestions. Below we respond to the comments in Weaknesses (W).

---

W1: Potential leakage due to using AdvBench to generate demonstration pool and the limited scale of AdvBench.

To prevent leakage or overfitting, we measure the cosine similarity between harmful behaviors and the target request using the sentence-transformers/all-MiniLM-L6-v2 model [1*,2*,3*]. We exclude demonstrations with a similarity score of 0.5 or higher to the target request. To further address concerns about AdvBench's limited scale, we conducted experiments on both AdvBench and HarmBench [4*].

As shown in $\\textrm{\\color{blue}Table C}$ of the Rebuttal PDF, I-FSJ maintains its effectiveness even after filtering similar harmful behaviors from the pool, demonstrating that its success is not due to replicating specific provided demonstrations.

We provide some case studies as following:

---

W2: Can special token be determined without the knowlege of target model.

The special tokens for open-source LLMs can be publicly accessed through their chat templates [5*, 6*]. In the case of closed-source LLMs, the special tokens are mostly unknown, despite attempts to extract them [7*].

To address this issue, we propose constructing a pool of public special tokens from open-source LLMs, and then searching within this pool for high-performing special tokens on closed-source LLMs. As shown in $\\textrm{\\color{blue}Figure A}$ of the Rebuttal PDF, we experiment on GPT-4 and observe that several public special tokens (e.g., </text>, </SYS>, </INST>) outperform the by-default one (\n\n). Furthermore, our findings indicate that there is some "transferability" with regard to special tokens, which could be an interesting research question.

Furthermore, as detailed in $\\textrm{\\color{blue}Table A}$ of the Rebuttal PDF, we show that our I-FSJ attack is effective on GPT-4, achieving $>90\\%$ rule-based and $>80\\%$ LLM-based ASRs with just 1-shot or 2-shot demos. We observe that both demo-level RS and the special token </text> (selected according to $\\textrm{\\color{blue}Figure A}$) can consistently improve ASRs against GPT-4.

---

W3: Adaptive defenses.

Yes, a red-teaming paper's goal is to propose a new attack that breaks down existing defenses and advocate for the design of adaptive defenses. Nearly every attack has its "adaptive defenses", but only after the attack is widely known in the community. Our I-FSJ can break almost all existing defense mechanisms, so we advocate for incorporating adaptive mechanisms into future defense design.

---

References:
[1*] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP 2019
[2*] https://github.com/UKPLab/sentence-transformers
[3*] https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2
[4*] HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
[5*] https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-2
[6*] https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3
[7*] https://twitter.com/krishnanrohit/status/1755122786014724125

Thank you for your valuable review and suggestions. Below we respond to the comments in Weaknesses (W).

---

W1: Should take ICA as the main target, rather than refining MSJ. The difference between the used baseline (FSJ) and ICA is not indicated.

According to the ICA paper [1*], even ICA (10-shots) has a lower ASR than our I-FSJ (2-shots) against Llama-2 on AdvBench. We summarize the results as below:

Furthermore, in $\\textrm{\\color{blue}Table A}$ of the Rebuttal PDF, we include new experiments on jailbreaking GPT-4 with our I-FSJ. We recap the results of ICA and I-FSJ against GPT-4 as below:

As seen, our I-FSJ significantly outperforms ICA. There are two main differences between ICA and the baseline FSJ:

• Each demo shot in ICA is longer. According to [1*], ICA (15-shots) exceeds Llama-2's 4096 context window, implying that each demo shot in ICA requires $>270$ tokens on average. In contrast, each demo in FSJ and our I-FSJ takes $<200$ tokens (could even be $64$ tokens as shown in $\\textrm{\\color{blue}Table B}$ of the Rebuttal PDF).
• ICA use multi-round chat templates. Each demo shot in ICA is a round of chat, and ICA assumes the ability to feed multiple rounds (multi-shots) into the LLM at the same time. In contrast, both FSJ and our I-FSJ combine multi-shot demos into a single chat, making them more suitable for jailbreaking close-source LLMs.

---

W2: The most important baseline, ICA, is missed in the experiments.

In addition to our response to W1, we attempt to re-implement ICA to provide a more complete comparison. However, since ICA does not open source its demo pool, we must implement it using the same demo pool as I-FSJ.

In $\\textrm{\\color{blue}Table B}$ of the Rebuttal PDF, we report the re-implemented ICA results against Llama-2 on AdvBench. To allow ICA to use more shots in the 4096 context window, we shorten demos to approximately 64 tokens for both ICA and I-FSJ. As seen, our I-FSJ (8-shots) achieves comparable ASRs to ICA (64-shots), resulting in $8\\times$ efficiency improvement.

---

W3: The first improved technique, injecting special tokens, though interesting, is of limited scientific contribution. More importantly, why these tokens can enhance the ASR is not well-explained or understood.

We believe that a paper that reveals previously unknown facts and makes the readers feel interested deserves credit for its contributions. We agree that "injecting special tokens" is an attack trick, but it also helps to achieve state-of-the-art ASRs against LLMs and advanced defenses, showing that these well-established alignment/defense mechanisms can be easily circumvented by a simple trick.

To explain why special tokens can enhance ASRs, we attribute the phenomenon to "Privilege Escalation", a common jailbreaking strategy in cybersecurity. Namely, special tokens such as [/INST] are used to wrap system and user messages, which indicate to the model that the wrapped messages have higher privilege (e.g., please refer to Figure 1 in [2*]). As a result, intentionally injecting special tokens may mislead the model into escalating the privilege of harmful demos and following them more closely.

---

W4: The second technique is anyway lacking novelty since the jailbreaking literature has already used the intention of random search (e.g., GCG and AutoDAN) to improve the jailbreaking prompt.

We completely understand your concern, but we never claim to be the first to use random search (RS), as it has already been a basic and widely used module in the jailbreaking literature. Instead, our novelty lies in proposing a better way to use RS (demo-level RS), resulting in significantly higher ASRs than, for example, GCG (token-level RS) and AutoDAN (sentence-level RS), as shown in Table 3. From our perspective, a good jailbreaking attack should be effective while remaining as simple as possible, allowing researchers to easily implement it and red-team their models.

---

References:
[1*] Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations
[2*] The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions

Dear Reviewer P1po,

Thank you for your valuable review and insightful suggestions. We have made significant efforts to write responses and conduct additional experiments based on your comments. Could you please let us know if our responses have alleviated your concerns? If there is any further feedback, we will do our best to respond.

Best,

The Authors

Dear Reviewer P1po,

Sorry for bothering you, but the discussion period will end in one day. All other reviewers have already returned detailed feedback on our rebuttal, so could you please take some time to let us know whether our responses have alleviated your concerns? Thank you!

Best,

The Authors

Dear authors,

Thank you for your response. I truly appreciate your efforts and time, especially for the experiment part. However, it's regrettable that my major concerns were not addressed, and some weren't even mentioned in your rebuttal.

• The main objective of this paper seems to be misleading. After reading the paper, especially the abstract:

Recently, Anil et al. [5] show that many-shot (up to hundreds of) demonstrations can jailbreak state-of-the-art LLMs by exploiting their long-context capability. Nevertheless, is it possible to use few-shot demonstrations to efficiently jailbreak LLMs within limited context sizes?

It gives me the feeling like the story that we already have a many-shot version attack, so in this work, we propose a few-shot one. However, given the publication of ICA as the few-shot version ICL-based attack, the story should be like, we already have a few-shot attack, so we want to propose an improved one. It is very clear that ICA is closer to this work than MSJ, but in the abstract, the authors escape ICA yet only mention MSJ, which, in my opinion, implicitly oversells the novelty and contribution of this work (I noticed the authors cited ICA in the main content, but the story in the abstract is not desirable). This concern was not even mentioned in the rebuttal.

• After reviewing the clarifications on the difference between ICA and FSJ in the rebuttal, it seems to me that the FSJ is essentially the same as ICA. The authors mentioned two differences, including the length of the prompt and the chat/prompt format, but these differences are not sufficient to distinguish these two methods. I question the validity of assigning a new name to the method simply because the prompt is shorter. Additionally, there is no evidence to support the claim that ICA can only be used in chat templates and cannot be applied to black-box models, as they reported ASRs on GPT-4. Therefore, I respectfully disagree with using a completely new name for this baseline method, as it not only disrespects the authors of ICA, but also may cause significant confusion, since a variety of works (including my own paper) already use the name ICA for background or baseline methods. In my opinion, it's perfectly acceptable to create new names for your own method (I-FSJ), but for existing methods, it is essential to align with common practices.

• There are other concerns on the technical part remain. For example, you claim I-FSJ (8-shots) achieves comparable ASRs to ICA (64-shots), resulting in $8\times$ efficiency improvement. But I-FSJ requires multiple queries and updates, while ICA only requires 1 single forward pass. Could you specify where the efficiency comes from?

Anyway, while my concerns mainly focus on the research practice aspects, I appreciate your efforts during the rebuttal. I strongly recommend the authors revise the manuscript based on the above comments for future versions.

Dear Reviewer P1po,

Thank you for your review and insightful feedback. We have responded to your concerns and promised to polish these parts in the final revision. If there are any unsolved concerns, we will do our best to clarify them before the discussion period ends.

If you are satisfied with our responses, would you like to raise your score? Thank you!

The Authors

Thank you for your valuable review and suggestions. Below we respond to the comments in Weaknesses (W).

---

W1: How does the attacker know the special tokens used in the LLMs, especially for closed-source models such as ChatGPT?

The special tokens for open-source LLMs can be publicly accessed through their chat templates [1*, 2*]. In the case of closed-source LLMs, the special tokens are mostly unknown, despite attempts to extract them [3*].

To address this issue, we propose constructing a pool of public special tokens from open-source LLMs, and then searching within this pool for high-performing special tokens on closed-source LLMs. As shown in $\\textrm{\\color{blue}Figure A}$ of the Rebuttal PDF, we experiment on GPT-4 and observe that several public special tokens (e.g., </text>, </SYS>, </INST>) outperform the by-default one (\n\n). Furthermore, our findings indicate that there is some "transferability" with regard to special tokens, which could be an interesting research question.

---

W2: Evaluating I-FSJ on closed-source models.

Based on your suggestions, we evaluate I-FSJ on GPT-4 with similar settings as in [4*]. We conduct our experiments using the OpenAI API ''gpt-4-1106-preview''. As detailed in $\\textrm{\\color{blue}Table A}$ of the Rebuttal PDF, we show that our I-FSJ attack is effective on GPT-4, achieving $>90\\%$ rule-based and $>80\\%$ LLM-based ASRs with just 1-shot or 2-shot demos. Furthermore, we observe that both demo-level RS and the special token </text> (selected according to $\\textrm{\\color{blue}Figure A}$) can consistently improve ASRs against GPT-4.

---

References:
[1*] https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-2
[2*] https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3
[3*] https://twitter.com/krishnanrohit/status/1755122786014724125
[4*] Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks