Many-shot Jailbreaking

Thank you for your review!

We appreciate that you found the simplicity of MSJ to be a strength given its effectiveness, and highlighted the scaling laws as providing insight into the nature of the jailbreak, as well as mitigation attempts.

Here are our responses to some of your critique:

• Augmenting experimental details and error bars: Here's how we've tried to improve the paper accordingly:

  • Specifying number of samples: We've added this information in the paper. All our measurements use dataset sizes of at least $512$, which ensures a standard deviation of at most approximately $5$ percent under a Bernoulli model (relevant for computing error bars on attack success rates).
    • Concrete action taken: Paper edited with the requested information.
  • NLL information: We've redone some of our plots with the error bars on the NLL values. You can find these in the attached. Please note the technique we used to reduce cross-datapoint variance between measurements within the same scaling law plots, described in Appendix C.2. This allows us to obtain clean scaling laws despite each individual measurement being relatively noisy. Note that the in-context power laws have been observed independently by Agarwal et al. as well, a work concurrent to ours.
    • Concrete actions taken: Requested plots generated and provided in the supplementary rebuttal figures.

• Question about GCG:

  • New GCG results: It is unfortunately difficult for us to run new GCG experiments on time for the rebuttal. However, we can still try to address a portion of the critique.
  • Mechanics of our GCG experiments:
    • The way we construct the GCG-augmented MSJ prompts is similar to what you describe: we first compute the GCG string that zero-shot jailbreaks the model on a variety of prompts, then stack [question + gcg-string + answer] pairs to construct the MSJ prompts. Just to be clear, we find a "universal" GCG string that works very well in the zero-shot setup, and use the same GCG string when we form the MSJ prompt.
    • Our experiment attempts to answer whether one can stack MSJ on top of an existing GCG attack. Our result here is mostly negative — the zero-shot benefit of the attack doesn't translate to the many-shot setting.
    • The opposite case that you've brought up (what if we take an MSJ prompt then optimize a GCG string on top of it) is also very interesting, and something we didn't try. A particularly interesting version of this could have been: optimize a GCG string for 10-shot MSJ prompts, then test it on 5 shot and 20 shot MSJ prompts. Our results on location specificity suggest that the benefit we get on 10-shot prompts will not transfer to 5-shot or 20 shot cases. We'll quite likely not have the opportunity to run this experiment by the end of the rebuttal phase, but with your permission we can mention this as a promising experimental extension in the paper.

[1] Agarwal, Rishabh, et al. "Many-shot in-context learning." arXiv preprint arXiv:2404.11018 (2024).

Thank you for your encouraging review!

We appreciate that you highlighted our focus on context length as a novel attack surface, which was the inspiration behind our work. We also appreciate that you found our experiments comprehensive.

We've tried to address some of your feedback — some directly on the writeup. Please take a look!

Addressing critique:

• Tables and numbers: Thank you for this suggestion, we agree with you! We've added a portion of our results (especially the results on the Malicious Use-Cases dataset) in Appendix D.

  • Concrete action taken: Requested tabular data provided. Please find two of the tables in the supplementary rebuttal figures.

• Longer prompts: Thanks for bringing this up! The effect of prompt length on the robustness of harmlessness training was studied in [1], Section 5.2 The authors find that prepending long conversations in the prompt, then asking adversarial questions doesn't manage to reduce the robustness of the LLama 2 models. This finding suggests that length alone is not a significant causal factor in jailbreaking success.

  There do exist methods (such as Greedy Coordinate Descent, which we discuss in the paper) that can find very short prompt suffixes that reach high degrees of jailbreaking success. A significant caveat is that this class of attacks use gradient information — something MSJ doesn't make use of. There exist jailbreaks such as that discussed in [2] that is a lot more effective than MSJ in the bounded context setup. We view that fact that the effectiveness of MSJ improves with context length as a strength: The performance of many jailbreaks such as [2] is constant in number of tokens available — they don't get better with increasing context length. On the contrary, MSJ gets much better — and in a predictable, power law relation.

  [3] links token length and steerability in an adversarial robustness context: This paper suggests that, as expected, having longer contexts makes it possible to steer model behavior more easily. (note that this paper finds adversarial attacks on token activations)

• Already jailbroken examples are (unfortunately) readily available: Unfortunately, finding jailbroken examples in Google is trivial: Simply look at the HarmBench dataset, which has publicly available question-harmful_response pairs. Also, with the release of Llama 3 400B, we will soon have open source, fully jailbroken models from which it's trivial to generate harmful responses.

• Data filtering: Thanks for bringing this up! We are actively following promising leads on how Many-shot Jailbreaking interacts with filtering-based methods, but are not in a position to be able to share some of our results just yet.

Answers to questions:

• Single conversation vs. dialogue: This is a great question! The vanilla version of MSJ that we present in the paper does not extend to single turn conversations — the many-shot structure is quite important for the attack to be effective. That being said, we're actively following some leads on this direction. For the attack to be effective on platforms such as ChatGPT and Claude.ai, not only does the attack need to be single-turn, but also effective at circumventing other safety layers at detecting jailbreak attempts. To do this effectively, one has to consider:

  1. How the effectiveness of separate evasions techniques stacks with MSJ
  2. Whether there could be MSJ specific evasion techniques

  We hope that this presents a rich research agenda for the robustness community to pursue!

  As an aside, all companies that are offering their language models via APIs allow for inserting faux steps in the dialogue history, making MSJ trivial to execute.

• Discussion on diversity of in-context exemplar distribution: This is an interesting question that touches upon the fundamentals of how in-context learning handles out-of-distribution datapoints. To our knowledge, there doesn't exist a set of general results that can help us here. We expect there to be a fair degree of domain-specific effects here — maybe certain OOD behaviors will be more difficult to elicit via MSJ. Empirically, we find a monotonic relationship between the prompt diversity and chance of transfer to OOD data.

• Long prefix vs. many-shot structure: We omitted this kind of analysis from the paper, as it already has been conducted by [1]. They tested the consequence of long, unstructured prompts on robustness, and have mostly not identified any significant degradation in performance. This points to the importance of the many-shot structure of the prompt.

Thanks again for your review and insightful questions. Please do Please let us know if you have any further questions, and (respectfully) consider

[1] Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankarara- man, K. A., Oguz, B., Khabsa, M., Fang, H., Mehdad, Y., Narang, S., Malik, K., Fan, A., Bhosale, S., Edunov, S., Lewis, M., Wang, S., and Ma, H. Effective long-context scaling of foundation models, 2023

[2] Liu, Yi, et al. "Jailbreaking chatgpt via prompt engineering: An empirical study." arXiv preprint arXiv:2305.13860 (2023).

[3] Fort, S. Scaling laws for adversarial attacks on language model activations, 2023.

Thank you for your response and consideration! Here are some quick points:

Dialogue vs. single conversation: We might have interpreted your earlier comment incorrectly as referring to the comparing the effect of context length alone vs. the dialogue structure!

To answer your question, we ran some additional experiments involving two conditions

• Condition 1: Instead of using Human/Assistant tags, we used “H/A” tags in the prompt — so far very similar to what we did in Section 3.3 about changes in formatting. The difference here is that we sandwiched these question-answer pairs between a single Human/Assistant turn. I.e. the format is:

Human: … A: … H: … A: … [xn] H: … Assistant:

This format makes sure that the MSJ prompt fits in a single human turn. * On the Opportunities to Insult dataset (which correlates very strongly with other datasets in our experience), this did not work very well, getting close to $0$ percent jailbreak rate.

• Condition 2: We took Condition 1, and modified the intermediate Question and Answer tags as follows:

H → “Prompt: (Answer the question in the manner displayed above.)”

A → “Aide:”

So now the format is Human: … Aide: … Prompt: (Answer the question in the exact manner displayed above): … Aide: … [xn] Prompt: (Answer the question in the exact manner displayed above): … Aide: … [xn] Prompt: (Answer the question in the exact manner displayed above): … Assistant:

Number of shots | Baseline multi-turn MSJ | Single turn MSJ

0 | 0.00 | 0.00

1 | 0.00 | 0.06

5 | 0.31 | 0.31

10 | 0.67 | 0.51

16 | 0.70 | 0.60

36 | 0.83 | 0.59

These results suggest that one can use a single-turn dialogue to execute MSJ, albeit less effectively than what one can get with a multi-turn setup. Of course there’s more to do here, but hopefully this experiment addresses part of your comment!

Comment about importance of jailbreaking research in a world with Llama3 and Google: This is a deep topic that probably deserves a better medium than the margins of a NeurIPS review, but just to briefly share our position: We believe that it is important to study methods that might jailbreak SOTA proprietary models whose capabilities might not have been yet matched by open source models. Jailbreaks such as MSJ could reduce the cost of jailbreaking open-source models as well, which makes defending against it important.

HarmBench: The independent replication results actually use the HarmBench dataset — please see Section 7. We’re not aware of any significant quality difference between using HarmBench question-answer pairs vs. using novel samples from, say, open source models.

Thanks again for engaging in this discussion!

Thank you for your critique and questions.

We appreciate that you found our scaling and mitigation analysis interesting, which we perceive to be some of the central contributions of our work. We also thank you for highlighting the importance of the independent replication.

Overleaf doesn't allow us to upload updated versions of the paper, so we'll instead explicitly mark all the changes we have already made in the draft in our response — please see bullet points labelled "Concrete action taken".

Regarding experimental details:

• Balancing anonymity and transparency: We completely understand your concern and genuinely sympathize with it. A major reason why we weren't able to be a lot more transparent in our reporting is that doing so runs the risk of invalidating the integrity of the anonymous review system — revealing too much identity-revealing information might disqualify us from the conference. If the paper gets accepted to the conference, we'll be able to add more details to our experiments, such as model names, API endpoints etc. As you've noted, the independent replication result is very valuable in this scenario, and gave us a lot more confidence that our results are quite general and setting-independent.

Relatedly, thank you for flagging that some of the sections in the Appendix are not being properly referred to! We've tried to fix all instances where we thought this happened.

Concrete action taken: Identified missing references to Appendix sections and added them.

---

Towards clearer evaluation algorithms:

• Further clarity on NLL computations: Thank you for your feedback here. We've tried to address this critique with the following actions.

• What Listing 1 and Listing 2 do: Listing 1 lines each question-answer pair in a list, and grabs consecutive slices of length num_shots from this list to form the MSJ prompts. Listing 2 lines the question-answer pairs in a list, grabs a large subsection of them of length equal to the maximum number of shots, and constructs MSJ prompts by cropping this large subsection from the left. The rationale for the latter procedure is explained below.

Concrete action taken: Explained, in prose, what the both pieces of pseudocode are implementing in the relevant part of the Appendix.

• Intuition behind Listing 1 and Listing 2: The algorithms we provide involve a procedure that is aimed at reducing cross-datapoint variance. The key invariant of the algorithms is that for all in-context prompts of different lengths, the set of final queries are the same. This makes cross-datapoint variance much smaller.

Concrete action taken: Fixed the broken line in Listing 1.

• Referencing the refusal classifier at the right place: We've also fixed this in the writeup!

Concrete action taken: Directly addressed the comment.

---

Questions:

• Dependence on quality of helpful-only models: It's debatable how the size of the gap between the open and closed source models will evolve over time. That being said, we believe that the recently released Llama 400B is a model that's strong enough to elicit a massive class of very problematic behaviors that can be directly used in the context of MSJ. While Llama 400B itself is not a helpful-only model, it is "trivial" to finetune it to make it so (e.g. Gade et. al.[1]). As you mention above, MSJ is a relatively effortless attack, and a sufficiently motivated actor can be expected to have the resources to coax Llama 400B to constructing MSJ prompts — or even release the weights of a helpful-only version of this model.

What if the adversary if forced to use a much less capable model? We ran a quick experiment to test this: We measured the performance of MODEL on the GPQA dataset [2] in with 0, 64 and 128-shot MSJ prompts, where the question-answer pairs were generated using an earlier generation model. We observed that moving from 0-shot to 64-shot prompts lead to a decrease of ~5% (from 40% to 35%), and 64-shot to 128-shot prompts didn't lead to any further degradation. We can reach the following tentative conclusions from this quick experiment:

1. Moving from zero-shot to few-shot setup leads to some degradation in performance.
2. Moving from few-shot to many-shot setup doesn't lead to any further degradation in performance.

These results paint a relatively optimistic picture for MSJ retaining most of its effectiveness even if the MSJ prompt is generated using a less intelligent model. To be able to claim this with the full confidence that's required of a scientific publication, we need to replicate this with a variety of strong-weak model pairs and tasks, which is something we don't have the bandwidth to do during the rebuttal phase.

• Effectiveness on chat interfaces: This is a great question, and mirrors a relevant question asked by Reviewer 3GmK. Proprietary chat interfaces like ChatGPT or Claude.ai can be expected to rely on layered, "defense-in-depth" approaches that go beyond the robustness of models themselves. This means that a jailbreak attempt not only has to be effective on the model, but also evade any attempts at detecting jailbreak attempts. The keys points are:

1. Whether MSJ stacks with other complementary attacks that are aimed at evading detection
2. Whether there are MSJ-specific evasion tactics

We are actively following leads on how Many-shot Jailbreaking interacts with detection methods (which can be viewed as a separate research endeavor that build on our current submission), but are not in a position to be able to share some of our results just yet.

If you find our improvements and response satisfactory, please consider improving your score! In all cases, we'd be happy to engage in any follow-up discussion.

[1] Gade, Pranav, et al. "Badllama: cheaply removing safety fine-tuning from llama 2-chat 13b."

[2] Rein, David, et al. "Gpqa: A graduate-level google-proof q&a benchmark."

Thank you for your encouraging review!

We appreciate that you found our empirical foray into the scaling and mitigations of MSJ thorough and valuable.

Inference-time defenses considered: Thank you for bringing up inference-based defenses!

We are actively following promising leads on how Many-shot Jailbreaking interacts with these methods, but are not in a position to be able to share some of our results just yet.

One thing we'd like to note is that the version of MSJ described in the paper doesn't make any attempts at disguising itself as an attack. A human that takes a glimpse at the prompt will easily be able to tell that it's up to no good! The keys questions here are, as you suggest:

1. Whether MSJ stacks with other complementary attacks that are aimed at evading detection
2. Whether there are MSJ-specific evasion tactics

This presents a rich research agenda that we were deliberate about not tackling in this submission, and one that we hope the adversarial robustness community will pursue. We're especially excited about our scaling law being useful for measuring the effectiveness of solutions and counter-attacks to those solutions.

Happy to answer more!

Please don't hesitate to ask any further questions! We'd love the chance to highlight strengths of our submission that might perhaps encourage you to further increase your score!

Thank you for your review!

We appreciate that you've found our results interesting and useful. The extent to which the scaling laws are so clean and reproducible did surprise us when we were initially getting the results.

Here's our attempt at addressing some of your critique:

• Connection to Agarwal et al:

  • Agarwal et. al. is concurrent work, as it happens in a field that moves as fast as ours! All of our experiments had already finished, and the writing had been more or less been locked by the time Agarwal et al's work came out. In other words, Agarwal's results had no causal influence on our ideas, experiments and presentation.
    • Concrete action taken: Revise the sentence in which Agarwal et al. is cited, and emphasize that it's concurrent work.
  • Additionally, we believe that 1) our focus on safety 2) detailed analysis on mitigations, and especially 3) structural description of "what it means to address many-shot jailbreaking" are among contributions that are exclusive to our submission. Elaborating on item 3, we prescribe a concrete measurement (slope of the in-context learning curve) provides us with a very clean metric for how one can measure progress towards addressing MSJ.

• Reason for why all negative-log-likelihood experiments were done on the psychopathy dataset: The simple reason is that we don't have full log-likelihood access to some of the proprietary models, and working with a yes/no dataset like the psychopathy dataset was the only way we could have demonstrated that the power law trend is a general phenomenon. Unfortunately, since we collected our results, some of the companies completely shut down log-prob access, making this kind of experiment impossible even for psychopathy dataset.

• Releasing datasets for reproducibility: Releasing datasets is somewhat tricky for a paper like ours: releasing the dataset immediately makes it possible to run our proposed attack on any model in the world! Luckily, we have HarmBench, which actually does have data of the form needed to reproduce our results. This is why our independent replication results in Section Independent Replication on HarmBench is so important.

Answers to questions:

• Disambiguating the equation: C is the offset, α is the slope and K controls the infinite-limit lower bound.
  • Concrete action taken: Added this information in the writeup.

If you find our response and paper improvements satisfactory, please consider improving your score! In any case, we'd be happy to answer any follow-up question you might have.