PANDAS: Improving Many-shot Jailbreaking via Positive Affirmation, Negative Demonstration, and Adaptive Sampling

We thank the reviewer for the positive feedback and detailed response. We are glad that the reviewer finds our claims well-substantiated and our experimental design sound and valid. In the revised paper, we will include a discussion on the manual inspection of model responses during evaluation, present additional experiments on the composition of defenses, and expand the discussion on model selection and the challenges of evaluating proprietary models. Below, we address specific concerns of the reviewer.

Manual inspection of model responses: We indeed verified responses from all models. We will clarify this in the revised version of the paper.

Compositions of multiple defenses: Since some defense methods have limited or no effect, and some even increase the ASR, we focus specifically on compositions that generally have a positive effect on reducing ASR. For MSJ, we combine SmoothLLM and Self-Reminder; at 64 shots, MSJ has an ASR-L of 36% and an ASR-R of 40%. For PANDAS, combining Self-Reminder and ICD-Exact results in ASR-L and ASR-R values of 70% and 82%, respectively. We observe that combining defenses can further decrease the effectiveness of jailbreaks.

Practical reasons to use PANDAS over MSJ: The modifications introduced by PANDAS, particularly PA and ND, are straightforward to implement and can be used as direct plug-ins to MSJ with minimal overhead. We will highlight this in the revised version of the paper.

On scaling laws: In our paper, we focused on using attention analysis to understand how PANDAS improves upon MSJ and did not explore scaling laws. However, we agree with the reviewer that investigating how PANDAS interacts with scaling behavior is an interesting direction for future work. Additionally, the scaling analysis in Anil et al. [1] was based on Claude 2, which was not specifically fine-tuned for robustness in long-context scenarios. As more recent models are likely trained with such robustness in mind, examining their scaling behavior would be particularly valuable.

Evaluations on reasoning models: Running MSJ and PANDAS is memory-intensive, and the large size of most reasoning models makes them even more challenging to evaluate. While preparing the original submission, we tested DeepSeek-R1-Distill-Llama-8B [2], a small reasoning model fine-tuned based on pre-trained Llama-3.1-8B, but found it to be easily jailbroken, and therefore omitted it from the paper.

Evaluations on proprietary models: Our primary goal is to demonstrate that PANDAS achieves improved jailbreaking effectiveness compared to MSJ. As we have limited access to credit for commercial models, we decided to focus our evaluation on the latest open-source models, all of which were released between May and December 2024, with most incorporating safety guardrails. Notably, Llama-3.1-8B is specifically fine-tuned for robustness in long-context scenarios. Our results indicate that PANDAS substantially improves jailbreaking performance on these open-source models.

To additionally validate our findings, we conduct evaluations on a commercial model, GPT-4o-mini, which has the lowest per-token cost among available commercial options (Claude’s and GPT’s). Using 128 shots and only a single restart on AdvBench, MSJ and PANDAS achieve an ASR of 0.19% (1 out of 520) and 2.12% (11 out of 520), respectively. For cost reasons, we do not perform Bayesian Optimization and adopt uniform sampling across all malicious topics. This result shows that, despite both methods showing low ASR, the effectiveness of MSJ can still be improved.

[1] Anil et al., Many-shot jailbreaking. In NeurIPS’24

[2] DeepSeek-A, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

[3] Zheng et al., Improved few-shot jailbreaking can circumvent aligned language models and their defenses. In NeurIPS’24

We thank the reviewer for the positive feedback and detailed response. We are glad that the reviewer finds our claims well-supported by empirical evidence and our experiment designs sound and valid. We will revise the paper by clarifying the quality of the malicious demonstrations, improving the overall presentations in Sec. 4.4., and expanding the discussion on model selection and the challenges of evaluating proprietary models. Below, we address specific concerns of the reviewer.

Quality of the malicious demo: For all malicious demonstrations, we use the Llama-Guard-3 and ensure all queries are evaluated as unsafe. Manual inspections are also performed.

Error in Equation 6: Thank you for pointing out the error and it indeed should be $R_i = 1-S_{i,i} = \sum_{j=1}^{i-1} S_{i,j}$. Also, the definition of $S_{i,j}$ can be found in Equation 5 at bottom of page 7.

Clarification on the n-shot MSJ breakdown: We include the target prompt as a part of the definition. That is, $N_{n+1}$ represents the beginning of the target prompts and $N_{n+2}$ marks the end of the entire prompt.

LLM-as-a-judge: Using LLM as a judge as a way to evaluate the effectiveness of jailbreak is a popular approach adopted in many recent works. In addition to using LLM alone, we also manually verify responses from all models.

Comparison with other methods: In this paper, we focus on methods in long-context scenarios. To the best of our knowledge, the only relevant prior work is i-FSJ (improved few-shot jailbreaking), which we extend to the many-shot setting.

Model Selection: Mazeika et al. empirically demonstrate that jailbreak effectiveness is consistent within model families despite differences in parameter count, with the main variation occurring across different model architectures. Based on this, we focus on models with around 8B parameters. This choice also allows us to evaluate with a large number of shots, which is important given the GPU memory demands of MSJ and PANDAS. Furthermore, our experimental setup follows recent work on few-shot jailbreaking.

Evaluations on proprietary models: Our primary goal is to demonstrate that PANDAS achieves improved jailbreaking effectiveness compared to MSJ. As we have limited access to credit for commercial models, we decided to focus our evaluation on the latest open-source models, all of which were released between May and December 2024, with most incorporating safety guardrails. Notably, Llama-3.1-8B is specifically fine-tuned for robustness in long-context scenarios. Our results indicate that PANDAS substantially improves jailbreaking performance on these open-source models.

To additionally validate our findings, we conduct evaluations on a commercial model, GPT-4o-mini, which has the lowest per-token cost among available commercial options (Claude’s and GPT’s). Using 128 shots and only a single restart on AdvBench, MSJ and PANDAS achieve an ASR of 0.19% (1 out of 520) and 2.12% (11 out of 520), respectively. For cost reasons, we do not perform Bayesian Optimization and adopt uniform sampling across all malicious topics. This result shows that, despite both methods showing low ASR, the effectiveness of MSJ can still be improved.

Cost performance tradeoff: When comparing long-context jailbreaking methods to others, we observe a trade-off between inference-time and compute-time cost. For example, popular jailbreak methods like GCG append a short suffix to the target prompt, introducing minimal overhead during inference compared to MSJ and PANDAS. However, GCG typically requires multiple model queries at inference time to optimize the suffix. In contrast, MSJ and PANDAS only require generating malicious demonstrations once, without repeated queries to the target model. These demonstrations can then be repeatedly sampled and reused to construct new jailbreak prompts, potentially reducing inference cost over time.

Mitigation of PANDAS: We focus on improving the original MSJ approach and use attention analysis to understand the effectiveness of PANDAS. In Table 3, we demonstrate PANDAS’ performance on models equipped with various defense methods. The results show that Self-Reminder and ICD-Exact are more effective in reducing ASR compared to other defenses. As requested by reviewer dnBC, we additionally evaluated the composition of defenses: combining Self-Reminder and ICD-Exact results in ASR-L and ASR-R values of 70% and 82%, respectively. We observe that combining defenses can further reduce the effectiveness of jailbreaks.

Clarification of Figure 4: As discussed below Figure 4, a refusal phrase is inserted after the first question, making the first exchange between the human and assistant a negative demonstration. In Figure 4, all other plots start at index 1, whereas the ND plot starts at index 0. This was an intentional choice to improve clarity by highlighting the presence of the ND example at the beginning of the sequence.

We thank the reviewer for the detailed feedback. We are glad that the reviewer finds our claims well-supported, our choice of datasets and baselines reasonable, and our experiment designs sound and valid. We will revise the paper by including additional attention analysis on other models, highlighting other important scientific contributions. Below, we address specific concerns of the reviewer.

Model Selection: Mazeika et al. empirically demonstrate that jailbreak effectiveness is consistent within model families despite differences in parameter count, with the main variation occurring across different model architectures [1]. Based on this, we focus on models with around 8B parameters. This choice also allows us to evaluate with a large number of shots, which is important given the GPU memory demands of MSJ and PANDAS. Furthermore, our experimental setup follows recent work by Zheng et al. [2].

Attention Analysis: For the attention analysis in Section 4.4, we focus on Llama-3.1-8B due to its popularity. However, the observation in Figure 4 holds across other models as well, and we will include these additional results in the revised version.

Scientific Significance: We thank the reviewer for acknowledging the empirical improvements of PANDAS over MSJ. The proposed modifications, particularly PA and ND, are not only motivated by existing jailbreaking literature, but also serve as supporting evidence for those underlying hypotheses. PA is inspired by the competing objective hypothesis introduced by Wei et al. [3], while ND draws from the idea of ‘learning from mistakes’ in in-context learning [4], as Anil et al. argue that MSJ’s success results from ICL mechanisms [5].

In addition, we would like to highlight other key contributions of our work. First, we investigate long-context vulnerabilities in models equipped with various defense strategies, showing that while some defenses are highly effective in general, they can be circumvented by MSJ and PANDAS. Second, our partition-based attention analysis shows a trend which new demonstrations increasingly attends to previous ones, and PANDAS further encourages this pattern to achieve higher jailbreak effectiveness. We will highlight these contributions more explicitly in the revised paper.

[1] Mazeika et al., Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. In ICML’24

[2] Zheng et al., Improved few-shot jailbreaking can circumvent aligned language models and their defenses. In NeurIPS’24

[3] Wei et al., Jailbroken: How does llm safety training fail? In NeurIPS’23

[4] Zhang et al., In-context principle learning from mistakes. In ICML’24

[5] Anil et al., Many-shot jailbreaking. In NeurIPS’24

We thank the reviewer for the detailed feedback. We are glad that the reviewer finds our method novel and our experiments are extensive. We will revise the paper by clarifying the improvements introduced by PANDAS, expanding the discussion on model selection and the challenges of evaluating proprietary models, and providing additional details on the attention analysis. Below, we address specific concerns of the reviewer.

Improvements in the long-conext setting (Claim 1): To clarify, our claim is not that PANDAS only works in long-context scenarios, but rather that it is designed to improve in long-context settings, while also offering improvements in shorter contexts. We also observe that improvements in few-shot does not always translate to improvement in the many-shot setting. For example, on i-MSJ, a method originally designed to improve in few-shot jailbreaking, for models such as Qwen and GLM, we observe that the ASR-L falls behind MSJ and we do not observe the same improvement with a large number of shots. On the other hand, we highlight that for all cases, PANDAS is consistently equal or more effective compared to baselines across all shots.

Model Selection (Claim 2): The main reason for focusing solely on 8B open-source models is that they allow us to perform inference with a large number of tokens given limited GPU resources. This choice is supported by the empirical findings of Mazeika et al. [1] and follows a similar model selection strategy as in Zheng et al. [2].

Evaluations on Commercial Models (Claim 2): Our primary goal is to demonstrate that PANDAS achieves improved jailbreaking effectiveness compared to MSJ. As we have limited access to credit for commercial models, we decided to focus our evaluation on the latest open-source models, all of which were released between May and December 2024, with most incorporating safety guardrails. Notably, Llama-3.1-8B is specifically fine-tuned for robustness in long-context scenarios. Our results indicate that PANDAS substantially improves jailbreaking performance on these open-source models.

To additionally validate our findings, we conduct evaluations on a commercial model, GPT-4o-mini, which has the lowest per-token cost among available commercial options (Claude’s and GPT’s). Using 128 shots and only a single restart on AdvBench, MSJ and PANDAS achieve an ASR of 0.19% (1 out of 520) and 2.12% (11 out of 520), respectively. For cost reasons, we do not perform Bayesian Optimization and adopt uniform sampling across all malicious topics. This result shows that, despite both methods showing low ASR, the effectiveness of MSJ can still be improved.

Clarification on the attention analysis (Claim 3): Our hypothesis is that the success of MSJ arises from the reinforcing instruction-following behavior as the number of demonstrations increases. The phrase “reinforcing instruction-following behavior” refers to the phenomenon where the model interprets the malicious question as another instruction to follow, rather than an unsafe request to be rejected, due to the consistent compliance pattern from early demonstrations.

PA and ND are designed to further encourage this effect. While measuring attention from the target prompt to previous demonstrations would directly show which examples it references, it would not reveal how the demonstrations themselves build upon one another.

By introducing a reference score between demonstrations, we capture how each demonstration “looks back” at earlier ones. A higher reference score suggests that later demonstrations are increasingly influenced by prior demonstrations, potentially strengthening the pattern of instruction-following established throughout the prompt.

Attention scores have been widely used as a proxy for understanding transformer behavior [3, 4]. In this work, we leverage this to study how contextual dependencies develop across long sequences, providing insight into how PANDAS improves over MSJ.

Evaluation of i-MSJ: We focus on AdvBench50 for the evaluation of i-MSJ primarily due to the extensive runtime required to perform 128 iterations of random search, following the setup in Zheng et al. [2]. It is also worth noting that even in the original paper by Zheng et al., results are reported only on the AdvBench50 dataset.

[1] Mazeika et al., Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. In ICML’24.

[2] Zheng et al., Improved few-shot jailbreaking can circumvent aligned language models and their defenses. In NeurIPS’24

[3] Oymak et al., On the role of attention in prompt-tuning, In ICML'23

[4] Quirke & Barez, Understanding addition in transformers, In ICLR'24