Leveraging the Context through Multi-Round Interactions for Jailbreaking Attacks

1. Lack of some detailed evidence to support the demonstration
2. Lack of comparison with the prevalent multi-turn attack Crescendo

• Lack of Novelty: The core idea of this paper closely resembles that of Crescendo [1], a multi-turn jailbreak method that starts with an innocuous query about a harmful task and gradually guides the model to generate harmful content through benign incremental steps. Given this similarity, the concept presented in this paper does not appear to introduce significant novelty over Crescendo.

  Furthermore, the implementation in this paper also mirrors Crescendo’s approach. Crescendo employs in-context learning with a series of carefully crafted examples—a strategy similarly adopted in this paper. For instance, Crescendo’s attack sequence begins with “introduce the history of [harmful target]” (Figure 1), while this paper uses a similar prompt, “Write a paper on the history of [harmful target],” as the starting point of its attack (Figure 1).

  It would be valuable for the authors to elaborate on how their approach diverges from or improves upon Crescendo's methodology. Are there any distinctive methodological or experimental results that could underscore the novelty of this work?

• Insufficient Experiments: The experiments in this paper are limited to a subset of the AdvBench benchmark, while other benchmarks, such as HarmBench [2] and SorryBench [3], offer more balanced and less noisy alternatives. Including results from HarmBench or SorryBench would enhance the robustness and comprehensiveness of the experimental evaluation, and the authors should consider incorporating these benchmarks in future work or as additional experiments.

• Lack of Comparisons with Other Multi-Turn Attacks: Besides Crescendo, several other multi-turn attack methods have recently emerged, including [4, 5, 6]. This paper would benefit from a concise comparison with these approaches, outlining key similarities or differences, and including a discussion of these methods in the related work section to provide a more comprehensive review of the landscape in multi-turn attacks.

1. Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack.
2. HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal.
3. SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal Behaviors.
4. Chain of Attack: A Semantic-Driven Contextual Multi-Turn Attacker for LLMs.
5. Speak Out of Turn: Safety Vulnerability of Large Language Models in Multi-Turn Dialogue.
6. Imposter.ai: Adversarial Attacks with Hidden Intentions Towards Aligned Large Language Models.

• The concept of multi-turn attacks on LLMs is already well-established in the literature (e.g., [1, 2, 3, 4]), with [2] particularly exploring the use of context to craft jailbreak sequences. Authors should clarify how this attack is different from the contextual attacks in [2]
• The method for generating benign prompts closely resembles that in [5], where in-context learning capabilities of the LLM are utilized to generate new attack prompts based on given examples. Authors should clearly state the key difference in their approach for generating prompts compared to [5]
• Section 4.1, which discusses evaluation metrics, mentions that the judge function relies entirely on human evaluation. However, there is insufficient clarity regarding the number of examples used to compute the jailbreak percentage and how human evaluation scores correlate with an automated LLM judge score. Authors should clearly specify the following points in regard to the paper:

1. The exact number of examples used to compute jailbreak percentages
2. Details on how many human evaluators were involved
3. Any analysis comparing human evaluation scores to automated LLM judge scores, if available

References:

1. Multi-turn and contextual jailbreak attacks research
2. CFA: Contextual attacks on LLMs
3. RedQueen: Multi-turn attacks
4. MART: Adversarial prompt sequences
5. PAIR: In-context attack prompt generation
6. Prompt-Guard-86M
7. Llama-Guard-3-8B

Novelty: The main results on 50 prompts do not show significantly more success than prior attacks. The open source models evaluated are not exciting, they’re somewhat outdated and not claimed to be robust to attacks. This work would benefit from attacks on more recent models such as Llama 3 or 3.1, or attacking a larger model/models with refusal training. The method is fairly simple, it boils down to prompting another LLM to mimic how a human jailbreaker would attack an LLM. It would be great to elaborate more on the novelty of this approach relative to recent work on multi-turn jailbreaking.

Soundness: A human judge was used on AdvBench without much detail given, this affects the soundness of the paper. Please elaborate more. Have you considered testing your method with a controlled judge, such as HarmBench [1], which has a trained static classifier? The transferability experiments (Table 2) feel contrived. (1) Attacks such as PAIR don’t assume a threat model to transfer the instance-level outputs of the attack, it is not a fair comparison. However, the attack itself only needs black box access, it doesn’t need to transfer the instance-level output. (2) The closed-source models tested have specific defenses implemented against popular attacks such as GCG, it is not a fair comparison to measure transferability. The authors should report the results in the original attack paper whenever possible, which shows a much stronger transferability attack against GPT-3.5 and GPT-4 models.

Defenses: Perplexity based defenses tested are specifically designed against GCG-style attacks, it is expected that they are not actually detecting harm. This work would benefit from showing strong results on general defenses such as DERTA [2] or Llama Guard [3], or more robustly trained models such as CYGNET.

Presentation: Page 1 line 45: “jailbreak prompts generated by automated attacks often do not perform well when transferred to other LLMs, which is a major bottleneck”. This contradicts the citation on GCG immediately prior, which is transferable to other LLMs. The reason why GCG performs worse in this paper when transferred is likely due to model providers providing specific defenses against it. This does automated attacks transfer poorly. Related work section only cites 3 papers involving jailbreaking LLMs, the rest are on background information such as in-context learning and autoregressive LMs that should be assumed as basic knowledge among this community. It would be good to further contextualize the novelty of this attack relative to prior work. Minor: Figure 1 has a typo “Repharse” -> “Rephrase”.

[1] Mazeika, M., Phan, L., Yin, X., Zou, A., Wang, Z., Mu, N., ... & Hendrycks, D. (2024). Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249. [2] Yuan, Y., Jiao, W., Wang, W., Huang, J. T., Xu, J., Liang, T., ... & Tu, Z. (2024). Refuse whenever you feel unsafe: Improving safety in llms via decoupled refusal training. arXiv preprint arXiv:2407.09121. [3] Inan, H., Upasani, K., Chi, J., Rungta, R., Iyer, K., Mao, Y., ... & Khabsa, M. (2023). Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674.