Derail Yourself: Multi-turn LLM Jailbreak Attack through self-discovered clues

Q2: Comparison with more multi-turn baselines:

A2: Following the reviewer’s suggestion, we evaluated CoA on Harmbench. However, we did not include CFA in our comparison because it is not open-sourced, and key implementation details (such as the system prompt) are unavailable. The results, as shown in the table, further validate the effectiveness of our ActorAttack, demonstrating superior performance over CoA against both open-source and closed-source models.

Q3: Incomplete evaluation in ablation study

A3: We appreciate the reviewer’s concern regarding the target models used in the ablation study. Here, we clarify the rationale behind our choices and provide additional evaluations to address the concern:

1. Rationale for Model Selection: We selected GPT-4o and Claude-3.5 as representative victim models because these LLMs are generally safer than alternatives like Vicuna, Qwen, or ChatGLM, as evidenced by their superior performance on safety benchmarks such as Harm-bench and Sorry-bench.
2. Additional Evaluations: To further address your concern, we conducted a more comprehensive evaluation to demonstrate the generality of our ActorAttack approach. Specifically, we included three additional models: Llama-3-8B-instruct, Qwen-2.5-7B-instruct, and GLM-4-9b-chat. In these experiments, we explored how varying the number of our attack clues affects effectiveness. The results, consistent with those in our paper, show that the proportion of successful attacks increases as the number of actors (attack clues) grows. It indicates that our ActorAttack can find more optimal attacks from our diverse attack paths. We will include these additional findings in the camera-ready version of our paper.

Q4: Comparison with more single-turn baselines.

A4: Thank you for the suggestion! We have added the results of ReNeLLM and MultiLingual to the table below. Our attack demonstrates stronger effectiveness compared to these single-attack baselines. In particular, these state-of-the-art single-attack baselines, including PAIR, PAP, and CipherChat used in our paper, perform poorly on Claude-3.5-sonnet, while our attack achieves a much higher success rate. It highlights the severity and pervasive nature of the safety vulnerabilities we identified. Due to the high computational cost of evaluating AutoDAN, its results are not yet complete. We will update this as soon as possible.

Thank you for your thoughtful review and for recognizing the strengths of our work! We are pleased to hear that you found the effectiveness of our proposed method and appreciated its utility in enhancing model safety. We have addressed your concerns below and are happy to continue the discussion to further clarify our contributions.

Q1: Our contributions over multi-turn baselines:

A1: Thank you for your thoughtful observation. However, we respectfully disagree with the concern about the novelty and technical contribution of our work. Grounded in sociological theory, our approach introduces a novel and systematic perspective to identify attack clues, enabling more diverse, effective, and efficient attacks compared to existing multi-turn baselines.

Key Contributions and Novelty

We address two critical challenges for successful multi-turn attacks, which differentiate our work from prior approaches:

1. Identifying Semantically Relevant and Diverse Attack Clues

• Existing approaches either rely on limited strategies like role-playing and scenario assumptions (CoA [1] and CFA [2]) or fixed communication templates plus human-crafted seed instances (Crescendo [3]) to generate attacks. Limited attack strategies and potential biases towards seed instances may lead to the diversity issue, ultimately limiting their effectiveness (Table 1, and Figures 7 and 8).
• In contrast, our approach is systematically supported by scientific theory. Inspired by Latour’s sociological network theory, we explicitly model the potential attack clues as a network:
  • Nodes represent attack clues that are semantically relevant to the harmful target.
  • Edges form stepwise attack paths connecting clues to the harmful target.
  • Our network has two distinct properties, which ensure the effectiveness of our attacks:
    • Higher diversity: we identify six distinct types of nodes, including both human and non-human entities, covering a broader spectrum of attack clues.
    • Inherent semantic relevance: different from existing works that infer attack clues from the seed instances, we generate a unique network for each harmful target, ensuring the derived clues are semantically relevant to the given target (Figure 2 and Figure 3(a)).
  • Our quantitative results (Table 1 and Table 4) demonstrate the enhanced diversity and effectiveness of our approach.

2. Efficient Convergence Toward the Harmful Target

• Existing works like Crescendo reactively adjusts attack paths during the attack process based on the target model's responses. By contrast, we pre-compute the attack paths (i.e., each edge of our network serves the attack path) and the inherent semantic relevance of our attack clues ensures the convergence of our attack paths.
• Experimental results show that our multi-turn attacks achieve high effectiveness without relying on feedback from the target model (Table 1), validating the efficiency of our method. Another evidence is that under the same attack budget, our attacks consistently achieve greater effectiveness than Crescendo (Figure 4). We also conducted a comparison between CoA and our method (see the table in response to Q5). These results demonstrate the efficiency of our method.

In summary, our work introduces a novel perspective and addresses the limitations of previous methods, achieving state-of-the-art performance in multi-turn attacks. We hope this explanation clarifies our contributions and the distinction from multi-turn baselines.

Q5: Comparison of state-of-the-art defense mechanisms

A5: Thank you for your valuable suggestion. In response, we have implemented and evaluated three distinct and powerful defense baselines: Rephrase [1], RPO [2], and Circuit Breaker [3], to comprehensively assess the effectiveness of our attack. Additionally, we included the performance of models fine-tuned using our multi-turn safety data. Since both Circuit Breaker and our defense mechanism rely on fine-tuning, we report the results specifically for Llama-3-8B-Instruct. The following table presents the Attack Success Rate (ASR) under these defenses.

Our findings show that Rephrase and RPO offer a partial reduction in ASR. However, Circuit Breaker greatly reduces the success rate of our attack, demonstrating the potential of safety alignment within the representation space. Moreover, fine-tuning with our safety data yields results comparable to Circuit Breaker, highlighting the value and effectiveness of the safety data we have constructed.

We also note that since Circuit Breaker implements its defense using only single-turn safety data, we believe that integrating it with our multi-turn safety data could further improve model safety. We will include this result in our camera-ready version.

Summary of Baselines:

1. Rephrasing [1] is a system-level defense that paraphrases each query in our attacks. It is a simple yet surprisingly effective strategy against jailbreaks.
2. RPO [2] is a system-level defense method that optimizes safety suffixes. It has shown strong transferability across multiple LLMs and is effective against a variety of jailbreaks.
3. Circuit Breaker [3] is a state-of-the-art, model-level defense that interrupts the internal representations responsible for harmful outputs, effectively halting the generation of unsafe content. We hope these results address your concerns and highlight the potential of our defense strategies in comparison to existing methods.

References:

1. Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline defenses for adversarial attacks against aligned language models. arXiv:2309.00614, 2023.
2. Andy Zhou, Bo Li, and Haohan Wang. Robust prompt optimization for defending language models against jailbreaking attacks. arXiv preprint arXiv:2401.17263, 2024.
3. Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving alignment and robustness with circuit breakers. arXiv preprint arXiv:2406.04313, 2024.

Thank you for your thoughtful and encouraging reviews! We are pleased to hear that you recognized the novelty of our adaptation of actor-network theory in multi-turn jailbreaks, and our method’s applicability to both black-box and white-box LLMs. We have addressed your concerns below and look forward to further discussion to refine and clarify our contributions. Additionally, we have uploaded the revised version of our paper, incorporating the changes based on your feedback.

Q1: Improving the writing of section 3

A1: Thank you for pointing out the writing issues in Section 3! To improve the clarity of the process by which we construct the actor network to discover attack clues, we have restructured Section 3.1 (line 192) into three distinct parts:

• Theoretical Grounding in Our Design
• Network Definition
• Network Adaptation to New Harmful Targets

Specifically, each node in the network represents an attack clue. For each harmful target, we generate a unique network, ensuring that the derived clues are semantically relevant to the given target. Our updated descriptions should make the process clearer to readers.

Q2 & Q4: Descriptions of notations used in algorithm pseudocode

A2: Thank you for your valuable feedback! To improve clarity, we have added a detailed description of the notations used in the algorithm pseudocode in the revised manuscript (line 160). Specifically, we clarify the roles of the four LLMs and provide explanations for symbols that were not previously introduced in the main text.

• The victim model $V_{\theta}$ represents the model being attacked.
• The attacker model $A_{\theta}$ generates multi-turn attacks.
• The judge model $J_{\theta}$ determines the success of the attack.
• The monitor model $M_{\theta}$ is responsible for our dynamic modification, as illustrated in Figure 3(c).

Additionally, except for the victim model, we employ the same LLM to implement the other three models. This clarification should make the relationships and roles of these models clearer to the readers.

Q3: Consistency of comparison with single-turn and multi-turn attack baselines

A3: Thank you for your helpful suggestion. In Figure 4, we compared the attack success rates of our method and Crescendo for GPT-4o and Claude-3.5-sonnet. However, we agree that comparing single-turn and multi-turn attack baselines within the same setup would make the results clearer and more meaningful.

In response, we have conducted a more thorough evaluation. This includes a new multi-turn baseline, CoA [1], which was suggested by other reviewers. The updated results, shown in the table below, demonstrate that our method, ActorAttack, achieves the highest attack effectiveness, both for individual models and on average. This highlights the significant safety vulnerabilities we have identified in current LLMs. We will update Table 1 in the camera-ready version to include these new results.

Reference:

1. Yang, Xikang, et al. "Chain of Attack: a Semantic-Driven Contextual Multi-Turn Attacker for LLM." arXiv preprint arXiv:2405.05610 (2024).

Q6: Including more multi-turn baselines for comparison.

A6: Thank you for the suggestion! We have included CoA as a baseline to further validate the effectiveness of our method. The results, as shown in the table, demonstrate that our ActorAttack significantly outperforms CoA against both open-source and closed-source LLMs on Harmbench.

Q7: Non-adequate evaluations in ablation study.

A7: Thank you for raising this concern. We selected GPT-4o and Claude-3.5 as target models because they are currently among the safest LLMs based on evaluations over established safety benchmarks such as Harmbench and Sorrybench. Additionally, due to the high API costs associated with using Crescendo, we conducted evaluations on a uniformly sampled subset of 50 instances from Harmbench rather than the full set.

To address the reviewer's concern, we expanded our evaluation by including two additional LLMs Llama-3-8B-instruct and Qwen-2.5-7b-instruct, and assessing the complete Harmbench dataset. The results, presented in the table, confirm our findings: ActorAttack consistently demonstrates higher effectiveness and better diversity compared to Crescendo.

• table 1. results of Llama-3-8b-instruct

• table 2. results of Qwen-2.5-7b-instruct

Q8: The rationality of using GPT-4o for judgment.

A8: Thank you for this suggestion! We believe that our choice of GPT-4o as the judgment model is reasonable and well-supported. Our design aligns with the practices of [1], [2], and [3], which implement GPT-4-based judges. Notably, [1] found through human studies that GPT-4 is effective and accurate in identifying harmful outputs. To further validate the rationality of using GPT-4o as the judge, we conducted additional human study experiments. As shown in the table below, our findings confirm that the GPT-4o judge aligns more closely with human judgments compared to alternatives like Llama-Guard and the OpenAI Moderation API. Specifically, Llama-Guard exhibits a higher false negative rate (misclassifying unsafe outputs as safe), while the OpenAI Moderation API shows a higher false positive rate (misclassifying safe outputs as unsafe). These results underscore the reliability and alignment of GPT-4o for this task.

References:

1. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023.
2. Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. arXiv preprint arXiv:2401.06373, 2024.
3. Qibing Ren, Chang Gao, Jing Shao, Junchi Yan, Xin Tan, Wai Lam, and Lizhuang Ma. Exploring safety generalization challenges of large language models via code. In The 62nd Annual Meeting of the Association for Computational Linguistics, 2024.

Q2: Weak connection to theory.

A2: We respectfully disagree. Latour’s theory forms the foundation of our attack framework:

Based on Latour’s distinction between the roles of nodes in a network, we identified six categories of attack clues and different attack clues led to diverse attack paths (illustrated in Figure 11-14). Moreover, Latour emphasizes that both human and non-human actors hold equally significant positions in the network. This insight broadens our range of attack clues by including non-human entities such as books, media, or social movements.

Therefore, this theoretical grounding is not merely a conceptual reference but a cornerstone of our methodology, enabling us to systematically and effectively model diverse attack clues.

Q3: Lack of clarity about the reasons and benefits of initial attack generation.

A3: Thank you for pointing this out. The primary benefit of using the attacker model for initial attack generation is efficiency. Our attacks exhibit strong effectiveness even without relying on feedback from the target model (as evidenced in Table 1).

Here’s why our method succeeds:

• First, our network not only provides the attack clues (nodes), but also the attack paths (edges), which model the stepwise progression from a clue to the harmful target. Our pre-computed attack paths are model-agnostic.
• Second, we hypothesize that LLMs trained on similar pre-training data (e.g., internet-crawled texts) exhibit consistent behavior, leading to similar responses to identical queries. Thus, we use the attacker model as the proxy of unknown target models and use a self-talk mechanism to generate initial attacks.

We will clarify this explanation further in the camera-ready version.

Q4: Dynamic modification only involves repeated generation to reduce toxicity, with no heuristic strategy guidance.

A4: Thank you for pointing this out. We would like to clarify that our dynamic modification strategy is not limited to toxicity reduction. Instead, we explicitly identified two classic cases in its design: (1) scenarios where the model refuses to answer due to toxicity concerns and (2) situations where the model is unaware of the answer to the given question. In the latter case, we discard the current attack path and re-sample a new one to restart the process (L249-256). Our simple design leads to a non-trivial improvement in attack performance, as demonstrated in Table 1. More importantly, the diversity and semantic relevance of our attack clues underpin the efficacy of our method, such that our dynamic modification does not need complex designs.

Q5: Why multi-turn attacks are better than single-turn attacks.

A5: As discussed in our introduction, single-turn attacks often explicitly encode harmful intent in the input prompts (e.g., through adversarial suffixes [1], encodings [2], or role-playing [3]). Current safety alignment algorithms are adept at identifying and mitigating the harmful intent at a single input level [4, 5] but overlook the risks posed by multi-turn contextual exploitation. In contrast, our multi-turn attacks succeed by setting up a harmless yet semantically related conversational topic to conceal our harmful intention. Though our attack queries look innocuous, under our guidance, the model’s responses progressively reveal harmful details. Since our harmful intent is progressively expressed via contextual interactions, our attacks can bypass current safety mechanisms.

References:

1. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023.
2. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36, 2024.
3. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419, 2024.
4. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 27730–27744, 2022.
5. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.

Thank you for your thoughtful and encouraging reviews! We are glad to hear that you found our paper well-arranged and readable, and appreciated the reasonableness of our experiments. We have carefully addressed your concerns below and look forward to continuing the discussion to clarify and strengthen our contributions.

Q1: Limited novelty: The overall idea is consistent with Crescendo.

A1: Thank you for your thoughtful observation. However, we respectfully disagree with the concern about the novelty and technical contribution of our work. Grounded in sociological theory, our approach introduces a novel and systematic perspective to identify attack clues, enabling more diverse, effective, and efficient attacks compared to existing multi-turn baselines.

Key Contributions and Novelty

We address two critical challenges for successful multi-turn attacks, which differentiate our work from prior approaches:

1. Identifying Semantically Relevant and Diverse Attack Clues

• Existing approaches either rely on limited strategies like role-playing and scenario assumptions (CoA [1] and CFA [2]) or fixed communication templates plus human-crafted seed instances (Crescendo [3]) to generate attacks. Limited attack strategies and potential biases towards seed instances may lead to the diversity issue, ultimately limiting their effectiveness (Table 1, and Figures 7 and 8).
• In contrast, our approach is systematically supported by scientific theory. Inspired by Latour’s sociological network theory, we explicitly model the potential attack clues as a network:
  • Nodes represent attack clues that are semantically relevant to the harmful target.
  • Edges form stepwise attack paths connecting clues to the harmful target.
  • Our network has two distinct properties, which ensure the effectiveness of our attacks:
    • Higher diversity: we identify six distinct types of nodes, including both human and non-human entities, covering a broader spectrum of attack clues.
    • Inherent semantic relevance: different from existing works that infer attack clues from the seed instances, we generate a unique network for each harmful target, ensuring the derived clues are semantically relevant to the given target (Figure 2 and Figure 3(a)).
  • Our quantitative results (Table 1 and Table 4) demonstrate the enhanced diversity and effectiveness of our approach.

2. Efficient Convergence Toward the Harmful Target

• Existing works like Crescendo reactively adjusts attack paths during the attack process based on the target model's responses. By contrast, we pre-compute the attack paths (i.e., each edge of our network serves the attack path) and the inherent semantic relevance of our attack clues ensures the convergence of our attack paths.
• Experimental results show that our multi-turn attacks achieve high effectiveness without relying on feedback from the target model (Table 1), validating the efficiency of our method. Another evidence is that under the same attack budget, our attacks consistently achieve greater effectiveness than Crescendo (Figure 4). We also conducted a comparison between CoA and our method (see the table in response to Q5 from Reviewer 2ps2). These results demonstrate the efficiency of our method.

In summary, our work introduces a novel perspective and addresses the limitations of previous methods, achieving state-of-the-art performance in multi-turn attacks. We hope this explanation clarifies our contributions and the distinction from Crescendo.