Diffusion Attacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreak

Rebuttal for DiffusionAttacker Submission (Third Reviewer)

We appreciate the reviewer's detailed feedback and constructive suggestions. Below, we address each point in turn:

---

1. Method Section Clarity and Symbol Consistency

Reviewer Comment: The method section lacks clarity; symbols are inconsistent, especially between Sections 3.1 and 3.2, and between Equations (2) and (3).

Response:
We have carefully revised the method section to ensure consistency and clarity:

• Unified symbols across Sections 3.1 and 3.2: Symbols like $x$ and $f(x)$ are now consistently used to represent input prompts, and $z$ denotes the dimension-reduced representation.
• In Equation (2), we clarified that $g(x)$ is the output of dimension reduction, yielding $z$ as shown in Equation (3).

Example changes:

• Previous: $z$ appeared undefined in relation to $g(x)$.
• Revised: Explicitly state $z = g(x)$, where $g(x)$ applies PCA transformation on $x$.

---

2. Figure Consistency and Grammatical Errors

Reviewer Comment: Figure 3’s first plot is identical to Figure 2, and captions are inconsistent. There are grammatical errors.

Response:

• Figure Corrections: We updated Figure 3 to replace the duplicated plot and revised the caption to accurately describe the new visualizations.
• Grammatical Revisions: We conducted a thorough review to correct grammatical issues throughout the paper, ensuring smoother readability.

---

3. Dataset Details (Section 4.1)

Reviewer Comment: No detailed description of the dataset (number of training/test samples) is provided.

Response:
We have added detailed information about the dataset:

• Training Set: 23,572 samples from Wikipedia and 21,835 PAWS paraphrase pairs.
• Test Set: 900 harmful prompts from AdvBench and HarmBench datasets.

Additional information includes:

• Training-validation split: 80%-20%.
• Pre-processing: Applied standard tokenization and filtering for data quality.

---

4. Step-by-Step Algorithm or Pseudocode

Reviewer Comment: The inclusion of pseudocode or an algorithm would clarify the method. The inference phase is not explained.

Response:
We have added a step-by-step pseudocode for DiffusionAttacker in the revised manuscript. Here is a brief version:

Algorithm: DiffusionAttacker
1. Input: Harmful prompt X, pre-trained DiffuSeq model
2. Initialize Gaussian noise z_T
3. For each denoising step t = T to 1:
    a. Predict mean and variance (µ_t, σ_t) from z_t
    b. Generate intermediate state z_t-1
    c. Apply Gumbel-Softmax to sample rewritten prompt Y_t
    d. Evaluate attack loss L_att and semantic loss L_sim
    e. Update z_t using gradient descent
4. Output: Adversarial prompt Y*

5. Parameter Selection (Equation 11)

Reviewer Comment: How are parameters $t$, $T$, and $M$ chosen?

Response:
We selected $t$, $T$, and $M$ based on empirical experiments:

• $T$ (Total steps): Set to 1000, balancing generation quality and computational cost.
• $t$ (Initial steps without gradient updates): 200; early steps contain minimal semantic information.
• $M$ (Gradient updates): 5 evenly spaced steps in $T - t$ range to reduce overhead while maintaining performance.

We conducted an ablation to justify these choices:

---

6. Examples of Generated Prompts

Reviewer Comment: Specify examples of generated jailbreak prompts.

Response:
Here are sample adversarial prompts generated by DiffusionAttacker:

These examples demonstrate how DiffusionAttacker preserves harmful intent while bypassing LLM safety filters.

---

We hope these clarifications address your concerns comprehensively. Thank you for your valuable insights and suggestions.

1. Matching Harmful Intent in Generated Responses

Reviewer Comment: Harmful responses might not directly match the specific harmful intent of the original question.

Response:
Thank you for this insightful comment. To address this, we conducted additional experiments to assess the semantic similarity between the generated prompts and the original harmful intent using BERTScore and measured task completion accuracy (TCA) by humans. We also compared our approach against other baseline methods, such as GCG and AutoDan, to highlight the effectiveness of our method:

The results indicate that while some generalization occurs, DiffusionAttacker generates prompts that align significantly better with the intended harmful intent compared to other baseline methods. The higher task completion accuracy and semantic similarity demonstrate that our method effectively preserves the original harmful intent.

---

2. Discussion of Potential Defenses

Reviewer Comment: There is limited discussion on potential defenses, such as SmoothLLM.

Response:
We appreciate this suggestion and have expanded the discussion in Section 5 to cover prominent defense strategies, including SmoothLLM [1] and RA-LLM [2]. We also conducted preliminary experiments to evaluate the robustness of DiffusionAttacker against these defenses:

Although these defenses reduce the success rate, DiffusionAttacker still bypasses them to a certain extent. Future work will explore stronger defense strategies and adaptive countermeasures.

---

3. Transferability Evaluation

Reviewer Comment: The attack was not tested on production models like GPT-4 or Claude-3.5.

Response:
Thank you for highlighting this point. Our primary focus is on red teaming open-source white-box models, where we can fully leverage the diffusion process and internal gradients. Transferability to black-box models like GPT-4 or Claude-3.5 is an additional aspect of our method, and we acknowledge that the attack's effectiveness is reduced in these scenarios. Unlike existing black-box jailbreak attacks, which often rely on static jailbreak templates and have low diversity, DiffusionAttacker emphasizes dynamic generation and adaptability.

While our approach shows limited success in black-box settings, it excels in white-box environments by creating highly diverse and semantically aligned adversarial prompts.

---

4. Concrete Examples of Generated Prompts

Reviewer Comment: The paper lacks concrete examples of generated prompts to illustrate their quality and effectiveness.

Response:
We have added specific examples in Appendix A to illustrate the quality and relevance of the generated prompts. Below are some examples:

These examples demonstrate how DiffusionAttacker preserves harmful intent while bypassing LLM safety filters.

---

We sincerely appreciate the reviewer’s valuable feedback, which has significantly improved the quality of our paper!

---

References:
[1] SmoothLLM: Defending large language models against jailbreaking attacks
[2] Defending against alignment-breaking attacks via robustly aligned LLM

Rebuttal for DiffusionAttacker Submission (Second Reviewer)

We appreciate the reviewer's detailed feedback and constructive suggestions. Below, we address each point in turn:

---

1. Dimension Reduction and Ablation Studies

Reviewer Comment: No justification or ablation study for using dimension reduction before the harmful/harmless classifier. Impact of different methods not discussed.

Response:
The justification for using dimension reduction before the harmful/harmless classifier is that it effectively reduces complexity without impacting classification performance. As shown in [1], reducing dimensions to as low as 4 retains accuracy.

We conducted additional experiments to compare the impact of different dimension reduction methods—PCA (Principal Component Analysis), t-SNE, and UMAP—on Attack Success Rate (ASR) and Classifier Accuracy.

Why PCA:
PCA captures the key variance in high-dimensional hidden states with minimal information loss and computational cost. t-SNE and UMAP, while powerful, are non-linear and computationally intensive, which can slow down real-time gradient updates.

---

2. Motivation for Using Diffusion Models

Reviewer Comment: Why use a diffusion model instead of another LLM for rephrasing?

Response:
Diffusion models offer unique advantages in generating adversarial prompts:

• Controlled Generation: Diffusion models iteratively refine their output, enabling fine-grained control at each denoising step. This ensures the rephrased prompt maintains semantic consistency while altering internal representations. As shown in [2], diffusion-based language models outperform autoregressive models in controllable text generation tasks.
• Global Context Optimization: Unlike autoregressive LLMs that generate text token-by-token, diffusion models optimize the entire sequence holistically. This leads to better fluency and coherence, critical for evading detection.

Empirical Comparison:
We compared DiffusionAttacker with an LLM-based rewriter (fine-tuned Llama3-8b-chat):

While LLMs are faster, DiffusionAttacker achieves higher ASR and better fluency, which are crucial for bypassing detection mechanisms.

---

3. Presentation and Figure Clarity

Reviewer Comment: Improve presentation by referencing Figure 1 and ensuring font readability in Figures 2 and 3.

Response:
Thank you for highlighting this issue. We have revised the manuscript to:

• Explicitly reference Figure 1 during the method explanation for better clarity.
• Increase the font sizes in Figures 2 and 3 to improve readability.

---

4. Qualitative Examples

Reviewer Comment: No qualitative examples of original and rephrased prompts. Examples would enhance clarity.

Response:
We agree that qualitative examples add clarity. Here are examples of original and rephrased prompts, along with the corresponding LLM outputs:

These examples demonstrate how DiffusionAttacker preserves the harmful intent while bypassing safety filters, leading to successful jailbreaks. We have included these examples in the revised manuscript.

---

We hope these clarifications address your concerns comprehensively. Thank you for your valuable insights and suggestions.

We appreciate the detailed feedback and constructive suggestions. Below, we address each point in turn:

1. Baseline Selection

Reviewer Comment: The chosen baselines (GCG, AutoDAN) are not the most powerful. Black-box methods could be more fluent, efficient, and effective.

Response:
We selected GCG, AutoDAN, and COLD-Attack as they share key methodological elements with DiffusionAttacker (white-box, proxy-target, gradient-guided). This ensures a fair comparison of like methods. To address the concern about more powerful black-box baselines, we have included results from Puzzler attacks on LLama3-8b-chat, but both Puzzler and WordGame have not release the code, so the result is by the code we reproduced.

Our method shows superior performance in fluency and diversity while achieving a higher ASR.

2. Evaluation Metrics

Reviewer Comment: Prefix-match ASR and the GPT-4o judge are inadequate. Fluency (PPL) and Self-BLEU might not be crucial metrics for jailbreaking prompts.

Response:
Firstly, in previous work, ChatGPT was often used to determine whether the output content was harmful[1,2,3]. This task is not difficult, and GPT4o can achieve a considerable level of accuracy, we randomly constructed 50 harmful and 50 harmless contents from[4], and the results of this model are consistent with those of humans.

To enhance our evaluation, we added two more metrics:

• Content-Relatedness Score (CRS): Measures semantic preservation between input and generated prompts.
• StrongREJECT: An evaluation method that scores the harmfulness of the victim model’s responses[5].

3. Design Choices

Reviewer Comment: Why use last-layer hidden states? Why DiffusionSeq over other architectures?

Response:

• Last-layer States: Captures the final decision-making context. Experiments with middle layers resulted in an ASR drop from 0.74 to 0.63. And taking the last layer also maintains the same settings as previous works[4]. If choosing the middle layer, different models may not be able to maintain a unified setting because the distinction between harmful/harmless in different models does not occur in the same layer.

• DiffusionSeq: Supports continuous, guided generation. Research has shown that diffusion language models are more controllable than transformer architecture models[6]. In jailbreaking, compared to encoder-decoder models, it achieved 13% higher ASR

---

Thank you for your valuable insights and for helping us strengthen our work.

[1] Autodan: Generating stealthy jailbreak prompts on aligned large language models

[2]ASETF: A Novel Method for Jailbreak Attack on LLMs through Translate Suffix Embeddings

[3]Jailbreaking black box large language models in twenty queries

[4]Prompt-driven llm safeguarding via directed representation optimization

[5]A StrongREJECT for Empty Jailbreaks

[6]Diffusion-lm improves controllable text generation