KDA: A Knowledge-Distilled Attacker for Scalable LLM Red Teaming

While I believe this paper and its ideas could be a valuable addition to the current literature, I think there are a number of major issues that drive me towards rejection.

Claims

In my opinion, the most glaring weakness of this paper is its claim that KDA satisfies ABCDE properties. The main issue is that KDA is trained on and in large part relies on attacks (AutoDAN, GPTFuzzer, PAIR) which the paper itself claims do not satisfy ABCDE. In particular, E is very important as being exempt from commercial LLMs is an important claim as it lists a large number of techniques which already satisfy ABCD but not E. In order to train KDA one must first generate attacks from all of the $N_{att}$ SOTA attack methods. Therefore, I would argue that if any of these methods uses a commercial LLM then you must also claim that KDA is not E. This in someway also challenges the claim of B as AutoDAN is not B. One could claim that the KDA technique is general and could be used with only E SOTA attackers; however, one could make a similar argument for AutoDAN, GPTFuzzer, and PAIR which to the best of my knowledge also are general and have no requirement on using commercial LLMs (although they do in their evaluation). So by this argument, AutoDAN, GPTFuzzer, and PAIR should be marked as E and the claim that KDA is the first method to satisfy all 5 ABCDE properties is no longer true.

The paper also claims KDA is coherent and diverse; however, the evaluation makes no effort into showing this. While given the method I am inclined to believe this claim, the paper needs to address these claims with experiments (i.e. showing some perplexity metric and/or giving some metric on diversity).

Evaluation

The evaluation seems incomplete and/or is poorly written. Some evaluation details are unclear, like what test set is being used to evaluate? The paper seems to indicate that there are 2000 data samples split into 60/20/20 train/validate/test. So are you evaluating the human judge and KDA on this test set of size 400? If so, in Table 2 how is it possible to get an accuracy of 87.33 or 88.67 (i.e. 364/400 = 0.885, 365/400 = 0.8875)?

Algorithm 1 says that they sample K attack prompts then check the response from the target and Section 5.3 seems to indicate that an average of just over 1 query is needed. This suggests that a majority of the time the first generated attack works. Which attack is that? Generated in the style of which attack name (see line 290)? If a majority of the time the first attack works, why do we need to train on multiple attack methods, is one enough?

There seems to be a lot of "cross pollination" between the training of KDA, HJ and the evaluation. There are 3 other attack methods chosen: AutoDAN, GPTFuzzer, and PAIR (AGP). KDA in the evaluation is trained on AGP, the HJ is (seemingly, it is unclear) trained on the same data generated by AGP, KDA is evaluated against AGP and HJ is evaluated against competitors on a dataset generated by AGP.

There are missing evaluations. As mentioned above, it needs to be shown that KDA is C, D. I think it would also be good to show how KDA performs against LLM defenses (like AutoDAN, Page 7 Effectiveness against defense). How does KDA perform against methods it is not trained on?

After claiming that KDA does not rely on commercial LLMs, the harmful dataset construction is done with multiple commercial LLMs (i.e. GPT-4, GPT-3.5). This harmful dataset goes on to be the foundation for the training data which KDA is trained on and which the HJ evaluator is trained on. Many of the choices made are unexplained/unclear, i.e. why are 200 instances of Harmful-1k sampled when all samples could be used? $N_{att} = 3$, there are 200 queries, and 2 target models. How does this become 2000 data samples? Additional questions like this included below.

Clarity

Along with the issues above, there are some other issues with clarity.

Line 226 gives a definition of an optimization problem, then claims that many attack methods approximate it but has shortcomings. There is no where in the paper that I see which then addresses the optimization problem. Does KDA approximate it? Additionally, are you maximizing over all $A$s? Is there some constraint on the length of $A$? Is there some penalty for being coherent?

In line 221 $R_A \sim q_T(\cdot|A)$ where A is an attack prompt. However, in line 300, KDA is trained by minimizing the cross entropy between $q_{KDA}(\cdot|X)$ and target prompt $A$ where $X$ is a prompt and also a function? There is some mismatch in the notation. Detailed questions listed below.

Structure

Given my comments above the paper is poorly structured. Currently, too much of the paper talks about related works and there is not enough space to clearly describe KDA or give adequate experiments. There are also a large number of figures which take up considerable space which I think could be combined or are not needed.

• Although most criticisms of existing works are accurate, the last one ``Lack of human-aligned jailbreak evaluator'' is vague. First, it is not clear how to define and evaluate human-aligned jailbreaking evaluator. For example, GPTFUZZER's judgment model is trained from a dataset with human annotations. Given that the labels are assigned by humans and the classifier has high accuracy, we can also state that it is aligned with humans. I am not sure why this work marks it as not aligned with humans.

• The proposed method is super clear to me. For example, when generating the fine-tuning dataset what $\mathcal{M}$ refers to, what are the attack methods used in this work? Also, what is the fine-tuning objective function? Next token prediction loss or SFT?

• The proposed method lacks technical depth and novelty. Directly fine-tuning another model to generate attack prompts is not new. [1] proposes a similar idea already. In [1], they consider more regularization terms to ensure the diversity of the generated prompts. Also, it is not clear to me why using the knowledge distillation method rather than using RLHF or DPO to train the model generates attack prompts against one or multiple target models. Such a method should be more flexible than distilling a given dataset because the model has the freedom to explore different attack generation strategies. Also, why not directly use the dataset created for the jailbreaking attacks, what are the main benefits of the distillation other than efficiency?

• The judgment model mainly focuses on whether the model output is harmful. It cannot decide whether the output is related to the input query. As discussed in [2], this will introduce false positives.

• The jailbreaking performance of the proposed method is similar to GPTFUZZER from Fig. 7. I would suggest authors provide a more details explanation of the results and better justify the benefit of their approach compared to GPTFUZZER.

• I would suggest the authors explore simple defenses given their dataset. They can try to train a simple guardrail model to see if it helps filter adversarial outputs.

[1] AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs [2] When LLM Meets DRL: Advancing Jailbreaking Efficiency via DRL-guided Search

• As noted by the authors, its effectiveness heavily relies on the presence of successful attack prompts.

• The improvement achieved by the Judge/Evaluator over basic text matching is marginal, considering the substantial effort required for human labeling and fine-tuning the LLM.

• Lack of comprehensive evaluation: i) The paper hasn’t tested generalizability to other benchmark datasets outside of Harmful-1K, which was used to train KDA.
  ii) Jailbreak performance in the main paper has only been compared with the 3 attack models that KDA leveraged in its training.
  iii) Similarly, jailbreak performance in the main paper has only been compared with the 3 target LLM models that successful attack prompts were collected from and then KDA was finetuned with. In Appendix A.1 -- the transfer attack is only performed on 100 samples and it is using an LLM from the same LLM model family that KDA was fine-tuned with. (eg. KDA has been fine-tuned with successful GPT-4 attacks and then this transfer evaluation is performed on GPT-4o which is the same model family).
  iv) Results from Table 2, Figure 6 and Figure 7 are from small test sets (~100 samples?). The jailbreak performance comparison in particular should be carried out on multiple datasets, including larger ones. (The large scale evaluation in Table 3 is only targeting Vicuna-7B, which all 4 models had a high ASR on in Figure 6, so this doesn't add to the jailbreak performance evaluation.).
  v) the number of queries per success not shown for the jailbreak performance comparison.

• No mention of ethics review or ethics board approval being obtained for this work.

Typos:
Bottom of page 9: “KDA achieves a perfect ASR while being at least 4 times faster than PAIR, 7 times faster than PAIR,…” one of the PAIRs should be GPTFuzz.

Lack of contributions. The paper claims three main contributions: a human-aligned evaluator, an attacker LLM model with its training design, and a jailbreak dataset. However, each of these contributions has existing counterparts in prior work, and the paper does not clearly demonstrate its advantages or uniqueness compared to these studies. Specifically:

1. Human-Aligned Evaluator: Previous works have proposed fine-grained jailbreak evaluators, such as those in Harmbench [1] and StrongReject [2]. The paper, however, only compares its evaluator with naive methods (e.g., text matching), a vanilla LLM (GPT-4), and defense guardrails (Llama-Guard-2). None of these baselines are specifically designed for jailbreak evaluation. Therefore, the advantages or unique features of the proposed human-aligned evaluator over existing methods remain unclear.
2. LLM Attacker: Similar methodologies have been employed in prior studies. For example, AmpleGCG [3] trains an attacker LLM using GCG [4] attack data, and AdvPrompter [5] trains an attacker LLM with AutoDAN [6] data. This indicates that training an LLM to generate jailbreak prompts is not a novel idea. Unfortunately, the paper does not include comparisons with these methods. It seems that the main difference is the utilization of more jailbreak methods for generating prompts during training. If so, the contribution may be limited.
3. Generated Dataset: While contributing datasets to the community is commendable, the proposed dataset is generated by an attacker model trained on existing attacks like GCG and AutoDAN. This raises the question: why not directly use the jailbreak data from these existing attack methods, such as the data generated by Harmbench [1], which includes a more diverse set of jailbreak methods? The uniqueness and added value of the proposed dataset are not evident, potentially diminishing the significance of this contribution.

[1] Mazeika, Mantas, et al. "Harmbench: A standardized evaluation framework for automated red teaming and robust refusal." arXiv preprint arXiv:2402.04249 (2024).
[2] Souly, Alexandra, et al. "A strongreject for empty jailbreaks." arXiv preprint arXiv:2402.10260 (2024).
[3] Liao, Zeyi, and Huan Sun. "Amplegcg: Learning a universal and transferable generative model of adversarial suffixes for jailbreaking both open and closed llms." arXiv preprint arXiv:2404.07921 (2024). [4] Zou, Andy, et al. "Universal and transferable adversarial attacks on aligned language models." arXiv preprint arXiv:2307.15043 (2023).
[5] Paulus, Anselm, et al. "Advprompter: Fast adaptive adversarial prompting for llms." arXiv preprint arXiv:2404.16873 (2024).
[6] Zhu, Sicheng, et al. "Autodan: Automatic and interpretable adversarial attacks on large language models." arXiv preprint arXiv:2310.15140 (2023).