Thanks for the response! I really hoped that the rebuttal was still coming :-)

W1: I do not think this is a weakness (novelty is a bad metric anyway). I just wanted to point out that the differences could be made more explicit, the framing could be improved. But I guess being this honest could hurt average reviewer score and the issue is not severe

W2: I agree the total number of experiments exceeds/meets the ICLR standards. However, I think without this experiment, the robustness results are, in hard words, "meaningless." Existing work showed that Circuit Breakers over refuse a lot making the models very impractical. This puts the impressive robustness results into a different perspective. Other works show that models can get good MT bench scores with not outputting anything. A model that outputs nothing could get the correct MMLU probs, good MT Bench score and good robustness. Thus, I really encourage the authors to add this experiment to a possible camera ready version. At the same time, I will not consider it in my score.

W3: I reviewed other papers and saw the method under a different name and thus see an issue with having the same method under a different name in different papers. I understand your reasoning but do you not agree that it is an uncommon thing to do? (Again I will not consider it in my score as its quite subjective)

W6: Acknowledged, as hyperparameter search would limit this kind of research to big tech I do not want to be to strict. Maybe you can put a statement in the paper how difficult it was to get the method to run? Or what kind of measures were necessary to identify promising hyperparams?

Q3: See W2. this is the only real open issue I have. I think its evident that some prior work took the issue of over refusal not seriously enough (its also a newish finding). In my opinion its very important. However, the paper has some independent finding that do not rely on this evaluation.

I agree with some of the other reviewers that the paper felt rushed (not the experiments but the writing). ICLR is one of the most prestigious venues in ML. Submissions that are considered for acceptance should not have these basic flaws and desk rejection would have been viable, in my opinion. On the other hand, nobody wins if we put the paper through another review process just because of things that can be changed in the camera-ready version. (now that it has been reviewed anyway)

Like I said in my initial review, I think the paper has a lot of merits. My concerns were sufficiently addressed. I changed my score accordingly and kindly ask the authors to put their paper through an independent proof reading for their next submission.

Thanks, we appreciate the review and feedback! In this review cycle, to save everyone’s time and effort, we are opting for a radically honest and terse response.

W1: We compared our LAT method to Xhonneux et al. (2024) in Table 2 and Casper et al. (2024) in Table 6 of the appendix. Overall, our claims to novelty are not conceptual. Instead, they are based on showing that LAT can combine with existing methods in a versatile way to strengthen defenses. For what it’s worth, unlike prior work on continuous adversarial training like these two, we also release LLMs with strong performance. We released 15 models, and 2 datasets alongside the paper’s public arXival. On Hugging Face, the models have over 700 downloads and the datasets have over 1100, suggesting a useful contribution to the community.

W2: We think this is a useful point. More elaborate evaluations would always be nice, but we followed existing methodology from prior work (Liu et al., 2023). And we successfully showed that LAT improved over baselines. As for R2D2 over-refusing, we did not use an off-the-shelf model, and we tuned the baseline to be strong (no apologies for that). There is always more that can be done, but we think that the experimental thoroughness of our work is a strength and not a weakness. For example, we know of no prior works that make progress on jailbreak refusal, backdoor removal, and unlearning at the same time. Meanwhile, we only know of one (Tamirisa et al., 2024) that makes progress on two of these at once. We also experiment with 3 different LLMs, 5 different datasets, 13 distinct training algorithms, and 21 distinct evaluation measures for performance/robustness/efficiency. Please consider this experimental breadth compared to related works in your final assessment.

W3: We sympathize with this, and we have discussed it amongst ourselves a few times. But we ultimately chose the naming convention that we did to reflect how RT, RT-EAT, and RT-EAT-LAT build off of each other in a way reflected by the names.

W4: Done. Thanks.

W5: Fixed. Thanks.

W6: We don’t include hypers in the paper, but we release code! And for what it’s worth, we worked hard to make the code useful. As we write this, we have 9 forks and 26 stars on GitHub. We also have over 700 model downloads and 1000 dataset downloads on HuggingFace. So we think that portability and reproducibility are empirically strengths of the project, not a weakness.

Q1: This is correct. We updated section 4.1 to describe EAT as layer-zero LAT.

Q2: We swept over hypers for all baseline methods and all LAT methods so that the deltas that we identify in our experiments are due to the method and not differential tuning. And we emphasize that we aren’t merely saying this – we took it seriously and didn’t cut corners. For example, we delayed the arXival of the paper because we found better baseline implementations of GA and RMU in section 4.3.2 than we had previously. Finally as for your question about hyperparameters, see our response to W6.

Q3: These three measures cover three key aspects of utility. MMLU measures knowledge through multiple choice question-answering. MT-Bench measures knowledge and fluency through open-ended question answering. Compliance measures, well, compliance. These three measures are fairly standard among comparable works (e.g., Zou et al., 2024)

Thanks for your response

W1: Its not necessary to explicitly mention embedding space attacks in my opinion but a general mention that similar adv. training approaches exist would be sufficient.

W2/Q3: While not the best quantitative evaluation the example looks convincing. I agree that the effort to make the research reproducible / accessible outweighs the one "missing" experiment and at the same time provides some of the evidence I was looking for.

IMO this is very unconventional, but I asked a colleague to verify some of the claims. Ultimately, the papers contribution is similar to the one in Xhonneux 2024. However, this paper provides experimental results on more diverse threats and explores the decision choice of attacking deeper latent layers. Both papers were apparently put on Arxiv at a similar time. On the one hand, I do not see why I should punish the authors for likely having bad luck in previous review processes. However, major concerns remain, such as trying to influence the reviewer's opinion by unverifiable claims that the work is popular among known industry labs (why should this be relevant in the first place. This will just put pressure on a reviewer) or the very rushed state of the initial paper.

I raised my score to 6 as my colleague confirmed some of the claims made by the authors.

Thanks, we appreciate the review and feedback! In this review cycle, to save everyone’s time and effort, we are opting for a radically honest and terse response.

W1: We don’t think that using particularly high-powered attacks are needed to prove the point we intend to prove. Here, our goal is to expand out the pareto frontier relative to SOTA baselines, which we do. While it is possible to achieve very high jailbreak successes on models with customized attacks, we know of no comparable jailbreak-robustness works that test against adaptive attacks like Andriushchenko et al. (2024). Are there any papers you'd like to point us to who make progress on jailbreaks or jailbreak defenses with significantly more rigorous evaluations?

W1: Evaluations can always be more elaborate, but we think that the experimental thoroughness of the paper is a strength, not a weakness. For example, we know of no prior works that make progress on jailbreak refusal, backdoor removal, and unlearning at the same time. Meanwhile, we only know of one (Tamirisa et al., 2024) that makes progress on two of these at once. We experiment with 3 different LLMs, 5 different datasets, 13 distinct training algorithms, and 21 distinct evaluation measures for performance/robustness/efficiency. We have also released code, 15 models, and 2 datasets alongside the paper’s public arXival. Please consider this experimental breadth compared to related works in your final assessment.

W1: For what it’s worth, we are an academic group, so it is nontrivial to work with increasingly large models. However, we know for certain that there is explicit interest in LAT at industry labs including Anthropic, FAR.AI, and GreySwan AI. We have recently met with one industry team who is consulting with us on scaling up LAT.

W2: What specific algorithms would you recommend we compare to? To the best of our knowledge, all fine-tuning-based backdoor cleansing methods from the literature involve either passive forgetting or training the model under reconstructed proxy triggers. In that sense, our experiments have methodological coverage. Finally though, we think that additional backdoor removal baselines are not needed to show what we intend to show. Our goal was to use LAT with many methods in a versatile way to help remove persistent failure modes. Altogether, we think that our results for jailbreaks, backdoors, and unlearning make a strong case for this.

W3: We would agree that a ~2% drop in HP performance would be small if the baseline were high. But we emphasize that the factor of improvement is still large when we drop performance e.g. from 0.042 to 0.027, or in the most dramatic case, from 0.116 to 0.039.

W3: Sections 4.3.1 and 4.3.2 differ by task. The first is focused on unlearning copyrighted text, and the second is focused on unlearning capabilities. We engage with different data and baseline methods in each which is useful for showing the versatility of LAT.

W4: We don’t include hypers in the paper, but we release code! And for what it’s worth, we worked hard to make the code useful. As we write this, we have 9 forks and 26 stars on GitHub. We also have over 700 model downloads and 1000 dataset downloads on HuggingFace. So we think that reproducibility and portability is empirically a strength of the paper, not a weakness.

Minor: Thanks for the notes on formatting. We have fixed these.

W1

We agree that strong attacks are good. But we emphasize that our LAT method won a fair fight against baselines. We think that the last comment seems to reflect a standard that no comparable works in the literature meet. For example, consider Tamirisa et al. (2024), Mazeika et al. (2024), Zou et al. (2024), or Xhonneux et al. (2024). None of these comparable papers seem to evaluate their attack/defense methods up to the standard described. We sympathize with your criticism, but we think that it is not one of the paper, but one of the field in general. We would ask you to take this into account in your final review. To count this against the paper, we think that it should be necessary to point to at least one specific example of comparable work on adversarial attacks or defenses that meets this evaluation standard.

Normally, we would be happy to continue our ongoing work on followup projects to this one. But if you think that adding very strong attacks into Table 2 would make the difference, we could revisit this and add a column for soft prompt attacks Schwinn et al. (2024a) which are very strong Schwinn et al. (2024b).

W3

If you have doubts about any claims related to the code or models we released, you could ask a colleague to anonymously verify them for you.

Naming

We sympathize with this point. But we are between a rock and a hard place. We used to mention "Targeted LAT" in the title and describe the contributions in terms of targeted LAT. However, we have gotten separate feedback on the paper against doing this because it caused past readers of our past draft to worry about the novelty of targetedness. Since our main contributions are about model performance, we opted to act on past advice to name the paper after just LAT. We do not think that this is a misnomer overall because "adversarial training" is already used in a general way to refer to targeted and/or untargeted approaches. Please take into account that we have gotten conflicting feedback on this in your final review. This is also a matter of wording and not technical contribution, so we think this point is relatively small.

Thanks! Notes below.

Why we mentioned Tamirisa et al.: Both papers make common use of MT-bench and MMLU. And our choice of these measures is based on how these are common. But us mentioning this was focused on these measures of general performance, so this may be of limited interest to you.

Why we mentioned Mazeika et al.: They use a lot of attacks but not adaptive ones like Andriushchenko et al.

For concreteness: I think we're interested in what attack or set of attacks you would say would be sufficient to add to table 2. And maybe it's something we could do! For example, soft prompt attacks Schwinn et al. (2024a) are very strong Schwinn et al. (2024b) and something we could add. RT-EAT vs RT-EAT-LAT could make for an interesting comparison. We could also add evaluate the models on attacks from these two files from Andriushchenko et al (2024). What do you think?

Thanks for the detailed response!

• W1, W2, W3, Q1, Q3: I appreciate your honest response, which has solved my concerns.

• Q2: It seems inappropriate to refer the reviewer to the arXiv version of this paper because it would violate the ICLR double-blind review policy. I acknowledge the contribution of this paper to the community. The experiments in this paper meet the standards of an ICLR paper. However, the question is: Could the author see more performance gain if they experimented with an "unprotected" model like Vicuna? It is more like a suggestion rather than a criticism.

Another minor suggestion is to be careful when you change your LaTex template style because Tabs 2, 6, and 7 overlap with the line numbers.

I have raised my score from 3 to 5. Thanks for your responses.

Thanks, we appreciate the review and feedback! In this review cycle, to save everyone’s time and effort, we are opting for a radically honest and terse response.

W1: Our work definitely is a form of adversarial training. But our claims to novelty and contributions do not hinge on the adversarial training aspect of LAT. Our goal and contributions center around improving on existing algorithms for refusal training, backdoor removal, and unlearning. We also stress that our comparison between LAT and R2D2 shows that our method can improve on adversarial training’s effectiveness and efficiency.

W2: There are two key differences between our work and Casper et al. (2024). First, we use a targeted attacker whose goal is to elicit a specific harmful behavior from the network. The attackers used by Casper et al. (2024) are untargeted and only seek to minimize training loss. Second, we use targeted LAT to improve over SOTA techniques for jailbreak robustness, backdoor removal, and unlearning. Casper et al. (2024) focus on proof-of-concept experiments instead of augmenting SOTA methods. In the appendix, Table 6 directly compares the two methods. It shows that our targeted LAT can greatly improve over the robustness of untargeted LAT from Casper et al. (2024). Meanwhile, Casper et al. (2024) argued that our targeted LAT approach would be a novel and useful way to expand on their work. The discussion section of their paper says: “performing LAT with targeted adversaries could be a way to make models highly robust to specific foreseeable failures”

W3: We think this is more of a criticism against the jailbreaking literature than our specific paper. We sympathize with this point. But we think that it does little to detract from the scientific contributions of the paper. We also note that some research has argued that web-scale data poisoning is practical (Carlini et al., 2023). Meanwhile, other research has worked on backdoors not because of practical threats but because they offer a challenging debugging problem for AI systems (Rando et al., 2024, Casper et al., 2024b).

Q1: Good observation. We use a different autograder that was designed to have a lower false-positive rate (see line 276). However, we also use the HarmBench autograder in Appendix Table 7 and find similar qualitative results as in the main paper.

Q2: We agree with the importance of working on many models. In followup work, we are running experiments on over 70 models. However, we think what we already have meets a higher standard of thoroughness than most comparable papers to. We experiment with 3 different LLMs, 5 different datasets, 13 distinct training algorithms, and 21 distinct evaluation measures for performance/robustness/efficiency. Please consider this experimental breadth compared to related works in your final assessment. We have also released code, 15 models, and 2 datasets alongside the paper’s public arXival. As we write this, we have 9 forks and 26 stars on GitHub. We also have over 700 model downloads and 1000 dataset downloads on HuggingFace suggesting a practical contribution to the community.

Q3: See also our response to W3. In this case, we think you are referring to our experiments with “proxy triggers,” right? If so, this is, by construction, a practical scenario. Consider the real-world challenge of removing a backdoor with an unknown trigger but a known response. For example, this was the setting that Rando et al. (2024) worked with. In this case, your first option could be to simply train the model to behave well on normal data until it forgot the backdoor. That was our default approach. Your second option could be to use some backdoor trigger reconstruction method to produce a “proxy trigger” and then use that to adversarially train the model. This was our “proxy trigger” approach. We used proxy triggers from Rando et al. (2024) that were constructed by a team in their competition who had no knowledge of the actual trigger at all. Overall, in both the default and proxy trigger approach, LAT improved the backdoor removal :)

We are very appreciate of the reply! Here are a few more notes on Q2:

Q2-A: We agree that it would violate double blindness for you to look at public materials we released. We definitely do not ask you to do this. But we can guarantee the details above about the public code and models are accurate. If you have concerns, we encourage you to ask a colleague to anonymously verify any details and confirm this with you.

Q2-B: In preliminary experiments, we did do jailbreak robustness experiments with another model: Zephyr-7b-beta. Back then, we were using a different experimental setup, and we were focusing on comparisons between R2D2 and LAT. However, the results were not qualitatively different from what we observed in our final experiments:

Our old Preliminary Results:

Overall, LAT outperforms R2D2 (qualitatively similar to what we found in the main paper). We ultimately did not keep working with Zephyr models, because, given the breadth and depth of experiments we conducted, we opted to focus on Llama3-8b models. To be honest, we think that adding experiments for a model like Zephyr would be good, but normally, we would probably just continue to prioritize on our followup research instead of revisiting this. However, if you think this would be very compelling and would make a big difference, we could commit to adding some rows for Xephyr or Vicuna to Table 2 before camera ready.

Thanks, we appreciate the review and feedback! In this review cycle, to save everyone’s time and effort, we are opting for a radically honest and terse response.

W1-W5 Evaluations can always be more elaborate, but we think that the experimental thoroughness of the paper is a strength, not a weakness, and surpasses a reasonable standard. For example, we know of no prior works that make progress on jailbreak refusal, backdoor removal, and unlearning at the same time — we only know of one (Tamirisa et al., 2024) that makes progress on two of these at once. We experiment with 3 different LLMs, 5 different datasets, 13 distinct training algorithms, and 21 distinct evaluation measures for performance/robustness/efficiency. Please consider this experimental breadth compared to related works in your final assessment. We have also released code, 15 models, and 2 datasets alongside the paper’s public arXival. As we write this, we have 9 forks and 26 stars on GitHub. We also have over 700 model downloads and 1000 dataset downloads on HuggingFace suggesting a practical contribution to the community.

W1: We don’t think that using particularly high-powered attacks would be needed to prove the point we intend to prove. Here, our goal is to expand out the Pareto frontier relative to SOTA baselines, which we do. While it is possible to achieve very high jailbreak successes on models with customized attacks, we know of no comparable jailbreak-robustness works that test against adaptive attacks like Andriushchenko et al. (2024). Are there any papers you'd like to point us to who make progress on jailbreaks or jailbreak defenses with significantly more rigorous evaluations?

W2: Unless we misunderstand your point, we think the paper already does what was mentioned here. We compared our LAT method to Xhonneux et al. (2024) in Table 2 and Casper et al. (2024) in Table 6 of the appendix. Meanwhile, we compare RT-EAT and RT-EAT-LAT on nine different measures. Our method wins on 7, ties on one, and loses on one. We think that this more than meets a reasonable burden of proof to argue that LAT is a valuable method.

W3: More elaborate evaluations would always be nice, but we followed existing methodology from prior work. Ultimately, we set out to beat baselines and succeeded in doing so, so we think that this meets a reasonable standard of evidence. See also the response to W1-W5 above.

W4: We agree this would be good, and we are currently working with llama3 models in follow-up work. But we do not think this would be a helpful use of time to revisit this work to add more experiments, especially since R2D2 is becoming less popular and has no public implementation tuned to llama3 models (to the best of our knowledge). There will always be more things that can be added. But holding a paper to doing all experiments with llama3 models (which were released mid-project) would be a somewhat artificial standard.

W5: What specific algorithms should we compare to? To the best of our knowledge, all fine-tuning-based backdoor cleansing methods from the literature involve either passive forgetting or training the model under reconstructed proxy triggers. In that sense, our experiments have methodological coverage. Finally though, we think that additional backdoor removal baselines are not needed to show what we intend to show. Our goal was to use LAT with many methods in a versatile way to help remove persistent failure modes. Altogether, we think that our results for jailbreaks, backdoors, and unlearning make a strong case for this.

W6: Yes, we find that WHP-C-LAT induces a general performance cost compared to WHP-C. But most robustness/unlearning/backdoor-removal methods induce similar costs. Pareto domination isn’t necessary to demonstrate usefulness. We have shown that LAT can push out the Pareto frontier.

W7: The perturbation norm that we used was just a hyperparameter. We swept over multiple configurations of hyperparameters for LAT runs and control runs alike.

On jailbreaking evaluation methodology

In EUWJ’s explanation of their final decision, they argued that the six jailbreak robustness evaluations we used were insufficient because the jailbreaks did not achieve near 100% success on the base Llama models as was shown by attacks in Andriuschenko et al. (2024). Our reply was that this was (1) irrelevant because our conclusions were based off of how we beat the baselines and (2) that other similar papers do not meet the standard outlined. We think this is an artificial standard and asked three separate times for EUWJ to point to a paper that meets this standard or to say what specific experiment we should run. In our final reply, we asked if they think we should add soft prompt attacks (Schwinn et al., 2024) or to evaluate on adversarial prompts available on Andriuschenko et al. (2024)’s GitHub. But we did not receive a reply before the end of discussions.

We ask the AC to take this into account when assessing EUWJ’s rating of 3. If this concern is decisive, we could commit to adding evaluations from soft prompt attacks Schwinn et al., 2024 or the adversarial prompts from Andriuschenko et al. (2024) before a camera ready version.

Finally, thanks again to all involved. We appreciate your volunteered time and ideas!