Efficient Adversarial Training in LLMs with Continuous Attacks

We thank the reviewer for their insightful comments and address each point/question in turn:

W1: The authors should conduct a controlled experiment to measure runtime differences between the approaches:

A1: This is a valid concern and we address it by measuring the wall time it takes to run $15$ steps of the algorithms in both cases, extrapolating from there also the total experiment time, which would be prohibitive to run for us. The wall times are in the ballpark as our forward pass comparison:

On an A100 a single step of the R2D2 took the 489.7 longer than a single step of C-AdvIPO. Extrapolating this results to the total number of step considered for training R2D2, its complete training would have taken 1991 times longer with the implementation provided in the repository.

W2: Important notation and methods are not always introduced at the right point in the paper:

A2: Thank you for pointing this out! We will improve the writing further, for instance by better defining the term IPO earlier in the manuscript and in more detail.

W3: The hardware descriptions are imprecise:

A3: Roughly 800 hours were on 80GB A100, the remainder were on the 40GB GPUs. Only 11.6 hours were used on V100 GPUs, which were only used for debugging.

W4: Llama2 should be added as a baseline:

A4: We thank the reviewer for the suggestion as Llama2 is a popular model. We have trained Llama2 with the CAT loss successfully, which considerably improves the robustness of the model with minor degradations in utility. See the overall response and Figure 2 of the attached PDF for the results.

W5: More defenses should be included in the evaluation:

A5: We’d be happy to include any defenses the reviewer finds relevant. Could you point us to which defenses you would like to see?

In the meantime, let us discuss some that come to mind to us and why we didn’t include them: Many of the other defenses mentioned are orthogonal to our paper, such as perplexity filters on the input or toxicity filters on the output. Other papers such as Rainbow Teaming are both orthogonal and not reproducible as the authors do not share code/data/models necessary to do so (Rainbow teaming for instance use a Meta-internal helpful-only Llama model). Other works only apply to Bert style models on don’t focus on generation. We also want to note that it is standard practice to compare adversarial training approaches with each other as the vast majority of empirical approaches have been broken by third-party evaluations, and adversarial training remains one of the only exceptions [4].

(4) Tramer et al., “On Adaptive Attacks to Adversarial Example Defenses" NeurIPS, 2020

We thank the reviewer for their insightful comments and address each point/question in turn:

W1: R2D2 should be evaluated on more models.

A1: Unfortunately, Zephyr R2D2 is the only model available trained with R2D2 and training more base models with R2D2 is prohibitively expensive (see Table 1 in the paper and the overall response). We have thus focused on showing that our method is able to achieve a good robustness/utility trade-off and compared it where we could to R2D2, outperforming it.

W1.1: The authors should test their approach on more diverse safe answers :

A1.1: We explored generating diverse safety answers to increase the utility robustness trade-off in preliminary experiments but did not see any improvements. The common hypothesis is that a model is considered safe as long as the model provides any safe answer to a harmful query [Zou et al., 2023]. As diverse toward answers did not improve our results, we opted for the simpler approach. Using multiple away targets (harmful affirmative responses) for every query appeared to be more impactful, which is why we trained with 12 different possible away responses $\hat{y}$ for every given query. Note that the safety response is never used during evaluation and is just a training objective in the algorithm.

W1.2: Stronger attacks should be used for evaluation:

A1.2: We thank the reviewer for the remark. These attacks were not in the original submission because they are contemporary to our submission (NeurIPS guidelines suggest that any paper published on arXiv less than 2 months before the submission deadline should be considered contemporary work). However, we believe the reviewer suggestion to include these new attacks is a fantastic opportunity to showcase the width of the robustness of our model. We added results for the simple adaptive attack and an ICL attack for as many models as we could run during the rebuttal and will complete the remainder for the final paper. Both attacks are highly effective against the base models but cannot break the adversarial trained versions even when using $10000$ attack iterations and random restarts. We thank the reviewer for their suggestion, which we believe further strengthens our method’s empirical performance. Please see the attached PDF Table 2 and overall response for the results.

W1.3: Ablation studies on the choice of the preference optimisation algorithm should be conducted:

A1.3: As the reviewer points out, several preference optimization (PO) losses could be used in our adversarial training approach. We conducted preliminary experiments using DPO and found that IPO leads to less overfitting to the toward response (where the model rejects every request). A full ablation of all the preference losses feels to us a bit outside of the scope of this paper whose key contribution is to show that continuous robustness can robustify the models towards discrete attacks. However, we will surely add our results for DPO in the camera ready version.

W2: The proposed methods introduce hyperparameters that need to be tuned:

A2: It is true our method introduces a few more hyper-parameters (epsilon, attack iterations, beta), we believe tuning them is worth the improved robustness-utility trade-off. We acknowledge this limitation in the paper and also point out that the main baseline (R2D2) suffers from the same issue, with GCG having many more additional hyper-parameters to tune. Morever, it is common for adversarial training algorithms to require at least tuning of the epsilon and attack iteration parameters. Most methods, such as TRADES, additionally include a regularization parameter to control the robustness accuracy trade-off (beta in our case).

Lastly, due to the cost of fine-tuning LLMs we were not able to conduct more than a few training runs for all models, which was sufficient to make the method work. Thus, we conclude that our approach is reasonably robust to hyperparameter choices.

W3: The presentation of the paper can be improved

A3: We thank the reviewer for their observation and added a description of \mathcal{L}' (the actual loss used for optimization after applying the cutoff) to the paper. We also reviewed every equation and made sure all the variables are defined.

Q1: Why do bigger models require smaller values of $\epsilon$?

Q-A1: We thank the reviewer for pointing this out and are happy to provide a hypothesis for this observation. We argue that larger models have been trained for longer, which leads to token embeddings that contain more information. Thus, smaller perturbations have a larger effect on the model. To investigate this theory, we conduct a continuous $\epsilon$-ball attack ($\epsilon = 0.05$) on the Phi and Gemma base models. Even though the initial robustness of the Phi model is larger, its robustness decreases substantially faster under attack. These findings seem to support our intuition. See the attached PDF for loss curves. We will add these experiments to the final paper.

Q2: Is it possible to add a regularization parameter to the adv. training algorithm like in TRADES?

Q-A2: Thanks for the nice question. Controlling the trade-off between robustness and utility is indeed crucial. For C-AdvIPO we ablate how robustness and utility depend on the $\beta$ parameter in Figure 3a of the paper. We demonstrate that small $\beta$ values increase robustness and decrease utility, which can be used to control the trade-off. Similarly, the trade-off can be controlled for C-AdvUL by choosing different weightings for the toward, away, and utility loss. Here a lower utility loss positively impacts the robustness while decreasing utility. We included the weighting parameters in Table 2 of Appendix A but did not clearly define them in the paper. We apologize for this oversight and will include the parameters in Equation (4) of the camera-ready version.

Thank you for your reply and feedback.

We chose to evaluate our method with a subset of the strongest attack available at the point of submission (according to Harmbench [1]). Beyond the ICL attack and the adaptive attack that we performed for the rebuttal, we will conduct additional evaluations with the approaches proposed in [2, 3] for the final manuscript. These attacks have shown high ASR against robust models.

We will add our ablation study on the behavior of $\epsilon$ for models of different sizes to the final manuscript. We are open to any further suggestions on ablation studies that would further improve our work.

We will add a more profound discussion about the differences between previous approaches in encoder-decoder models and the presented adversarial training algorithm. More specifically, we will:

• Elaborate on the different training settings, i.e., post-training stage for LLMs and pre-training in encoder-decoder models
• Differences in the optimization target, i.e., robustness in classification settings vs. preference optimization or alignment finetuning
• Differences in the optimization goal, i.e., improved generalization vs. adversarial robust alignment.

We want to emphasize that robustness in classification problems differs substantially from adversarial robust alignment. In the classification setting, models are trained to be robust to minor input perturbations with respect to their prediction. In contrast, in the alignment setting, we are not concerned with the stability of predictions. Rather, we train the model to refrain from generating "toxic" outputs altogether. To achieve this, losses need to be adjusted for this task (such as combining different objectives or using preference optimization algorithms). None of the previous methods was designed for LLM alignment and they can not be employed for this task without further changes.

Lastly, the majority of these approaches were used during the pre-training stage of encoder-decoder models. Applying these algorithms during the pre-training stage of an LLM exceeds our computational budget by orders of magnitude.

[1] Zou et al., "HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal" (Feb. 2024)

[2] Thompson T and Sklar M., "Fluent Student-Teacher Redteaming" (July 2024)

[3] Liao et al., "AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs" (May 2024)

We thank the reviewer for their insightful comments and address each point/question in turn:

W1: Is adversarial training suitable to robustify LLMs in the context of diverse threats?

A1: Thanks for initiating this discussion. We agree that the perturbations set in LLMs are much less constrained than in Computer Vision. However, both past work in computer vision [1, 2] and existing work in LLMs [3] indicate that variations of latent adversarial training can improve robustness against diverse threats. Our experimental results demonstrate that latent space adversarial training extrapolates robustness well beyond the continuous attack we train on, such as jailbreaking attempts and suffix attacks. Moreover, adversarial training is one of the only methods that stood the test of time and delivered reliable robustness improvements. In contrast, the majority of heuristic preprocessing approaches and other techniques were later broken. Overall, we don’t claim that AT can solve the problem entirely, but it presents a piece of the puzzle in robustifying the current generations of LLMs and is, therefore, worth studying. We are happy to engage in further discussions regarding this point.

W2: Could the authors include stronger attacks in their evaluation?

A2: Thanks for the references. These attacks were not in the original submission because they are contemporary to our submission (NeurIPS guidelines suggest that any paper published on arXiv less than 2 months before the submission deadline should be considered contemporary work). However, we believe the reviewer suggestion to include these new attacks is a fantastic opportunity to showcase the width of the robustness of our model. We agree that a thorough evaluation is crucial in this domain. We have added two attacks. One adaptive attack (as proposed by TJPv, which has shown very strong results on open source and proprietary models) and ICL. Our models increase the robustness against both attacks considerably compared to the base models, with most adversarially trained models achieving 100% robustness. See the PDF Table 2 and the overall response for the full results.

W3: A sanity check should be performed with continuous embedding space attacks to explore the limits of the robustness of the models:

A3: We agree with the reviewer that this is an important sanity check and have added this as an experiment. Against unbounded attacks, all models exhibit 0% robustness; against epsilon-ball attacks, the adversarial trained models show higher robustness than the base model (see also PDF and general comment).

W4: The connection of adversarial training to machine unlearning should be highlighted:

A4: We thank the reviewer for the great suggestion! Machine unlearning in the face of adversarial attacks is indeed closely related to the setting we consider here, we will amend the related work to highlight this. However, while the unlearning literature has proposed many new losses, such as NPO, they alone do not provide robustness to adversarial attacks (see the paper appendix B.2 Table 6 and attached PDF Table 1). We do believe that future work may look at how our adversarial training method may used in machine unlearning as well, but a full evaluation of this is beyond the scope of this paper.

Q1 Why are there no experiments done on the Llama series of models?:

Q-A1: No particular reason except that it’s utility performance is worse than the models we considered, but we have added LLama2 with our CAT method, which considerably increases the robustness of Llama2 (see the overall response and attached PDF Figure 2).

(1) Casper et al., “Defending Against Unforeseen Failure Modes with Latent Adversarial Training” 2024

(2) Laidlaw et al., "Perceptual Adversarial Robustness: Defense Against Unseen Threat Models.” ICLR, 2021

(3) Dai et al., "Formulating robustness against unforeseen attacks." NeurIPS, 2022

We thank the reviewer for their quick response and for increasing their score!

We agree that further attack evaluations would improve our work and will look into adding decomposition attacks to our work. We welcome any further feedback to improve our paper and are thankful for your feedback.