MixAT: Combining Continuous and Discrete Adversarial Training for LLMs

We thank Reviewer $\textcolor{purple}{KJJ1}$ for their thorough review and positive assessment of our work! We are happy to hear that the reviewer finds our work and MixAT to be non-trivial, relevant, and an essential part of the LLM robustness toolbox. Below, we individually address the reviewer’s questions and comments.

Q16: The Abstract claim about the vulnerability of continuous attack training methods to discrete attacks seems overstated. Could the authors comment on this?

We thank the reviewer for pointing this out and agree that our original wording may have overstated the relationship between continuous relaxations and discrete attack vulnerabilities. Our intention was not to claim a direct causal link, but rather to highlight an empirical observation: in our experiments, models trained with continuous adversarial perturbations (e.g., in the embedding space) may still be susceptible to a diverse set of discrete attacks, even if individual attack success rates are low.

This was actually one of the initial motivations behind the introduction of the ALO-ASR metric, which captures the worst-case failure across a suite of attacks. We believe this complements existing robustness metrics and underscores the importance of evaluating model defences against diverse, realistic threats, even when the models show robustness against individual attacks.

We will revise the sentence in the abstract to better reflect this nuance and avoid implying a general claim, as follows:

Despite their effectiveness and generalization capabilities, training with continuous perturbations does not always capture the full spectrum of vulnerabilities exploited by discrete attacks.

We are happy to further discuss this with the reviewer in case they have particular ideas on how to best describe our observed phenomenon.

Q17: Would the authors consider discussing recent work from anthropic on constitutional classifiers, and recent work from Che et al. in the Related Work Section?

Certainly. We thank the reviewer for suggesting this and are happy to include an additional paragraph on model-augmenting (Guards, Classifiers, …) defenses against LLM jailbreaks as they become an increasingly important part of practical model deployments. Further, we are currently looking into incorporating the experiment from Che et al. in our own work.

Q18: Could the authors extend the caption of Fig 6 from App. to explain the acronyms?

Thank you for pointing this out! We will revise the caption to explain all acronyms in the updated version of the work. We will also make sure that the remaining figure captions contain full descriptions of the associated experiments.

Q19: Would the authors consider replacing Table 1 with heat-colored text or a heatmap?

Sure. We thank the reviewer for this great suggestion. We have considered various variants for the main table, including splitting, but for the initial version, we settled for this variant. Color-coding the values in the table might be a great way to increase readability, and we are currently experimenting with various ways to incorporate this in the next revision of the work.

Q20: Would the authors consider adding two more columns to Table 1 giving the mean delta for utility score over baseline and the mean delta for ASR below baseline?

We considered including an average Utility column in Table 1, but decided against it because the utility metrics vary widely in scale and sensitivity across different tasks and models. Averaging them could obscure important differences and lead to misleading conclusions (since the XSTest scores show the highest variance, the average would mostly follow the XSTest trends). Instead, we decided to first and foremost report individual scores to allow for a more accurate and nuanced comparison across attacks (allowing readers to select the metrics important for them - e.g., multiple choice answering vs text generation performance vs safety-specific utility benchmarks). However, we can consider changing utility scores with deltas from the baseline as this might look more informative, though the issue with averaging the deltas would be the same (XSTest would still have the highest variance).

On the other hand, for a unified robustness metric we have proposed the At Least One ASR (ALO-ASR) instead of Average ASR. ALO-ASR provides a more realistic view than just averaging the scores obtained by different attacks, for two reasons: 1. ALO-ASR better reflects the real threat model where a malicious actor can try multiple attack techniques for each sample of text, until obtaining at least one harmful output. 2. ALO-ASR provides a more realistic weighting (we no longer over-emphasize easier samples where methods constantly succeed) across different harmful requests with varying hardness, resulting in a clearer model comparison.

We hope that adding color-coded meaning to the tables will help visualise both the individual results and the overall (averaged) trends. However, if the reviewer considers this would improve the understanding of our claims and results, we are happy to include the requested Average Utility and Average ASR scores in the updated manuscript.

We thank Reviewer $\textcolor{blue}{S3HQ}$ for their review and suggestions. We are happy to hear Reviewer $\textcolor{blue}{S3HQ}$ finds our problem well motivated and our corresponding evaluation comprehensive. Below, we individually address the reviewer’s outstanding comments.

Q12: Is MixAT technically novel? It seems that it is only a combination of existing methods.

Yes, we believe MixAT is technically novel. While our method builds upon existing works and ideas, we want to emphasize that MixAT does not represent a trivial combination of continuous and discrete adversarial training (which we evaluate as a separate baseline (DualAT) in our work), a point also raised by Reviewer $\textcolor{purple}{KJJ1}$. Instead, MixAT proposes joint discrete and continuous attacks (See Figure 1 in the paper), significantly widening the attack domain and providing a natural extension over approaches that so far have only been used separately in adversarial LLM training. Notably, while prior work in adversarial training in other domains, such as image classification, has shown that combining different attack strategies during training improves the generalization of the defense capabilities of the network, MixAT is the first work to propose this for adversarial LLM training, achieving state-of-the-art results.

Q13: Could you elaborate on why you compared "MIXAT + GCG" to other baselines that weren’t trained with GCG attacks?

Sure. We want to clarify that MixAT + GCG is not our main result but rather an ablation study intended to explore the impact of training with a more diverse set of attacks. Importantly, R2D2 is also trained with GCG samples, so including MixAT + GCG in the comparison helps contextualize performance across methods that leverage similar data. Our end goal is to build models that generalize to a wide range of attacks, and this ablation highlights that increasing attack diversity during training can be an effective step toward that goal.

Q14: Could the authors provide some experiments for varying the hyper-parameters of the continuous PGD attack?

Yes. Below, we show how scaling the $\epsilon$ (0.5x, 2x), the number of steps (0.5x, 2x), and the step-size (0.5x,2.0x) of the continuous PGD attacks used in MixAT affects the robustness and performance of the trained models. Note that for a fair comparison we scaled the step-size along with the perturbation budget $\epsilon$ and we also scaled the step-size inversely with respect to the number of steps.

Overall, we observe consistent results on the utility benchmarks with only minor and to be expected deviations from the original values. For robustness, this trend continues, however, we find that generally the chosen tradeoff of original MixAT performs on the top end of all tested configurations, with especially high-step low learning rate experiencing a stark drop in ALO robustness. We plan to conduct further hyperparameter experiments here to analyze the trends better, and we will include them in the next revision of the paper.

Q15: Could the authors include an average Utility or average ASR column in Table 1?

We considered including an average Utility column in Table 1, but decided against it because the utility metrics vary widely in scale and sensitivity across different tasks and models. Averaging them could obscure important differences and lead to misleading conclusions (since the XSTest scores show the highest variance, the average would mostly follow the XSTest trends - e.g., multiple choice answering vs text generation performance vs safety-specific utility benchmarks). Instead, we only report individual scores to allow for a more accurate and nuanced comparison across attacks.

On the other hand, for a unified robustness metric our work proposes the At Least One ASR (ALO-ASR) instead of Average ASR. ALO-ASR provides a more realistic view than just averaging the scores obtained by different attacks, for two reasons: 1. ALO-ASR better reflects the real threat model where a malicious actor can try multiple attack techniques for each sample of text, until obtaining at least one harmful output. 2. ALO-ASR provides a more realistic weighting (we no longer over-emphasize easier samples where methods constantly succeed) across different harmful requests with varying hardness, resulting in a clearer model comparison.

If the reviewer believes that average Utility and average ASR would improve the understanding of our claims and results, we are happy to include the requested scores in the updated manuscript.

We thank Reviewer $\textcolor{green}{WP5s}$ for the thorough review. We are happy to hear Reviewer $\textcolor{green}{WP5s}$ finds our work contributes substantially to the body of knowledge around adversarial training of LLMs, and that they find our experimental evaluation convincing and comprehensive. Below, we address the reviewer’s outstanding questions.

Q7: In the Static MixAT experiment, is the Continuous attack part the only one that is made static? Could the authors provide some intuition on why training with static continuous attacks work? Is the token embedding layer frozen during fine-tuning with LoRA adapters?

Yes, indeed in the Static MixAT experiment, the PAP-generated attacks are already model-agnostic, so it only makes sense to consider the Continuous part of the attacks as being static. Our intuition is that even when training with static continuous adversarial examples generated for the base model, these samples still have an adversarial value for the model that is being trained, mostly because our model is being fine-tuned from a pretrained model and therefore it is not substantially different from the base model. Informally, at least for the first few training steps, the difference between the model trained with statically vs dynamically generated adversarial examples should not be meaningful. As the training progresses, the two models should diverge and the statically generated attacks should get less potent, as the model starts overfitting them. However, our fine-tuning procedure is very short compared to the full pre-training of the base model, and therefore, we hypothesise that the model does not have enough time to drift significantly from the dynamically trained model. This is in contrast with most works in adversarial training for Computer Vision applications, where the models are adversarially trained from a random initialization for which training with static attacks would be meaningless.

Finally, the token embedding layer is indeed frozen during LoRA training, keeping the continuous embedding attacks aligned with the model. We note that, it is also possible to freeze the token embedding layer even when fine-tuning without LoRA.

Q8: Would training MixAT + GCG on statically generated GCG suffixes still show improvements? Would this reduce the computational time?

Thank you for the interesting question! In theory, one would expect that training MixAT + GCG on statically generated GCG suffixes should still show improvements. Prompted by the reviewer's suggestion, we performed this experiment and show the results in the tables below.

Our results show that incorporating static GCG samples into MixAT training indeed shows similar benefits with the dynamically generated ones.

However, note that statically generating all the GCG samples for the whole training set takes 33.5 hours on H200 GPU for Llama 3 8B and 36.6 hours for Zephyr, which makes this singular experiment significantly more expensive than our dynamic MixAT + GCG training. On the other hand, once we have generated the static GCG samples, we can run multiple training runs incorporating them. See, for example, Q1 of Reviewer $\textcolor{orange}{zGMj}$, where we trained a variant of MixAT that only uses GCG samples and Continuous Attacks.

Q9: How do you evaluate a response as harmful or not? Do you use one of the classifiers that comes with HarmBench?

Yes. Our evaluation uses the cais/HarmBench-Llama-2-13b-cls model released alongside HarmBench on HuggingFace to determine which responses are toxic. We will clarify this in the updated manuscript.

Q10: Could the authors provide any hypotheses for why model quantization slightly increases robustness?

Yes. We think that model quantization creates sharper ‘decision boundaries’ between refusal and harmful responses. This might cause slightly harder optimization problems when searching for adversarial examples. On the other hand, quantized models also have slightly lower utility scores, which might impact the model’s ability to correctly output harmful responses. We find this question very interesting and inspiring, but a thorough analysis on the impact of quantization on model robustness is outside the scope of our work.

Q11: Should we put more weight on XSTest than generic benchmarks that do not test the safety capabilities of the model?

We think this is a highly relevant question. Correctly evaluating LLMs in general, and robust LLMs in particular, is still a very open research problem. We agree that XSTest provides a very meaningful signal for assessing a model’s ability to distinguish between real and fake harmful requests, and we found it very useful in our evaluations. However, XSTest is essentially a binary classification task, where any non-refusal answer is scored, regardless of its quality, which may not capture the full complexity of model behavior. We believe it’s also important to ensure that the model retains its ability to generate correct and relevant answers, which is why we also consider more established benchmarks in general LLM evaluation. Ultimately, we hope that our quite extensive evaluation can provide important pointers on how LLM robustness evaluations should develop.

We thank Reviewer $\textcolor{orange}{zGMj}$ for the insightful review and are happy to hear that they found our work well motivated and effective, as well as our experimental results convincing. Below, we individually address the reviewer’s questions and comments.

Q1: Could the authors compare the PAP-MixAT with a variant that uses another discrete attack as seeds for the continuous attacks? Is MIXAT's success tied specifically to PAP or to the general principle of discrete seeding?

We thank the reviewer for the interesting question. Prompted by the reviewer's response, we additionally train Zephyr and Llama3 MixAT models with statically generated GCG discrete attacks as seeds instead of PAP (using dynamically generated GCG samples would make the training too long and expensive). We present the results below:

Our results demonstrate that using GCG samples as seeds for the continuous attacks can improve the model robustness over the CAT baseline, but it becomes clear that the lower diversity of the GCG samples is detrimental to the generalization of robustness properties compared to rephrasing attacks such as PAP, TAP, and PAIR. In our MixAT + GCG ablation, we have shown that combining PAP, GCG, and Continuous attacks can further improve robustness, so we hypothesize that the diversity of the attacks used for training is crucial for wide robustness. On the other hand, the comparison between MixAT and our baseline DualAT shows that the stronger mixed (discrete + continuous) attacks are better for training than simply combining the two training losses in a multi-objective manner. We will include this discussion in the next revision of the paper.

Q2: Could the authors provide some experiments for varying the hyper-parameters of the continuous PGD attack?

Yes. Below, we show how scaling the $\epsilon$ (0.5x, 2x), the number of steps (0.5x, 2x), and the step-size (0.5x,2.0x) of the continuous PGD attacks used in MixAT affects the robustness and performance of the trained models. Note that for a fair comparison we scaled the step-size along with the perturbation budget $\epsilon$ and we also scaled the step-size inversely with respect to the number of steps.

Overall, we observe consistent results on the utility benchmarks with only minor and to be expected deviations from the original values. For robustness, this trend continues, however, we find that generally the chosen tradeoff of original MixAT performs on the top end of all tested configurations, with especially high-step low learning rate experiencing a stark drop in ALO robustness. We plan to conduct further hyperparameter experiments here to analyze the trends better, and we will include them in the next revision of the paper.

Q3: Could the authors test MIXAT's robustness against fundamentally different attack modalities, such as those based on role-playing, complex scenarios, not directly related to paraphrasing?

Certainly. First we would like to emphasize that rephrasing attacks like TAP and PAP often automatically come up with role-playing scenarios or scenarios where the toxic action request is justified with a particular urgency, necessity, or cultural appropriateness (See Appendix B in [1] for example). Such scenarios are also very common in datasets like HumanJailbreaks, which we also experimented with in Table 1.

Based on the reviewer’s suggestion and to show the generalization of our defense to completely different types of attacks (i.e., neither classical adversarial examples (GCG) nor paraphrasing/role-playing jailbreaks (TAP, PAP, HumanJailbreak)), we conducted two additional experiments - one using code attacks from the QueryAttack dataset [2] on Qwen 2.5 14B models; and one using ASCII art-based attacks from the ArtPrompt dataset [3] on Llama 3 8B models. We chose the model families per attack in such a way that the attacks were particularly potent on the base models to allow us to compare the defense methods meaningfully. We present our results in the tables below:

QueryAttack:

ArtPrompt:

For both experiments, we consider an attack successful if their supplied LLM Judges scored the LLM response with toxicity of 3 out of 5 or higher. For ArtPrompt, we report the results on their best-performing font (vitc-h-gen), while for the QueryAttack, we report the ALO-ASR on the full set of their 9 code modalities. Importantly, we find both MixAT and MixAT+GCG extremely robust to these attacks, not producing even a single toxic answer, unlike CAT and LAT. Further, we see that, in general, the robustness from GCG attacks and role-playing attacks seems to transfer successfully to these settings as well. We will include these results in our revised manuscript.

[1] Mehrotra, et al. "Tree of attacks: Jailbreaking black-box llms automatically.", Neurips 2024
[2] Zou, et al. "QueryAttack: Jailbreaking Aligned Large Language Models Using Structured Non-natural Query Language.", arXiv 2025
[3] Jiang, et al. "Artprompt: Ascii art-based jailbreak attacks against aligned llms.", ACL 2024

Q4: Could the authors elaborate on the intuition for why the combined perturbation space (R(X) + $\delta$) is so much more effective than the union of the two spaces used in DUALAT?

The reviewer makes an interesting observation. Intuitively, we found two ways to interpret this phenomenon. First, we note that naturally the space of attacks covered by MixAT is much larger and, unlike the DualAT combination, includes less redundancy (where continuous and discrete give similar signals). Second, and perhaps more importantly, it is natural to assume that especially points around actual (discrete) jailbreak attacks give an informative signal w.r.t. model robustness. That is to say, by applying continuous attacks around a benign data point, we mostly train the model to behave well on inputs where the model already performs well. By shifting this to points that lead to abnormal model behavior, we expect a much stronger signal (in the continuous space around the discrete point) while having to spend less effort optimizing the continuous perturbation. We will include this discussion in the next revision of the paper.

Q5: Was the time-cost increase of MixAT + GCG training primarily due to the GCG sample generation, or did the training on these combined embeddings also converge more slowly?

Yes. The increased costs of MixAT + GCG are primarily due to the costs of generating GCG samples during training. We keep the same training schedule (number of steps, number of epochs) for all our MixAT experiments, including MixAT + GCG.

Q6: Could the authors explain why MixAT's training regimen is less detrimental to model utility compared to CAT?

Sure. One limitation of CAT is that it relies only on the original set of harmful samples for training, which can be a relatively low-diversity dataset. As a result, the model may start overfitting to superficial patterns in those prompts, such as specific words or phrasing, without fully learning to recognize harmful intent. This often leads to overly cautious behavior, including refusals on benign prompts, as seen in the drop in utility scores and performance on XSTest. In contrast, MixAT includes discrete PAP-style samples during training, which introduces more diverse and realistic adversarial examples, helping the model generalize better and strike a stronger balance between robustness and utility.