Other defenses + gen setting

Thank you for the reframing suggestion. We indeed focus on classification (see below and WX3L rebuttal). We believe that our overall points will hold (1: model scale has limited impact on robustness by itself, but improves safety training, and 2: the relative effectiveness of attack and defense compute is more important than point estimates), but the overall offense/defense balance may change for different attacks/defenses or in the generative setting.

Having analyzed classification tasks, we wanted to sanity check at least one generative task. Our Qwen2.5 StrongREJECT results are consistent with point 1, as larger Qwen2.5 models appear to benefit more from safety training. This said, since Qwen2.5 does not have a helpful-only variant, there could be confounding factors (see WX3L rebuttal).

We only intended to make strong claims about adversarial training (and to a lesser extent, perplexity filtering, which is bypassed by BEAST) in classification, and will try to make our focus more clear in the paper. If there is anywhere in particular you think would benefit from attention, please let us know and we will address it.

More safety tuned LLMs

Thanks for the comment. Note that our StrongREJECT experiment is in fact on Qwen2.5 Instruct, which underwent Harmlessness and Debiasing training as part of RLHF [1].

As you mention, the bulk of our analysis was on finetuned base models which have not undergone safety training. This was intentional. For one, our result that “while model scale on its own only slowly improves robustness, model scale enhances safety training” was only observable because we started from models without safety training. See WX3L rebuttal for more discussion.

[1] Yang et al., "Qwen2.5 technical report", 2024, page 6-7

Effect of different robustification approaches

This is a good question. While we chose to implement a fairly simple adversarial training method, it is likely that different algorithms would give somewhat different results.

More than the specifics of our approach, we tried to emphasize the importance of offense/defense balance of when studying robustification algorithms: ASR goes up with attack compute and down with defense compute, and so the relative strengths of these changes are much more informative than any point estimate of ASR.

Classification vs HarmBench

Two main reasons for classification:

1. It enables us to systematically study trends in adversarial robustness over 3 orders of magnitude of model sizes. We find LLMs first become competent at key generative tasks like chat dialog at ~1B params, which would have greatly limited the range of model sizes for studying variation in robustness.
2. It allows us to unambiguously tell if an attack was successful or not. In the generative setting, evaluation is less clear. Historical attack evaluations focus on keyword/phrase matching, which does not guarantee bad behavior. LLM-as-a-judge (also used in HarmBench) improves on this but is not perfect: in early experiments, we often disagreed with the LLM judge (more details in WX3L rebuttal).

Finally, we wanted to run a large number of experiments/seeds. This would have been significantly more expensive in the generative setting due to needing to use larger models for generation, plus the judgement cost.

All this said, we believe the generative setting is a natural direction for future work.

No Llama/Gemma

Thank you for this question.

Analytically

• Using Pythia enables apples-to-apples comparisons between different models to isolate the effects of scale on its own, as opposed to scale + different amounts of data, safety training, etc.
• We can only observe the separate effects of model scale and model scale + safety training because we start with models that have not been safety trained

Practically

• Gemma 3 only ranges from 1B–27B in model size, only varying by 1.5 orders of magnitude
• Llama requires efficiency techniques to be practical for larger scales. Using quantization or LoRA would undermine the comparability between models at different scales
• Starting at the 7.6M parameter scale and going bigger lets us afford experiments with multiple seeds over 3 OOMs of model sizes

No BERT

Thank you for this question. Our Pythia and Qwen2.5 classifiers had high accuracy, so we did not see the need to improve classifier strength. We think studying encoder models could be interesting future work.

Different attacks/defenses

Interesting question! It depends which results you are talking about.

We expect that the finding that scale alone confers limited robustness but boosts safety training will hold.

The offense/defense slopes could significantly change for different attack/defense combinations (exciting future work). Indeed, a central goal of the paper is to promote a scaling lens. It is by plotting these slopes that one could show a future defense technique to be more efficient than any current attack (or vice versa)!

Focus on classification

Thank you for this point, it is well-taken. We decided to focus on classification tasks for two main reasons:

1. it allowed us to study 3 orders of magnitude of model sizes—about 1.5 orders of magnitude more than we could have in the generative setting (since in our experience generative models only get reasonably good at the 1B size)
2. it gave an unambiguous signal for attack success or failure (much better than token matching, see [1], and more reliable than llm-as-a-judge [2], which in our experience often gave assessments of attack success/failure that we didn’t agree with)

The generative setting brings with it other challenges too: in order to do the tasks in StrongREJECT or HarmBench, we need to use in-context learning or to prompt an Instruct model. Is the model's performance then thanks to its safety training, or other parts of its Instruct training? Finally, Instruct models have often undergone training that we don’t know details of, making the FLOP-relative analysis and comparison across model sizes much more difficult than in the Pythia (and to some extent, Qwen2.5 base) classification setting.

Having decided to focus on classification, we tried to do a spread of relevant tasks. Spam and IMDB, while easy tasks, are still real-world-relevant. Also, Helpful and Harmless are quite challenging. If you know of any frontier classification tasks in NLP, we would be grateful to hear about them.

[1] Zhu et al., AdvPrefix: An Objective for Nuanced LLM Jailbreaks, 2024

[2] Raina et al., Is LLM-as-a-judge robust?..., 2024

Already established claims

Thank you for pointing this out. Like you, we did expect to see strong-to-weak transfer, and confirmed that it occurs across all model sizes.

Going further, we unexpectedly only found weak-to-strong transfer for small models. We also investigated changing the threat model, showing that large models generalize better from suffix attacks to infix attacks than smaller ones. This underscores our conviction that investigating robustness across model scales is crucial, rather than looking at a single point estimate.

That said, we could move Figure 5 to the appendix and use the space for more core plots, as you suggest below. Would you agree with this approach? Also, we would be grateful if you could please let us know if there are other results in the main body you think would best be moved to the appendix to make more space.

Offense-defense balance

Thank you for engaging so attentively with our plots!

• We agree with the “1 order of magnitude” point. Eyeballing the extrapolation in Figure 1, to reach attack-defense parity (y = x + 0) we would likely need to go 4-6 orders of magnitude larger in model size, which is beyond the current largest known models. In your mind, would making this “impracticality for current models” more clear in the paper (perhaps even in the abstract) address your concern? We are trying in this paper to make it clear we aren’t trying to give a “practical” suggestion for how to make models more robust, but rather exploring the trends (and promoting a scaling trend perspective more broadly, which we believe is important for evaluating attacks and defenses in general).
• The choice of 2% exactly was somewhat arbitrary (1% or 3% would have worked too—we can include those plots in the appendix if you are interested), but it was constrained on both sides. We needed a value large enough such that the measurement of how much compute it takes to reach that attack success rate is not dominated by noise (eg, if we had put 0.1% as the ASR threshold, then only a few datapoints would need to be successfully attacked in order to reach that ASR, and there would be more noise in the curve). On the other side, it needed to be small enough that we eventually reach that ASR across all model sizes, even after the models have undergone many rounds of adversarial training. Because the models eventually become quite robust with adversarial training, going larger than 2% would mean we start missing datapoints because there is no amount of attack compute in the regime studied that achieves a 2% ASR (indeed, even 2% is not reached by some models in Figure 1, which you can see by the lack of error bars around some models/datapoints).
• We did every analysis on Spam and Harmless for Pythia and Qwen2.5, and more tasks where possible (See Appendix, especially D). We notice we are missing the Pythia Harmless Offense-Defense plot and will add it to App D (apologies!).

Focus on core analysis

Thanks for this interesting suggestion. We could cut Figure 5 or 6 (or both) and replace with core analysis plots. Is there any plot you’d be particularly happy to see in the main body? Perhaps parts of Figure 33 and 35 (to show the sample-efficiency vs flop-efficiency distinction)? Alternatively, we could include more of the attack scaling or adversarial training plots.

Other defenses besides adversarial training?

Great question, this is exactly the kind of future study that we would like to explore! Indeed, one of our core points is that, because ASR goes up with attack compute and down with defense compute, looking at a single point estimate can be deceiving. Instead, we propose the scaling lens of focusing on offense/defense balance as you suggest.

Input filtering

We implemented a perplexity filter, though we only mention it in passing (line 247), using it to check that our BEAST implementation defeats it.

It would be interesting to study scaling properties of model-based filters like LlamaGuard [1] more directly, and would be relevant in the generative setting [2]. In contrast, using a perplexity filter does not lend itself to a study of scaling.

Rewriting and retokenization

Like perplexity filtering, rewriting and retokenization are one-off increases in cost, where there is no way to spend more compute to make the defense stronger. In the same way that BEAST defeats perplexity filtering, we believe that future algorithms could likely circumvent these defenses. We also suspect these defenses are unpalatable to frontier model developers due to possible performance degradation.

Therefore, we focused on scaling trends for a fixed set of attacks and defenses where compute can be smoothly scaled, in contrast to the cat-and-mouse setting of novel filters and filter circumventions.

Test-time scaling

This is an interesting question and has recently been explored in [3], where the authors find that increasing test-time compute reliably improves robustness in a generative setting.

Developing more effective methods and not duplicating work

We would recommend the following:

• Do adversarial training in latent space [4], which is much more compute-efficient than in token space
• Use LLM-based input and output filters [1, 2]

[1] https://ai.meta.com/research/publications/llama-guard-llm-based-input-output-safeguard-for-human-ai-conversations, 2023

[2] https://www.anthropic.com/news/constitutional-classifiers, 2025

[3] https://openai.com/index/trading-inference-time-compute-for-adversarial-robustness, 2025

[4] Casper et al., Defending against unforeseen failure modes with latent adversarial training, 2024

[Minor] The paper does not introduce new technical ideas. However, I still think it is valuable.

We agree on both fronts :). One possible small technical contribution is setting up the mildly intricate adversarial training pipeline (details in Appendix D2) but this is a minor contribution at most. We will also make the code public for others to use.

[Minor] Interpretation of landscape?

Thank you for raising this. Having read the paper so many times, it’s hard for us to catch issues like this, so this is very helpful feedback. We will go over the intro and related work sections to improve flow and add interpretation.

Is adversarial training actually scalable?

Great question, we will attempt to explain this better and give a takeaway message in the paper.

We find that adversarial training is scalable in the sense that holding the attack, threat model, and attack compute budget fixed, additional adversarial training reliably improves robustness. However, in our studied settings, the offense/defense slope is <1 across model sizes, so the defender will lose to the attacker if both scale proportionally. In our case, the defender would need to develop a more efficient defense algorithm to move the slope >1 if they want to outpace the attacker.

Larger models generalize better to different threat models, so additional pretraining compute mostly shows up in improved generalization from adversarial training.

Future offense/defense slopes will be different

Thank you for this. We agree that future attacks and defenses will lead to different slopes (and intercepts). The point we most want to emphasize in the offense/defense section is how the scaling lens is necessary to understand how an attack or defense will perform, vs only looking at a point-estimate which might become irrelevant with a different allocation of compute to offense and defense. We will make this more clear in the paper.

Only StrongREJECT for gen

Thank you for this point, we will include a note in the paper of why we focused on classification and other generative settings were not tried.

We focus on classification to study a wider range of model sizes, for unambiguous ASR measurement, and due to other challenges with evaluating generative models (see WX3L rebuttal for more details).

We settled on StrongREJECT rather than HarmBench because it fit more naturally into our model evaluation pipeline. Since the two benchmarks share similarities and our paper focused on classification, we decided not to test on HarmBench too. This said, we would be interested in future studies focusing fully on the generative setting!

Lacks 70B+ models, no GPT-class or DeepSeek

Thank you for these points, they are well-taken.

Model Scale: Computational limitations constrained our study:

1. The adversarial training defense studied in the paper is very compute intensive, already taking thousands of H100 hours to complete for all seeds and model sizes.
2. To finetune 70B+ models we would need to use LoRA and quantization, which would make the comparison with smaller models less apples-to-apples. Are ASR differences from model scale, or because of LoRA or quantization?

We believe that our core claims are well-substantiated by our experiments as they stand:

1. Model scale alone confers limited robustness, but model scale improves adversarial training
2. ASR goes up/down with attack/defense compute, so point estimates of robustness are less meaningful than offense/defense balance

Model families: We used Pythia for an apples-to-apples comparison of ASR for model sizes across 3 OOMs. With other model families, different sizes use different training procedures, making it hard to tell if the difference is from using more or less data, different amounts of safety training, etc.

We agree that a study on frontier models would be valuable, but it would require a different regime of computational resources than what we have access to in order to control appropriately for the factors mentioned above.

Lack of generative tasks

This point is also well-taken!

Our decision to focus on classification tasks was intentional:

1. It allows us to study robustness over 3 orders of magnitude for high performing models, since classification models achieve high performance with small model sizes (~14M parameters). In the generative setting, it is difficult to get good performance with <1B parameters, leaving us with 3-4 models spanning ~1 OOM before running into comparability issues arising from LoRA, quantization, etc., and undermining our ability to identify longer trends.
2. Classification provides an unambiguous attack success rate, since we know the label. In the generative setting, the standard approach is LLM as a Judge which is not guaranteed to agree with a human evaluator [2], and often disagreed with our judgments.

Our Qwen2.5 StrongREJECT sanity-check for the generative setting in Figure 2 is consistent with our hypothesis that model scale enhances safety training. For the initial study, we wanted to control for as many variables as we could and find trends over as wide a spread of sizes as possible. We believe a follow-up study squarely focused on the generative setting would be worth doing.

[2] Raina et al., Is LLM-as-a-judge robust? Investigating universal adversarial attacks on zero-shot LLM assessment, 2024.

Technical details of adv training

We agree it would be helpful to have a more formal overview of the algorithm in the main body (vs Appendix D2). We will also add information about the different adversarial training hyperparameters we tried.

• We wanted to make Algorithm 1 as readable as possible, but could make it more formal if you think that would be of value
• What do you mean by “perturbation magnitude”? All of our attacks are adversarial suffix (or infix, or prefix), so there isn’t such a clear notion of deviation from the original datapoint (unless you’re talking about number of attack iterations or suffix length?)

Model editing vs. adversarial training

Thank you for bringing this defense to our attention.

We chose to focus on adversarial training because it is an industry standard and gives us a natural way of scaling defense compute (we also implemented a perplexity filter to check BEAST can bypass it). Model editing does not have a compute scaling component so does not naturally fit into the same kind of study.

We are not claiming that adversarial training is the best defense method, but rather that model scale on its own confers limited robustness, but model scale enhances adversarial training. In follow-up work, exploring model editing with a wider set of defense techniques would be interesting.

Practical efficiency improvements

Relevant question! We would recommend:

• More efficient adversarial training using attacks in latent space [3, 4]
• Input-output filtering [5]

We can add this to the conclusion if you think that would be valuable.

[3] Schwinn et al., Soft prompt threats…, NeurIPS 2024

[4] Casper et al., Defending against unforeseen failure modes with latent adversarial training, 2024

[5] Inan et al., Llama guard: Llm-based input-output safeguard for human-ai conversations, 2023

Scaling for backdoor poisoning?

Great question! Indeed they can, see [6]. Not only is it a successful attack, it becomes more effective against larger models. This suggests that (a) careful data curation and (b) strong safeguards over finetuning APIs are crucial for frontier models.

[6] Bowen et al., Data Poisoning in LLMs: Jailbreak-Tuning and Scaling Laws, 2024.