Evaluating Adversarial Defense in the Era of Large Language Models

1. The attack is constrained to be (a series of) single character perturbations -- in otherwords standard looking typos, but the average spellcheckers are not mentioned. I understand that open source models like Llama don't include preprocessing and that we largely don't know what proprietary systems like ChatGPT do with user input (in the way of preprocessing). But in the vein of the attacks and defenses considered in this work I think a standard spell-check or grammar editor is a reasonable baseline (see more in the questions below).

2. The threat model is confusing to me in this context. I think the authors defined the terms "white-box" and "black-box" and use them consistently, but what setting would a defense be needed where the defender knows the type of attack, but does not have access to the weights of the model? (Better phrasing in the questions below.) Overall, I think the threat model is contrived.

3. The conclusions are foggy. I understand that in works that consider a range of methods toward the same end -- in this case defense against typos -- that sometimes the conclusion is that some methods work in some situations and others work better in other situations. But I didn't draw the same conclusion as the authors describe in the Conclusion and Discussion section. Firstly, they seem to write off white-box defenses as impractical, and while this claim has merit it seems to odd to introduce the white box method after the black box only to back pedal and say these aren't practical. Also, I'd like some clarity on the threat model so I can better understand when the black box methods are, in fact more practical.

4. The attack may actually change the labels. The authors don't adequately address how often the attack changes the semantic meaning of the input. This is a critical step in this project since some of the downstream tasks are related to semantic meaning. Historically, adversarial robustness work discusses "clean label" attacks or defines "adversarial" as imperceptible to humans or as something that doesn't change the semantic meaning, and enforces this constraint with some norm (l-inf is common in vision). In this work, how do we know the perturbations aren't changing the semantics?

I don't think the following weakness keeps me from accepting this work, but it is the cause of my Presentation score above, which asks to consider the contextual place among related work. I do think this is a weakness and I think the authors should consider addressing some of the more recent work on LLM robustness in the related work section. The particular related work referenced below is only a small subset of existing work on LLM robustness and it does generally consider a slightly different adversarial objective -- but I still feel that without mentioning the work from the past summer this paper is lacking.

1. This paper feels out-of-date. The details in the reviewer guide say that work published on or after May 28, 2023 does not need to be compared to but the authors may reference or discuss it. In this instance -- in this outrageously fast paced field I think we have a particularly difficult situation. The title of this work "Evaluating Adversarial Defense in the Era of Large Language Models" and the work on the topic that came out over this past summer make it feel to me like it is quite behind the times. To be clear, I'm not critical of the authors -- I routinely find my own work in the same category -- but as a reviewer, I am hesitant.
   1. Universal and Transferable Adversarial Attacks on Aligned Language Models by Zhou et al. 2023 Presents a far stronger attack than typo-style manipulations and it has been cited 45 times (according to Google Scholar at the time of writing this review).
   2. Follow up work on that attack method that proposes defenses to it has also come out.
      1. Baseline Defenses for Adversarial Attacks Against Aligned Language Models by Jain et al. 2023 has been cited 14 times (according to Semantic Scholar at the time of writing this review). This work looks similar to the work at the focus of this review as it highlights several defenses and adaptive attacks (like the 'strong' attack).
      2. Certifying LLM Safety Against Adversarial Prompting by Kumar et al. 2023 proposes certifiable defenses and has also been cited already.

1. While typing errors are prevalent in real-life, they represent a limited form of adversarial attack with limited impact. As a result, the transferability and generalizability of the proposed defenses are poor.

2. The key concern about the paper is the lack of novelty. Novelty appears limited to combining existing approaches, the authors should more clearly describe their specific novel contributions.

3. In the white-box scenario, direct adversarial training of LLMs is costly and suffers from catastrophic forgetting. The cost of adversarial training is also unclear.

Major Weaknesses:

• The robustness of the proposed defences is not evaluated properly. It seems likely that an adaptive attack can fool the model. For instance, in the case of using a smaller model to correct the text with wrong spelling, the attacker could try to generate an attack such that the original black box model as well as the smaller model can be fooled simultaneously. Similarly in the case of self defense, the attacker can try to fool the model by appending the text with the appended prompt. Therefore, the authors should consider a more realistic scenario and then try to evaluate their defence by considering these conditions.

• Similarly in the case of white box attacks, there is no proper justification to believe why crafting an adaptive attack is not possible. It has already been shown that fine-tuning doesn't change the mechanisms of the model, though it might change the model's behavior[1]. As a result it is unexpected that fine-tuning would make the model robust. This robustness might be seen only on the fine tuning distribution of samples and against weak attacks.

Minor weaknesses:

• I feel that the scope of the problem addressed is a bit narrow and the authors should try to investigate a more general version of adversarial attacks rather than only investigating the attacks which aim to misspell the sentences.

• It would be nice if the authors try to focus on one defence method and provide a comprehensive analysis on it, instead of proposing multiple defences. This will help in improving the focus of the paper.

[1] Qi, Xiangyu et al. “Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!” ArXiv abs/2310.03693 (2023)

• I feel the overall contribution might be limited. The safety threat of character-level word substitution has been found and studied long before. The defense techniques are also not novel. In light of this, the submission is like expanding the study breadth in LLMs. Moreover, the study is particularly focused on character substitution attacks. In natural language domain, the robustness threat comes from much broader aspects. Even for input perturbation attacks, character substitution is just a specific type. There are several others like word substitution, word removal, word insertion, sentence paraphrasing, etc, c.f., AdvGLUE. The conclusions from one threat model may not be able to improve our standing on the general topic of robustness in LLMs.