Robustness Over Time: Understanding Adversarial Examples’ Effectiveness on Longitudinal Versions of Large Language Models

Thank you for your elaborate review. In the following, your comments are first stated and then followed by our point-by-point responses.

My primary concern with this paper is its limited scope in terms of technical innovation.

We believe we introduced a new evaluation methodology, i.e., adversarial robustness should be evaluated over time. The other three reviewers highly appreciate this new angle of research. Developing new attacks/benchmarks is of general interest but would not solve (and is not relevant to) the problem that LLMs are vulnerable even to existing attacks in existing benchmarks. We also believe that uncovering the LLMs’ vulnerability to existing (simple) attacks poses more severe threats than newly designed (advanced) attacks.

The benchmarks employed are commonly used, potentially even in the training of the GPT models under scrutiny.

This is a good point, and we will add a related discussion in the paper based on the following. In the LLM era, an LLM's training data may (partially and inadvertently) include the evaluation/test data. However, we point out that our evaluation data is not included in the training data of both GPT and LLaMA models because:

• Regarding GPT-3.5, the GPT-3.5 training data is up to Sep 2021 according to OpenAI's website (see https://platform.openai.com/docs/models/gpt-3-5). The AdvGLUE dataset, however, was published in Nov 2021, and the PromptBench dataset was published in 2023. Therefore, It is safe to conclude that there is no possibility of training the GPT-3.5 model using these datasets.
• Regarding LLaMA-1/2, their paper (https://arxiv.org/abs/2302.13971) states they used the GitHub dataset with Apache, BSD, and MIT licenses. However, the AdvGLUE's license is CC BY-SA 4.0. Thus, Meta cannot choose ADVGLUE to train the model. In addition, PromptBench is a brand new dataset generated using different LLMs from 5 months ago, one month before the LLaMA-2 was released. Therefore, it is very unlikely for Meta to use this dataset for training the LLaMA-2 model. More generally, adversarial data are much less likely than clean data to be included in any natural dataset because they are specifically optimized. Our finding that LLMs are vulnerable to our adversarial data also suggests that they are not used for training.

The objective behind updating from GPT-3.5 v0301 to GPT-3.5 v0613 may not exclusively target robustness enhancement. Thus, expecting substantial improvements in robustness may not be reasonable.

We agree that robustness is not exclusively important, and we do not claim that. However, the security of LLMs is an emerging research topic, and achieving adversarial robustness is a key requirement. For example, OpenAI has established the preparedness team to build a robust framework for monitoring, evaluation, prediction, and protection against the dangerous capabilities of frontier AI systems [4]. Following this research trend, we provide new insights that updated LLMs are vulnerable even to existing attacks.

[4]. https://openai.com/blog/frontier-risk-and-preparedness

Thanks for your constructive comments and suggestions; they are exceedingly helpful in improving our paper.

I don't think introducing robustness evaluation over time is a significant novelty.

We advocate for a distinctive perspective in measuring LLM performance by examining the robustness of models updated over time. Note that LLMs are publicly available for millions of users. Robustness is, therefore, an important aspect for owners of LLMs since minor user input errors could result in unforeseen consequences. Unfortunately, our empirical findings reveal a prevailing oversight in addressing this concern. We acknowledge that the LLaMA framework incorporates robustness considerations within its training data. Nonetheless, our results indicate that their evaluations may overestimate the robustness of the updated model since even existing attacks considered in our work perform better on the updated model. Therefore, there remains a necessity for a holistic evaluation against adversarial attacks of the LLMs across various longitudinal versions.

I assume the proposed benchmark will be used for future models

It is an interesting point to consider the future compatibility of our evaluation methodology. This is indeed why we focus on the methodology of over-time robustness evaluation independent of any dataset. Specifically, we indeed adopt existing datasets in our work. Following the same methodology, in the future, we could also choose other (existing) datasets that do not overlap with the training data, e.g., the dataset with a CC BY-SA 4.0 license or constructed by ourselves. It is a generally very trending topic to construct or select non-overlap datasets for LLM evaluation nowadays [5, 6]. We would like to discuss this topic further in the revised version.

I am not sure what level of improvement we should anticipate. If the model is updated for function calling, what should we expect about the robustness?

We thank the reviewer for pointing out the function calling service. This service provides a promising angle to incorporate our new finding that a new version of the model may not be more robust than its older counterpart. Specifically, based on our evaluation results, the users can choose the most robust version via function calling instead of sticking to the newest version that is less robust. On the other hand, for platforms that do not provide the function calling service, we would still expect that the new model would be more robust than the old model. In sum, we will provide the specific context when we say expected robustness and especially discuss the application of function calling.

[5].Zhou, Kun, et al. "Don't Make Your LLM an Evaluation Benchmark Cheater." arXiv preprint arXiv:2311.01964 (2023)

[6]. https://twitter.com/keirp1/status/1724518513874739618

Thanks for your constructive comments and suggestions; they are exceedingly helpful in improving our paper.

The authors should investigate and provide insights into the robustness of these models in the context of different parameter sizes.

Thanks for the valuable suggestions. Our current research is focused on the time dimension since we consider it the most straightforward dimension for comparing the robustness of two closely related models. We would like to consider the suggested extensions for future work, i.e., more subject matter domains, dimensions, and model parameter settings.

The authors should elaborate more on the process of generating adversarial examples by different surrogate language models.

We have added more details on how to generate the adversarial samples by different surrogate language models in Appendix.

Results of LLaMA-30B

This is indeed a typo, and we have removed it from the paper.

Could the authors provide an explanation for the underlying reason that caused the LLMs not to exhibit heightened robustness over time? Additionally, could they discuss potential strategies to address this issue?

The training details of LLMs are generally not available to the public. However, a simple observation is that the model trainer tends to consider model accuracy over robustness, partially because achieving adversarial robustness requires considerable additional effort. We have added the content to suggest the model trainer to use enhancing adversarial robustness methods in model upgrades in our discussion.

Have the authors shared their findings with OpenAI or Meta to report this issue?

We have not done it yet, and we would like to do it once the review process is finished.

Thank you for your valuable feedback and pointing out the many positive aspects of our work. Below, we’ll address the negative aspects you mentioned.

Enhancing Adversarial Robustness in Model Upgrades.

This is a good point. We have discussed the strategies in the paper to suggest the model trainer to use enhancing adversarial robustness methods in model upgrades.

Considering New Features in Model Evolution.

This is a promising direction for our future work. We have added the content about the new feature of GPT-3.5 in the paper. For the browse with bing of ChatGPT, it may be expected that new features are also vulnerable to existing attacks and may also introduce new attack surfaces that make LLMs even more vulnerable.

Thanks for your encouraging words and constructive comments. We sincerely appreciate your time reading the paper, and our point-to-point responses to your comments are below.

Hope the author can provide a more detailed description of "zero-shot ICL learning" and "few-shot ICL learning" in Figure 2.

If the reviewer means the description of the two terms, we will further clarify that we follow [1,2,3] on this terminology. Zero-shot learning means that the query includes only the description and the question but without any demonstrations, while few-shot learning means that the query also includes a few demonstrations. We have added the explanation in the caption of Figure 2. If the reviewer means the description component in the query, please see the detailed definitions in Section 2.1.

[1]. Yongqin Xian et al. Zero-Shot Learning - A Comprehensive Evaluation of the Good, the Bad and the Ugly. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.

[2]. Tom B. Brown et al. Language Models are Few-Shot Learners. In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS, 2020.

[3]. Jason Wei et al. Finetuned Language Models are Zero-Shot Learners. In International Conference on Learning Representations (ICLR), 2022.

What does the second column "Adversarial Query" in Table 1 mean? Please clarify.

Adversarial Query refers to the query that contains the adversarial content in any of its three components (description, question, and demonstrations), as defined in Eq. (1) and Eq. (2). Specifically, each component can be clean (denoted as C) or adversarial (denoted as A). Therefore, in Table 1, the adversarial query AC means a zero-shot learning-based query that consists of an adversarial description and a clean question, and AAC means a few-shot learning-based query that consists of an adversarial description, an adversarial question, and clean demonstrations. We will describe it more clearly in Section 5.1.

"For instance, on the SST2 dataset, when applying BertAttack (Li et al., 2020) to create the adversarial description, the Robust Test Scores (see Section 4.3) for both versions of GPT-3.5 are almost 0." Which table or figure is being described specifically? Please clarify.

Sorry, we indeed made a mistake here. It should be ''For instance, on the SST2 dataset, the average result of Robust Test Scores of zero-shot learning for both versions of GPT-3.5 dropped from 85.093% and 87.390% to 37.210% and 20.652%, respectively (see Figure 3).''