An LLM can Fool Itself: A Prompt-Based Adversarial Attack

Many thanks for your comments! Please find our replies below.

[Reply to Q1] No prior studies have studied how to ask the LLM to directly generate effective adversarial data via prompts. We are the first to propose a non-trivial attack prompt to effectively prompt the LLM to directly generate adversarial examples that can fool itself.

So far, prompt engineering and in-context learning (e.g., few-shot inference) have been widely applied to solve various optimization tasks [1,2,3,4], but they have not yet been applied to utilize the LLM to solve the task of generating adversarial examples. We are the first to design an effective attack prompt that can elicit the victim LLM to output adversarial examples that can fool itself.

Our design of the attack prompt template is non-trivial and reasonable. No prior studies have designed such attack prompts. We are the first to map the mathematical formulation of the conventional textual attacks (e.g., attack objective function and attack optimization method) into an attack prompt in a textual form. Empirical results validate the effectiveness of our attack prompt.

[Reply to Q2] We highlight our novelty and significant contributions as follows.

1. Technically, PromptAttack introduces a novel paradigm of adversarial attack that prompts the LLM to directly generate the adversarial examples in a black-box manner, which exempts complex optimization procedures.

   • Conventional adversarial textual attacks require humans to propose complex optimization procedures (e.g., calculating prediction logits in a white-box manner [5] and computationally-expansive back-propagations [6]) to solve the attack objectives. PromptAttack, without complex optimization procedures, asks the LLM to directly generate adversarial examples via an attack prompt in a black-box manner, which is efficient and simple. We believe we opened a new research direction in studying how to let the LLM directly generate strong adversarial examples via attack prompts.
   • No prior studies in in-context learning (e.g., few-shot inference) [1,2,3,4] have studied whether the LLM can solve the task of generating adversarial examples via prompts. We make the first effort to validate that the LLM can solve the task of generating adversarial examples via an attack prompt.
   • No prior studies in prompt engineering have designed such attack prompt templates. We are the first to propose an effective attack prompt template inspired by conventional adversarial textual attacks.

2. Empirically, PromptAttack is the state-of-the-art tool for evaluating the adversarial robustness of the (black-box) LLM in terms of effectiveness, adaptability, scalability, and efficiency.

   • Effectiveness. As shown in Table 2, PromptAttack achieves a significantly higher ASR compared to prior SOTA methods (AdvGLUE [8] and AdvGLUE++ [7]). It indicates that PromptAttack can discover more failure modes of existing (black-box) LLMs.
   • Adaptability. Compared to transferable adversarial examples used in AdvGLUE and AdvGLUE++ that are fixed once generated, PromptAttack can adaptively generate the adversarial examples against the latest LLM via the attack prompt. In this way, PromptAttack can provide a reliable evaluation.
   • Scalability. AdvGLUE and AdvGLUE++ use ensemble attacks which require white-box access to the LLM [5,6]. PromptAttack only requires querying the LLM in a black-box manner. Therefore, PromptAttack is applicable to various LLMs, even the black-box API (e.g., GPT-3.5).
   • Efficiency. PromptAttack, without expensive computation (e.g., back-propagation), only requires black-box queries for each data points to generate an adversarial variant. We find that PromptAttack against GPT-3.5 consumes about 2 seconds to generate an adversarial variant for each data point, which validates the efficiency of PromptAttack.

---

References

[1] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, NeurIPS 2022.

[2] language models are zero-shot reasoners, NeurIPS 2022.

[3] Large language models are human-level prompt engineers, ICLR 2023.

[4] Large Language Models as Optimizers, ICLR 2024 submission.

[5] Bert-attack: Adversarial attack against bert using bert, EAAI 2020.

[6] SemAttack: Natural Textual Attacks via Different Semantic Spaces, ACL 2022.

[7] Decodingtrust: A comprehensive assessment of trustworthiness in gpt models, ArXiv 2023.

[8] Adversarial glue: A multi-task benchmark for robustness evaluation of language models, NeurIPS (Dataset track) 2021.

Many thanks for your comments! Please find our replies below.

[Reply to W1] The task description is used for in-context learning, which illustrates the detailed objective of the optimization task in the form of natural language. We provide a detailed illustration in Appendix B.7 and provide examples of the task description for solving the SST-2 task in Table 20.

[Reply to W2] We demonstrate that PromptAttack is more computationally efficient in evaluating robustness of the LLM than AdvGLUE and AdvGLUE++.

PromptAttack against GPT-3.5 only requires querying GPT-3.5 without any GPU resources, which consumes about 2 seconds to generate the adversarial example for each data point. In contrast, AdvGLUE and AdvGLUE++ consume much more running time and GPU memories to generate transferable adversarial examples since they require complex optimization procedures (e.g., calculating the predicted logits [1] and do back-propagations [2]) in their ensemble attacks against an ensemble of language models. The computational consumption is estimated on RTX A5000 GPUs.

[Reply to W3] We provide the robustness evaluation of Palm 2 using PromptAttack in a black-box manner.

The results show that GPT-3.5 is more adversarially robust than Palm 2 under PromptAttack.

Note that Google Bard is powered by Palm 2. We do not provide the results of Bard and Claude under PromptAttack since Google Bard API and Claude API are available to only a limited number of users [11,12]. Although we can access Bard and Claude via the web page, it would be impractical for us to test inputs one by one on a web page without the API.

[Reply to Q1] In Table 3, we utilize the ensemble strategy. In Table 4, we do not utilize the ensemble strategy.

The ensemble strategy collects the adversarial examples generated by 9 different types of perturbation instructions together. Then, PromptAttack with the ensemble strategy (PromptAttack-EN) selects the adversarial example that can successfully fool the victim LLM as the output. In this way, PromptAttack-EN significantly improves the ASR.

[Reply to Q2.1] We explain why PromptAttack works from the perspective of in-context learning.

In-context learning [3,4,5,6,7,9,10] has been shown as a powerful method that asks an LLM to solve the optimization task via a well-designed prompt. The prompt describes the optimization task in the form of high-level natural language. Benefiting from the LLM's ability to understand natural language, the LLM can solve the optimization task [9].

For example, [7] lets the LLM directly generate a good task instruction according to the prompt that aims to find a task instruction that optimizes the task accuracy. [9] lets the LLM directly generate a good solution for the mathematical question according to the prompt that aims to find a solution that optimizes the accuracy.

Similarly, PromptAttack can be regarded as a novel type of in-context learning, which lets the LLM directly generate a new sentence (i.e., adversarial example) given the attack prompt that aims to find a sentence that fools itself. The LLM's powerful comprehension ability enables itself to solve the task of generating adversarial examples. It leaves future work to provide a deeper and theoretical understanding of why in-context learning works including [3,4,5,6,7,9,10] and our proposed PromptAttack.

[Reply to Q2.2] To the best of our knowledge, there is no defense against adversarial attacks in in-context learning. We try to utilize the superior in-context learning methods [4,5,6,7,10] and adversarial training (AT) [8] as defensive strategies and report the performance under the defense.

First, we consider the superior in-context learning methods including few-shot in-context learning [4], chain-of-thought (CoT) [5,6], and automatic prompt engineering (APE) [7,10] as the defensive strategies. The performance is evaluated on GPT-3.5 using the SST-2 dataset. The results show that the few-shot strategy is effective, but CoT and APE are not effective in defense.

Second, although AT [8] is effective in defending against adversarial attacks, we cannot directly apply AT to LLMs due to computational prohibition. Thus, we conducted AT on BERT and RoBERTa, which validated the effectiveness of AT in robustifying language models.

The experimental results are copied from Table 16 in Appendix B.6. We used adversarial examples generated by PromptAttack against GPT-3.5 to evaluate the ASR on standardly and adversarially trained language models.

---

[Update more results under possible defense]

Concurrent works [13,14] are particularly designed for defending against jailbreak attacks, which is not applicable to defending against conventional adversarial attacks. Another concurrent work [15] uses the perplexity (PPL) filter to filter out low-quality prompts, thus defending against possible adversarial prompts, which is applicable to defending against adversarial examples. Here, we provide the ASR with or without the PPL filter to show that the PPL filter [15] is effective in defending against adversarial examples. PromptAttack is still a strong adversarial attack under this defense [15].

---

References

[1] Bert-attack: Adversarial attack against bert using bert, EAAI 2020.

[2] SemAttack: Natural Textual Attacks via Different Semantic Spaces, ACL 2022.

[3] Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, ACM Computing Surveys, 2023.

[4] What Can Transformers Learn In-Context? A Case Study of Simple Function Classes, ArXiv 2022.

[5] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, NeurIPS 2022.

[6] language models are zero-shot reasoners, NeurIPS 2022

[7] Large language models are human-level prompt engineers, ICLR 2023.

[8] Freelb: Enhanced adversarial training for natural language understanding, ICLR 2019.

[9] Large Language Models as Optimizers, ICLR 2024 submission.

[10] Use Your INSTINCT: INSTruction optimization usIng Neural bandits Coupled with Transformers, ICLR 2024 submission.

[11] https://www.googlecloudcommunity.com/gc/AI-ML/Google-Bard-API/m-p/538517.

[12] https://www.anthropic.com/earlyaccess

[13] https://openreview.net/forum?id=xq7h9nfdY2

[14] https://openreview.net/forum?id=wNere1lelo

[15] https://openreview.net/forum?id=0VZP2Dr9KX

[Reply to Q3] To the best of our knowledge, there is no defense against adversarial attacks in in-context learning. We try to utilize the superior in-context learning methods [4,5,6,7,10] and adversarial training (AT) [8] as defensive strategies and report the performance under the defense.

First, we consider the in-context learning methods including few-shot in-context learning [4], chain-of-thought (CoT) [5,6], and automatic prompt engineering (APE) [7,10] as the defensive strategies. The performance is evaluated on GPT-3.5 using the SST-2 dataset. The results show that the few-shot strategy is effective, but CoT and APE are not effective in defense.

Second, although AT [8] is effective in defending against adversarial attacks, we cannot directly apply AT to LLMs due to computational prohibition. Thus, we conducted AT on BERT and RoBERTa, which validates the effectiveness of AT in robustifying language models.

The experimental results are copied from Table 16 in Appendix B.6. We used adversarial examples generated by PromptAttack against GPT-3.5 to evaluate the ASR on standardly and adversarially trained language models.

---

[Update more results under possible defense]

There are three concurrent works [13,14,15] that aim to defend against adversarial prompts. [13,14] particularly designed for defending against jailbreak attacks, which is not applicable to defending against conventional adversarial attacks. We utilized the defensive strategy in [15] which uses the perplexity (PPL) filter to filter out low-quality prompts, thus defending against possible adversarial prompts. Here, we provide the ASR with or without the PPL filter to show that the PPL filter [15] is effective in defending against adversarial examples. The results further validate that our PromptAttack is still a strong adversarial attack under this defensive strategy [15].

---

References

[1] Bert-attack: Adversarial attack against bert using bert, EAAI 2020.

[2] SemAttack: Natural Textual Attacks via Different Semantic Spaces, ACL 2022.

[3] Decodingtrust: A comprehensive assessment of trustworthiness in gpt models, ArXiv 2023.

[4] What Can Transformers Learn In-Context? A Case Study of Simple Function Classes, ArXiv 2022.

[5] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, NeurIPS 2022.

[6] language models are zero-shot reasoners, NeurIPS 2022

[7] Large language models are human-level prompt engineers, ICLR 2023.

[8] Freelb: Enhanced adversarial training for natural language understanding, ICLR 2019.

[9] Adversarial glue: A multi-task benchmark for robustness evaluation of language models, NeurIPS (Dataset track) 2021.

[10] Use Your INSTINCT: INSTruction optimization usIng Neural bandits Coupled with Transformers, ICLR 2024 submission.

[11] https://blog.research.google/2022/04/pathways-language-model-palm-scaling-to.html

[12] https://www.cnbc.com/2023/05/16/googles-palm-2-uses-nearly-five-times-more-text-data-than-predecessor.html

[13] https://openreview.net/forum?id=xq7h9nfdY2

[14] https://openreview.net/forum?id=wNere1lelo

[15] https://openreview.net/forum?id=0VZP2Dr9KX

Many thanks for your comments! Please find our replies below.

[Reply to Q1] We highlight our novelty and significant contributions as follows:

1. Technically, PromptAttack introduces a novel paradigm of adversarial attack that prompts the LLM to directly generate the adversarial examples in a black-box manner, which exempts complex optimization procedures.

   • Conventional adversarial textual attacks require humans to propose complex optimization procedures (e.g., calculating prediction logits in a white-box manner [1] and computationally expansive back-propagations [2]) to solve the attack objectives. PromptAttack, without complex optimization procedures, asks the LLM to directly generate adversarial examples via an attack prompt in a black-box manner, which is efficient and simple. We believe we opened a new research direction in studying how to let the LLM directly generate strong adversarial examples via attack prompts.
   • No prior studies in in-context learning (e.g., few-shot inference) [4,5,6,7,10] have studied whether the LLM can solve the task of generating adversarial examples via prompts. We make the first effort to validate that the LLM can solve the task of generating adversarial examples via an attack prompt.
   • No prior studies in prompt engineering have designed such attack prompt templates. We are the first to propose an effective attack prompt template inspired by conventional adversarial textual attacks.

2. Empirically, PromptAttack is the state-of-the-art tool for evaluating the adversarial robustness of the (black-box) LLM in terms of effectiveness, adaptability, scalability, and efficiency.

   • Effectiveness. As shown in Table 2, PromptAttack achieves a significantly higher ASR compared to prior SOTA methods (AdvGLUE [9] and AdvGLUE++ [3]). It indicates that PromptAttack can discover more failure modes of existing (black-box) LLMs.
   • Adaptability. Compared to transferable adversarial examples used in AdvGLUE and AdvGLUE++ that are fixed once generated, PromptAttack can adaptively generate the adversarial examples against the latest LLM via the attack prompt. In this way, PromptAttack can provide a reliable evaluation.
   • Scalability. AdvGLUE and AdvGLUE++ use ensemble attacks which require white-box access to the LLM [1,2]. PromptAttack only requires querying the LLM in a black-box manner. Therefore, PromptAttack is applicable to various LLMs, even the black-box API (e.g., GPT-3.5).
   • Efficiency. PromptAttack, without expensive computation (e.g., back-propagation), only requires black-box queries for each data point to generate an adversarial variant. We find that PromptAttack against GPT-3.5 consumes about 2 seconds to generate an adversarial variant for each data point, which validates the efficiency of PromptAttack.

[Reply to Q2] We provide an analysis on the scale of the LLM w.r.t. the attack performance and computational overhead as follows.

$^*$The number of parameters of GPT-3.5 is provided by this website, which could be inaccurate since OpenAI does not publish the accurate number of parameters of GPT-3.5.

1. As the scale of the LLM (measured as the number of model parameters) increases, the ASR of PromptAttack first increases and then decreases.

   The increase of ASR from Llama2-7B to Llama2-13B could be owing to that the LLM’s comprehension ability increases as the model becomes larger, which increases the quality of generated adversarial examples (see detailed analysis in Section 4.1). The decrease of ASR from Llama2-13B to GPT-3.5 could be owing to that GPT-3.5 has more parameters, which leads to stronger robustness.

2. PromptAttack is much more computationally efficient than AdvGLUE and AdvGLUE++. We show that AdvGLUE and AdvGLUE++ require much more computational resources to generate transferable adversarial examples in the following table.