Automatically Eliciting Toxic Outputs from Pre-trained Language Models

Thanks for your constructive comments and suggestions! We hope our response resolves your concerns.

Weakness1

As pre-trained language models advance in many tasks, addressing safety concerns becomes increasingly necessary and imperative. Our research explores the potential risk of publicly available language models and critically assesses their vulnerability. We believe that AI research always comes with social and ethical issues, and our work might contribute to the enhancement of language model safety in the future. We have supplemented an ethnical statement section in our paper. We really appreciate the suggestion!

We have carried out experiments on another sentence dataset and the results are reported in the response to all reviewers.

Weakness2

As pre-trained language models are seeing increasing use across a wide variety of applications, a lot of problems in language models have been discovered and studied, e.g. toxicity, hallucination, bias, etc. Due to the different sources of these problems in language models, it is very difficult to study them under the same framework. Most existing researches also view these problems as different vulnerabilities in language models and study them separately. In this paper, we mainly focus on the toxicity and security risks of language models. Bias like gender and racial discrimination is also included in some toxic expression.

We have tried to automatically construct prompts to generate statements that violate common sense such as examples in [1] without changing parameters. However, all existing algorithms do not work well in this task. The possible reason is that toxic outputs of language models output are due to the toxic examples in the training data [2], while misinformation and hallucinations can be caused by architecture flaw and suboptimal training objective [3]. Therefore, most current researches divide these potential risks into different types for study.

As discussed in Section 5.2 in our paper, the success rate of the attack is strongly correlated with the perplexity of the target output. In other words, our method can achieve a good performance if the target text has high fluency on the language model.

We think it is meaningful to propose a framework that can elicit different kinds of potential risks at the same time. We will further study this problem in future work.

Weakness3

Following established experimental settings of adversarial prompts [4], we construct test dataset containing 1/2/3 tokens from Civil Comments Dataset. The adaptation of the tokenization method for the test data is performed to align it with the requirements of various language models. As longer output text requires a longer prompt to elicit, relatively short test data help reduce the computational cost of the optimization. To better alleviate your concerns, we have carried out experiments on a new sentence dataset to alleviate your concerns. The experimental results are reported in the response to all reviewers.

[1]：Meng, Kevin, et al. "Locating and editing factual associations in GPT." Advances in Neural Information Processing Systems 35 (2022): 17359-17372.

[2]：Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633–2650, 2021.

[3]：Huang, Lei, et al. "A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions." arXiv preprint arXiv:2311.05232 (2023).

[4]：Jones, Erik, et al. "Automatically Auditing Large Language Models via Discrete Optimization." arXiv preprint arXiv:2303.04381 (2023).

Thanks for your constructive comments and suggestions! We hope our response resolves your concerns.

Weakness1

We adopt DPP to model the negative correlations between quality and diversity in beam selection. Calculating the determinant of the kernel matrix precisely requires a high computational cost, previous algorithm that provide exact implementation has O($M^4$) complexity, where $M$ denotes the total number of candidate items [1]. However, the approximate greedy algorithm used in our implementation has O($N^2M$) complexity when selecting $N$ items out of $M$ [2]. Therefore, actually in ASRA, the running time of DPP selection procedure is extremely short and almost negligible. We have calculated the average running time in our datasets. One iteration takes around 0.732s on GPT-J and 0.834s on LLaMA in total, while the DPP algorithm only takes 0.002s on these language models. We provide the average running time of one iteration in the algorithm and the corresponding throughput of different methods on LLaMA in the following table:

Weakness2

Given any length of toxic text, our algorithm will search for prompt to elicit it from the language model. Obviously, longer toxic text is harder to elicit from language models, thus requires a sequence with more tokens to be optimized as the prompt. As prompt length increases, all adversarial algorithms face a higher computational cost.

Following established experimental settings of adversarial prompts [3], we construct test dataset containing 1/2/3 tokens from Civil Comments Dataset. The adaptation of the tokenization method for the test data is performed to align it with the requirements of various language models. As longer output text requires a longer prompt to elicit, relatively short test data help reduce the computational cost of the optimization.

To better alleviate your concerns, we have carried out experiments on a new sentence dataset to alleviate your concerns. The experimental results are reported in the response to all reviewers.

Question

DPP model is used to model the negative correlation between quality and diversity to avoid redundant search in the algorithm. This property of algorithm is very important in solving the task. We make attempts on a famous algorithm for unsupervised summarization, Textrank [4]. The success rate of the algorithm on LLaMA is provided as follows.

The algorithm which mainly models similarity between different candidates to select items does not work well in this task.

[1]：J. Gillenwater, A. Kulesza, and B. Taskar. Near-optimal MAP inference for determinantal point processes. In Proceedings of NIPS 2012, pages 2735–2743, 2012.

[2]：Laming Chen, Guoxin Zhang, and Eric Zhou. Fast greedy map inference for determinantal point process to improve recommendation diversity. Advances in Neural Information Processing Systems, 31, 2018

[3]：Jones, Erik, et al. "Automatically Auditing Large Language Models via Discrete Optimization." arXiv preprint arXiv:2303.04381 (2023).

[4]：Mihalcea, Rada, and Paul Tarau. "Textrank: Bringing order into text." Proceedings of the 2004 conference on empirical methods in natural language processing. 2004.

Thanks for your constructive comments and suggestions! We hope our response resolves your concerns.

Weakness1

Following established experimental settings of adversarial prompts [1], we construct test dataset containing 1/2/3 tokens from Civil Comments Dataset. The adaptation of the tokenization method for the test data is performed to align it with the requirements of various language models. As longer output text requires a longer prompt to elicit, relatively short test data help reduce the computational cost of the optimization.

We have carried out experiments on a new sentence dataset to alleviate your concerns. The experimental results are reported in the response to all reviewers.

Weakness2

By introducing a BERT-based toxicity classifier, we compute the unigram toxicity score of each token in the vocabulary. We force the model not to generate tokens with relatively high toxicity scores, containing around 3, 000 tokens. We call this the toxicity constraint. The performance of different algorithms with and without the toxicity constraint is illustrated in the table below. We perform experiments on GPT-2 and GPT-J.

As illustrated in the table, the toxicity constraint doesn't have a large impact on the performance of our algorithm.

Weakness3

ASRA calculates an approximate probability for each token in the vocabulary in step 1. The approximate value is computed with the average first-order approximation value. As we discuss in Section 3 in our paper, the approximation depends on $t$ random tokens and can be computed efficiently with one gradient back propagation and matrix multiplication between the gradient matrix and the embedding table matrix. The matrix multiplication has a complexity $O(ld_{e}|\mathcal{V}|)$, where $l$ denotes the sequence length, $d_e$ denotes the dimension of word embedding and $|\mathcal{V}|$ denotes the size of the vocabulary. We have recorded the computational time of this process in each iteration. The approximation step occupies 0.62s of in the experiments on GPT-2 with a vocabulary size of 50,257. Therefore, although the time consumption increases as the vocabulary size becomes larger, the algorithm can still be applied in practical scenario.

[1]：Jones, Erik, et al. "Automatically Auditing Large Language Models via Discrete Optimization." arXiv preprint arXiv:2303.04381 (2023).

Thanks for your constructive comments and suggestions! We hope our response resolves your concerns.

Weakness1

In our paper, we choose GBDA, Autoprompt and ARCA as baselines. Although Hotflip is an old method in adversarial attack, latter algorithms have made optimizations on the basis of Hotflip which achieve a big leap in performance. Notably, methodologies such as Seq2Sick [1] introduce subtle perturbations to input sequences, thereby disrupting the efficacy of language models across diverse tasks. The primary objective of this algorithm is assessing the robustness of language models through the introduction of such perturbations.

However, the motivation of the algorithm is to deviate language models from the correct output to construct adversarial samples in various text generation tasks. In our task, we want to elicit the language model to produce specific content that is considered as toxic or unsafe. The optimization objectives of these two categories of algorithms are inherently distinct. As a result, direct comparisons within the confines of the same experimental setting is challenging.

Therefore, we add a discussion about this algorithm in the Related Work section instead. We really appreciate the suggestion!

Weakness2

We supplement the performance of our algorithm against two defense approaches, including DAPT [2] and DExperts [3]. We follow the implementation in the original paper and test the performance of our algorithm against the two defense approaches on GPT-2. The results are listed as follows:

According to the experimental results, ASRA can also elicit some toxic cases even against defense approaches. Generally, DAPT does not improve the robustness of the pre-trained language model against our adversarial attack, while DExperts achieves a relatively good performance. The attack and evaluation of defense methods need further research in the future.

Weakness3

Most existing toxicity evaluation is conducted by evaluating model responses with human-written inputs. The evaluation method of safety of pre-trained language models depends on the practical scenario. Common evaluation assumes that users query language models with some questions or dialogue. However, malicious attackers can attack pre-trained language models by intentionally constructing adversarial prompts. Ensuring the safety of language models requires evaluation at different levels. This paper explores how our algorithm can assess the safety of pre-trained language models when facing intentional adversarial challenges.

Weakness4

We agree with your concern that our approach might be used by malicious attackers to attack public large pre-trained language models, leading to toxic content generation or privacy leakage. We believe that AI research always comes with social and ethical issues, and our work might contribute to the enhancement of language model safety in the future. We have supplemented an ethnical statement section in our paper. We really appreciate the suggestion!

Question

Our algorithm is not specific to text generation tasks like summarization, but to test the security and toxicity of pre-trained language models. The objective of our algorithm is to elicit certain content from the language model. Therefore, it is challenging to conduct experiments on various text generation tasks like [1]. As we discuss in Section 5.2, the success rate of the attack is strongly correlated with the perplexity of the target output. In other words, our method can achieve a good performance if the target text has high fluency on the language model.

[1]：Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples.

[2]：Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models

[3]：DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts