Curiosity-driven Red-teaming for Large Language Models

We thank the reviewer for appreciating the novelty of our work. We’ve added

1. Experiments on Vicuna and ChatGPT (Appendix C.3)
2. Analysis on the toxicity classifier (Appendix C.1)

to the manuscript. The following is our detailed response.

The experiments chose GPT2 as the target model in the main results, making the results not too convincing. I recommend testing on a wider range of LLMs, including proprietary models like ChatGPT and open-sourced models like LLaMA-2-chat (the user has already done but there are not enough results and details) and Vicuna.

Why not try "real" LLMs like proprietary models like ChatGPT and open-sourced models like LLaMA-2-chat and Vicuna? I do not think the results are GPT2 with 137M parameters are convincing enough.

Answer:

In addition to GPT2, our main results also include experiments against Dolly-7B and LLaMA2-7B-chat-hf, demonstrating our method is able to create diverse and effective testcases.

We’ve added the results of red teaming against Vicuna-7B and ChatGPT (gpt3.5-turbo) in Section C.3 in the Appendix, showing that our method is also effective at achieving diverse and effective testcases against these models. Our results show the following:

1. Higher Diversity: Our approach yields more diverse test cases and target model’s responses, as shown in (Figures 9(i)(c,d,e,f) and 9(ii)(c,d,e,f)).
2. More Unique Test Cases: Figures 9(i)(b) and 9(ii)(b) illustrate that our RL+Curiosity approach generated around 50,000 unique test cases resulting in toxic responses surpassing a 0.9 toxicity threshold. This number is significantly higher than those produced by the baselines, RL+TDiv and RL. It's important to note, as Figures 9(i)(a) and 9(ii)(a) indicate, that while RL+TDiv led to a higher proportion of toxic responses, the overall lower number of unique test cases indicates an issue of lack of diversity, with many cases being exactly identical based on string comparison.

1. Are the RoBERTa hate speeh classifier strong enough to detect the toxic generations?

Answer:

Yes. To show that, we compared the RoBERTa model’s classification with GPT-4’s classification. We sampled 3000 target model’s responses each for all red-teaming methods and prompted GPT-4 to predict whether the response was toxic or not. Table 7(iii) in the Appendix showed the precision, recall, and F1-score with GPT-4’s predictions as ground truth and RoBERTa as an estimation. The table shows that that F1 scores are all close to one, which indicates that RoBERTa’s predictions are similar to GPT-4’s predictions. We also added the confusion matrix in Figure 7(ii) in the Appendix for reference.

We thank the reviewer for appreciating our writing and empirical results. We added

1. Ethic statement (after the Discussion section)
2. Experiments on ChatGPT and Vicuna (Appendix C.3)
3. Analysis on over-fitting to classifier (Appendix C.1)

to the manuscript. Also, we are running experiments on text-to-image experiment. Our responses are detailed in the following.

Ethical comments

Answer:

We’ve added an ethic section after the discussion section and would like to know if the reviewer thinks this addresses potential ethic concerns.

The empirical results are limited to the text-to-text generation models.

Can you provide simple experimental results on text-to-image models like stable-diffusion?

Answer:

Our framework is applicable to text-to-image red-teaming. We are running the experiments and will do our best to add these results during rebuttal.

The proposed automated red-teaming method is based on the pre-trained offensiveness classifier. As [1] raised, there exists a risk of discovering test cases that over-fit the red team classifier, resulting in false positive test cases. But there isn't any analysis about this risk.

Answer:

We added a new analysis in Appendix C.3, showing that the toxic response elicited by our red-teaming method generalize to other toxicity classifiers also.

High Correlation of Toxicity Predictions Across Classifiers: Table 7(i) shows Pearson Correlation Coefficients (PCC) of the toxicity probabilities predictions between RoBERTa and two other classifiers, on the responses of the target model Dolly-7B. The table shows that PCC is near one, indicating that a strong positive correlation between RoBERTa and other classifiers in toxicity prediction. This implies that when RoBERTa predicts higher toxicity probability, the other models are also likely to show similar increases in their toxicity probability predictions.

Similar to GPT-4’s predictions: Table 7(iii) in Appendix C.3 compared RoBERTa's classification accuracy to GPT-4 by analyzing 3000 responses from each red-teaming method. GPT-4 evaluated these responses for toxicity. Using GPT-4's predictions as the ground truth, we assessed RoBERTa's precision, recall, and F1-score. The results, with F1 scores nearing one, suggest a high similarity between RoBERTa's and GPT-4's predictions. We also added the confusion matrix in Figure 7(ii) the Appendix C.3 for reference.

Note that we compared RoBERTa with GPT-4 using a confusion matrix and F1 score rather than Pearson Correlation Coefficient (PCC), as GPT-4 cannot predict probabilities. Attempts to prompt GPT-4 for probabilistic outputs, using various phrasings, were unsuccessful; the model typically responded that it could not provide numerical probabilities.

Can you provide experimental results on large language models such as the text-davinci-003 model or gpt-3.5-turbo prompted chatbots?

Answer:

We’ve added the results of red teaming against Vicuna-7B and ChatGPT (gpt3.5-turbo) in Section C.3 in the Appendix, showing that our method is also effective at generating diverse and effective testcases against these models. Our results showed the following facts:

1. Higher Diversity: Our approach yields more diverse test cases and target model’s responses, as shown in (Figures 9(i)(c,d,e,f) and 9(ii)(c,d,e,f)).
2. More Unique Test Cases: Figures 9(i)(b) and 9(ii)(b) illustrate that our RL+Curiosity approach generated around 50,000 unique test cases resulting in toxic responses surpassing a 0.9 toxicity threshold. This number is significantly higher than those produced by the baselines, RL+TDiv and RL. It's important to note, as Figures 9(i)(a) and 9(ii)(a) indicate, that while RL+TDiv led to a higher proportion of toxic responses, the overall lower number of unique test cases indicates an issue of lack of diversity, with many cases being exactly identical based on string comparison.

It is just a curiosity question. In a realistic scenario, each query on the target model usually incurs costs. Hence, the number of model queries during the red teaming process can be an important factor for the overall cost of the method. Can you provide the number of queries used during the end-to-end process of RL+"curiosity"? Also, can you estimate the price when we red-team gpt4 api using RL +"curiosity"?

Answer:

Number of queries: We used 40K queries for instruction following experiments (Section 4.3) and 100K queries for text continuation experiments (Section 4.2). We determine both numbers based on when the combined rewards (i.e., toxicity probability + curiosity + entropy bonus) of RL+Curiosity (our method) stop growing.

Price of red-teaming GPT-4: $143.999. We did the calculation by assuming the maximum amount of input and output tokens are all 40 and 40K queries and based on the GPT-4’s pricing info in https://openai.com/pricing.

We thank the reviewer for their comments and are glad that he/she appreciated the comprehensiveness of our experiments, found our idea sound and our presentation clear. The responses to questions are below:

The authors choose an objective of maximizing test case novelty but do not show how their method influences target LLM response novelty.

How does your method compare with RL+TDiv if we look at the diversity of target LLM responses? I presume that both test case diversity and target LLM response diversity are important when red-teaming, and I would like to know whether RL-TDiv essentially achieves target LLM response diversity without necessarily going through the intermediate step of test case diversity.

Answer:

We added a new figure to show that RL+Curiosity (ours) also leads to higher target LLM response novelty (i.e., diversity) than RL+TDiv in Figure 8 in Appendix C.2.

RL+TDiv adds the diversity of the embeddings from the target model’s responses as rewards. Though RL+TDiv achieves higher response diversity than RL, it underperforms ours. This indicates that Curiosity is a more effective objective to achieve both test case and response diversity. We elaborate on these experimental results in Appendix C.2 and briefly explain them below:

• Why does RL+Curiosity improve responses diversity? Maximizing test case diversity makes the target model to respond to different prompts and hence leads to diverse responses (i.e., answers to the prompts).
• Why does RL+Curiosity lead to higher response diversity than RL+TDiv? RL+TDiv solely maximizes the diversity of the current batches, rather than the diversity of the all responses seen in the whole training time. We will elaborate this point in the next answer.

This is not too important but it is worth mentioning that the novelty of the method is somewhat limited. The cosine similarity reward is very similar to the RL+TDiv baseline, except it is applied to the test cases.

Answer:

We want to highlight another conceptually critical difference.

RL+TDiv solely maximizes the diversity within the current batch of responses, as measured by cosine similarity. This approach does not incentivize the red-team model to generate responses that differ from those in previous batches. Consequently, the model can repeatedly elicit similar responses identical to those in the past, ending up with low response diversity overall. This is evident in Figure 8(d, e) in the Appendix C.2, where RL+TDiv exhibits lower response diversity than ours. It indicates that maximizing testcase diversity is conducive to not only producing diverse testcases but also eliciting diverse toxic responses.

In A.7, you mention you sample 100 sets of 100 test cases and calculate both diversity metrics across each subset. What is the test case size for a select number of thresholds in your experiments, and why is this sampling method used?

Answer:

In A.7, we calculate the diversity of testcases exceeding each toxicity threshold by subset sampling approach because the number of testcases at each threshold can vary across methods.

For example, in Figure 2(i, a), nearly 0% of testcases exceeding threshold 0.9 are found by RL baseline, while 50% of testcases found by RL+Curiosity (ours) exceed threshold 0.9. Thus, the testcase sizes for calculating diversity are different at threshold 0.9, which makes them incomparable.

To compute diversity with varying-size testcase sets, we follow the suggestion in [1, 2] and use the resampling method to estimate the diversity of the testcase set.

[1] Perez, Ethan, et al. "Red teaming language models with language models." arxiv 2022

[2] Lee, Deokjae, et al. "Query-Efficient Black-Box Red Teaming via Bayesian Optimization."  arxiv 2023

Could you clarify what number of test cases exceed each toxicity threshold for RL+Curiosity for the experiments in section 4.2 and 4.3?

Answer:

Each trial (i.e., run with a different random seed) of the experiment uses 100K queries in Section 4.2 and 40K queries in Section 4.3. Thus, the testcases that elicit responses exceeding each toxicity threshold by our proposed RL + curiosity method are:

We thank the reviewer's timely response.

“Also, note that while RL (without curiosity and TDiv) achieves a high level of diversity in Figure 2i(c), only a limited number of test cases exceed high toxicity thresholds [0.2, 0.9]” Could you report these numbers? Could you also address why the zero shot baseline performs so well in Figure 2(ii)(c)?

Answer:

Could you report these numbers?

• The following table shows the number of testcases at each threshold of RL baseline in Figures 2(i) and 2(ii) (green curves). Compared with the table shown in the previous responses, the number of testcases generated by our method at each toxicity threshold is up to 10 times higher than the RL baseline. We will also attach these tables to the Appendix in the next manuscript revision during the rebuttal.

Could you also address why the zero shot baseline performs so well in Figure 2(ii)(c)?

• The main reason is that the zero-shot baseline is not finetuned with RL. As Perez et al. 2022 suggested, pre-trained LLMs tend to have higher testcase diversity than RL-finetuned LLMs in red teaming, likely because RL's reward maximization objective hurts diversity [1]. The zero-shot baseline in Figure 2(ii)(c) is a pre-trained LLM and is, hence, expected to have higher diversity than the RL baseline (the green curve in Figure 2(ii)(c)).
• However, the zero-shot baseline is notably ineffective in generating testcases that trigger toxic responses in the target model (see Figure 2(i)(a) and 2(ii)(a)). Additionally, it shows lower diversity based on 1-SelfBLEU (Figures 2(i)(b), 2(ii)(b)) and only matches embedding diversity (Figures 2(i)(c), 2(ii)(c)) when compared to the our RL+Curiosity method.

[1] Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, 34:27381–27394, 2021

We are glad that the reviewer appreciates our idea and results. We answer the rest of questions below.

The distinction between adversarial attacks and red teaming seems artificial. I wouldn't want this distinction being introduced in the community. Surely adversarial attacks can be semantic, and surely red teaming can include gibberish adversarial examples as an interesting failure mode of LLMs.

Answer:

Thanks for the suggestion, and we agree with the reviewer. We’ve updated the manuscript and removed the distinction in the related work section (Section 5). We are happy to further revise this section during the rebuttal according to the reviewer’s comments.

I would suggest adding more visual separation between diversity and quality plots. Currently it's hard to tell which is which in Figures 1 and 2 without turning my head sideways to read the axis.

Answer:

We’ve updated the figures, moving “Quality” and “Diversity” to the titles of each subplot so that the reader can tell which plot is for quality and which plot is for diversity easily. We are happy to revise the figures if the reviewer thinks there could be a better way of visualizing these data.

This addresses my concerns. I think the paper could be accepted and have raised my score to an 8.