Explore, Establish, Exploit: Red Teaming Language Models from Scratch

Thank you for the feedback and comments. We are glad to hear that you found the paper to be well-written and that the CommonClaim dataset will be valuable. Below are replies to weaknesses and questions.

W1 & W4 – Re: Comparisons to prior work and missing references

New discussions of related work: Thank you for the suggestions about prior work. In particular, we think that it will be useful for us to cite Xu et al. (2021)([2]) as an example of the effectiveness of a filtering baseline. We added this. Meanwhile, we have added Lee et al. (2023a) [(3]) and Ziegler et al. (2022) as examples of human-generated attacks against language generators, and Lee et al. (2023b)([4]) next to our discussion of Shi et al., (2022) and Kumar et al. (2022) which use related langevin-dynamics-based approaches to automated red teaming.

In comparison to prior works that rely entirely on humans to generate adversarial prompts, our work is novel in its use of automated attacks in the Exploit step. We agree that human-in-the-loop attacks are interesting and important. However, we believe that automated attacks are an important complement to these methods. In practice, we expect that a combination of both will be used. One advantage of our approach is that a fixed amount of human effort in the Establish step can be used to enable very large amounts of cheap, automated attacks. Another advantage is that sometimes it might be easier to define a target category of text than to attack a model to produce that target category. Also automated methods and humans simply tend to produce different types of prompts, so these different methods for attacking networks can be complementary.

Our attack method adds to the method used by Perez et al. (2022). We are sorry that the paper was unclear about our contribution. Our diversity objective for RL is novel. While Perez et al. talk about the importance of diversity in RL-based attacks, they do not use our diversity objective (see, e.g., Figure 2 of their paper or the last paragraph of their section 2.2.) We have updated our discussion of the ablation experiments for this diversity term to clarify that the ablation experiment was with the same method as Perez et al. (2022) and Deng et al. (2022). We also added vocabulary size analysis to better quantify mode collapse. Across 100 samples, the prompt vocabulary size decreased from 116 to 3 and 81 to 9 for the toxicity and untruthfulness red teaming experiments without our diversity method.

We added a discussion of other prior techniques to improve diversity and noted that they are not applicable to RL attacks. We added mention of how Lee et al. (2023b)([4]), Shi et al., (2022), and Kumar et al. (2022) use an analogous but non-RL technique for this. Thank you for pointing this out!

See also our global response. We give an overview the paper’s novel contributions.

W2 – Re: Quantitative analysis

We would like to clarify that we have a fair amount of quantitative evaluations of our method. These include analysis of how much our attacks increased toxicity, mode collapse without our diversity technique, label agreement between humans and models, and the success of the prompt generators at eliciting harmful responses according to each classifier we used. We discuss these in detail in the global response. Are there any other quantitative experiments you would like to see us add?

We are running a new experiment in which we omit the diversity term used in the first step (Explore) for toxicity experiments. Thank you for the suggestion.

Thank you for your kind responses. Global response and your responses sufficiently addressed most of my concerns on your paper.

There was a line of research about detoxifying pre-trained language models [1,2,3]. My previous comment “However, filtering in training data or outputs can degrade the model performance.” in W3.1 is based on [1], which shows that filtering toxic contents in training data can help detoxify the LM but also can result in performance degradation.

I appreciate the inclusion of additional references, such as [4], in the revised version of the paper, which adequately addressed my concerns regarding this issue.

In W3.2, I requested you to justify the "why from scratch" part. Your explanation, that the context of toxicity can be very diverse, was helpful in understanding the necessity. However, as you agreed, we still do not need to red-team a new model from scratch if we already have a dataset and a classifier sharing the same context. In this regard, I still think that the proposed red-teaming method has limited-applicability. For example, since the authors released a valuable dataset CommonClaim in this paper, we can red-team other new models for the purpose of fact-checking by re-using this dataset and classifier, not from scratch.

Therefore, I still think that the proposed method will only be used in limited situations from the perspective of red teaming. However, from the perspective of utilizing the automatic red-teaming method to collect datasets and train classifiers corresponding to toxicity in certain new contexts, it seems like a nice research approach. In this regard, I raised my scores to 5.

But, I think it would be better to emphasize the collect datasets and train classifiers parts, more than the red-team part in the paper.

[1] Challenges in Detoxifying Language Models, Welbl et al., EMNLP findings 2021.

[2] Detoxifying Language Models Risks Marginalizing Minority Voices, Xu et al., NAACL 2021.

[3] Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP, Schick et al., TACL 2021.

[4] Pretraining language models with human preferences, Korbak et al., ICML 2023.

Thank you for the update. The revised introduction appears to be clearer in conveying the contribution of your work. However, I still have difficulty understanding which point of your work is more beneficial than [1]. I have some additional questions for clarification.

In your previous responses, you commented that:

Both types of methods are clearly valuable, but one useful property of our approach is that a fixed amount of human effort can then allow for highly scalable automated adversarial example generation.

Can you provide more details or evidence to support this statement?

Based on your responses, the primary distinction between your work and [1] appears to be the presence of annotation during the exploitation step, with your approach offering a lower annotation cost compared to previous partially automated human-in-the-loop methods [1,2]. However, [1] addressed the problem of high annotation cost by only annotating candidates in the gray area, where filter predictions are ambiguous. Also, it is worth noting that [1] also can be fully automated by stopping annotation and using only filter predictions to label candidates.

In this regard, if you were to allocate the same annotation budget in both approaches, can you clarify why your approach is beneficial than [1] in such a scenario?

[1] SQuARe: A Large-Scale Dataset of Sensitive Questions and Acceptable Responses Created Through Human-Machine Collaboration, Lee et al., ACL 2023. [2] WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation, Liu et al., EMNLP findings 2022.

Thank you for the feedback and comments. We are glad to hear that you found the paper to be well-written and to address an important gap in the literature. Below are replies to weaknesses in order.

W1 – Re: Concerns about only using GPT-2 and GPT-3

We agree that working with Llama, Claude, and Bard-type models would be useful for establishing clearer relevance. We added this to our discussion of future work.

We also emphasize that our method is black-box, and the RL attacks that we use in the exploit step could be replaced with other attack methods. In the adversarial attacks literature, findings with individual methods/models often quickly become outdated, but these two facts make us expect that our work may have some implications that may continue to be valuable further in the future.

W2 – Re: adding to the discussion of limitations

We have added to Section 2, and Section 5 to discuss these. Thank you for pointing these out. We discuss how alternatives could have been used other than the RL attack we used (see the “Red-teaming with automated searches for natural language prompts” paragraph). We added references to works on AI oversight (Bai et al., 2022), AI feedback (Lee et al., 2023), active learning (Zhen et al., 2022), and weak supervision (Boecking, 2020) to discuss how human involvement might be effectively scaled.

Q1 – Re: “human input can be used to determine the set of labels used to train the classifier…Can the authors describe a bit more what that process looks like in practice?”

Abstractly, the establish step can be divided into three substeps. After the red team has expired the data, they pick a label set, obtain labels, and then train a classifier for harmful text on those labels. So in our case, the answers that we obtained from crowdworkers were not used to select the label set. The label set was selected by us after our own analysis of the data.

Consider how this compares to prior approaches. Naively (e.g. without developing a contextual understanding of failure modes), it may seem reasonable to use a classifier trained on the CREAK dataset of true and false statements. But as we find, this does not work well, presumably because of the distributional differences and the lack of a “neither” label.

Q2 – Re: Concerns about dysfluencies in prompts

We agree that the results in Table 4 have dysfluencies and repetition, but we emphasize two points, and we updated the discussion of these results accordingly.

• Prompts appearing as fluent English is not required for them to be adversarial. For example, Zou et al., 2023 studied jailbreaks with triggers such as “restored into one sentence grammar using proper colon”. We only focus on developing diverse and adversarial prompts. Making plain-English ones is not a goal of ours, however, it would be possible to do with a stronger KL penalty on the generator’s outputs.
• We agree that there is room to further improve the diversity of the attack distribution. However, our method is much better than the baseline. Qualitatively, the results in Table 4 are much more diverse than the results from the baseline in Table 6. And without our diversity reward, 61 out of the 100 sentences we sampled were all identical.

Thank you again for your time and help! We are looking forward to the rest of the discussion period.

Thank you for the feedback and comments. We are glad to hear that you found the paper to contribute an effective solution to a very important problem. We do our best to address your concerns below. Please let us know if there is anything else that we can do to help.

W1 — Re: Straightforwardness of the method

We agree that this is a natural way to approach the problem. However, there has not been prior work to show the benefits of a contextually defined target for automated red teaming. Prior works on red teaming have tended to neglect the role of preference formation and human factors in AI oversight, and simple filtering-based baselines. To the best of our knowledge, ours is the first work on automated attack methods that addresses these.

The Explore and Establish steps were the reason why our red teaming was more successful than the experiment with the pre-existing CREAK data. Based on prior work (e.g. Perez et al. (2022)) one might suspect that the best way to approach automatic red-teaming for false statements is to use an off-the-shelf classifier for true vs false statements. However, we show that using a classifier trained on the CREAK dataset, was not effective (see Section 3.2.1 and Appendix D). Contextual red teaming was needed.

W2 – Re: Concerns about using the target model’s latents

Sorry for the lack of clarity – we updated Section 2 and Figure 2 to clarify that we do not need the target model’s latents. We apologize that we were unclear about whether we need the target model’s latents. We only use these when available. We have updated our description of the methods to say that when latents are not available, we use another model’s embeddings as done in Section 3.2. Internal activations of GPT-3-text-davinci-002 were not available via API, so we instead used embeddings from GPT-3-text-ada-002, a dedicated text encoder.

W3 – Re: Making comparisons to other methods

We want to clarify that we compare our method to conventional RL-based attacks and quantitatively analyze the results. We improve on the RL attack method from Perez et al. (2022) and Deng et al. (2022) with our approach to producing diverse adversarial prompts. In our control experiments that did not include the diversity technique, the prompt generators exhibited mode collapse. We are sorry that this was not clear, and we updated our description of these experiments to point out that this was the method used by these two prior works. Is there another experiment that you would recommend that we run?

Comprehensive benchmarking is important, but out of scope. In this paper, our goal was to run experiments to validate the benefits/performance of the improvements that we make to red teaming and RL attacks. We agree that other approaches to automated red teaming are interesting to compare, but we think that this would best be explored in separate work. We note that there currently are two competitions being run to do this, and we are contributing to this by testing our RL attacks with one of them.

We have added a discussion of benchmarking to our future work section and clarified the goal of our experiments/evaluations.

W4 – Re: Concerns about dysfluencies in prompts

We agree that the results in Table 4 have dysfluencies and repetition, but we emphasize two points, and we updated the discussion of these results accordingly.

• Prompts appearing as fluent English is not required for them to be adversarial. For example, Zou et al., 2023 studied jailbreaks with triggers such as “restored into one sentence grammar using proper colon”. We only focus on developing diverse and adversarial prompts. Making plain-English ones is not a goal of ours, however, it would be possible to do with a stronger KL penalty on the generator’s outputs.
• We agree that there is room to further improve the diversity of the attack distribution. However, our method is much better than the baseline. Qualitatively, the results in Table 4 are much more diverse than the results from the baseline in Table 6. And without our diversity reward, 61 out of the 100 sentences we sampled were all identical.

Thank you again for your time and help! We are looking forward to the rest of the discussion period.

Thank you for the feedback and comments. We are glad to hear that you found the design and implementation of our experiments to be good. Here are replies to weaknesses and questions in order.

W1 – Re: Adversarially training the target model after attacking it.

We agree that adversarial training is an important next step. In the “What comes after Explore/Establish/Exploit?” paragraph, we discuss this and others. Enabling this is a motivation of ours.

Our contributions are focused on red teaming but we believe they are sufficient and valuable in their own right. Please see our global response for a summary of contributions, quantitative results, and qualitative results. Prior works on red teaming such as Perez et al. (2022) have not performed adversarial training either. Meanwhile, our work is the first to our knowledge to use automated attacks to red team a model of a size comparable to GPT-3 for untruthful text. We also release the CommonClaim dataset. We updated the “Future work” paragraph to discuss how we do not perform any outer applications of the dataset, classifier, or adversarial data.

W2 – Re: Comparisons to other methods such as Perez et al. (2022a).

We want to clarify that we compare our method to the RL-based one used by Perez et al. (2022) and quantitatively analyze the results. We improve on the RL attack method from Perez et al. (2022) and Deng et al. (2022) with our approach to producing diverse adversarial prompts. In our control experiments that did not include the diversity technique, the prompt generators exhibited mode collapse. We are sorry that this was not clear, and we updated our description of these experiments to point out that this was the method used by these two prior works. We also added vocabulary size analysis to better quantify mode collapse. Across 100 samples, the prompt vocabulary size decreased from 116 to 3 and 81 to 9 for the toxicity and untruthfulness red teaming experiments without our diversity method. Is there another experiment that you would recommend that we run?

W3 – Re: Ablation studies.

We want to clarify that we performed ablation experiments. We also have a new one underway. Sorry that this was unclear. We updated our descriptions of these tests in the paper to clarify this. Thank you for this suggestion. The ablation tests were ones in which we removed the diversity term in the exploit step (Section 3.1.1, Section 3.2.1, Appendix B), replaced our false statement classifier with an off-the-shelf one (Section 3.2.1, Appendix D), and replaced our false statement classifier with one trained on chatbot labels (Section 3.2.1, Appendix E). We are currently running a new ablation test in which we omit the diversity subsampling from the Explore step. Please see the global response for all details. Are there other ablation tests you recommend we do?

Q1 – Re: CommonClaim in supplement.

We updated the supplementary materials to include CommonClaim. We apologize for not including this in the supplement. Thank you for pointing this out.

Q2 – Re: “...the goal of this work is to red-team from scratch. However, in step 2 of the proposed framework, we still need to choose a label set…”

Could you please clarify this concern? In our framework, the process of choosing a label set is how we define the target for the red team. For example, in our truthfulness red-teaming, we defined the categories as part of the process, based on interaction with the model outputs.

Our results show the benefits of contextual red teaming in comparison to a baseline that uses an off-the-shelf truthfulness classifier based on the pre-existing CREAK dataset. When we red team for truthfulness in Section 3.2, we identified the need for a third additional category of “neither” true nor false to account for statements that were opinions, vague, obscure, uncommon knowledge, or otherwise hard to categorize as true or false by common knowledge. This third label is an advantage that our approach had over the control experiment with the CREAK dataset which used a classifier trained on a pre-existing dataset that did not include a “neither” category.

Thank you again for your time and help! We are looking forward to the rest of the discussion period.

Ablation experiments/baselines

• To measure the impact of our diversity technique, we ablated the diversity term in the toxicity and truthfulness experiments. We observed mode collapse without this term. See above (Section 3.1.1, Section 3.2.1, Table 5, Appendix B).
• To compare our contextual EEE approach to one that simply uses a classifier off the shelf, we compare using our untrue text classifier to one trained on the pre-existing CREAK dataset. We find that the CREAK classifier fails to elicit anything related to untrue claims (Section 3.2.1, Appendix D, Table 4, Table 7).
• To compare human labels to chatbot labels, we compare using our untrue text classifier trained on human labels to one trained on GPT-3.5 labels. We find that the classifier trained on chatbot labels fails to elicit anything related to untrue claims (Section 3.2.1, Appendix E, Table 4, Table 8).

We are running a new experiment – results to come soon

To test the influence of diversity subsampling in the Explore step, we are rerunning the full toxicity red teaming experiment (all 3 steps) without diversity subsampling in the first step. Thank you to tVNe for pointing this out. Updates to follow.

Updates to the paper and supplemental materials (details in individual responses)

• Updating the description of our contributions in the introduction.
• Updates to the limitations and future work paragraphs in response to all four reviewers.
• We updated Section 2 and Figure 2 to clarify that our method does not rely on white-box access to the target model. When its latents are not available to produce embeddings (i.e. the experiments in Section 3.2), we use another embedding model instead. Thank you to weY9 for pointing this out.
• We added vocabulary size analysis to better quantify mode collapse. Across 100 samples, the prompt vocabulary size decreased from 116 to 3 and 81 to 9 for the toxicity and untruthfulness red teaming experiments without our diversity method.
• Adding CommonClaim to the supplemental materials in common_claim.csv. Thank you to weY9 for pointing this out.
• Expanding on our discussion of dysfluencies in the generated prompts. Thank you to FuAh for pointing this out.
• Adding discussion of several related works. Thank you to tVNe for pointing this out.