FLIRT: Feedback Loop In-context Red Teaming

We thank the reviewer for their feedback.

Regarding the first point made by the reviewer, We see two sub-questions here that we address separately.

on attack effectiveness: First, we agree that “attack effectiveness” metric by itself is not very useful as the model may learn to repeatedly “generate” the same/similar effective prompts. This is exactly why we report the diversity metric in addition to attack effectiveness and propose an objective where we combine attack effectiveness with the diversity metric, to force FLIRT to generate prompts that are novel and effective at the same time. In this regard, we are not quite sure about the reviewer’s statement “It doesn't sound reasonable to penalize methods that discover more adversarial prompts (though some of the prompts are not effective)”. The diversity part of our objective assures that FLIRT generates novel prompts, and the effectiveness part of the objective ensures that the new prompts are effective against the target model. Without the latter, FLIRT would generate a very diverse set of prompts, but those prompts would not be effective against the target model, thus defying the purpose of red teaming.

on different diversity metric: The second part of the comment is about having different categories of unsafe generation (e.g., sexual, violent), and then studying the diversity of the prompts in terms of different types of unsafe generation. This notion of category-wise diversity is interesting. However, we believe the diversity metric we study in the paper is better for our purposes as it captures a more general notion of diversity regardless of the task/categories in hand. Furthermore, it might be hard to categorize certain prompts into specific categories in which case the proposed evaluation metric might not be robust and reliable. In addition, Figure 6 (refer to the appendix) contains some experiments on using a different diversity score by reporting Rouge-L score distributions as an additional study on the diversity of the generated prompts which should answer your concern.

On the effect of seed prompts: As mentioned in section 3, “In addition, we perform numerous controlled experiments to better understand the effect of seed prompts and how they differ from the generated prompts in the Appendix.”, Figure 5 in the appendix has a study on this. In Figure 5 we test our framework with different sets of seed prompts each containing different number of unsafe prompts and report the attack effectiveness rate. We demonstrate that even with 1-2 unsafe examples, the framework is able to generate a large percentage of successful examples. In addition, Figure 6 shows that generations are different than the seed prompts according to their Rouge-L distributions.

Why are we better than SFS: Both factors play a role. As shown in Figure 5, the initial prompts have an effect (as we increase the number of successful prompts the attack becomes more effective; however, note that even with 1-2 successful initial prompts, we are able to obtain high attack success rates). In addition, as shown in the main body of experiments, the way we incorporate the feedback (successful prompts) plays a significant role (results are different in a FIFO based approach compared to LIFO, scoring, LIFO-scoring). This suggests that the feedback and the incorporation of the successful prompts also play a significant role and can change the results significantly such that each approach imposes its own trade-offs. Hence, both the initial prompts and the feedback (the way that the successful prompts are added) play a role.

On text-to-text evaluators: We only used toxigen for text-to-text experiments and perspective api for text-to-image experiments and not text-to-text. The reasons why we did this was:

1. To make the distinction between text-to-text and text-to-image experiments which are disjoint experiments and the result for one does not affect the result for another.
2. In text-to-image experiments our classifier was applied on the input prompt while in text-to-text experiments it was applied on the output prompt; hence, for the ease of notation in section 2.3 and clarity of writing, we used two different classifiers so that we have two different notations to make this distinction.
3. To show that our approach is general enough to apply to different classifiers hence we used different classifiers to showcase this capability and that we are not only limited to perspective api but a wider range of classifiers.

We acknowledge that there is always the risk of false negatives/positives in using any automatic classifier and hence clearly mentioned this in our paper. In addition, we perform a dedicated experiment to this in our ablation study 5, where we did experiments in which we show that our results are robust to existing noise coming from these automatic classifiers.

We thank the reviewer for their comments.

On the weaknesses of our work: We kindly ask the reviewer to provide more context/substance for their criticism of the soundness of our paper. We believe we have provided extensive experimental results and evidence to validate our claims; please see Sec. 3. We would appreciate if the reviewer explicitly states what aspects of our results are weak, and what additional evidence needs to be provided.

We thank the reviewer for pointing to the relevant work, we have added it in the revised draft. We note that while both BRT and FLIRT are read teaming methods, the underlying technical approaches are very different. BRT uses bayesian optimization to edit and find successful examples, while FLIRT is based on iterative in context learning with a feedback loop. FLIRT is more efficient as it uses in-context learning without requiring an additional expensive training. FLIRT also does not require extensive access to datasets to be trained over as it is using in-context learning and can operate using a few seed prompts.

Comment on other kinds of diversity metrics: Figure 6 in the appendix has some experiments using Rouge-L score. We consider the same evaluation strategy as the self-instruct paper.

On SFS operating similarly to FLIRT: SFS is a retrieval based approach in which they retrieve examples from the pool of zero-shot examples for the few-shot generations. If the pool of zero-shot generations are not successful, regardless of the temperature value, the approach would not find successful examples. On the other hand, FLIRT (especially the scoring approach) uses a feedback loop which improves upon its few-shot demonstrations starting from only a few (three or four) demonstrations in each successful iteration. In this case, if a new generation is more successful, FLIRT will consider it as its demonstration and keep improving on it in the next iterations. This has two advantages, first we do not require a large pool of zero-shot examples to be generated to begin sampling from as we are reusing our generations as demonstrations. Second, FLIRT can improve upon its demonstrations by incorporating the scoring coming from the feedback loop to further improve its demonstrations and hence the generations.

On trade-off curve: We included the trade-off curve in Figure 7 of our revised paper.

We thank the reviewer for the feedback.

1. We added citation and discussion around the mentioned references in our revised paper. The RL-based works in (Deng et al, Perez et al, and Casper et al) require a relatively large dataset of prompts for training. The motivation behind our work is to learn to generate adversarial prompts without access to such dataset, using a seed set of a few adversarial prompts instead. That said, one can in principle use the prompts generated via our framework (FLIRT) for doing RL, although this will not be a comparable baseline. In fact, ( Perez et al.) explored a similar approach, where they trained a red LM on data generated by their SFS approach, using both SFT and RL. From the reported results in Figure 2 from Perez et al. we observe that our approach is in par with the RL approaches, with the additional benefit of being more efficient as it does not involve training. Since efficiency is one of our primary goals, we find it reasonable and fair to compare our methods to a baseline such as SFS that require no training and is using few shot setup. In addition, Zou et al. proposes a white-box jailbreaking attacks on text-to-text models which is not fully aligned to our setup that is a black-box setup on multimodal models in the context of red-teaming.

2. As the reviewer mentions, the proposed in-context reinforcement learning-based method sounds similar to our scoring attack, in that FLIRT uses the score as a type of feedback signal to learn effective (and diverse) prompts during an iterative learning process. Doing this with in-context RL sounds like an interesting future direction to explore. We assume the actions and rewards would correspond to selecting prompts and effectiveness of those prompts respectively. However, it is not very clear what would be the states of the RL agent, and the corresponding policies. If the reviewer has some insights on this they could share, we’d greatly appreciate that.

3. We respectfully disagree with the assessment that “humans do most of the heavy lifting”. While we seed the model with a few human-provided prompts, FLIRT learns how to discover novel, effective prompts automatically. Please see Fig 6 (in the appendix) where we show how different the generated prompts are to the seed prompts according to the Rouge-L score distribution which suggests that our framework is able to discover new/novel instances. Furthermore, even if the seed prompts are not very effective, FLIRT will still learn to generate effective prompts; please see Fig. 5. where we do experiments with different sets of seed prompts that have different number of effective prompts in them and show that even with 1-2 effective seeds, the framework is able to generate various effective prompts.

4. Thank you for the great pointers. We have applied them to our revised paper.

We thank the reviewer for the feedback and the pointer to the paper. We have included a reference to the paper in our revised paper. We would like to note that according to ICLR’s guidelines “authors are not required to compare their work to concurrent work within the 2 month time frame as well as unpublished work.” This paper requested by the reviewer became public days before the ICLR submission date and is unpublished. However, we have included a citation of this work to acknowledge it in our paper.