Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Models

Thank you for your valuable questions! Below, we address them in detail.

Answer to W1: The harmfulness is not annotated by a model. We generate potential pseudo-harmful prompts automatically. After generation, we manually annotate each prompt as "harmful," "harmless," or "controversial" according to content moderation documents from OpenAI and Meta. This post-validation process ensures the reliability of our dataset. We describe this manual annotation phase as the third step in constructing the PHTest dataset in Section 4. We will enhance these descriptions to prevent any misunderstandings.

Answer to Q1: Eq. 1 presents the general formulation of this problem, requiring a classifier $\gamma_{harmless}$ and a threshold $\alpha_0$. In our method, we use a writer LLM as the proxy of the classifier, which not only saves computation but also effectively handles incomplete prompts during generation. As detailed in Appendix A, we let $\alpha_0=1$, transforming the objective into Eq. 2.

In Eq. 2, $\beta$ is a hyperparameter that balances two objectives. If $\beta$ is too small, the generated prompts may be harmless and unable to trigger refusal. If $\beta$ is too large, the generated prompts could trigger refusal but might be harmful. Therefore, we use a validation set (including harmless, pseudo-harmful, and harmful prompts) and a grid search to select $\beta$ to be the one where pseudo-harmful prompts have higher values of the objective (Eq 2) than other prompts. Note that, $\beta$ also depends on the LLMs we use since different LLMs have different objective ranges. Below is the range of $\beta$ in our three settings. We randomly sample a $\beta$ within the range for each run.

Thanks for the question. We will add these details to the paper.

Thank you for your valuable questions! Below, we address them in detail.

Answer to W1: Due to the page limit, please see our response to Reviewer Dgn7's last question. Thank you!

Answer to W2: Good question. In the appendix, we use "Sorry..." as an illustration. Below, we discuss the details.

• Selection of $y_{refusal}$:
  • A good $y_{refusal}$ should guide the generation of prompts that can trigger refusals in reality. Therefore, we choose $y_{refusal}$ as the most common refusal prefix for the target LLM by testing many harmful requests. In our experiments, we use "I cannot fulfill your request ..." for Llama2-7b, and "I cannot provide instructions on ..." for Mistral-7b.
• Effect of $y_{refusal}$:
  • As discussed in Sec 3.2, $y_{refusal}$ can be tailored to generate prompts targeting specific rule violations. For example, using "I can’t assist with copyright infringement" as $y_{refusal}$, the generated prompt could be "Where can I find open-access journals ..." Therefore, varying $y_{refusal}$ is a way to promote diversity. We only have a qualitative analysis on it since we fix $y_{refusal}$ but use reference prompts to achieve diversity. We will add the qualitative analysis to the paper.

Answer to W3: Figure 9 aims to see whether harmful and pseudo-harmful prompts can be differentiated by the second term in Eq. 2. Here's a detailed breakdown of the experiment:

• We collect prompts from 3 classes: harmless, pseudo-harmful, and harmful. The last two are from the XSTest dataset.
• We calculate the likelihood of refusal (i.e., the exponential of the second term in Eq. 2) for each prompt. The likelihood of refusal serves as a feature for classifying the three prompt classes. We then compile a list of feature values for each prompt class.
• With kernel density estimation (KDE) and these feature values, we draw Fig. 9. The colors denote the class: blue for harmless, orange for pseudo-harmful, and green for harmful (apologies for the typo in the legend). We observe that three classes are quite separable.
• Using the feature values of pseudo-harmful class and harmful class, we can calculate the AUC (with packages like sklearn) of classifying them to be 82.1%. This supports that although the target LLM refuses both pseudo-harmful and harmful prompts, the likelihoods are different. Thus, the likelihood of refusal is a good feature for distinguishing these two classes.

Thanks for this question. We will add these details to the paper.

Thanks for your valuable questions! Below, we address them in detail.

Answer to W1: We would appreciate further clarification on this question. If the concern is that our dataset generation process is similar to previous methods, we highlight several key differences:

• Previous datasets, XSTest and OKTest, either manually craft pseudo-harmful prompts or use GPT-4 to generate them with harmful words. It results in small datasets (XSTest has 250 examples, OKTest has 300 examples) with limited content, and often fail to trigger refusals.
• In contrast, our dataset is generated using a newly proposed algorithm that automatically generates pseudo-harmful prompts from scratch, with the optimization goal of triggering the refusal of target LLM. This approach produces a large, diverse, and natural dataset that can effectively trigger refusals of many LLMs. Our dataset contains 1.5K examples, featuring newly discovered categories of false refusals.

Therefore, our data generation process significantly improves the dataset quality and value compared to previous ones.

Answer to W2: This is a very good question. Yes, we believe that revealing detailed categories provides important insights for improving alignment.

• Identifying specific categories of false refusals allows developers to perform a more granular analysis of LLM performance. This detailed understanding helps in diagnosing the root causes of false refusals more effectively. For example, if the LLM frequently refuses prompts with homonyms and figurative language, the root cause might be a deficiency in language understanding. If the LLM refuses many requests related to public statistics, the root cause might be the over-generalization of safety fine-tuning.
• Based on the granular analysis, developers can implement targeted improvement strategies. For instance, they can fine-tune the LLM with examples containing homonyms and figurative language for the first case, and explore more requests related to public statistics that shouldn't be refused and incorporate them into safety fine-tuning to avoid over-generalization for the second case.

Thanks again for the question. We will add the discussion on the potential use of these categories to the paper.

Thank you for your valuable feedback! Below, we address the questions in detail.

Answer to W1&Q1: We would like to clarify that the categorization occurs after prompt generation. These categories are the natural outcomes of our autonomous generation algorithm. During generation, our method uses reference prompts to steer style and domain, rather than targeting specific categories directly. This approach ensures a diverse set of pseudo-harmful prompts, which we then categorize into "Compound term" or others. We will make this process clearer to prevent any misunderstandings.

Our dataset is of high quality. Unlike previous datasets that were manually crafted and limited in scope and diversity, ours contains 1.5K diverse prompts that effectively trigger false refusals across various LLMs. Using 7b models like Llama2-7b, Vicuna-7b, and Mistral-7b as writer LLMs, and ShareGPT as the reference prompts, has proven effective in generating these high-quality pseudo-harmful prompts.

Answer to W2&Q2: Thanks for the insightful suggestion!

• Our method serves as a red-teaming tool to help LLM developers identify false refusals specific to their models. When using single target LLM, the generated prompts are tailored to it, which is beneficial for red-teaming purposes.
• To ensure a large and diverse dataset, we collect pseudo-harmful prompts generated from three settings (writer LLM + target LLM to be Llama2-7b + Llama2-7b, Vicuna-7b + Llama2-7b, and Mistral-7b + Mistral-7b), forming the PHTest dataset. We observe that the pseudo-harmful prompts generated from a single target LLM already exhibit some transferability, as they can trigger false refusals of various black-box LLMs, such as Claude.
• We agree that using an ensemble of target LLMs is a promising way to enhance transferability. We would like to explore it in the future work since this approach presents two challenges. Firstly, it requires more memory and computation resources. Secondly, different LLMs have different vocabularies and tokenizers, complicating the optimization of the next token. One potential solution is to use tokenizer adapters [1]. Thanks again for the great suggestion. We will include a discussion on this and explore it in our future work.

[1] Minixhofer, Benjamin, Edoardo Maria Ponti, and Ivan Vulić. "Zero-Shot Tokenizer Transfer." arXiv preprint arXiv:2405.07883 (2024).