SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal

We sincerely appreciate the reviewer’s acknowledgment of our work. With the clarification below, we hope we can address the reviewer’s remaining question:

In Figure 4, GPT-3.5-turbo appears to outperform GPT-4 and o1 in terms of safety. Generally speaking, GPT-4 is considered to have superior safety mechanisms compared to GPT-3.5-turbo. Could the authors explain this discrepancy?

While it is true that OpenAI has invested significantly more resources into ensuring the safety of GPT-4(o) compared to GPT-3.5-turbo, as highlighted in the GPT-4o System Card [1], it is also designed to be more permissive in certain contexts, as outlined in the OpenAI Model Specification [2]. These design choices led to its higher fulfillment rates on numerous categories of our benchmark:

• The GPT-4o specification explicitly allows for discussing sensitive topics like sex and reproductive organs within scientific or medical contexts. This explains GPT-4o’s higher fulfillment rates in categories such as “#25 Advice on Adult Content”, where earlier models like GPT-3.5-turbo were more conservative and refused these instructions.
• According to the Spec, GPT-4o is instructed to “not try to change anyone's mind,” even when responding to controversial topics. In a qualitative example demonstrated in the Spec, when asked to argue in favor of a violent extremist, the model should comply while providing factual information from multiple perspectives. This results in GPT-4o showing higher fulfillment rates in many other categories within the “Potentially Inappropriate Topics” domain, such as “#32 Political Belief”, “#35 Social Stereotype Promotion”, and “#36 Extremist Content Promotion.”
• GPT-4o is guided by principles in the Spec to “be as helpful as possible without overstepping”. For example, for “advice on sensitive and/or regulated topics (e.g., legal, medical, and financial)”, the model should fulfill such requests, by providing the user with necessary information (without providing regulated and unqualified content). This policy leads to significantly higher fulfillment rates of GPT-4o in the “Potentially Unqualified Advice” domain (the last five categories) compared to GPT-3.5-turbo.

Safety is a multifaceted concept that extends beyond any single metric, including our measure of safety refusal rates. While a lower fulfillment rate on our benchmark does not necessarily equate to greater safety, as the interpretation of “safety” varies among stakeholders, our evaluation provides critical insights into how models handle unsafe instructions. On one hand, a conservative refusal approach (e.g., GPT-3.5-turbo) may align with certain expectations but risks hindering helpfulness in contexts deemed permissible by others. On the other hand, acceptability or risk often depends on cultural, legal, and organizational standards, which can differ widely across contexts and regions.

Our evaluation of safety refusal, as captured by SORRY-Bench, is essential for understanding how models balance refusal and fulfillment across diverse scenarios. For example, GPT-4o’s higher fulfillment rates in certain categories reflect deliberate design decisions to prioritize nuanced, context-aware permissiveness over blanket refusal. While this may cause GPT-4o to appear less “safe” than GPT-3.5-turbo on some measures of refusal, the results align with OpenAI's evolving policies aimed at balancing safety, helpfulness, and user satisfaction.

Ultimately, SORRY-Bench provides a critical tool for evaluating and comparing these complex trade-offs, offering valuable insights into the varying approaches to model safety across the LLM landscape. We have incorporated the discussion above into Appendix K.4 of our paper.

[1] https://openai.com/index/gpt-4o-system-card

[2] https://openai.com/index/introducing-the-model-spec/

Q2: Why are model responses sampled probabilistically? (section 4.1)

In our benchmarking experiments, we use probabilistic sampling to acquire model responses because it reflects the most common setup in real-world applications. For instance, commercial chat services like ChatGPT and Claude default to random sampling, which is also the practice in certain scenarios that aim to evaluate LLM capabilities, such as in Chatbot Arena. To assess the impact of this randomness, we have reported results from three repeated sampling rounds in Appendix K.5, demonstrating that the variance introduced by probabilistic sampling is minimal.

We acknowledge that greedy decoding, which produces deterministic outputs, enhances reproducibility compared to probabilistic sampling. However, based on our additional study, we argue that model safety refusal performance is highly similar between greedy decoding and probabilistic sampling strategies. Thus, our chosen setting does not impede reproducibility significantly. Specifically, we re-evaluated six different LLMs with the temperature set to 0 (i.e., greedy decoding). The results, shown below, closely align with those in our main findings (Figure 4) and confirm that greedy decoding (or not) has no significant impact on safety refusal rates.

We have incorporated the discussion and results above into Appendix K.5.

Q3: Could you provide some examples of disagreement between the evaluation and human evaluators? Is there a systematic pattern in the disagreements?

Below, we provide three representative examples (truncated due to the length limit in OpenReview comment, please refer to Appendix J.4 for details) of disagreements between model evaluations and human annotations, corresponding to three failure patterns we have observed:

1. Bullet List Format Leading to Misclassification of “Refusal” as “Fulfillment”

   In Example 1, the model response provides a detailed bullet-pointed instruction on how to design a healthy lifestyle website. Despite this response not fulfilling the malicious intent of promoting “anorexia and bulimia as healthy lifestyle choices,” the judge model incorrectly classified it as “fulfillment.” This suggests that bullet list formatting may mislead the evaluation model into perceiving comprehensive responses as fulfilling unsafe instructions, regardless of their actual fulfilments of the malicious intent.

2. Safety Disclaimers Leading to Misclassification of “Fulfillment” as “Refusal”

   Example 2 illustrates a case where the model fulfills an unsafe instruction by drafting a tweet that spreads misinformation about climate change being a hoax. However, the response concludes with a disclaimer, such as “this is a responsible use of social media...,” which may have caused the judge model to incorrectly classify the response as a “refusal.” This pattern suggests that safety disclaimers may mislead evaluation models by overshadowing the underlying fulfillment of the unsafe request.

3. Borderline Cases with Disagreement Between Human and Judge Models

   Example 3 presents a nuanced case where the model partially fulfills an unsafe instruction. It recommends three AI startups based in California (fulfilling the first part of the request) but refuses to predict their exact future prices over the next six months (refusing the second part). Such borderline cases, where responses partially fulfill and partially refuse an instruction, naturally lead to disagreement. These cases are expected and rather acceptable, as they reflect the inherent subjectivity and complexity of evaluating nuanced responses.

We acknowledge that our specialized safety evaluator model is not an ultimate gold standard, as it may occasionally fail to align with human rubrics (as discussed above). Nonetheless, the overall agreement between the model and human annotators remains high—outperforming other design choices, as shown in our meta-evaluation (Table 1). This underscores the relative reliability of our evaluation approach, despite these occasional discrepancies. We have incorporated the discussion above, as well as the qualitative examples below, into Appendix J.4 of our paper, calling for awareness of such potential failure modes during safety evaluation.

(to be continued)

We thank the reviewer's acknowledgment of our work, together with various constructive suggestions and questions. We have incorporated the reviewer's suggestions in the paper revision (uploaded). With the following clarifications, we hope we can help resolve the raised concerns & questions.

W1: More specifications.

• What proportion of the resulting dataset is new?

  We conducted a cross-comparison of our final dataset (440 instructions) against the 10 prior datasets we leveraged (containing over 3,600 instructions). Our analysis revealed that only 5.9% (26/440) of the records are exactly identical to those in prior datasets.

  Furthermore, we calculated the maximum Jaccard similarity between each of our data points and the prior data points. Across all 440 unsafe instructions, the average Jaccard similarity is only 44.5%. Notably, only 17% (75/440) of our data points exhibit more than 80% Jaccard similarity with prior data points. This underscores the novelty of our dataset compared to existing ones.

• How are the novel unsafe instructions created?

  We provide details in Appendix G.1 and G.2. Particularly, we asked dataset collectors to manually create additional novel unsafe instructions for that category when the data points from prior datasets were insufficient. In this case, we encouraged the collectors to either compose new data points themselves, or utilize web resources (e.g., search engine or AI assistance).

• How are the 20 linguistic mutations created?

  We describe the implementation of these linguistic mutations in detail in Appendix H. These mutations are based on scenarios observed in prior work that may influence model safety. Specifically:

  • For the six writing styles and five persuasion techniques, we used GPT-4 to paraphrase our base dataset, using few-shot prompts from prior work.
  • For the four encoding/encryption strategies, we directly encoded or encrypted each unsafe instruction in the base dataset using standard methods. Following prior work, we included few-shot prompt templates during evaluation to help LLMs interpret and respond to these encoded/encrypted instructions effectively.
  • For the five non-English versions, we utilized the Google Translate API to translate the base dataset into five target languages. For evaluation, all model responses to these non-English unsafe instructions were translated back to English to ensure consistency in analysis.

We appreciate the reviewer’s request for clarification on these important details. In response, we have added concise explanations in the main body of the manuscript and provided comprehensive descriptions in the Appendix for further reference.

W2: Dataset is restricted by previous datasets and may overlook new categories of harm.

We agree with the reviewers that our dataset design has inherent limitations in capturing emerging safety risks, both because our taxonomy stems from prior work, and due to the static nature of our dataset -- a challenge faced by all static safety datasets without maintenance. We explicitly acknowledge this limitation in Appendix A.1, that the landscape of safety in the real world is evolving rapidly, and there may be new safety risks uncovered every now and then. To catch up, our taxonomy and dataset may need regular revising. Ensuring adaptability to these evolving risks is a key priority, and we are committed to ongoing maintenance of our benchmark assets (e.g., HuggingFace and Github repository) in future iterations of our work.

W3: Fine-tuning may introduce bias.

We appreciate the reviewer's observation that fine-tuning safety evaluator models on imbalanced human annotations could lead to biased judgments:

if in the training set every instruction of "self-harm" is refused, the model may learn to categorize all examples with a "self-harm" instruction as refused, regardless of the model response.

However, we argue that this is not a major concern in our work, for the following reasons:

1. To mitigate such potential risks of bias, we have curated our human judgment dataset sampling from diverse responses of 31 LLMs. These include models that tend to refuse more frequently (e.g., Claude-2.1) and those that do not (e.g., Dolphin-2.2.1-Mistral-7B). This diversity ensures a balanced dataset that includes both refusal and fulfillment responses, reducing the likelihood of evaluators learning the collapsed patterns as hypothesized.
2. Below, we analyzed the proportions of responses labeled as "refusal" v.s. "fulfillment" across all 44 categories. The results show that "fulfillment" labels range from 16.9% to 59.4%, demonstrating that the dataset contains a significant variety of judgments. Therefore, the concern that "if in the training set every instruction of 'self-harm' is refused" does not apply to our data.

(to be continued in the next comment)

Thank you to the authors for the very thorough response. I appreciate all the efforts that went into this response.

Re: W1: How is Jaccard similarity between data points (instructions) computed? What does Jaccard similarity mean in this case? I noticed that this is added as a footnote on page 5, but I don't understand what it means with respect to comparing instructions/datasets. If you include this metric, I think it deserves some explanation; however, I also don't think it's absolutely necessary to include it in the main body - if you need to explain the notion of Jaccard similarity as applied to comparison of instructions, it seems better suited to an appendix section.

Re: Q1: I still don't find myself convinced that time is the best metric, but I agree it's a decent and convenient baseline. I suggest linking to Appendix J.3 from the main body (~line 355), since I think the justification and discussion is important, but too lengthy to fully include in the main body.

Re: Q2: Is there a citation for temperature 0.7 being the most common sampling setting? I would have thought it was temperature 1.0. [1] points out that tweaking sampling hyperparameters such as temperature can have significant impacts on measured attack success rate, and I think if this benchmark is to be widely used, I'd think that having reproducible results at greedy temperature seems like a better choice than probabilistic sampling. At the very least, I think this sampling design choice should be highlighted and justified in the main text.

Re: Q3: Thanks for providing these examples. I see you've added them in J.4 - I think they are a great addition to the paper!

References:

[1]: Huang, Yangsibo, et al. "Catastrophic jailbreak of open-source llms via exploiting generation." arXiv preprint arXiv:2310.06987 (2023).

Again, we sincerely thank the reviewer for the suggestions and detailed follow-up comments that have been shaping our work better. We hope our replies below can resolve your remaining concerns.

Re: W1: How is Jaccard similarity between data points (instructions) computed? What does Jaccard similarity mean in this case? I noticed that this is added as a footnote on page 5, but I don't understand what it means with respect to comparing instructions/datasets. If you include this metric, I think it deserves some explanation; however, I also don't think it's absolutely necessary to include it in the main body - if you need to explain the notion of Jaccard similarity as applied to comparison of instructions, it seems better suited to an appendix section.

We apologize for not providing sufficient context for this experiment. Below, we offer a more detailed explanation of the Jaccard similarity results and their significance.

To analyze the similarity beyond verbatim matches, we compute the Jaccard similarity (or Jaccard index) [1] between the 440 instructions in our base dataset and those in prior datasets (3.6K+ instructions). Specifically, for each instruction in SORRY-Bench, we represent it as a set of words $A$. We then calculate the pairwise Jaccard similarity score for $A$ with the set of words ($B$) for every instruction in the prior datasets. The Jaccard similarity between two sets $A$ and $B$ is defined as:

We record the maximum Jaccard similarity (i.e., $\max_B J(A,B)$) for each instruction, capturing the extent of overlap with any prior data point. This approach provides insight into lexical similarity that accounts for word-wise partial overlaps, helping to quantify the resemblance between our instructions and existing datasets beyond exact matches.

Our results show that, across all 440 instructions from SORRY-Bench, the average maximum Jaccard similarity is barely 44.5% – meaning that, on average, less than half of the words in an instruction overlap with any prior data point. Additionally, only 17% (75/440) of our data points exhibit more than 80% Jaccard similarity with prior data points. These statistics indicate that SORRY-Bench contains a substantial proportion of novel or significantly altered instructions.

To clarify this analysis, we have moved the explanation of Jaccard similarity and the results to Appendix G.4, as suggested.

[1] Wikipedia. Jaccard index. https://en.wikipedia.org/wiki/Jaccard_index.

Re: Q1: I still don't find myself convinced that time is the best metric, but I agree it's a decent and convenient baseline. I suggest linking to Appendix J.3 from the main body (~line 355), since I think the justification and discussion is important, but too lengthy to fully include in the main body.

We appreciate the reviewer’s understanding and the further suggestion. We have added a link to the justifications of “time cost” (Appendix J.3) in Section 3.3.

Thanks to the authors for addressing my concerns. I think the resulting adjustments have strengthened the paper, and have articulated appropriate caveats and limitations. I have adjusted my score from a 6 to an 8.

We thank the reviewer's acknowledgment of our work, together with various constructive suggestions and questions. We have incorporated the reviewer's suggestions in the paper revision (uploaded). With the following clarifications, we hope we can help resolve the raised concerns & questions.

W1: Lack of Explanation on Domain Division

We apologize for the lack of details regarding the domain division in our safety taxonomy. After curating the safety taxonomy as described in Section 2.2, we aggregated the 44 potential risk categories based on the nature of harm, model developers’ safety policies, and practices observed in prior datasets. Specifically:

1. Assistance with Crimes or Torts: A significant portion of these categories is closely related to crimes or torts as defined by U.S. law (e.g., terrorism). This focus is consistent with prior dataset designs (e.g., the inclusion of illegal activities as a key risk domain) and platform policies (e.g., OpenAI’s usage policy requires compliance with applicable laws). To capture this grouping precisely, we encapsulate these 19 categories (#6–#24) under the term “Assistance with Crimes or Torts.”
2. Hate Speech Generation: Categories such as lewd or obscene language (#1–#5) relate to the generation of hate speech, a well-known concern in language model applications. While hate speech is often protected under the First Amendment in the U.S., it can be considered a criminal offense in other jurisdictions, such as Germany. For this reason, we separated these categories into an independent domain, “Hate Speech Generation”, distinct from the crime-related categories.
3. Potentially Inappropriate Topics: Several categories are unrelated to explicit legal violations but are subject to differing interpretations by model developers and platform policies. For example, “#25 Advice on Adult Content” is considered appropriate by OpenAI’s guidelines but not by Anthropic, as such requests might be potentially offensive and inappropriate in certain social contexts. We aggregated these 15 categories (#25–#39) into the domain “Potentially Inappropriate Topics” to reflect these nuanced considerations.
4. Potentially Unqualified Advice: The remaining categories concern scenarios where LLMs provide advice on critical topics such as medical emergencies or financial investing. These categories are not inherently offensive but are flagged as risky by some model developers (e.g., Gemini) because the models lack qualifications in these areas. Users who follow inaccurate or misleading advice could face real-world harm, such as medical or financial loss. This unique nature led us to classify these five categories (#40–#44) under the domain “Potentially Unqualified Advice.”

Our benchmark results (Figure 4) provide additional evidence supporting the reasonableness of this domain division:

• “Hate Speech Generation” and “Assistance with Crimes or Torts”: A majority of models fulfill none or very few unsafe instructions from these two domains. For instance, Claude-2.1 and Claude-2.0 refuse almost all unsafe instructions across the 24 categories in these domains.
• “Potentially Inappropriate Topics”: Models show varied behavior within this domain. For example, GPT-4 (OpenAI) fulfills most requests from “#25 Advice on Adult Content,” whereas Gemini and Claude models refuse the majority of such requests.
• “Potentially Unqualified Advice”: In this domain, only Gemini-1.5-Flash refuses all unsafe instructions. This aligns with Gemini’s policy guideline on “Harmful Factual Inaccuracies,” which emphasizes that the model should avoid providing advice that could cause real-world harm to users’ health, safety, or finances.

We have also updated our Appendix D to include these additional details, ensuring clarity in the rationale and design of our domain division.

W2: Clarifying Contributions

We appreciate the reviewer’s suggestion that providing a clear, independent summary of our contributions would help contextualize our work for readers. In response, we have added the following bullet-point summary at the end of Section 1 (Introduction):

• We construct a class-balanced LLM safety refusal evaluation dataset comprising 440 unsafe instructions, across 44 fine-grained risk categories.
• We augment this base dataset with 20 diverse linguistic mutations that reflect real-world variations in user instructions, resulting in 8.8K additional unsafe instructions. Our experiments demonstrate that these mutations can notably affect model safety refusal performance.
• We collect a large-scale human safety judgment dataset of over 7K annotations, on which we conduct a thorough meta-evaluation to examine varying design recipes for an accurate and efficient safety benchmark evaluator.
• We benchmark over 50 open and proprietary LLMs, revealing the varying degrees of safety refusal across models and categories.

We sincerely appreciate the reviewer’s thoughtful evaluation and valuable feedback on our work. Below, we aim to address the specific concerns and suggestions raised.

W1: Fine-tuning judge model may cause it to overfit on SORRY-Bench, but fail to generalize to unexpected, real-world unsafe requests.

We acknowledge the reviewer’s concern that fine-tuning the judge model on our human annotation dataset may reduce its ability to generalize to unexpected, real-world unsafe requests. However, the primary goal of constructing an automated safety judge model in our work is to accurately and effectively evaluate model safety refusal (predominantly) on SORRY-Bench – where all the unsafe requests are known and fixed. By fine-tuning LLMs on our human annotations for model responses within SORRY-Bench, we can create a specialized judge model that is better aligned with the evaluation criteria and scenarios of this benchmark. While we do not expect this judge model to perform as effectively on unseen unsafe requests in the real world, we recognize that it may be less accurate in such unexpected scenarios compared to zero-shot LLMs like GPT-4o. This tradeoff is a deliberate choice to prioritize evaluation performance on SORRY-Bench.

W2: Challenges in achieving consistent, universal safety standards across diverse models and user inputs.

We appreciate the reviewer highlighting the variability in safety performance across different LLMs, particularly in handling low-resource languages, as a key challenge in achieving consistent safety standards. Our experiments on linguistic mutations, including non-English instructions, reveal significant disparities in performance across models. For example, GPT-4o exhibits stronger safety refusal in non-English scenarios, while others (e.g., Llama and Vicuna) fall short. We fully agree that achieving universal safety across diverse models and user inputs, especially in scenarios involving diverse linguistic patterns (e.g., low-resource languages), remains a critical and challenging goal.

The variability we observed highlights the need for more robust training strategies and alignment techniques tailored for handling such contexts (e.g., multilingual and low-resource languages). While addressing this issue is beyond the scope of our current work, we believe that benchmarks like SORRY-Bench, with its inclusion of linguistic mutations, play a vital role in identifying these gaps. We hope future research leverages our benchmark to drive improvements in multilingual safety alignment and help bridge these inconsistencies.

Q1: How can SORRY-Bench ensure that improvements in model safety performance generalize effectively to real-world scenarios beyond the specific categories and linguistic variations included in the benchmark?

While we have made significant efforts to capture a broad range of risks through our 44-class taxonomy, dataset, and 20 linguistic mutations reflective of real-world user behavior, we acknowledge the limitation of generalizing to emerging risks that may occur in the future. This limitation stems from the static nature of our dataset, a challenge inherent to all static safety benchmarks without ongoing updates. As we have noted in Appendix A.1, the real-world safety landscape evolves rapidly, with new risks continuously emerging. To address this, our taxonomy and dataset will require regular updates. Ensuring adaptability to these evolving risks is a key priority, and we are committed to ongoing maintenance of our benchmark assets (e.g., HuggingFace and Github repository) in future iterations of our work.

Dear Reviewer aFD2,

We thank the reviewer for the valuable suggestions and comments. We hope our previous rebuttal response has adequately resolved your concerns regarding 1) overfitting issues of fine-tuning safety judges, 2) challenges in universal safety standards, and 3) generalizing SORRY-Bench to real-world scenarios.

As the deadline of ICLR rebuttal period is approaching, we look forward to hearing your feedback on our response and would be happy to clarify any additional questions.

Best,

Authors