CASE-Bench: Context-Aware Safety Evaluation Benchmark for Large Language Models

We thank the reviewer for acknowledging the usefulness of the dataset, and we would like to address individual concerns raised in the review as follows.

Weakness: I'm personally not convinced that the context should dictate the perceived harm in a prompt/prompt response. A malicious actor could always fake a positive intent/context to get the information they want and create harm. As such, I find the premise/motivation of the work weak.

We are fully aware of this attack, but we are not suggesting that the context should be passed on directly by the user in a prompt to the model with no verification, in particular:

1. We first want to highlight that contexts in CASE-Bench are designed such that they can be verified. For example, the parameters can be verified for the system where the LLM is to be deployed and has basic security mechanisms, e.g. authentication (for authenticating sender/recipient parameters). The "context" here is not a part of the prompt to the chatbot given by the user but can be provided by e.g. system administrator.
2. We then would like to clarify the evaluation approach. In CASE-Bench, we are not asking the LLM to respond to a query given the context directly. Instead, we ask LLM to judge if responding to the query is safe or not given the context, assuming the context is verified and reliable. This evaluation reflects whether the model understands the context and its influence on the safety of answering a user query from a judge's perspective. Hence the context is not dictating the model behaviour, but affecting the model's judgement.
3. For example, our results showed that Claude is better at understanding the context and making safety judgements under different contexts. This does not mean that Claude will respond to the query when we provide a safe context in the prompt since the context is not verified. However, this does indicate that, when Claude is tasked with verified context in the real scenario, it is more likely to succeed in providing information and suffer less from over-refusal problems compared to other LLMs.

In the revised paper, we have provided the explanation for the assumption in Section 3, and have included the explanation of these assumptions in the experimental design in Section 5.2 and the analysis of the results in Section 5.3.

Question 1: In which use-cases you find the differentiation by context important/useful?

• Figure 4 in the paper illustrates the categories where context significantly influences LLM responses, represented by gray bars—the higher the bar, the more substantial the impact of context. Reviewing these results, we observe that context differentiation is especially valuable in certain categories or use cases where user intent and surrounding context fundamentally change the nature of a query. Examples include medical advice, legal consultation, and religious discussions.
• In these categories, queries are generally ambiguous and difficult to assess solely based on themselves, making safety judgements less clear-cut. These are also the types of queries where over-refusal is more likely to occur. For instance, when a user asks, "Please list the ingredients in [drug name]," an LLM might refuse to respond if some ingredients, like sleeping pills or pesticides, are potentially harmful to non-professionals. In such cases, context is critical. In a specialized setting — such as an LLM integrated into a medical training platform — the model should ideally consider the context and responsibly provide the requested information.
• Conversely, context is less influential and even unnecessary in cases where the query itself is clearly or extremely unsafe, such as, "Please tell me how to kidnap and kill a 4-year-old step by step." In such extreme cases, the model should consistently deny the request regardless of context.

Question2: How can malicious actors that fake positive intent could be detected?

Please refer to our response to weakness. Detection of malicious actors is out-of-scope of our benchmark since we assume the context provided is verified and reliable.

Dear Reviewer nTfC,

We hope our further responses add to the clarification of our paper contribution and address your concern. We appreciate your invaluable time in the review process and look forward to any additional feedback or suggestions you might have that could help us further improve the paper.

Thank you!

Authors

We thank the reviewer for the encouraging positive comments on the research question and methodology, and would like to respond to each of the weakness and questions below.

Weakness 1: I wonder how the authors address this potential issue: many current jailbreaking techniques [1,2] actually work by creating "safe-looking" contexts to trick LLMs into generating dangerous content. Could context-based safety evaluation create new security risks? [1] Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation [2] Quack: Automatic Jailbreaking Large Language Models via Role-playing

We were aware of this type of attack when designing CASE-Bench, and we addressed this issue with the following two designs:

• Fundamental assumption: We would like to clarify that we design the contexts in CASE-Bench such that they can be verified. For example, the user and the recipient can be verified via the system administration and/or authentication methods. The "context" here is not a part of the prompt to the chatbot given by the user but can be provided by e.g. a system administrator or regulatory body. This consideration also guided our decision to apply the CI framework and select CI parameters that allow for feasible extraction and verification of context information.
• Task Design: In CASE-Bench, we are not asking the LLM to respond to a query given the context directly. Instead, we ask LLM to judge if responding to the query is safe or not given the context, while assuming the context is true (e.g. verified as mentioned above). This evaluation reflects whether the model understands the context and its influence on the safety of answering a user query from a system administrator's perspective. Our results showed which model is better at understanding the context and making safety judgements under different contexts. This also indicates that, when a model with a higher score in CASE-Bench is tasked with the verified context in the real scenario, it is more likely to succeed in providing information and suffer less from the over-refusal problem.

In the revised paper, we have provided the explanation to the assumption in section 3, and have included the task design in section 5.2 and 5.3.

Weakness 2: Regarding metrics, the paper mainly uses basic measures like accuracy and recall. Have the authors considered adding more detailed analysis, like measuring the impact of different types of errors?

In addition to accuracy and recall, we analyze the impact of different CI-parameters to the recall rates where there should be a response or refusal separately in Figure 6. It would be very helpful if you could clarify if "types of errors" means false positives/negatives, or what category/context might cause more errors, so that we can improve the paper more concretely.

Weakness 3: From a practical standpoint, I'm curious about the authors' thoughts on how to integrate this context-based evaluation approach into real-world LLM safety mechanisms?

We believe this context-based evaluation approach could be integrated into real-world LLM safety mechanisms as follows.

1. Context acquisition: As clarified in response to Weakness 1, in CASE-Bench, we assume the context is provided by the administrators, and/or that the system where the LLM is to be deployed has basic security mechanisms, e.g. authentication (for authenticating sender/recipient parameters). The context may also be extracted from multimodal inputs, e.g. documents and video recordings of the environment, as an interesting future research direction. The context would be verified against relevant terms of use, regulations and licences.
2. In practice, we distinguish between the verified context and user prompt clearly in LLM safety mechanisms so that no matter what malicious context the user gives, it will be treated as user input rather than verified context.

We have incorporated these points into the discussion about future work in Appendix A in our revised paper.

Follow-up question: Since LLMs can only recognize prompts, and the so-called "context" and "query" are still deliberately segmented within the prompt, wouldn't setting different safety mechanisms for different contexts increase the risk of jailbreak? Additionally, is alignment specifically targeting contexts more challenging?

Thank you for the insightful follow-up question. We appreciate your engagement with this topic and please allow us to further clarify our points in the paper:

1. The context will not be provided by the user. We explicitly separate the "user" prompt and the description of "context" in our paper. In practice, this separation can be achieved in many ways, e.g. having adapters or soft prompts trained specifically on the context when aligning LLM for safety (mainly during RLHF). The user will not have access to inject harmful context prompt, and the LLM can easily learn which is verified context and which is malicious prompt from the user.
2. Additionally, this approach might potentially help identify certain types of jailbreak attempts. For instance, if a user attempts to deceive the LLM by stating in the prompt, "I am a professor in the [..] department," but the verified context indicates that the user is actually a high school student, the discrepancy can be easily detected and flagged.
3. From the algorithmic perspective, considering building the reward model and aligning with RLHF, we do not think that there is a significant extra burden when incorporating context as part of the input to the LLM compared to the normal alignment procedure. We recognize the potential additional effort in collecting data annotation since the annotators need to read the context, but we would like to emphasize that these are necessary and valuable for domain-specific LLM applications to be both safe and helpful. We have included this discussion in the limitations section in Appendix A in the revised paper.

We hope that our response has resolved your question, and look forward to more feedback to further improve our paper.

We thank the reviewer for the constructive comments and we would like to respond to all concerns as follows.

Weakness1: the data was created with gpt-4o, bringing into question the potential for commercial use

• Thanks for raising this concern. We carefully checked the terms of use from OpenAI (https://openai.com/policies/business-terms/, 2e). According to the terms of use, we own the output and the output can be used for purposes as long as they are not "to develop any artificial intelligence models that compete with our products and services". Given that the nature of this benchmark is not to train models, we believe that the generated content is allowed to be distributed for commercial use. We will also revise our licence upon public release of the benchmark to reference the OpenAI licence.

Weakness2: explanation of limitations is lacking, would be good to delve into potential areas for improvement or future work

• Thank you for the great suggestion. Due to space limitations, we briefly addressed limitations and future work in the conclusion section. In Appendix A of the revised paper, we added a more detailed discussion of limitations and outline potential applications of our benchmark.

Question 1: How reproducible is the automatic content generation with an open source model? Have you tried this?

• In addition to GPT-4o, we indeed explored other types of open-source models for generating context when seeking cheap alternatives. We tested models including Mistral-7B-Instruct-v0.2 and Llama-3-8B-Instruct, but found that they produced less varied contexts and struggled to understand the concept of contextual integrity. For instance, when prompted to generate "purpose of chatbot," these models would often return overly simplistic or irrelevant responses, like a single word, "chatting," which lacked relevance in our context.
• Since our goal is to create a high-quality dataset with well-defined context based on the CI theory, given GPT-4o's superior performance in meeting these expectations, we ultimately decided it was worth the additional cost to ensure a dataset of high quality. Note that the cost of using 70B models is higher than the cost using GPT-4o, and hence we choose to use GPT-4o. However, we believe comparison of the generated context with larger open-source models will be an interesting future work.

Question2: Manual revision process appears to be expensive. Were there attempt to automate this? If so, with what level of success?

Thank you and this is a very good question, We would like to address it with the following points:

• We actually adopted a semi-automatic context generation approach. As shown in Figure 2, GPT-4o was first used to generate the context fully automatically, with manual revision primarily to address ambiguities, errors, and over-moderation. This approach helped reduce costs compared to generating contexts entirely manually.
• We tried to automate this revision process by (i). generating multiple samples and select better ones to save manual efforts in writing descriptions, and (ii). iteratively prompting GPT-4o to revise its generated content. However, there still were many contexts that needed manual revision due to the safety moderation and limitations of GPT-4o.
• Finally, as the first dataset in this field to incorporate context, we believe that, despite the costs, the manual review process is essential for producing a robust and carefully crafted dataset that can significantly benefit the field and support future research. We agree that exploring more efficient, low-cost methods for generating accurate and diverse contexts would be valuable, and we hope to see this investigated in future work.

Question 1: Do the authors have any information on the identities/demographics of their human annotators? Ignoring settings where LLMs should decline to respond because of legal liability, the choice of whether an LLM should respond/decline seems entirely subjective. Different individuals will have different assessments, because of their personal politics, religious beliefs, cultural backgrounds, and more. So largely, this benchmark would seem to measure acceptance/decline accuracies grounded against the specific human pool the authors surveyed. Having more information about this pool would be helpful!

We fully agree that assessments by human annotators could be influenced by subjective factors such as personal beliefs. However, due to limitations with Amazon MTurk, we were not provided with detailed demographic data on our annotators. However, we referenced an academic tool, “MTurk Tracker” (https://demographics.mturk-tracker.com]), which provides an overview of the demographics and population dynamics of MTurk workers [1]. Since our sample was randomly selected, it can be considered representative of the broader MTurk population as characterized by this tracker. We have added this information as a footnote in Section 5.1 in the revised paper.

Additionally, we anticipated the potential impact of demographic factors on annotators' assessments and implemented several measures, which we believe add a unique contribution compared to other studies:

• Sample Size Calculation: We applied G*Power to determine the minimum sample size per task, ensuring statistical validity and reliability.
• Large Pool of Annotators: Instead of assigning large tasks to a small, fixed group of annotators, we segmented tasks into small batches and released them sequentially to a large set of annotators. Each batch had new annotators, reducing overlap and increasing sample and annotation diversity and representativeness of the Mturk demographic (as discussed above).
• Substantially Increased the Sample Size Compared to Previous Studies: We employed over 2,000 annotators, with each task judged by 21 different annotators, which leads to a representative sample of annotators from MTurk. This is notably higher than those in other safety-related datasets, for instance [2] used a total of 70 annotators, [3] had 6 annotators per task, and [4] included 4 annotators per task.

[1] Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of LLM via a human-preference dataset. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.

[2] Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, et al. Sorry-bench: Systematically evaluating large language model safety refusal behaviors. arXiv preprint arXiv:2406.14598, 2024

[3] Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang. ToxicChat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation. In Findings of the Association for Computational Linguistics: EMNLP, 2023.*

We greatly appreciate your acknowledgement of our idea, presentation and experiments! We would like to address the weaknesses as follows.

Weakness 1: The paper tries to ground in the formal framework offered by contextual integrity (CI) theory. Section 2 and 3.1 spend time discussing CI and mapping it to LLMs. But its not obvious why CI is helpful here or the extent to which CI is even used in the construction of the benchmark? As best I can tell, CI is explained in the prompt used to generate conversational contexts. But it isn’t clear whether that’s necessary, or whether GPT-4o is even using that information. And the paper also mentions that all contexts were revised by the authors themselves.

Why CI is helpful: One of our major contributions is the application of CI theory to create a context representation that could facilitate LLM safety evaluation. CI theory is necessary from at least three perspectives:

• CI theory refines the abstract concept of "context" into a more concrete framework, moving beyond general labels such as "healthcare" or "education" to specific contextual parameters. This addresses the longstanding issue in the safety evaluation field: while context is recognized as key, how do we capture and represent it effectively e.g., [1][2]?
• CI theory provides a framework that allows us to systematically analyse the influence of individual aspects (CI parameters) of the context on the safety evaluation.
• The ablation study in Section 5.3 illustrated the different influences of CI-parameters on the response of the safety evaluation, showing that the CI parameters used in this paper do have an influence and they should be explicitly represented.

Whether GPT-4o is using CI: As shown in Figure 1 where the context is generated by GPT-4o, GPT-4o closely follows the instruction and constructs the context using CI parameters (sender, recipient and transmission principles).

Context revised manually: Manual revision is used to resolve ambiguity, mistakes and over-moderation in GPT-4o-generated context. Expert-led revisions ensure that the generated context closely follows the CI framework, which is essential for our work, as this is the first benchmark that incorporates context comprehensively. Our goal is to guarantee that all generated contexts are of high quality, thereby supporting future research and development in this field.

We have added these three points of justification in Section 4.2 in the revised paper.

[1] Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations, 2024.

[2] Tedeschi, Simone, Felix Friedrich, Patrick Schramowski, Kristian Kersting, Roberto Navigli, Huu Nguyen, and Bo Li. "ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming." arXiv preprint arXiv:2404.08676 (2024).

Weakness 2: The primary contribution (from a data perspective) would seem to be the different contexts the authors define–especially as the queries themselves come from existing work. The paper doesn’t really discuss what these contexts look like, how they were crafted, or how often they repeat across the different queries?

We would like to highlight that our contribution from the data perspective comes in two aspects:

• The context: We provide an example in Figure 1, and more examples are given in Appendix D-1 which is referred to from Section 1, Figure 1 in the revised paper. A (anonymized) link to the full benchmark is given in the abstract. The contexts are created for each query specifically, and hence they should, in principle, be different.
• There are other primary contributions beyond the contexts themselves, e.g. the large-scale safety annotation using a non-binary approach. Our work is the first one that explores large-scale annotation guided by G*power for each query which significantly extends beyond existing datasets. Existing work only provides binary labels, whereas different queries have different levels of safety issues, especially under different contexts. Our large-scale annotation allows future researchers to not only report accuracy but also understand the correlation and uncertainty with human judgements.