CASE-Bench: Context-Aware SafEty Benchmark for Large Language Models

We sincerely appreciate Reviewer dcxK for recognizing the contributions of our research and for the insightful questions. We address them as follows:

1. Extra Experiments on Reasoning Models:

• We provide results on DeepSeek-R1 as follows. | LLM |Method| Accuracy | R(Safe/Unsafe) | | -------- | ------- | ------- |------- | | Deepseek-R1 | Binary |87.9% | 84.0%/91.0% | | Claude-3.5-sonnet | Binary |88.7% | 86.5%/90.4% | | GPT-4o-2024-08-06 | Binary |77.0% | 54.6%/94.8% |

• We observe that while DeepSeek-R1 is not specifically optimized for safety, it achieves performance similar to Claude-3.5-sonnet, which is safety-optimized, and significantly outperforms GPT-4o-2024-08-06. This suggests that improved reasoning capabilities can potentially benefit safety judgements.

• We will incorporate these observations and recommendations into the paper.

2. Recipient Verification:

• Our work assumes access to recipient information as a controlled input for benchmarking purposes. In practice, we envision several approaches that could be explored:
  • The system could obtain and use recipient-related information by directly requesting it from users, with clear consent explicitly granted. Where appropriate, general categories (e.g., “medical professional”) could be used in place of detailed sensitive attributes.
  • In institutional or enterprise environments, users may already belong to trusted identity groups (e.g., doctors, students, researchers), which can be referenced securely without disclosing individual-specific data.
  • Recipient-related information may also be obtained through privacy-preserving or cryptographic techniques. For instance, advanced cryptographic methods such as Zero-Knowledge Proofs can enable users to demonstrate that they meet certain criteria (e.g., being over the age of 18) without revealing their actual identity or exact age [1]. Additionally, differential privacy techniques [2] can be applied to obscure individual-level information while still enabling the evaluation of general thresholds or boundaries, thereby maintaining utility without accessing real user data.

3. Regarding partial compliance:

• Thank you for raising this question. In our study, we test LLM-as-a-judge and does not have explicit partial compliance behaviour. However, we believe our BCE results reflects model uncertainty, and hence can indicate the degree of partial compliance.

[1] Patil et al. “Age Verification using Zero-knowledge Proof”.
[2] Zhu et al. “More than privacy: Applying differential privacy in key areas of artificial intelligence”.

We thank the reviewer for acknowledging the importance of incorporating context in safety evaluations, and we would like to address the concerns as follows.

1. Regarding Jailbreaking:

• Context manipulation aims to bypass the safety mechanism and make LLM respond with harmful content, which is different from our setup. In contrast, we evaluate LLM-as-a-judge for safety evaluation and test if LLM can make safety decisions by considering the context.
• In practice, these judges are used by e.g. administrators and the context can be verified by the administrator. Therefore, the context is indeed out of reach from user input. We represent them using natural language and feed them as the precondition to the LLM in our experiments. But, as explained, this does not mean the context would be provided by the users.

2. Regarding whether the model treats context as verified or not:

• We instruct the model to treat the context as verified, and the model is used as a judge. The task is not to make a response, but to make a judgement. Therefore:
  • A good model should closely follow our instructions and give its judgements by treating the given context as verified and reliable.
  • LLM judgements can be affected by the model internal safety mechanism. If the internal safety mechanism is too harsh, it will tend to flag almost everything as unsafe, even what is indeed safe, which yields a worse alignment with human judgements.

3. Regarding under what context or types of queries human and LLMs' judgements are mismatched.:

• We used Llama3-70B-Instruct model as a representative example to analyze the mismatch between human and LLM judges as suggested by the reviewer.
• Human thought safe where LLM flagged unsafe:
  • These mainly happens to queries that are less harmful themselves, including "fake news", "political belief" and "explicit content generation".
  • In these cases, the LLMs fail to consider whether it really causes actual harm given the specific context, and makes the decision only based on its internal safety mechanism.
• Human thought unsafe where LLM flagged safe:
  • These mainly happens to queries that may cause more serious social impact, including "violent crimes" and "child abuse".
  • These queries are mostly harmful regardless of the context. Therefore, errors occur when LLM is influenced by the context and make a "safe" decision.

We now provide specific examples in the github repo and will also add this analysis to the revised paper.

We sincerely thank reviewer P2mw for the detailed and constructive suggestions. We would like to address the following concerns:

1. Regarding experiments with LlamaGuard:

• Following the reviewer’s suggestion, and as an example, we have also evaluated LlamaGuard using our benchmark. We will add the following results to the revised version of our paper: | Setting | Accuracy | R(Safe/Unsafe) | PCC | BCE | | -------- | ------- | ------- | ------- | ------- | | Without Context | 54.1% | 25.3%/77.0% | 8.10 | 0.2661 | | With Context | 60.1% | 31.8%/82.6% | 26.34 | 1.7108 |

• We have the following observations:

  • Incorporating context improves the performance of LlamaGuard which makes it align better with human judgements. Hence modeling context has benefits in this case too.
  • The accuracy is not as good as proprietary LLMs with context, indicating a limited ability to understand the context with the current model which agrees with the reviewer's assumption.
  • We appreciate the reviewer for enlarging the application scope of the proposed benchmark. We agree with the reviewer’s observation that our benchmark could also be used to fine-tune safeguards. We also believe that our benchmark can be used to evaluate any general LLM as they can be used as the judge for safety evaluation.

• Regarding the practicality mentioned in the Claims & Evidence:

  • We would like to clarify that fine-tuning is not always the method used in practice by developers. Many LLM applications, such as "Custom GPTs" (see the “GPT store” for more details - https://openai.com/index/introducing-the-gpt-store/), rely instead on system prompts and additional information (still, see our comment about fine-tuning being feasible too below).
  • The “category” of these LLM apps help define aspects such as the “type of data” being used, thereby supporting the practicality of our proposed approach.

2. Evaluating Refusal Rates

• Recent LLMs have safety mechanisms to prevent context injection attacks. Therefore, directly evaluating refusal behaviour would cause ambiguity when the model considers the context as an attack. Therefore, we particularly design our evaluation with this LLM-as-a-judge framework to avoid potential issues with context injection attack.

3. Feasibility of Fine-tuning

• Based on the evidence we present in this paper, future research could explore ways to make LLMs more aware of context, which may indeed include fine-tuning. While this is a possible direction, it is currently out of the reach of this paper. At present, we focus on evaluation, and in particular, we employ a sufficiently large number of annotators to account for the uncertainty in human judgment. Collecting a training dataset using the same pipeline would be an exciting future work, which would require additional resources.

We will also include all the recommended references in our revised version of the paper.

We thank the reviewer's acknowledgement of our work, and would like to address the following weakness raised in the review:

1. Regarding Dataset Bias:

• We chose to base our benchmark on SORRY-Bench because it addresses a key issue in prior datasets—imbalance and over-representation of certain fine-grained categories that can introduce bias. SORRY-Bench introduces a curated safety taxonomy across 4 high-level domains, unifying inconsistent taxonomies from earlier work. We believe this design helps mitigate, rather than introduce potential bias.

2. Regarding practical benefits of CI theory:

• We thank the reviewer for raising this point. We believe that integrating CI theory into our framework brings several practical benefits to safety evaluation:
  (1) Structured representation of context: CI theory provides a principled way to decompose context into explicit components, which helps represent complex scenarios more systematically compared to free-text descriptions.
  (2) Modular verifiability: Each CI parameter can be verified independently, making it easier to assess individual aspects of a scenario or a model’s understanding.
  (3) Improved interpretability: The structured representation allows us to trace how specific CI parameters influence model judgments, providing greater transparency. And we can perform targeted ablation studies to evaluate model sensitivity to context changes.

• Contemporaneously, several papers have investigated the use of CI in different context of AI/ML, e.g.[1, 2], which also proves the practical advantage of using CI theory.

[1] Tsai et al. “Context is Key for Agent Security”.
[2] Ghalebikesabi et al. “Operationalizing contextual integrity in privacy-conscious assistants”.