SafetyAnalyst: Interpretable, transparent, and steerable LLM safety moderation

We sincerely thank Reviewer Hbfe for their detailed feedback and for highlighting areas where our explanations can be improved. However, there appear to have been several misunderstandings, which we address below.

1. The reviewer asked: “A smaller student LM (by the way what kind of LM are you using here? BERT-based?)”. Our base student LM is Llama-3.1-8B-Instruct, which we state multiple times in the main text:

   • Line 83: “via supervised fine-tuning of Llama-3.1-8B-Instruct”
   • Line 126: “we fine-tuned an open-weight LM (Llama-3.1-8B-Instruct) to specialize”
   • Line 207: “LM (Llama-3.1-7B-Instruct) to specialize in the tasks” (typo).

2. The reviewer stated: “SafetyReporter was never mentioned before Table 1”. We introduced SafetyReporter in Figure 1, both in the figure and its captions: “two specialist models — one to generate harms and one to generate benefits (together named SAFETYREPORTER)”. Regarding “What is the difference between SafetyReporter and SafetyAnalyst?”, please refer to Figure 1’s captions and the paragraph below Table 1 in Section 2.2.

3. The reviewer claimed: “All the paper claims made in the method section refer to a table in the appendix”. We request clarification on “the method section”, as our manuscript lacks a section named “Methods”. Please specify which of the two appendix tables (Tables 4 and 5) is being referenced.

4. The reviewer wrote: “In Section 2.3, the authors claim that they propose a new aggregation algorithm. I have the feeling that this is just a mere multiplication between some predefined weights. Are these weights somewhat learned? Are you also constantly updating $f$, $g$, and $h$ when $W$ and $\gamma$ are updated?” Section 2.3 clarifies that the feature weights are fitted to a given label distribution by maximum-likelihood estimation, meaning they are optimized, not predefined or constantly updated. We will revise this section for better clarity.

5. The reviewer asked: “Who's the teacher and who's the student model in Table 2?” In Line 209, we note “teacher models (SOTA LLMs)”, referring to those listed in Lines 76-77: “SOTA LLMs (GPT-4o, Gemini-1.5-Pro, Llama-3.1-70B-Instruct, Llama-3.1-405BTurbo, and Claude-3.5-Sonnet)”. Lines 214-215 specify the students as “The two student models that specialize in harm and benefit feature generation are collectively named ‘SAFETYREPORTER.’”

6. The reviewer wrote: “It makes little sense to say that a model's performance is $F1 \geq 84.7$. Is the $F1 = 84.7$ as what it is shown in the table? What are you trying to say in Lines 290-292?” Aggregation models trained on harm-benefit trees generated by all models achieved high classification performance, measured by $F1$, AUPRC, and AUROC, reported in Table 2. We request that the reviewer please clarify why “it makes little sense to say that the model’s performance is $F1≥84.7$”—we used this number since it is the lowest $F1$ among all models reported in Table 2, so the $F1$ scores in Table 2 are all $≥84.7$.

7. The reviewer asked: “Why is the conclusion a brief paragraph? Make it a section where you summarize your paper and future works.” The Discussion section includes the Conclusion subsection, summarizing our findings and future works. We can rename this section to “Conclusion” if preferred.

8. The reviewer wrote: “The experiments feel cut short where there is no clear connection between the harm-benefit-trees and the performances. Where do harm-benefit-trees come into play here. Do the authors really need these trees?” Sections 2.3 and 2.4 detail how the features in the harm-benefit tree are aggregated numerically and translated into a safety label that was used for evaluation in our experiments (i.e., the harm-benefit tree serve as the input to the aggregation algorithm, which outputs a harmfulness score that is then converted into a binary label). Nonetheless, we agree that further studies showing the usefulness of different features in the harm-benefit trees would be compelling, so we are working on supplementing the manuscript with ablations of different types of features in the trees (e.g., actions, effects, etc.).

9. The reviewer wrote: “In the appendix Table 5, it is not clear what the authors are measuring here to assess the agreement with human annotators.” Lines 814-816 explain: “To obtain the agreement rates, we computed the proportion of positive ratings (e.g., very plausible, somewhat plausible, and reasonable) among all positive and negative ratings.” We will move this explanation to the table captions for clarity.

We hope this clarification resolves the misunderstandings and respectfully ask Reviewer Hbfe to reconsider their assessment in light of this explanation while we work on revising the manuscript. Once we have updated the manuscript, we will comment again with a point-by-point response to all reviewers’ comments. Meanwhile, we appreciate your time and effort in revisiting the above points.

1. Table 3 reports averages of F1 scores on all datasets. Stating that an average score is better than the other doesn't say much. In fact, if you look at WildGuard and SafetyReporter, the former is constantly better than the latter on each dataset. It just has a performance drop in SORRY-Bench which makes the average F1 plummet. This is a classic scenario where averages aren't trustworthy and, in this scenario, make SafetyAnalyst the second-best performing method after GPT-4, when, instead, WildGuard should be. Here, I would suggest the authors to perform a Friedman test [1] with a post-hoc Bonferroni-Dunn test to assess whether SafetyReporter is better than the rest of the SoTA models without considering GPT-4. Here, one has to use multiple runs over the same dataset -- say 10+ -- to have several F1 scores for each dataset and then perform the Friedman test to tell whether the overall F1 averages are different among the SoTA methods and SafetyReporter. Then, one performs the Bonferroni-Dunn test where the control group is SafetyReporter and assess whether its average F1 score is statistically and significantly different from the rest. For each of the other SoTA methods, except GPT-4, this test gives a p-value which shows us if the method is "better" than the other. Using a p-value of would be sufficient. I expect that SafetyReporter is better than all but WildGuard, which undermines the paper's claim that SafetyReporter outperforms existing LLM safety moderation systems on prompt harmfulness classification.

2. There's a 6.2 point difference in terms of F1 scores (GPT-4 has 81.6 and SafetyAnalyst has 75.4). This would be justifiable if SafetyAnalyst were more interpretable than the black-box GPT-4. I fail to see why one would prefer SafetyAnalyst rather than GPT-4o.

1. Cool.
2. In my experience, you shouldn't introduce methods in captions. There isn't any single place where SafetyReporter is in the text before the Figure. Please introduce SafetyReporter in the text as well. What is the difference between SafetyReporter and SafetyAnalyst? Please respond here concisively.
3. It is still a methods section, although the title of it is not "Method" per se. I'm referring to both Tables. However, we're not here to discern useless stuff (i.e., "hey which Table is the reviewer referring to"). My concern is that all the claims of novelty are deferred to the appendix, which makes the paper wierd to read.
4. Cool.
5. Cool.
6. I'd still go with an F1 equals to bla bla bla here. It confuses the reader to see a $\geq$. But, thanks for the clarification.
7. Yeah, rename it, and extend it to at least 2 paragraphs.
8. Cool, I'd like to see these ablations before the discussion period ends. Is this feasible?
9. Nice. Repeating stuff in the table caption is very useful when parsing the table in isolation to the text.

We sincerely thank Reviewer hQ2Z for their thoughtful feedback and detailed critique of our manuscript. We have carefully considered all comments and conducted additional analyses, revising the manuscript accordingly. All changes are highlighted in red in the revised manuscript. We hope these updates address the reviewer's concerns and strengthen the manuscript. Below, we provide a point-by-point response to the reviewer's comments:

1. Literature Review
   The reviewer recommended improving the literature review. To address this, we have expanded Section 4 to include references to additional relevant work, such as TrustLLM and SafeLLM. In addition, given that our system is designed to be an innovation in safety content-moderation, we now add detailed descriptions of the other comparable safety content-moderation systems in Appendix D.1. These additions provide a broader context and further situate our contributions within the existing body of research.

2. Inference Time Evaluation
   We have reported inference time evaluation results for SafetyReporter and WildGuard in Section 3.2 of the revised manuscript. To address concerns about improving inference time for SafetyReporter, we recommend reducing the complexity of the harm-benefit tree. To support this, we provide ablation studies (Appendix D.4) on the features of the harm-benefit tree in WildJailbreak and WildGuardTest, showing how these features contribute to feature aggregation. We acknowledge that these results may not generalize across all datasets or use cases, and, thus, include all features in the harm-benefit tree in the current manuscript for generality.

3. Robustness Against Jailbreaking and Prompt Injection
   The reviewer inquired about the robustness of the proposed solution against jailbreaking and prompt injection attacks. To address this, we highlight that SafetyReporter was fine-tuned on harm-benefit trees generated from both vanilla and adversarial prompts sampled from WildJailbreak. This dataset includes synthetic vanilla harmful and benign prompts as well as adversarial prompts created using various combinations of jailbreaking techniques. The performance of SafetyReporter on the adversarial subset of WildGuardTest demonstrates its effectiveness against adversarial attacks (see Table 2; 4th column from the left). Furthermore, in Appendix D.3, we provide a breakdown of classification accuracy for SafetyReporter and other baselines on fine-grained prompt categories in SORRY-Bench. Here, SafetyReporter exhibited the highest robustness to persuasion techniques including Authority Endorsement, Evidence-based Persuasion, Expert Endorsement, Logical Appeal, and Misrepresentation.

4. In-Context Reward Hacking
   The reviewer asked, “Considering the problem of in-context reward hacking, how the proposed method could help us to avoid such issues?” We kindly request clarification on what is meant by “in-context reward hacking” in this context. To our understanding, the SafetyReporter framework does not incorporate a reward model or RLHF, so it is unclear how this issue directly applies to our work. We are happy to address this point further once additional clarification is provided.

5. Clarification on Novelty
   The reviewer suggested leveraging mechanistic interpretability to increase the novelty of our work — an example of which is Anthropic’s research into surfacing concepts latent in their language model. We agree that using mechanistic interpretability to form “concepts” and enhance LMs’ reasoning is an interesting and important way of increasing the transparency and interpretability of AI systems. However, our approach leverages a different and completely novel (and computationally cheaper) method of making an LM-based content moderation system more interpretable. In our approach, we employ chain-of-thought prompting over a semi-structured harm-benefit feature space to generate harm-benefit trees associated with extents, likelihoods, and severities, which are then aggregated by an interpretable algorithm. All of this guides the model's reasoning about prompt safety, as illustrated in Figures 1 and 2. To emphasize the novelty and contributions of our work, we have added a Contributions paragraph to the end of Section 1 and expanded on the interpretability, transparency, and steerability of our system compared to other baselines in Section 3.3.

We hope these updates address the reviewer’s concerns. We are grateful for the constructive feedback and respectfully encourage the reviewer to consider raising their rating.

We sincerely thank Reviewer SruY for their positive feedback and thoughtful critique of our manuscript. We have conducted additional analyses and revised the manuscript accordingly to address the reviewer's insightful comments. All changes are highlighted in red in the revised manuscript. We hope these updates strengthen our work and respectfully encourage the reviewer to consider raising their rating. Below, we provide detailed responses to each of the reviewer’s comments:

1. Request for more discussion on the differences between existing content moderation systems

The reviewer suggested providing further discussion on the differences between existing content moderation systems. To address this, we have expanded the subsection titled “Existing LLM content moderation systems” in Section 4 and added Section 3.3 in the main text, which elaborates on the advantages of interpretability, transparency, and steerability in our system compared to the baselines. Additionally, we included detailed descriptions of the baseline models in Appendix D.1, where we highlight their differences and contextualize them in relation to our system.

2. Evaluation of steerability and alignment to different datasets

We agree with the reviewer that providing a demonstration of how our system can be aligned would bolster our claims of steerability. In response to this concern, we now demonstrate the capacity of the model to be steered using a case study (see Appendix E).

In future work, we are eager to demonstrate steerability more comprehensively. However, we encountered a few blockers when attempting to tackle this challenge using the safety moderation benchmarks that are included in the paper (and others like them). Existing prompt safety datasets are designed to cover many safety categories comprehensively and capture safety intuitions that the majority of people share. That is, both the prompts and the responses are drawn from similar distributions, so the benefit of steering a content moderation system to align to one of these datasets over another is rather limited. On the other hand, different individuals are likely to make at least somewhat different judgments across a range of safety-related cases and we do expect that our system should be able to be steered to capture individual-level variation. However, extant prompt safety datasets lack identifying information that would allow us to determine which answers are given by which participants. Given these limitations, adequately demonstrating the steerability of SafetyReporter fully would require substantial new annotation data. This is an exciting direction for future work.

3. Inference time evaluation results

We have reported inference time evaluation results for SafetyReporter and WildGuard in Section 3.2 of the revised manuscript. To address concerns about improving inference time for SafetyReporter, we recommend reducing the complexity of the harm-benefit tree. To support this, we provide ablation studies (Appendix D.4) on the features of the harm-benefit tree in WildJailbreak and WildGuardTest, showing how these features contribute to feature aggregation. We acknowledge that these results may not generalize across all datasets or use cases, and, thus, include all features in the harm-benefit tree in the current manuscript for generality.

4. Performance insights on SORRY-Bench

The reviewer inquired about the reasons behind performance differences between GPT-4 and SafetyReporter, which was driven by SORRY-Bench. In Appendix D.3, we offer a detailed breakdown of classification accuracy across different models and prompt categories:

• GPT-4 outperformed SafetyReporter by successfully detecting a subset of Encoding and Encrypting prompts (Atbash and Caesar ciphers).
• SafetyReporter demonstrated the most robustness to persuasion techniques (Authority Endorsement, Evidence-based Persuasion, Expert Endorsement, Logical Appeal, and Misrepresentation).
• WildGuard failed to identify potentially unsafe prompts in certain non-English categories (Marathi, Malayalam, and Tamil).

These observations are summarized in Appendix D.3 to provide clarity on the comparative strengths and weaknesses of the models. Note that it would be possible to prompt or train SafetyReporter to look for and decode ciphers if that were of interest for a particular use case — and which would improve performance on SORRY-Bench.

We hope that these revisions and clarifications adequately address the reviewer’s comments. Thank you again for your valuable feedback, which has greatly helped us improve the quality of our work.