Weakness 3

As we have also responded to the 3rd reviewer's comments, we employed two simple prompt methods from several related studies [3] [4] to conduct experimental comparisons. Given that these methods are aimed at hate speech and toxic content, and due to time constraints, we selected three harmful text datasets for evaluation, as shown below (reporting F1-score):

This new evaluation result shows that simple prompting can relatively shape an LLM to obtain satisfactory performance. Meanwhile, JudgeRail maintains superior performance across all datasets and has a significant advantage in terms of latency.

Weakness 5

Actually, our proposed JudgeRail can be applied to GPT models. We chose not to include GPT models in our original experiments since we aim to propose a solution that uses open-source LLMs, which have comparable model sizes to existing LLM-based moderation tools, such as LlamaGuard3 and ShieldGemma. Nevertheless, to better present the generalization capability of JudgeRail, we have evaluated the performance of using GPT4 with JudgeRail on three datasets and present the results in the following table. We selected these datasets since their data sizes are relatively small and cause relatively lower costs of calling the GPT4 API.

As shown in the above table, GPT4 equipped with JudgeRail -- GPT4(JR) obtains comparable performance to our best-performing Gemma2(JR) on AdvBench and OpenAI Moderation datasets, while performing worse on the HateCheck dataset. By examining the samples mistakenly classified by GPT4(JR), we find that, while HateCheck primarily focuses on hate speech, some of its samples labeled as Non-hate still contain offensive materials. This type of content is often recognized as harmful by GPT4. This aligns with our previous findings mentioned in the paper regarding the ambiguity in the label system and the inaccuracies in dataset labels.

Question 1

JudgeRail can be implemented on the GPT models and we have presented the evaluation results for GPT4 equipped with JudgeRail in the response above.

Question 2

Thank you for your valuable question. We would like to demonstrate that JudgeRail can be applied to text-to-image models. We have evaluated JudgeRail with several text-to-image prompt datasets from prior research [5]. The results are shown below (reporting accuracy; the Template prompts dataset contains 30 samples, the Lexica prompts dataset contains 404 samples, and the other two datasets each contain 500 samples.):

We will provide a description on utilizing JudgeRail for recognizing harmful prompts used for attacking text-to-image generation models and present the updated evaluation results in the appendix.

Question 3

Since we cannot reproduce ToxicDetector as previously mentioned, we can only compare to the latency reported in [2]. Specifically, we first referenced the results from [2] for the Perspective API, which reports a latency of 0.8 seconds. This latency is close to our measurements for the latency of calling Perspective API, which is around 1 second. Based on this consistent latency result, we observed that the ToxicDetector reported in [2] has a latency of 0.078 seconds, and our average latency ranges from 0.088 to 0.102 seconds. This demonstrates that ToxicDetector is faster than the proposed method by around 13%.

Question 4

Please refer to the response for weakness 3.

[1] Realtoxicityprompts: Evaluating neural toxic degeneration in language models

[2] Efficient Detection of Toxic Prompts in Large Language Models

[3] Yongjin Yang, Joonkee Kim, Yujin Kim, Namgyu Ho, James Thorne, and Se-Young Yun. 2023. HARE: Explainable Hate Speech Detection with Step-by-Step Reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5490–5505, Singapore. Association for Computational Linguistics.

[4] He X, Zannettou S, Shen Y, et al. You only prompt once: On the capabilities of prompt learning on large language models to tackle toxic content[C]//2024 IEEE Symposium on Security and Privacy (SP). IEEE, 2024: 770-787.https://arxiv.org/pdf/2308.05596

[5]Yiting Qu, Xinyue Shen, Xinlei He, Michael Backes, Savvas Zannettou, and Yang Zhang. 2023. Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Models. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS '23). Association for Computing Machinery, New York, NY, USA, 3403–3417. https://doi.org/10.1145/3576915.3616679

Thank you for your valuable feedback.

Weakness 1

Thank you for your suggestion. We will discuss the limitations from the following points: First, due to latency considerations, our current in-context learning mechanism is relatively simple. More complex mechanisms, such as Retrieval-Augmented Generation (RAG), will be explored in future work. Additionally, we have shown that the label system has a clear impact on detection performance, while existing label systems have some degree of semantic ambiguity. This inspires us to consider designing a more refined label system with better-separated semantic representations. Finally, while the detection capabilities of JudgeRail essentially depend on its underlying LLM, JudgeRail also benefits from this as the underlying model evolves in its capability.

Weakness 2&4

Thank you for the reference, and we will include it in our related work. Following your suggestion, we employed the RealToxicityPrompts [1] dataset for evaluating several models used in our paper and the recommended ToxicDetector [2]. Since the ToxicChat dataset is also a toxic prompt dataset, we have included the test results for both datasets, as shown in the following table (reporting F1-score with the best performance):

As shown in the above table, the reported performance of ToxicDetector outperforms other models on the RealToxicityPrompts by a large margin. We noticed from [2] that ToxicDetector is trained and evaluated on this RealToxicityPrompts dataset. As we have done for SplineLLM to respond to the second reviewer's comment, we would like to further evaluate the generalization performance of ToxicDetector. However, we find that this method has not been released to the public, making it difficult to evaluate ToxicDetector [2] on other datasets.

In the meantime, we have noticed that both [2] and our work have adopted Perspective API for comparison, and the performance of Perspective API on RealToxicityPrompts has been reported. We have evaluated Perspective API on RealToxicityPrompts with the same sample size to reproduce the reported result. However, as shown in the above table, our reproduced performance is significantly worse than the reported result. Such inconsistency makes it difficult to draw a reasonable comparison with the recommended ToxicDetector.

Nevertheless, we have evaluated our proposed method, along with other two LLM-based moderation models, on this RealToxicityPrompts dataset. We have also presented our previous evaluation results obtained on ToxicChat in the above table, as both datasets contain prompt-type harmful text samples.

Thank you for your valuable feedback and suggestions.

Weakness 1

Thank you very much for your valuable suggestions. As we have also responded to the 2nd reviewer's comments, for the comparative experiment with and without the logit rectification, we will continue to test with larger scales, spanning 500 to 1000 samples, with an equal number of samples randomly sampled from all five datasets used in this work. However, we would like to highlight that we introduce the logit rectification mechanism primarily to extract valid formatted decisions, even when an LLM generates outputs that do not conform to JudgeRail's output format specification. In other words, the logit rectification mechanism serves as an efficient and effective error-handling mechanism. Therefore, we used the 100 samples to validate that using logit rectification, compared to recognizing model decisions from its unstructured generated content via sophisticated parsing mechanisms, has minimal impact on the moderation performance.

We followed your suggestion and employed two simple prompt methods from several related studies [1] [2] to conduct experimental comparisons. Given that these methods are aimed at hate speech and toxic content, and due to time constraints, we selected three harmful text datasets for evaluation, as shown below (reporting F1-score):

This new evaluation result shows that simple prompting can relatively shape an LLM to obtain satisfactory performance. Meanwhile, JudgeRail maintains superior performance across all datasets and has a significant advantage in terms of latency.

[1] Yongjin Yang, Joonkee Kim, Yujin Kim, Namgyu Ho, James Thorne, and Se-Young Yun. 2023. HARE: Explainable Hate Speech Detection with Step-by-Step Reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5490–5505, Singapore. Association for Computational Linguistics.

[2] He X, Zannettou S, Shen Y, et al. You only prompt once: On the capabilities of prompt learning on large language models to tackle toxic content[C]//2024 IEEE Symposium on Security and Privacy (SP). IEEE, 2024: 770-787.https://arxiv.org/pdf/2308.05596

Weakness 2 and writing

Thank you for your valuable feedback regarding the organization of the experiments section. We will revise this section by adding clear subheadings or bolded paragraphs headings to better structure the observations and improve clarity. The typo "a LLM " will also be fixed to "an LLM". Thanks again for your kind advice.

Questions:

Thank you for raising this question, and we are glad to provide further clarification on our thoughts.

Since an open-source LLM trained and released for generic text generation tasks is commonly pre-trained on a vast amount of data, we believe the model already understands certain common knowledge, such as those used in composing harmful text. Indeed, by examining most harmful text datasets, we find that the content moderation task primarily involves common sense and everyday language, with rather limited requirements for domain-specific knowledge. As such, we think these open-source models may be able to obtain satisfactory performance even without the JudgeRail framework.

According to the result shown in the above table, as expected, an LLM equipped with simple prompting techniques demonstrates good moderation performance, while JudgeRail demonstrates a notable improvement on the dataset and also has a significant advantage in terms of latency. However, during this newly added experiment, we found that an LLM often does not output according to the format instructed in the prompt. This highlights the need for a mechanism, such as the proposed logit rectification, to complement the prompts, thereby better converting an LLM into a powerful content moderation tool.

We appreciate your comments and will revise our manuscript accordingly.

Weakness 2

(1) We chose to evaluate Perspective API by its binary classification performance, as Perspective API does not have a released, associated labeled multi-label dataset for fine-grained comparison. Therefore, to evaluate and compare JudgeRail in a multi-label setting, we selected the OpenAI Moderation API, which offers a labeled multi-label dataset.

(2) We apologize for the misleading presentation. We would like to specify that we introduce the logit rectification mechanism primarily to extract valid formatted decisions, even when an LLM generates outputs that do not conform to JudgeRail's output format specification. In other words, the logit rectification mechanism serves as an error-handling mechanism, avoiding the need for post-output parsing-based error handling methods. Therefore, we used the 100 samples to validate that using logit rectification, compared to recognizing model decisions from its unstructured generated content via sophisticated parsing mechanisms, has minimal impact on the moderation performance.

(3) Following the above clarification, we would like to specify that we did not use the selected 100 samples to build a decoding vocabulary. The decoding vocabulary size depends on the label system used. For example, when using the Perspective API label system, the vocabulary can range from "0"-"6". When using the OpenAI Moderation API label system, the vocabulary can range from "0"-"9" and "A"-"E".

For the comparative experiment with and without the logit rectification, we will continue to test with larger scales, spanning 500 to 1000 samples, with an equal number of samples randomly sampled from all five datasets used in this work.

We will clarify the aforementioned misleading points in Section 3.2 of the revised manuscript.

Writing:

Thank you for the advice. We will revise the description in the corresponding section of the manuscript.

Question:

Thank you for raising this question, and we are glad to provide further clarification on our thoughts.

First, we believe that an LLM trained and released for generic text generation tasks is commonly pre-trained on a vast amount of data, enabling the model to already understand certain common knowledge, such as those used in composing harmful text. Indeed, by examining most harmful text datasets, we find that the content moderation task primarily involves common sense and everyday language, with rather limited requirements for domain-specific knowledge.

Furthermore, by surveying other literature, we have noticed that some prior research[2] has highlighted the questionable effectiveness of fine-tuning LLMs, as it primarily changes the style of the LLM's output rather than its knowledge.

The combination of these latter two observations and the former intuition guided our design of the prompting framework. Our goal is to leverage this common sense more effectively. Therefore, we chose to assign an LLM a commonly known Judge character, with the commonly acquired principle of "presumption of innocence", and validated our design through thorough experiments.

[1] Adams, C., Jeffrey, S., Julia, E., Lucas, D., Mark, M.,Nithum, and Will, C. Toxic comment classification challenge, 2017.

[2]Lin B Y, Ravichander A, Lu X, et al. The unlocking spell on base llms: Rethinking alignment via in-context learning[C]//The Twelfth International Conference on Learning Representations. 2023.

Thank you for your review and the insightful comments.

Weakness 1

We initially focused on comparing with content moderation methods that are more suitable for practical use cases, leading us to select moderation tools and models (such as Perspective API, OpenAI Moderation API, ShieldGemma, and LlamaGuard3) released by companies with real moderation demands. We acknowledge the importance of empirically comparing with research-oriented moderation methods. Following your suggestion, we have attempted to include comparisons with SplineLLM and RigorLLM.

SplineLLM proposes to utilize the internal representations of LLMs to characterize a given prompt and generation. In SplineLLM, content moderation is a subtask, where an instance of SplineLLM is trained and tested on the same dataset, Jigsaw [1]. Meanwhile, RigorLLM integrates several different processes and components into its proposed framework, including energy-based hidden embedding data augmentation, optimization-based prompt suffix generation, and a fusion-based model combining robust KNN with LLMs. This indicates that RigorLLM has a relatively more sophisticated implementation. Indeed, when evaluating RigorLLM, we encountered difficulties in deploying RigorLLM, facing running and configuration issues. Due to limited time, we have not yet obtained the results. We will continue to work on reproducing it.

In contrast, we have successfully reproduced the SplineLLM approach and included the results in our experiments, as shown in the following table in markdown format. For AdvBench, we report the accuracy, while for other datasets, we report the F1-score. The latency is measured in seconds.

As previously mentioned, SplineLLM is trained and evaluated on the same dataset. As shown in the above table, the generalization of SplineLLM is limited when evaluated across different datasets. Furthermore, the performance of SplineLLM is more closely aligned with Martin-ha, ToxRoberta, and S-nlp, with a significant difference being its performance on AdvBench, which approaches that of several LLM-based moderation solutions. Due to the simplicity of SplineLLM, its serving latency is noticeably lower than that of LLM-based moderation solutions, yet still significantly higher than Bert-based moderation solutions.

Thank you for your valuable comments.

Weakness 1

We would like to clarify the design of our label system and will provide a more detailed description in the revised manuscript. To avoid arbitrary selection of harmful categories, we have adopted the categories defined by Perspective API and LlamaGuard3, which offer more fine-grained classifications compared to other reviewed moderation tools.

To be specific, the P1, P2, ..., P6 correspond to the Perspective API's harmful categories (in Table 1, where 1.Toxicity, 2.Severe toxicity, ..., 6.Threat). We incorporate all 6 categories in our label system when testing the content datasets. And S1, S2, ..., S14 correspond to LlamaGuard3's harmful categories (in Table 1, where 1.Violent crimes, 2.Non-violent crimes, ..., 14.Defamation). We incorporate all 14 categories in our label system when testing the prompt datasets.

Weakness 2

Thank you for providing this reference [1]. We acknowledge that in Jury Learning [1], dissenting voices are integrated by modeling individual annotators and allowing practitioners to define the jury composition. Additionally, we have identified another related work, Digital Juries [2], which proposes a civics-oriented approach for adjudicating content moderation cases. However, both approaches necessitate deploying multiple models, potentially consuming significant GPU memory resources and introducing a complex decision-making mechanism, which may result in additional latency.

In contrast, we named our framework JudgeRail, inspired by the role-playing scheme commonly adopted in developing jailbreak prompts. While one can assign an LLM a harmful character, which may be challenging due to more mature safety alignment, we can also assign an LLM a helpful character to combat harmful content. This character must be fair, driving us to select the principle of "presumption of innocence," which naturally fits the "Judge" character and is a common-sense principle that most LLMs can understand and follow.

Moreover, while sharing the spirit of introducing knowledge from the judicial system, practicality is a key consideration in JudgeRail. This motivates us to design a generic prompt framework that works with individual LLMs and incorporates the logit rectification mechanism, which accelerates processing and efficiently handles out-of-scope generation issues.

We will revise the manuscript to more clearly articulate our novelty and incorporate the newly provided references.

Question

Thank you for pointing out the less clarified statement. We will rephrase our presentation to highlight our observation regarding low-precision models. Specifically, our evaluation results, collected from three LLMs including Gemma2(JR), ShieldGemma, and LlamaGuard3, indicate that adopting their low-precision counterparts introduces rather limited performance impact. We find this observation intriguing, as it contrasts with studies [3,4] that have demonstrated the crucial role of model precision in generative tasks. This differing impact on model performance, driven by low-precision models, leads us to wonder that decision-making moderation tasks may have distinct requirements for model precision. We will provide a clearer presentation in the revised manuscript.

[1] Mitchell L. Gordon, Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S. Bernstein. 2022. Jury Learning: Integrating Dissenting Voices into Machine Learning Models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 115, 1–19. https://doi.org/10.1145/3491102.3502004

[2] Jenny Fan and Amy X. Zhang. 2020. Digital Juries: A Civics-Oriented Approach to Platform Governance. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376293

[3]Li S, Ning X, Wang L, et al. Evaluating quantized large language models[J]. arXiv preprint arXiv:2402.18158, 2024.

[4] Gong Z, Liu J, Wang J, et al. What Makes Quantization for Large Language Model Hard? An Empirical Study from the Lens of Perturbation[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(16): 18082-18089.