We are delighted by the reviewer's recognition of the well-motivated focus, the clarity of the writing and figures, as well as the potential impact of FineHarm and SCM. Below, we address the concerns and suggestions in detail:

---

1 Details of test data and OOD testing (W1)

The train-test split follows random split and thus our test is under the i.i.d setting. To test models' capacity under the OOD setting, we further tested SCM (trained on FineHarm) on two existing, non-overlapping benchmarks, Toxic-Chat [1] and ToxiGen [2], and compared the performance with the OOD baselines derived from [1, 2], i.e., HateBERT and ToxDectRoBERTa.

Toxic-Chat:

ToxiGen:

Under the OOD setting:

• SCM has a stronger generalization ability than the baselines in the original papers [2,3].
• On Toxic-chat, the performance of SCM-7B under OOD testing is comparable to the i.i.d. performance reported in [2] (F1=72.1).

[1] ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation. EMNLP’23 Findings

[2] ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection. ACL’22

2 Baseline comparison and supplementary experimental results (W2/W3/W4)

• We added more existing methods, including open-source (HateBERT, ToxDectRoBERTa-Large, and LlamaGuard-3-8B) and closed-source ones (Google Perspective and OpenAI Moderation API).
• We increased the maximum allowed $k$ from 10 to 50, and the optimal macroF1 of SCM-0.5B improved from 95.64 (when k=4) to 96.08 (when k=13).

The results with more details (MAR/FAR/k) are shown below, which show that the proposed SCM using partial detection can still rival the performance of the added baselines using full detection.

3 Further clarification of the paper's content (Q1/Q2/Q3/Q4)

Thank you for pointing out the unclear expressions, which has been of great help to us. We have revised the relevant expressions in the subsequent version and will provide further clarification below.

Q1: For the token-level labeling, does the proposed heuristic annotation approach use the delete method in Appendix B.1?

The delete method was an unsuccessful attempt, and the approach we have adopted does not involve any deletion operations. The ultimately adopted steps are described in Section 3.2 of the main text.

Q2: What is the relationship between the holistic scorer and the token scorer? Do they share the model architecture or weights?

The token scorer and holistic scorer are two linear layers for a hidden_size-to-2 mapping (corresponding to harmful or not) with only a slight difference (the holistic scorer does not set a bias while the token scorer does). They do not share weights.

Q3: For Equation (4), are there any typos in steps 2 and 3? For example, in step 2, should it be $\vee P\left(f_{h o l}\left(h_{n}\right)\right) \cdot P\left(\left(f_{t o k}\left(h_{i}\right)\right)_{(i=1)}^{n}\right)$?

This equation holds due to the equivalence in propositional logic between implication and a combination of negation and disjunction. An implication $A \Rightarrow B$ is only false in one scenario: when $A$ is true, but $B$ is false. In all other cases, the implication holds.

The truth-value table is as follows, which lists all possible truth values for $A$ and $B$ and compares the two expressions:

Notice that the columns for $A \Rightarrow B$ and $\neg A \vee B$ are identical, indicating the logical equivalence. So we have steps 2 and 3 in Eq. (4).

Q4: What is the stopping rule to get the results in Figure 4? Is delay-k equal to 1?

The termination position depends on the $k$ value when the best performance in Table 2 is achieved. It can also be directly seen from the results for the second reply.

---

Thank you again for your insightful feedback.

We believe these revisions will strengthen the paper's clarity. Please let us know if further clarifications are needed.

Best regards

Thank you for highlighting the significance of the problem we addressed and the value of the FineHarm dataset and SCM to the community. Below, we address your concerns and suggestions in detail:

---

1 Differences between SCM and Speculative Decoding (W1)

Thank you for providing an insightful perspective. We agree that SCM shares similarities with speculative decoding in terms of the partial context they are applied and the token-level evaluation with the assistance of a (commonly smaller) model. However, SCM and speculative decoding are also very distinct, which we’d like to clarify through three core dimensions:

• Different motivation. The design of speculative decoding is to accelerate the generation process, improving the inference efficiency. In contrast, the core goal of SCM is to conduct a real-time assessment of potential risks in the output stream, and it does not influence the token decoding of the monitored LLM.
• Different functional positionings of the external model. In speculative decoding, the introduced small LM is used to draft tokens, and the verifier is the large model itself; however, in our method, the monitored model (regardless of its size) will draft tokens, and the small LM we introduce (i.e., SCM) is used for verification (the harmfulness).
• Different standpoints for cost optimization. Speculative decoding optimizes the single-token generation stage by replacing the inference of the large LM with the smaller one, while SCM considers the costs in the overall generation perspective (via stopping the output early by observing fewer tokens, thus eliminating the unnecessary cost caused by detection after complete outputs).

Due to the differences in motivation, the roles of models, and the standpoints for cost optimization, we believe that the design of SCM is deeply rooted in the comprehension of harmful output monitoring scenarios and is non-trivial even if speculative decoding exists. We plan to share such a technical discussion with readers and cite related works in the revised version.

2 Ablation of holistic loss design (W2)

After removing the Holistic Scorer, the detection performance drops, possibly due to the absence of future knowledge injection during learning that reflects the harmfulness to the entire response.

3 Out-of-distribution (OOD) testing (W3)

Since the test data shares the same source or distribution as the training data, we further tested SCM (trained on our FineHarm) on two existing, non-overlapping benchmarks, Toxic-Chat [1] and ToxiGen [2], and compared the performance with the OOD baselines derived from [1, 2], i.e., HateBERT and ToxDectRoBERTa.

Toxic-Chat:

ToxiGen:

Under the same OOD setting:

• SCM has a stronger generalization ability than the baselines mentioned in their papers.
• On the Toxic-chat dataset, the performance of SCM-7B under out-of-distribution testing is comparable to the in-distribution testing value mentioned in the paper (F1=72.1).

[1] ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation. EMNLP’23 Findings

[2] ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection. ACL’22

4 Baseline comparison (W4)

We added more existing methods, including open-source (HateBERT, ToxDectRoBERTa-Large, and LlamaGuard-3-8B) and closed-source ones (Google Perspective and OpenAI Moderation API). The results are shown below, which show that the proposed SCM using partial detection can still rival the performance of the supplemented baselines using full detection.

5 Latency of detection (Q1/Q2)

Let $t_{gen}$ be the generation delay, $t_{det}$ be the detection delay, and $T$ be the length of the generated sequence. If termination occurs at position $k \leq T$ due to partial detection, then:

1. For synchronous generation and detection (SCM), the delay for a single token is $\max(t_{gen}, t_{det})$. The time taken until termination is $k \times \max(t_{gen}, t_{det})$. If $k$ follows uniform distribution, the average time is $\mathbb{E}(t)=\frac{1}{T}\sum_{k=1}^T[k \times \max(t_{gen}, t_{det})] = \frac{T+1}{2} \times \max(t_{gen}, t_{det})$. [Recalling our empirical results, the distribution peak would be smaller than the midpoint, and thus the avg time would be lower.]

2. For detection after all generations are completed (Full detection), the total time taken is $t=T \times t_{gen} + t_{det}$.

We tested the latency under the same setups:

• Without using any acceleration methods.
• With the test sequence length increasing from 50 to 100.
• Each test sequence is repeated 10 times to report the min/max/avg latency.

Generation latency ($t_{gen}$) of Qwen:

Detection latency ($t_{net}$) of SCM:

Notice that $t_{gen}>t_{det}$ under the same scale, so the average time of SCM is $\frac{T+1}{2}\times t_{gen}$ and the time of full detection is $T \times t_{gen} + t_{det}\approx T\times t_{gen}$. SCM is faster since $\frac{T+1}{2}<T$ for any $T>1$. If the generation model is larger, the condition $t_{gen}>t_{det}$ still holds. [We omit the situation where the detection model is larger than the generation model, as it is rare in reality.]

Detecting the previous token during the process of generating the next token does not introduce additional time consumption, which means that only a 1-token delay is needed to ensure that the streaming monitor runs smoothly in parallel and barely affects the user experience.

---

Thank you again for your insightful feedback.

We believe these revisions will strengthen the paper's clarity. Please let us know if further clarifications are needed.

Best regards

Thank you for highlighting the significance of the problem we addressed, the value of the FineHarm dataset to the community, and the strong empirical results achieved by SCM. Below, we address your concerns and suggestions in detail:

---

1 Annotation issue (W1)

Thank you for raising this question, which gives us a chance to further elaborate on the motivation behind the heuristic annotation.

Due to the absence of fine-grained labels, it is hard to train harmful output moderators with incomplete outputs. Our annotation aims to find a cost-acceptable way to provide a reasonably good starting point for subsequent models’ learning, even if it may not be as accurate as the expensive human labeling. After testing and giving up the LLM-based deleting approach, we ultimately turned to the relatively conventional way relying on the POS prior, which was then experimentally shown to be useful. The potential noise from heuristic annotations was then tolerated by the proper coordination of data and learning strategies in SCM’s training. As shown in Fig. 5, SCM finally identified harmful words with a reasonable combination of prior knowledge from the annotations and its own learned experience.

By intentionally relaxing the quality requirement of annotations, SCM exemplifies a data-and-model solution that balances the performance and the annotation cost. Among them, the heuristic annotations play a key (though imperfect) role.

2 Key contribution (W2)

We sincerely believe that the key contribution is not merely an application of multi-task learning with fine-grained labels.

• For the idea, though ProtectAI and GuardrialsAI support streaming safety monitoring, their implementations are not intended to pursue an early-stopping in nature. Our work identifies the importance of early-stop moderation, reviews existing possible solutions (including ProtectAI’s and GuardrialsAI’s), and formulates the early moderation with actionable settings (i.e., the Delay-k notations).

• For the techniques, we provide a data-and-model solution that tackles the remaining challenges to building such a moderator, including the cheap acquisition of fine-grained token-level annotations and the more effective adaptation of the moderator to the incomplete semantics, resulting in FineHarm and SCM as we introduced. The solution considers the multiple factors (annotation costs, compatibility, etc.) for real-world applications, which we believe is not too straightforward and mature to be a less technical contribution. As we mentioned in the discussions, we hope that SCM could be an inspiration for real-world attempts.

3 Computational overhead (W3/Q2)

3.1 FLOPS

We use the fvcore library to calculate the number of floating-point operations (FLOPS) required for the model to infer a token. The FLOPS required by the SCM Detection Model is less than that required by the Qwen Generation Model.

3.2 Latency

Let $t_{gen}$ be the generation delay, $t_{det}$ be the detection delay, and $T$ be the length of the generated sequence. If termination occurs at position $k \leq T$ due to partial detection, then:

1. For synchronous generation and detection (SCM), the delay for a single token is $\max(t_{gen}, t_{det})$. The time taken until termination is $k \times \max(t_{gen}, t_{det})$. If $k$ follows uniform distribution, the average time is $\mathbb{E}(t)=\frac{1}{T}\sum_{k=1}^T[k \times \max(t_{gen}, t_{det})] = \frac{T+1}{2} \times \max(t_{gen}, t_{det})$. [Recalling our empirical results, the distribution peak would be smaller than the midpoint, and thus the avg time would be lower.]

2. For detection after all generations are completed (Full detection), the total time taken is $t=T \times t_{gen} + t_{det}$.

We tested the latency under the same setups (without using any acceleration methods, with the test sequence length increasing from 50 to 100, and each test sequence is repeated 10 times to report the min/max/avg latency).

Generation latency ($t_{gen}$) of Qwen:

Detection latency ($t_{net}$) of SCM:

Notice that $t_{gen}>t_{det}$ under the same scale, so the average time of SCM is $\frac{T+1}{2}\times t_{gen}$ and the time of full detection is $T \times t_{gen} + t_{det}\approx T\times t_{gen}$. SCM is faster since $\frac{T+1}{2}<T$ for any $T>1$. If the generation model is larger, the condition $t_{gen}>t_{det}$ still holds. [We omit the situation where the detection model is larger than the generation model, as it is rare in reality.]

Detecting the previous token during the process of generating the next token does not introduce additional time consumption, which means that only a 1-token delay is needed to ensure that the streaming monitor runs smoothly in parallel and barely affects the user experience.

4 Baseline comparison (W4)

We added more existing methods, including open-source (HateBERT, ToxDectRoBERTa-Large, and LlamaGuard-3-8B) and closed-source ones (Google Perspective and OpenAI Moderation API). The results are shown below, which show that the proposed SCM using partial detection can still rival the performance of the supplemented baselines using full detection.

5 Out-of-distribution (OOD) testing (W5)

For the OOD setting, we tested SCM (trained on our FineHarm) on two existing, non-overlapping benchmarks, Toxic-Chat (EMNLP’23 Findings) and ToxiGen (ACL’22), and compared the performance with the OOD baselines derived from them, i.e., HateBERT and ToxDectRoBERTa.

Toxic-Chat:

ToxiGen:

Under the same OOD setting:

• SCM has a stronger generalization ability than the baselines mentioned in their papers.
• On the Toxic-chat dataset, the performance of SCM-7B under out-of-distribution testing is comparable to the in-distribution testing value mentioned in the paper (F1=72.1).

6 Annotation ablation (Q1)

We trained a content moderation model (CLF below) using QwenForTokenClassification, which was trained with naively labeled tokens: Each token is labeled as "harmful" or "not harmful" based on whether the response-level label is "harmful". Due to the space limit, we report the results at a 1.5B scale. This shows that such a naive labeling method introduces more noise, leading to a decline in model performance.

We see similar findings at the other two scales. We will include the complete results in the revised version.

---

Thank you again for your insightful feedback, which has helped us greatly improve the quality of the paper.

We believe these revisions will strengthen the paper's clarity. Please let us know if any further clarification is needed.

Best regards

We would like to thank the reviewers for acknowledging the contributions of our work in terms of motivation, data-and-model solution, and experiments, as well as for providing detailed evaluations and constructive feedback. Below, we address the concerns and suggestions in detail:

---

1 Baseline comparison (W1)

We added more existing methods, including open-source (HateBERT, ToxDectRoBERTa-Large, and LlamaGuard-3-8B) and closed-source ones (Google Perspective and OpenAI Moderation API). The results are shown below, which show that the proposed SCM using partial detection can still rival the performance of the supplemented baselines using full detection.

2 The generalizability of SCM (W2)

2.1 Data diversity in the test set

The sources of harmful responses in FineHarm consist of two parts:

• Harmful responses from the uncensored model based on the harmless data in WildGuard and WildJailbreak.
• Harmful responses provided by WildGuard, covering 9 LLMs (GPT-4, OLMo-7B-Instruct, GPT-3.5, Vicuna-7b-v1.5, Llama3-8B-Instruct, Mistral-7B-Instruct-v0.2, and 3 Dolphin variants) [1].

Therefore, FineHarm contains harmful responses from multiple LLMs to ensure the source diversity.

2.2 Out-of-distribution (OOD) testing

For OOD setting, we tested SCM (trained on our FineHarm) on two existing, non-overlapping benchmarks, Toxic-Chat [2] and ToxiGen [3], and compared the performance with the OOD baselines derived from [2, 3], i.e., HateBERT and ToxDectRoBERTa.

Toxic-Chat:

ToxiGen:

Under the same OOD setting:

• SCM has a stronger generalization ability than the baselines in the original papers [2,3].
• On Toxic-chat, the performance of SCM-7B under OOD testing is comparable to the i.i.d. performance reported in [2] (F1=72.1).

[1] WildGuard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of LLMs. NeurIPS’24

[2] ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation. EMNLP’23 Findings

[3] ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection. ACL’22

3 Hyperparameter sensitivity analysis of $\beta$ (Q1)

Due to the length limit, the following table shows the impact of $\beta$ on the results of the SCM-1.5B, and it can be seen that the macro F1 generally shows a trend of first rising, then stabilizing, and then fluctuating slightly as the weight of logical constraint changes. When a larger weight is adopted (greater than 1), the penalty for logical inconsistencies tends to be saturated.

The other two models (0.5B/7B) show a consistent trend. We will include the results in the subsequent version.

---

Thank you again for your insightful feedback.

We believe these revisions will strengthen the paper's clarity. Please let us know if further clarifications are needed.

Best regards