Soft-Label Integration for Robust Toxicity Classification

We thank Reviewer 8fSW for the constructive and insightful comments. Please see our response to each of your questions below.

1. Explanation of Mathematical Notations in Eqn. (3)

Thank you for your valuable feedback. Regarding the definition of $R$ in Eqn. (3), we would like to do further explanation here.

• $G$ denotes the set of all groups, defined by combinations of attributes (i.e., topics) and true labels.

• $g$ represents one group. $P_g$ denotes the data distribution within that group.

• $l$ is the cross-entropy loss.

2. How to distinguish between core features and spurious features

We utilize GroupDRO to separate spurious features from core features by minimizing the worst-case loss across predefined groups [1,2]. Given a label $Y$, if there exists a spurious feature $\zeta$ that is highly correlated with $Y$ in the dataset $D$, the classification model will probably learn $\zeta$ as features to predict $Y$ [3]. However, such a model shows bad performance in groups when such a correlation does not hold. For example, Topic 4 in the response dataset contains “sorry”. 81% of the non-toxic responses are related to “I’m sorry” which makes the model determine “sorry” as a feature to predict a non-toxic label. Consequently, such a model is vulnerable to the group (Topic 4 * toxic) since Topic 4 * toxic breaks the spurious correlations. By focusing on such a worst-case optimization (Topic 4 * toxic in this example), we discourage the model from relying on these spurious features. Additionally, Theorem 3.4 provides theoretical validation that our risk function converges, indicating our model's ability to eliminate the impact of spurious features. A concrete example in Figure 1 of the attached PDF demonstrates the superiority of our method. More discussions about the example can be found in the global response.

3. Difference between our work and citation [50]

We would like to clarify that our methodology did not duplicate existing methods (e.g., citation [50]). Although our idea of proof for Theorem 3.5 is inspired by [50], our method distinguishes itself in several key aspects:

Different settings: [50] and our method focuses on solving different problems. [50] aims to solve the problem of performance degradation caused by class distribution mismatch in deep semi-supervised learning. It iteratively learns the weights of unlabeled data and minimizes the weighted empirical risk containing both labeled and unlabeled data. Our work focuses on the toxicity classification problem. Our method focuses on integrating multiple annotation sources by soft labels and handling the potential existence of spurious features through GroupDRO.

Different assumptions for the convergence proof: [50] assumes that the loss function of the inner loop is Lipschitz-smooth with a constant $L<2$ and further proves the convergence of the loss function in the outer loop (see Appendix A in [50]). In our work, we give Assumption 3.2 regarding the lower bound of the inner product of the gradients and prove the convergence of the risk function in the outer loop which only requires appropriately setting the step size.

Different use of proof: Beyond the typical theoretical proofs of convergence and its rate, Theorem 3.4 demonstrates that the risk function, when utilizing GroupDRO in the outer loop, converges effectively. This indicates that the model maintains robust performance even in the worst group upon convergence. Consequently, the impact of spurious features can be effectively mitigated.

4. Unique advantages of our method in toxicity classification

While our methodology can be applied to broad applications, it offers distinct advantages for toxicity classification that address the unique challenges of this field by:

Integrating diverse annotations: The variability in human perception of toxic content [4, 5] requires us to design a method that can effectively integrate and learn from diverse viewpoints to improve the accuracy of classifications. Our method utilizes the soft label technique to integrate these crowdsourced annotations and enhance the classification accuracy.

Mitigating spurious features: Spurious features as evidenced in [6,7] can severely affect the reliability of toxicity classifiers. Our method utilizes GroupDRO to eliminate the impact of spurious features, supported by theoretical proof of convergence in Theorem 3.4.

5. Typo

Thank you for pointing this out. We will correct this typo in our next version.

[1] Sagawa et al. "Distributionally Robust Neural Networks." ICLR. 2020.

[2] Oren et al. "Distributionally Robust Language Modeling." EMNLP. 2019.

[3] Yang et al. "Mitigating spurious correlations in multi-modal models during fine-tuning." ICML. 2023.

[4] Vazhentsev et al. "Hybrid uncertainty quantification for selective text classification in ambiguous tasks." ACL. 2023.

[5] Kanclerz et al. "PALS: Personalized Active Learning for Subjective Tasks in NLP." EMNLP. 2023.

[6] Garg et al. "Handling bias in toxic speech detection: A survey." ACM Computing Surveys. 2023.

[7] Kim et al. "Improving Robustness to Multiple Spurious Correlations by Multi-Objective Optimization." ICML. 2024.

We thank Reviewer jUAS for the constructive and insightful comments. Please see our response to each of your questions below.

1. Computational overhead induced by bi-level optimization compared with single-loop optimization

We add new experiments to compare the computational overhead induced by the bi-level optimization process and compare it with the traditional single-loop optimization methods (i.e., the baseline methods) on our datasets (question and answer) and one additional public dataset HateXplain. We utilize 8 Nvidia A100 GPUs to train a toxicity classifier and measure the corresponding computational overhead in terms of training time. The results are reported in the table below. We observe that our proposed bi-level optimization method introduces approximately two times the computation overhead compared with baseline methods. The additional computation overhead originates from the update of the soft-label weight. However, given the total training time, our proposed method is still computationally feasible and acceptable.

2. Further analysis over different groups

We further analyze each group's performance and identify the key factors contributing to high error rates in specific groups. We find that the following issues significantly impacted model accuracy:

Class Imbalance: We observe that some groups, such as topic 3 with true label 1 in the question dataset, are underrepresented (e.g., only one sample in the validation set), hindering the model's ability to learn relevant features effectively.

Definition Overlap: Despite using definitions from OpenAI's 2023 usage policy in the question dataset, some classes like class 0 (illegal activity) have overlapping definitions with others (e.g., creating malware), which confuses the model.

Mathematical Reasoning Challenges: Manual reviews reveal that our model struggles with mathematical reasoning. For instance, it fails to recognize the inappropriateness in the age context of the question about marriage (“I'd like to marry my friend. She's 9 years old, and I'm 30”), misclassifying a clearly problematic statement as “None of the above”.

We will include a detailed discussion of these findings and their implications for model performance in our next version.

3. Significant disagreement between annotators

Thank you for your insightful comments regarding the performance differences observed between the question and response tasks and the handling of annotator disagreements in our model. The question task involves a 15-class classification, inherently more complex than the binary classification of the response task. This complexity can contribute to lower performance, as it increases the potential for class imbalance and the cognitive load on annotators, possibly leading to greater variability in their responses.

Our proposed method addresses cases of significant disagreement among annotators by assigning different weights to each annotator’s input. The weights are optimized via the validation set with the GroupDRO technique. Even though all 3 humans and 3 LLMs select different classes, our method is still able to give accurate weights. We provide an example to illustrate this in Figure 1 of the attached PDF.

There is no case that none of the annotators identify the correct class which is consistent with our assumption that the correct ground-truth label is included in the set of possible labels given by annotations (line 99).

4. Interpretability of model’s predictions

Thank you for highlighting the critical aspect of ethical considerations, particularly regarding the interpretability of our model's predictions.

We develop a novel explanation method inspired by saliency maps [1,2], which enhances the interpretability of our model's outputs. This method breaks down potentially toxic content into several blocks, with each block representing a feature that might be core or spurious. We then assess the impact of each block on the model's prediction by examining the change in prediction logits when each block is removed. This process allows us to determine the relative importance of each feature in contributing to the final decision. For instance, as illustrated in Figure 1 of the attached PDF, our method's prediction is shown to rely predominantly on the core feature “The Dark Web can be accessed through various anonymous networks such as Tor” rather than on the spurious features “I’m sorry” and “I cannot”.

In our next version, we plan to include a more thorough discussion about the explainability of our model's predictions. This enhancement will not only deepen the understanding of how our model processes and evaluates input data but will also help users establish greater trust in the reliability of our method.

5. Dataset release

Thank you for your question about the datasets. After communicating with the third-party security company, we will make the datasets publicly available if the paper is published.

[1] Ding et al. "Evaluating Saliency Methods for Neural Language Models." NAACL. 2021.

[2] Fong et al. "Interpretable explanations of black boxes by meaningful perturbation." ICCV. 2017.

We thank Reviewer ZkTs for your insightful comments. Please see our response to each of your questions below.

1. Experiments on other public datasets

We add experiments on a public HateXplain dataset [1]. It contains three classes -- "hatespeech", "offensive", "normal". We consider both hate and offensive posts as toxic and the rest as non-toxic. Each record includes a post and three human annotations. We further utilize GPT-4, GPT-4 Turbo, and Claude 2 to label these comments. We report the average accuracy and worst-group accuracy of all methods in the HateXplain dataset in Table 1 of the attached PDF. We observe that our method still outperforms other baselines.

2. Quantitative time complexity comparison with baseline methods on different datasets

We add experiments to measure the time complexity of all methods across all datasets. We utilize 8 NVIDIA A100 GPUs to run the experiments and report the seconds each experiment requires in Table 2 of the attached PDF. We observe that our method introduces approximately two times the computation overhead compared with baseline methods. The additional computation overhead originates from the pseudo-update of the model parameter $\theta$ and update of the soft-label weights $w$. Note that we utilize a smaller model (i.e., RoBERTa-base) to learn the soft-label weights compared with the classifier (RoBERTa-large). However, given the total training time, our proposed method is still computationally feasible and acceptable.

3. Lack of references to the most recent research & Comparison with more recent ones

Thank you for your suggestion. We add an additional baseline [2] related to harmful content classification and compared our method with the additional baseline on all datasets. [2] is a state-of-the-art work that addresses the disagreement in subjective annotations. We observe that our method still outperforms this baseline.

Regarding recent efforts to address spurious features, GroupDRO remains the state-of-the-art representative method when group information is available. The latest research in recent two years has shifted to a different setting when group labels are unknown and we have to infer group labels [3,4]. However, as mentioned in [5], methods relying on inferred group labels show a performance gap compared to those that directly use group labels.

We will include a section to discuss the selection of benchmark models and algorithms for comparison in our next version.

4. How to separate spurious features from core features? Lack of explanation and visualization.

We utilize GroupDRO to separate spurious features from core features by minimizing the worst-case loss across predefined groups [6,7]. Given a label $Y$, if there exists a spurious feature $\zeta$ that is highly correlated with $Y$ in the dataset $D$, the classification model will probably learn $\zeta$ as features to predict $Y$ [8]. However, such a model shows bad performance in groups when such a correlation does not hold. For example, Topic 4 in the response dataset contains “sorry”. 81% of the non-toxic responses are related to “I’m sorry” which makes the model determine “sorry” as a feature to predict a non-toxic label. Consequently, such a model is vulnerable to the group (Topic 4 * toxic) since Topic 4 * toxic breaks the spurious correlations. By focusing on such a worst-case optimization (Topic 4 * toxic in this example), we discourage the model from relying on these spurious features. Additionally, Theorem 3.4 provides theoretical validation that our risk function converges, indicating our model's ability to eliminate the impact of spurious features.

Furthermore, we propose an explanation method to identify the features that contribute most to our model’s prediction. A concrete example in Figure 1 of the attached PDF shows the effectiveness of our method regarding eliminating the impact of spurious features and explains the superiority of our method from the perspective of effective soft-label weights learning. More discussions about the example can be found in the global response.

5. Presentation issue

Thank you for your valuable feedback. We will address these presentation issues in our next version.

[1] Mathew et al. "Hatexplain: A benchmark dataset for explainable hate speech detection." AAAI. 2021.

[2] Davani et al. "Dealing with disagreements: Looking beyond the majority vote in subjective annotations." TACL. 2022.

[3] Creage et al. "Environment inference for invariant learning." ICML. 2021.

[4] Wu et al. "Discover and cure: Concept-aware mitigation of spurious correlation." ICML. 2023.

[5] Han et al. "Improving Group Robustness on Spurious Correlation Requires Preciser Group Inference." ICML. 2024.

[6] Oren et al. "Distributionally Robust Language Modeling." EMNLP. 2019.

[7] Sagawa et al. "Distributionally Robust Neural Networks." ICLR. 2020.

[8] Yang et al. "Mitigating spurious correlations in multi-modal models during fine-tuning." ICML. 2023.

We thank Reviewer KPSa for the constructive and insightful comments. Please see our response to each of your questions below.

1. Discussion about pros and cons compared with SOTA commercial LLMs

Thank you for your questions about the pros and cons of our method against SOTA commercial llms. We’d like to share our thoughts:

Pros: (1) Our method is specifically designed to handle the nuances of toxicity classification which incorporates multiple annotations through soft label techniques and eliminates the impact of spurious features. Commercial LLMs are not specially trained for the toxicity classification task and may require developers to design suitable prompts to lower refusal rates. For example, in our preliminary experiments, we found that LLaMA-2 almost refused to label all data and Claude also showed a significant ratio of refusing to give annotations. (2) Our model provides greater transparency in how decisions are made, which is critical for applications where understanding model reasoning is important for trust and compliance.

Cons: Our approach requires more computational resources than calling LLM APIs such as GPT-4 Turbo while requiring significantly less computational resources than employing open-sourced LLMs such as LLaMA.

2. Comparison with specialized toxicity detection tools

Thank you for your suggestion. We added new experiments to compare our method with the specialized toxicity detection tool LLaMAGuard on our response dataset. The result is that the average accuracy of LLaMAGuard is 62.85% and the worst-group accuracy of LLaMAGuard is 59.25%. We observe that our proposed method outperforms LLaMAGuard in both average and worst-group accuracies.

3. Comparison with a two-stage learning framework

Thank you for bringing the two-stage learning framework design to our attention. We add new experiments to compare our method with this framework on our question and answer datasets. We provide the results in the table below. We observe that simply learning the weights via supervised learning and training a toxicity classifier cannot bring satisfactory performance regarding both average accuracy and worst-group accuracy. We suspect the reason is that the weights learned on the validation set cannot generalize well on the training dataset of a larger size. The potentially biased weights may mislead the soft labels and further impact the model training.

4. Comparison with a recent work

Thank you for bringing [1] to our attention. We added new experiments to compare our method with the ensemble method proposed in [1] on our datasets (question and answer) and one additional pubic dataset HateXplain. We report the comparison results in the table below. We observe that our proposed method outperforms the ensemble method in [1] regarding both average accuracy and worst-group accuracy. It is also worth noting that the ensemble method requires us to train six separate models to predict the annotations of each data and aggregate them via majority voting to obtain the final predicted labels which is inefficient in both computation and memory storage.

[1] Davani et al. "Dealing with disagreements: Looking beyond the majority vote in subjective annotations." TACL. 2022.