Decoupling Noise and Toxic Parameters for Language Model Detoxification by Task Vector Merging

Can your method scale efficiently with larger models such as GPT-3 or GPT-4, and what computational resources are required for such implementations?

It’s hard to adapt closed source LLMs such as GPT-3, GPT4 because our approach needs to merge parameters from several models, which is not available in closed source LLMs. However, We believe our method can be applied to open-sourced larger models efficiently. Our approach requires the same amount of computation as the existing task vector negation method, except for merging multiple detoxified models, because total number of iterations of training multiple models on the split datasets is adjusted to be the same as the number of iterations for training a single model on the entire dataset is the same as total number of iterations of training multiple models on the split datasets. The process of merging multiple models requires significantly less computation than the training process and can be performed using only a single CPU. Moreover, this can be done more effectively by using off-the-shelf libraries such as Mergekit.

Is there a significant increase in training or merging time compared to single-model detoxification approaches?

As we mentioned in our previous answer, the training time is the same compared to the single-model detoxification method and our approach only requires additional time for the merging process. However, we believe that the merging time is relatively minor compared to the training time. Moreover, using the off-the-shelf libraries for merging models can make this process even more efficient.

Paper lacks comparison to other strong model self-detoxification approaches...

We did not compare our method with strong self-detoxification approaches as mentioned by the reviewer. Those methods require creating extensive toxic and non-toxic datasets for each model, which is data-intensive. Our task vector negation method allows efficient detoxification using only existing toxic datasets. We measured the performance of a typical detoxification method using only toxic datasets and confirmed our method’s superiority under the assumption of the same fine-tuning data availability.

Due to length limitations, we were unable to answer all of the reviewer’s questions and cannot include references.
We appreciate the reviewer’s understanding and suggest referring to our responses to the other reviewers

Dear Reviewer VAs3,

We sincerely apologize that we could not provide an enough response on the Rebuttal due to the length limitation. In this comment, we will address the parts that were lacking or unanswered in the Rebuttal. We hope this will help for Reviewers.

We wish to express our appreciation to the reviewers for their insightful comments on our paper. Please find our responses below.

The method's efficacy is demonstrated primarily through reductions in toxicity scores and model degradation rates, but there's a lack of discussion on whether the segmentation of datasets and subsequent model merging might lead to overfitting on the specific characteristics of the datasets used (i.e., Civil Comments and Toxigen). What other aspects of these datasets are being exploited in this process? Without further clarity, this could impact the robustness of the method when applied to real-world, diverse datasets (Section 4.1).

What alternative metrics for model degradation may be used, that were not explored here?

As the reviewer mentioned, we acknowledge the possibility of overfitting to other characteristics of the dataset beyond toxicity. However, as we mentioned in this paper, our method has been shown to enhance a common feature (detoxification) and suppress distinct features (data format, etc.) in segmented datasets, which we believe will result in better generality compared to the existing task vector negation method. To further clarify the degeneration of other aspects, we have included additional evaluations on various downstream tasks [1, 2, 3, 4, 5, 6, 7]. We choose these downstream tasks for the evaluation of model degeneration following this detoxification research [8]. We measure this by averaging accuracy on the downstream tasks. We provide a short description for each downstream task on our comment to Reviewer tRLA. Moreover, we have presented the main experiment results including the downstream evaluations in the table (please see our Rebuttal to Reviewer tRLA). As a result, We found that our approach can achieve better detoxification than the existing task vector negation method while maintaining similar model performance.

Despite the empirical rigour in general, this paper does not explore a wider collection of datasets (or languages) nor does it provide a robust collection of baselines (i.e., existing detoxification methods, especially newer or more complex architectures) against which results are compared.

Due to the limited number of existing toxic datasets, as the reviewer mentioned, we were not able to conduct experiments with a wider variety of datasets. However, to ensure the robustness of our method across datasets, we included the Toxigen in addition to the Civil Comments dataset used in previous detoxification research. Furthermore, we conducted experiments with Llama2-7b and presented the results in our Rebuttal to Reviewer vKrR. The results in that table show that the proposed method is also effective on the recent and larger model.

Regarding the comparison with existing detoxification methods, we did not compare our proposed method with existing strong self-detoxification approaches [8, 9]. However, these self-detoxification methods require the creation of extensive toxic and non-toxic datasets from a model. These datasets need to be generated for each model, requiring a large amount of data for each detoxification. However, the task vector negation method adopted in the proposed approach allows for efficient detoxification using only existing toxic datasets.

Thank you for your advice. We will update our paper following your advice.

[1] HellaSwag: Can a Machine Really Finish Your Sentence? by Zellers et al.

[2] WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations by Pilehvar and Camacho-Collados

[3] PIQA: Reasoning about Physical Commonsense in Natural Language by Bisk et al.

[4] WinoGrande: An Adversarial Winograd Schema Challenge at Scale by Sakaguchi et al.

[5] LAMBADA: Backward Chaining for Automated Reasoning in Natural Language by Kazemi et al.

[6] RACE: Large-scale ReAding Comprehension Dataset From Examinations by Lai et al.

[7] BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions by Clark et al.

[8] Exploring the limits of domain-adaptive training for detoxifying large-scale language models by Wang et al.

[9] CMD: a framework for Context-aware Model self-Detoxification by Tang et al.

Dear Reviewer tRLA,

We sincerely apologize that we could not provide an enough response on the Rebuttal due to the length limitation. In this comment, we will address the parts that were lacking or unanswered in the Rebuttal. We hope this will help for Reviewers. We wish to express our appreciation to the reviewers for their insightful comments on our paper. Please find our responses below.

The primary issue with the paper is its writing. The exact methodology is unclear and hard to follow. The experimental setup does not clearly state the experimental setup, choice of metrics for evaluation, and justification for choices made in the experimental design.

We sincerely apologize for any difficulties in following our experiment settings. Based on the advice of many reviewers, if our paper is accepted, we plan to make the following changes and additions in the camera-ready version:

1. Correct typos and grammatical errors.
2. Clearly describe the purpose and intent of each experiment.
3. Provide more detailed information on the evaluation methods and settings for each experiment (Please see our response below).
4. Include a precise definition of toxicity and examples for each toxicity category (Please also see our comment to Reviewer vKrR).

Once again, we deeply apologize for the unclear explanations that made it difficult to comprehend this paper. If there are any other comments in addition to what we have mentioned, please let us know. Thank you very much.

Beyond writing, the first point of confusion is how toxicity is evaluated. How is avg. max toxicity and toxicity probs are calculated, as shown in Table 1.

We are truly sorry for the difficulty caused by not providing clear explanations for the metrics in the table caption. Following the previous study [1], We evaluate the model’s toxicity using Perspective API [2] in Table 1. We specifically use the nucleus sampling strategy [3] to produce 25 generations of 25 tokens each for the same prompt. This allows us to determine the maximum toxicity score for each prompt and compute the average as the Expected Maximum Toxicity. Additionally, we calculate the probability of generating toxic continuations (toxicity score above 0.5) within the 25 generations for each prompt, referred to as the Toxicity Probability score.

We prompt all models with “I don’t care if this is controversial” and we use 1K prompts labeled as ‘normal’, and randomly sampled from RealToxicPrompts [1] for the toxicity evaluation. Although the explanations for these metrics are included in the paper, we have realized that it is challenging for readers to understand without them being under the table caption. We will address this issue with the other writing problems the reviewers have pointed out, in the camera-ready version.

Degeneration of the model is evaluated with perplexity on wiki-103. This metric alone cannot sufficiently support the claim that this method leads to lesser degeneration. For example, what happens to other capabilities of the models? It would also be important to evaluate the generation themselves, as perplexity could be a very noisy proxy for generation quality. Specifically, human evaluation of generations would be highly recommended.

Although we also agree with the reviewer’s opinion that human evaluation is the most accurate evaluation, we were unable to conduct human evaluation due to time constraints. However, following the feedback from the reviewers, we have included additional evaluations on various downstream tasks [4, 5, 6, 7, 8, 9, 10]. We choose these downstream tasks for the evaluation of model degeneration following this detoxification research [11]. We measure this by averaging accuracy on the downstream tasks. We provide short descriptions for each downstream task in the next comment due to the length limitation.

Degeneration of the model is evaluated with perplexity on wiki-103. This metric alone cannot sufficiently support the claim that this method leads to lesser degeneration. For example, what happens to other capabilities of the models?...

Following the feedback from the reviewers, we have included additional evaluations on various downstream tasks. We choose these downstream tasks for the evaluation of model degeneration following this detoxification research. We measure this by averaging accuracy on the downstream tasks.

We have presented the main experiment results including the downstream evaluations in the table below.

We indicated the average scores for downstream tasks as the ‘Utility’ score in the rightmost column. Our model’s degeneration after detoxification was similar to the existing task vector negation method but achieved better detoxification. We will include the detailed evaluation results of all detoxified models in the final version if our paper is accepted.

Due to length limitations, we were unable to answer all of the reviewer’s questions and cannot include references.
We appreciate the reviewer’s understanding and suggest referring to our responses to the other reviewers

Continuing from the previous comment, we have provided a brief description of the downstream tasks below.

Hellaswag [4]: Tests the model’s ability to choose the most plausible continuation of a given context, assessing commonsense reasoning.
WiC (Word-in-Context) [5]: Evaluates the model’s ability to determine if a word has the same meaning in two different contexts, testing lexical semantics.
PIQA (Physical Interaction QA) [6]: Evaluates the model’s physical commonsense reasoning by selecting the most reasonable answer to everyday task questions.
WinoGrande [7]: Evaluates the model’s understanding of pronoun resolution in sentences with ambiguous references, testing commonsense reasoning.
LAMBADA [8]: Evaluates the model’s ability to predict the last word of a given passage, testing its broad contextual understanding and coherence.
RACE [9]: Tests the model’s reading comprehension skills on passages with multiple-choice questions, used primarily for middle and high school-level texts.
BoolQ [10]: Tests the model’s ability to answer yes/no questions based on a given passage, assessing its fact-checking and reasoning abilities.

In the rebuttal section, we have presented the main experiment results including the downstream evaluations in the table. We truly apologize for not being able to present the results clearly due to the length limitation.

Thank you for your advice. We will update our paper following your advice. Again, thank you for giving us the opportunity to improve our manuscript with your valuable comments and queries.

[1] RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models by Gehman et al.

[2] Perspective API by Conversation AI

[3] The curious case of neural text degeneration by Holtzman et al.

[4] HellaSwag: Can a Machine Really Finish Your Sentence? by Zellers et al.

[5] WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations by Pilehvar and Camacho-Collados

[6] PIQA: Reasoning about Physical Commonsense in Natural Language by Bisk et al.

[7] WinoGrande: An Adversarial Winograd Schema Challenge at Scale by Sakaguchi et al.

[8] LAMBADA: Backward Chaining for Automated Reasoning in Natural Language by Kazemi et al.

[9] RACE: Large-scale ReAding Comprehension Dataset From Examinations by Lai et al.

[10] BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions by Clark et al.

Dear Reviewer tRLA,

As of now, we have not received any responses from the reviewer. If the reviewer has any questions or comments about the rebuttal or the results of the additional experiments, we would be happy to answer them. We apologize for the last-minute reminder during your busy time.

Dear Reviewer o8PJ,

We sincerely apologize that we could not provide an enough response on the Rebuttal due to the length limitation. In this comment, we will address the parts that were lacking or unanswered in the Rebuttal. We hope this will help for Reviewers. We wish to express our appreciation to the reviewers for their insightful comments on our paper. Please find our responses below.

Perplexity is not currently considered the best metric for generation performance. The model could produce outputs with high confidence while outputting low-quality text. It would be interesting to measure the performance of the model on some LLM benchmarks to ensure the lack of performance degradation.

Following the feedback from the reviewers, we have included additional evaluations on various downstream tasks [1, 2, 3, 4, 5, 6, 7]. We choose these downstream tasks for the evaluation of model degeneration following this detoxification research [8]. We measure this by averaging accuracy on the downstream tasks. We provide a short description of these downstream tasks on our comments to Reviewer tRLA. Moreover, we have presented the main experiment results including the downstream evaluations on our rebuttal to Reviewer tRLA’. As a result, We found that our approach can achieve better detoxification than the existing task vector negation method, while maintaining similar model performance.

The models evaluated are relatively small or non state-of-the-art. It would be interesting to see if this technique scales on bigger models. Have you tried this method on larger models? Does it work at scale?

We believe that our approach will also work on larger and state-of-the-art models. To demonstrate this, we applied our proposed method to a larger and more recent model, Llama2 - 7b, and presented the results in the table (please see the table in our Rebuttal to Reviewer vKrR). The results in that table show that the proposed method is also effective on the recent and larger model.

Thank you for your advice. We will update our paper following your advice.

Again, thank you for giving us the opportunity to improve our manuscript with your valuable comments and queries.

[1] HellaSwag: Can a Machine Really Finish Your Sentence? by Zellers et al.

[2] WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations by Pilehvar and Camacho-Collados

[3] PIQA: Reasoning about Physical Commonsense in Natural Language by Bisk et al.

[4] WinoGrande: An Adversarial Winograd Schema Challenge at Scale by Sakaguchi et al.

[5] LAMBADA: Backward Chaining for Automated Reasoning in Natural Language by Kazemi et al.

[6] RACE: Large-scale ReAding Comprehension Dataset From Examinations by Lai et al.

[7] BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions by Clark et al.

[8] Exploring the limits of domain-adaptive training for detoxifying large-scale language models by Wang et al.

The paper contains several typos ("detoxify the each model", "paramter").

We sincerely apologize for any difficulties in following our experiment settings. Based on advice of many reviewers, if our paper is accepted, we plan to make the following changes and additions in the camera-ready version:

1. Correct typos and grammar errors.
2. Clearly describe the purpose and intent of each experiment.
3. Provide more detailed information on the evaluation methods and settings for each experiment (Please also see our response below).
4. Include a precise definition of toxicity and examples for each toxicity category (Please also see our response from Reviewer vKrR).

The paper does not provide a clear definition of toxicity or the kind of phenomena evaluated. It would be interesting to know some examples of toxicity evaluated and how accurate is Detoxify on those examples. What kind of toxic phenomena is evaluated? Words that are considered toxic or sentences that are toxic due to the context?

We sincerely apologize for any confusion caused by not clearly defining toxicity. The definition of toxicity is subjective and may vary slightly across different studies, but we used the toxicity definition from the Perspective API. According to the Perspective API, which is commonly used to assess toxicity in many papers, toxicity is defined as “a rude, disrespectful, or unreasonable comment that is likely to make you leave a discussion.” The Perspective API categorizes toxicity into four levels and classifies toxic sentences into four categories. The Perspective API uses data labeled by humans according to their toxic levels and categories. This data is used to train a classifier, which then measures the toxicity of text. If our paper is accepted, we plan to include examples of generated sentences from both the baseline and our detoxified models, categorized by the toxicity levels and the categories. Once again, we apologize for not providing a clear definition and promise to include a more precise definition in the paper.

Due to length limitations, we were unable to answer all of the reviewer’s questions and cannot include references.
We appreciate the reviewer’s understanding and suggest referring to our responses to the other reviewers

Dear Reviewer o8PJ,

As of now, we have not received any responses from the reviewer. If the reviewer has any questions or comments about the rebuttal or the results of the additional experiments, we would be happy to answer them. We apologize for the last-minute reminder during your busy time.

Dear Reviewer vKrR,

We sincerely apologize that we could not provide an enough response on the Rebuttal due to the length limitation.
In this comment, we will address the parts that were lacking or unanswered in the Rebuttal. We hope this will help for Reviewers.

We wish to express our appreciation to the reviewers for their insightful comments on our paper. Please find our responses below.

The idea of using task vectors is not novel and have been widely explored in the literature. Moreover, learning a task vector for a text style transfer task is also not novel. Therefore, the contribution

As the reviewer has pointed out, we conducted detoxification by utilizing task vector negation. However, we believe that our main contributions are as follows. 1) We proposed a novel detoxification method by merging multiple task vectors from fine-tuned models with split toxic datasets, and 2) We discovered that when models trained on split datasets are merged, the merged model experiences a smaller amount of parameter shift from the pre-trained model than the standard task vector negation method. We assume that by applying our method, the important attribute (detoxification) is maintained within the merged vectors, while other attributes (text format, etc.), which we believe might be present in the split datasets and related to model degeneration, are suppressed.

Lack of other merging strategies. Authors do not consider any other merging strategies except linear. It would be beneficial to consider other merging strategies from related work. These methods were proven to be better than the simplest linear merging.

The detoxification results are shown in the table in the Rebuttal section. We mentioned the performance (Utility Score) on downstream tasks and described these tasks in our comments to Reviewer tRLA. We hope this will be helpful.

The paper does not provide a clear definition of toxicity or the kind of phenomena evaluated. It would be interesting to know some examples of toxicity evaluated and how accurate is Detoxify on those examples.

We sincerely apologize for any confusion caused by not clearly defining toxicity. The definition of toxicity is subjective and may vary slightly across different studies, but we used the toxicity definition from the Perspective API [2]. According to the Perspective API [2], which is widely used to assess toxicity in many papers [1, 3, 4], toxicity is defined as “a rude, disrespectful, or unreasonable comment that is likely to make you leave a discussion.” The Perspective API categorizes toxicity into four levels and classifies toxic sentences into four categories. The Perspective API uses data labeled by humans according to their toxic levels and categories. This data is used to train a classifier, which then measures the toxicity of text. If our paper is accepted, we plan to include examples of generated sentences from both the baseline and our detoxified models, categorized by the toxicity levels and the categories. Once again, we apologize for not providing a clear definition and promise to include a more precise definition in the paper.

Paper lacks comparison to other strong model self-detoxification approaches. With the current evaluation results it is unclear, whether the proposed method is better than other strong approaches.

In this paper, we did not compare our proposed method with existing strong self-detoxification approaches [3, 4], as mentioned by the reviewer. However, these self-detoxification methods require the creation of both extensive toxic and non-toxic datasets from a model. These datasets need to be generated for each model, requiring a large amount of data for each detoxification. However, the task vector negation method adopted in our proposed approach allows for efficient detoxification using only existing toxic datasets. We measured the performance of a typical detoxification method (i.e., gradient ascent) using only toxic datasets and compared its performance with our proposed method, confirming that our method is superior to those under the assumption that the fine-tuning data available are the same.

[1] Editing Models with Task Arithmetic by Ilharco et al.

[2] Perspective API by Conversation AI

[3] CMD: a framework for Context-aware Model self-Detoxification by Tang et al.

[4] Exploring the limits of domain-adaptive training for detoxifying large-scale language models by Wang et al.

[5] TIES-Merging: Resolving interference when merging models by Yadav et al.

Authors do not consider any other merging strategies except linear. It would be beneficial to consider other merging strategies from related work. These methods were proven to be better than the simplest linear merging. Though pointed out in Future work section, this limitation appears to be crucial for the work to be considered thorough enough.

Following the opinions of the reviewer, we applied Ties-Merging to our proposed method, which is widely used as a merging method other than linear merging. Specifically, we used Tie-Merging to merge multiple detoxified models from split datasets in our approach. We have presented the results including this detoxification method to Llama2-7b in the table below.

In the table, “Linear” refers to linear merging, and Ties-Merging represents a detoxification model using Tie -Merging. Following the advice from the reviewer, we have also included the utility score, which indicates the average scores on downstream tasks. The results in the table show that our approach with Ties-Merging also achieved better detoxification performance while maintaining a similar level of model performance (Fluency, Utility scores).

Due to length limitations, we were unable to answer all of the reviewer’s questions and cannot include references.
We appreciate the reviewer’s understanding and suggest referring to our responses to the other reviewers

Dear Reviewer vKrR,

As of now, we have not received any responses from the reviewer. If the reviewer has any questions or comments about the rebuttal or the results of the additional experiments, we would be happy to answer them. We apologize for the last-minute reminder during your busy time.

Dear all, The discussion period will close soon. Please respond to the authors' rebuttal and let us know if you have any further concerns.