Large Language Models can Become Strong Self-Detoxifiers

We thank the reviewer for comments on the clarity of our method and strong coverage of experiments. We give our detailed response to your questions in the following:

1. Formatting. We thank the reviewer for pointing out, we have corrected the font and capitalize each word in our titles.

2. Model $f_v(c,x)$. We let $f_v(c,x)$ be the closed-form Bayes optimal classifier which takes the form $f_v(c,x)=sign \left(w_v^T\left(g\left(c\oplus x\right)-b_v\right)\right)$, where $w_v = \Sigma^{-1}\left(\frac{\mu_1-\mu_2}{2}\right), b_v = \frac{\mu_1+\mu_2}{2}$, $\Sigma$ and $\mu_1,\mu_2$ are estimated according to line 206-209 from toxic/non-toxic context. We gave more details in paper section 3.1. In practice, this close-form solution yields 75.3% accuracy on Jigsaw Unintended Bias in Toxicity Classification dataset with Llama 3.1-8b-Instruct sentence embeddings and 74.2% with Llama 2-7b sentence embeddings. On 7979 toxic prompts vs benign prompts in RTP, this close-form solution yields 86.9% accuracy with Llama 3.1-8b-Instruct sentence embeddings and 82.1% with Llama 2-7b sentence embeddings. Our design choice of deterministic linear rule is inspired by vast self-supervised representation learning literatures that use linear evaluation for representation quality. Another benefit we see from this formulation is the efficiency via closed-form updates of constrained decoding for token-by-token generating, to steer the generation toward low-toxicity output, as specified in Sec. 4.6 runtime and memory usage.

3. Perplexity increase. The perplexity generally increases as the toxicity rate decreases, and this is a known trade-off face by detoxification literatures. For example, as shown in Table 13, other external-model-free methods lowered the toxicity level at the cost of much larger increase in perplexity (e.g. ToxificationReversal and self-debiasing yield 6.6X and 2.1-3.4X bigger perplexity). One aim of SASA is exactly to find a better trade-off with minimum overhead. In Figure 2, we showed the toxicity-perplexity trade-off plot on 4 test cases together with the strongest external-model-dependant method, RAD, and see that SASA's trade-off is generally comparable with RAD's and closes the performance gap without external signals.

   We point the reviewer to Table 7, 10-12 for the generated text examples. Although there is increase in the perplexity as the toxicity rate decreases, the overall perlexity is still small and the quality of generated texts is high.

4. Citations in Tab 1. We have added citations to Table 1 accordingly.

We thank the reviewer for acknowledging our rebuttal and for raising the score. Please let us know if there are any other questions or concerns. Thank you!

7. Detoxifying various types of harmful content. We provided in manuscript Appendix A.3 Table 9 the detoxification result with each attribute probability on the challenging RTP using Llama-2-7b. Specifically, the attributes include severe toxicity, identity attack, insult, profanity, and threat. From the table, we can see that SASA has demonstrated significant reductions in all types of harmful content, showcasing its effectiveness across diverse toxic categories. Our experiment on BOLD datsaet Table 14 also highlights SASA's usage in mitigating gender biased language. Originally, Llama decoded sentences for female group have generally higher toxic rate (0.066 vs 0.031), implying the LM being somewhat biased against female. With controlled decoding, SASA mitigates this gender bias well and reach balanced toxic rate (0% for both male and female groups).

8. Changing sampling distribution and the resulting generated text. We show an example in the following when we actively steer the generation away from being toxic. Given prompt "I hate you. You are being like a drama queen who seeks attention all the time. I wish you", the original top-p sampling retains nonzero probabilities for "die" with $p=0.03$, along with other 20 candidate tokens. With SASA, the continuation with "die" had the largest negative margin $-1.6$ compared with others and its sampling probability becomes 10 times smaller ($p'=0.003$).

   The perplexity generally increases as the toxicity rate decreases, and this is a known trade-off face by detoxification literatures. For example, as shown in Table 13, other external-model-free methods lowered the toxicity level at the cost of much larger increase in perplexity (e.g. ToxificationReversal and self-debiasing yield 6.6X and 2.1-3.4X bigger perplexity). One aim of SASA is exactly to find a better trade-off with minimum overhead. In Figure 2, we showed the toxicity-perplexity trade-off plot on 4 test cases together with the strongest external-model-dependant method, RAD, and see that SASA's trade-off is generally comparable with RAD's and closes the performance gap without external signals.

9. Key assumptions in our theoretical proof. The optimization problem we formulated can be seen as minimizing a convex function on a convex domain and stationary points (nullify gradient of lagrangian) are global optimiziers. The proof does not need any assumption.

10. SASA detoxification performance on different text lengths. In our experiments, we detoxify non-toxic RTP, challenging RTP, BOLD, and AttaQ datasets. Here, we analyze the lengths of these datasets in this figure (https://imgur.com/a/VDhdcsX) and report their average max toxicity over the samples within the range. We observe no obvious increase in the toxicity level of the longer sequences, implying SASA's efficacy for longer prompts. There is no theoretical limit for SASA. However, depending on how the LM is able to produce meaningful sentence representations for long texts, SASA effectiveness could be bottlenecked by the LM capacity. We have not experienced this in our experiments.

3. Sensitivity analysis across data sources. We have shown promising detoxifiction results on Real-toxicity-prompts (RTP), BOLD, and AttaQ, where RTP is selected from the OPENWEBTEXT CORPUS (OWTC) and contains English web texts scraped from outbound URLs from Reddit that associated with news or blog articles; BOLD is collected from Wikipedia pages and is scraped according to demographic domains; AttaQ is built based on HH-RLHF and hence has conversational data around 20 suggested topics by Anthropic. We note that putting together a sorted prompt set that covers wide categorized prompts from diverse data sources requires tremendous effort and is beyond the scope of our detoxification paper. While most baselines focus on only RTP, we tried to cover more diversity by including BOLD and AttaQ, and wish our promising results on those datasets can reassure the reviewer of the applicability of SASA.

   Specfically, we reduced the Avg. Max Toxicity by 2 times on challenging RTP (from 0.87 to 0.426), 3.2 times on non-toxic RTP (from 0.323 to 0.101), 3.3 times on AttaQ (from 0.468 to 0.142), and almost 9.3 times on BOLD (from 0.214 to 0.0229). Comparatively, RAD reduced the Avg. Max Toxicity by 1.8 times on challenging RTP (from 0.87 to 0.481), 2.3 times on non-toxic RTP (from 0.323 to 0.143), 1.8 times on AttaQ (from 0.468 to 0.264), and 4.3 times on BOLD (from 0.214 to 0.0496). Moreover, as discussed in Section 4.4 and Table 14, we have also shown SASA to be effective in mitigating subgroup toxicity - we showed that Llama decoded sentences for female group have generally higher toxic rate (0.066 vs 0.031), implying the LM being somewhat biased against female. With controlled decoding, SASA mitigates this gender bias well and reach balanced toxic rate (0% for both male and female groups).

4. Human evaluation. Conducting a user study or human evaluation is unfortunately beyond our scope due to our resource limitations. Moreover, our organizational policy also prohibits conducting human evaluations that contain harmful contents. We have given some examples of the generated texts in Table 7, 10-12 in our manuscript if the reader wish to inspect the performance qualitatively.

5. Quantitative assessment of the computational overhead. At each prediction one has to see the impact of each token in the vocab say on toxicity and this results with O(|V|) size of the vocab evaluation. In practice, we speed this process up by modifiying only top-p values of original logits (this strategy was also used in RAD). In the paper line 478-514, we have also discussed the wall-clock runtime and memory usage (2.6X faster, 1.4X less memory usage compared with RAD), demonstrating SASA's efficiency compared to the baseline. Compared with the standard autoregressive sampling, taking Llama-2-7b as an example, with single V100, the standard autoregressive sampling takes 2.9 hours to complete RTP 10k prompts; comparatively SASA takes 3.1 hours to complete these prompts without engineering optimizations.

6. More failure analysis. We thank the reviewer for initiating this discussion. We believe there are a few potential triggers and reasons that might cause SASA generation to remain toxic. For example, if the general tone of the prompts is too toxic to begin with, the top-p candidate continuations might all be toxic, then SASA could only find the potentially least toxic one but can not go beyond the available candidates. This issue can be mitigated by increasing p such that there are more candidate tokens to choose from, which on the other hand comes at the cost of latency overhead. Besides, SASA generation might also remain toxic as a consequence of the false positives with the Bayes optimal classifer. Specifically, on 7979 toxic prompts vs benign prompts in RTP, our close-form solution yields 86.9% accuracy with Llama 3.1-8b-Instruct sentence embeddings and have a breakdown confusion matrix with TP=3878. FP=418. FN=634, TN=3049. If a candidate continuation is wrongly assigned a positive margin, then by SASA's mechanism this continuation will be encouraged.

We thank the reviewer for the positive feedback and we are happy to see the reviewer agrees that SASA addresses a critical issue in LLM deployment with a lightweight solution that is applicable to various LLM applications. We give our detailed response to your questions in the following:

1. Toxicity-perplexity tradeoff. Generally, constrained decoding is for open-ended generation tasks when there are many valid token continuations. On tasks that are most related to domain knowledge or require factuality, one should prioritize to minimize the divergence to reference distribution (the second term in our optimization objective) by applying a small $\beta$. To the extreme when $\beta=0$, SASA becomes the standard decoding strategy. For tasks such as real-toxicity-prompt that have prompts prone to generating racist, sexist, or otherwise toxic language, the general rule of thumb is to choose a $\beta$ such that the resulting perplexity is not too many times bigger then the original. As the perplexity is naturally a monotonic funciton with $\beta$, thus users can decide the largest $\beta$ that yields reasonable perplexity for their own tasks. In practice, we try to limit our choice of $\beta$ such that the resulting perplexity is smaller than 2X the original perplexity. Balancing this tradeoff between perplexity and toxicity is a known challenge to all detoxification literature, and SASA serves as an external-model-free alternative to the existing model-dependant solutions with comparable tradeoffs. On the note of the "compromised coherence" by (combining word filtering with) SASA, originally we wrote that "Thus, despite the promising decrease in toxicity brought by the introduction of word filtering, the increase in perplexity also suggests the potential compromised generation coherence." in our manuscript Section 4.6. For example, at $\beta$ = 500, the perplexity rises from 7.195 with SASA alone to 19.657 with SASA+word filtering. We give some generated examples in the following to allow qualitative analysis, from which we see that SASA+WF continuations can sometimes drift away from the prompt, leading to overall compromised coherence of the generated story.

2. Interpretability of subspaces. Our use of attriubte subspace is motivated and backed by several recent literatures around the internal separability of LM representations. For example, [1] has also discussed linear classifiers for each attention head across all layers and showed toxic/no-toxic examples can be well-separated. Also, in [2], it has been studied that visualizations of LLM true/false statement representations also reveal clear linear structure. Here, we show an example of the t-SNE plot (https://imgur.com/a/cikNYqw) of 100 RTP toxic/non-toxic prompts' sentence embeddings by Llama-3.1b-Instruct. It can be seen that, there exists linear strategies that separate the two classes sufficiently well given that there is good separation between the two classes.

[1] Samuel Marks and Max Tegmark, "The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets", First Conference on Language Modeling, 2024.

[2] Yu Li and Han Jiang and Chuanyang Gong and Zhihua Wei, "DeStein: Navigating Detoxification of Language Models via Universal Steering Pairs and Head-wise Activation Fusion", First Conference on Language Modeling, 2024.

Dear Reviewer sk9R,

In light of the rebuttal timeline, we hope the reviewer can look at our rebuttal and share your feedback, if any. Should there be any outstanding questions or concerns, we are dedicated to providing our take and are ready for further discussion if needed. Thank you!

We thank the reviewer for the comments on the lightweightness of SASA and its effectiveness on tasks. We give our detailed response to your questions in the following:

We don't fully understand reviewer's concern about generalization capability. While we will try our best in the following to re-emphasize some facts around detoxification tasks, we sincrerly hope the reviewer can let us know if there are remaining concerns on this point.

• (i) In our manuscript, we have shown promising generalization ability through experiments on RTP, BOLD, and AttaQ where the same subspace was used to adjust the sample probability. Specfically, we reduced the Avg. Max Toxicity by 2 times on challenging RTP (from 0.87 to 0.426), 3.2 times on non-toxic RTP (from 0.323 to 0.101), 3.3 times on AttaQ (from 0.468 to 0.142), and almost 9.3 times on BOLD (from 0.214 to 0.0229). Comparatively, RAD reduced the Avg. Max Toxicity by 1.8 times on challenging RTP (from 0.87 to 0.481), 2.3 times on non-toxic RTP (from 0.323 to 0.143), 1.8 times on AttaQ (from 0.468 to 0.264), and 4.3 times on BOLD (from 0.214 to 0.0496). Moreover, as discussed in Section 4.4 and Table 14, we have also shown SASA to be effective in mitigating subgroup toxicity - we showed that Llama decoded sentences for female group have generally higher toxic rate (0.066 vs 0.031), implying the LM being somewhat biased against female. With controlled decoding, SASA mitigates this gender bias well and reach balanced toxic rate (0% for both male and female groups).
• (ii) As the reviewer pointed out, the perplexity generally increases as the toxicity rate decreases, and this is a known trade-off face by detoxification literatures. For example, as shown in Table 13, other external-model-free methods lowered the toxicity level at the cost of much larger increase in perplexity (e.g. ToxificationReversal and self-debiasing yield 6.6X and 2.1-3.4X bigger perplexity). One aim of SASA is exactly to find a better trade-off with minimum overhead. In Figure 2, we showed the toxicity-perplexity trade-off plot on 4 test cases together with the strongest external-model-dependant method, RAD, and see that SASA's trade-off is generally comparable with RAD's and closes the performance gap without external signals.
• (iii) We point the reviewer to Table 7, 10-12 for the generated text examples. Although there is increase in the perplexity as the toxicity rate decreases, the overall perlexity is still small and the quality of generated texts is high.

5. Common insights between SASA and personalization alignment [3] and factuality enhancement [4], and guidance on similar ideas in a new domain. SASA, PAD [3], DoLa[4] share the common goal of improving language model outputs during inference without additional training or reliance on external models. The core insights and shared principles include

   • They are all test-time methods that steer outputs toward different desired attributes, such as reduced toxicity, enhanced personalization, or improved factual accuracy. This training-free strategy allows easy adoption and does not change the original LM parameters.
   • Their core assumptions are all around the informativeness of LM internal states or representations. For instance, SASA identifies toxic subspaces within the model's sentence embedding space, while DoLa contrasts logits from different layers to enhance factuality.

   To give guidance on similar ideas in a new domain, we suggest a close look at the model's internal representations to identify patterns associated with the target attribute. As large models come with more and more emergent capabilities, the hope is that more attributes/notions/definitions are already encapsulated in the internal states of the model, as evidence in PAD, DoLa, and SASA. We will include our discussion and citations to the manuscript.

6. Issues with a well-trained critic. The first issue arises with this approach is the requirement for hosting the critic model during inference time. The second issue essentially stems from the first since there is also additional inference cost besides memory overhead (also discussed in manuscript line 478-514). The third issue regards the reliance of alignment on specific critic models, which might backfire and/or add costs for users if the critic is propietory. Comparatively, SASA works directly on any opensource models, where the chat version is released, but the reward model (the well-trained critic) used to align the model is often not released, which prevents developers from accessing and using the critic for test-time adjustment. Finally, SASA can also be viewed as a complimentay detoxification method to the existing ones. The fact that it is lightweight paves the way to perform detoxification collaborately similar to what we have shown in Appendix A.6 with non-parametric critic word filtering.

7. Views on using long chain-of-thought reasoning/planning and self-reflection to tackle toxicity issue in models. We appreciate the reviewer’s insightful suggestion to explore long chain-of-thought reasoning or planning as a means to address issues in LMs. While CoT reasoning has demonstrated effectiveness in structured reasoning tasks, we believe its direct applicability to toxicity mitigation may be limited: Toxicity often arises from biases or harmful associations embedded within the model’s representations, rather than from errors in multi-step reasoning processes. Therefore, CoT, which focuses on step-by-step logical generation, is less likely to target the root causes of toxic behavior effectively. That said, we recognize the potential of hybrid approaches that incorporate long CoT and self-reflection to detoxify during generation - self-evaluative steps that explicitly guide the model to critique its own outputs for harmful content. However, this will incur memory and speed overhead (due to generating and storing more tokens). The human labor needed to construct the CoT prompt should also be considered, as in CoT prompt commonly used for reasoning tasks does need to include reasoning trace in it. As such, we believe that lightweight decoding-time methods, such as SASA which does not require much manual intervention rather than choosing a desired $\beta$, remain a practical and more promising solution for toxicity mitigation.

We thank the reviewer for the positive feedback on our paper and for seeing SASA to be highly innovative both theoretically and practically. Indeed, SASA is a lightweight alternatives to existing toxicity mitigation techniques and we are happy to see the reviewer finds our motivation clear, our proposal (SASA) theoretically supported, and our experimental validation extensive. We give our detailed response to your questions in the following:

1. Evaluation on common sense knowledge. Typically these alignment proposals (e.g. RAD, Self-Debiasing, Toxification Reversal, Rectification, etc) don't consider utility checks such as mmlu since $\beta>0$ is only to be turned on when one needs benign open-ended generation, not on multiple choice tasks. As a test-time alignment methods, we do not make changes to the original LM to obtain a proxy model, which is a key difference to most backdoor attack elimination methods and self-distillation that involve updates of the original model. As our alignment method is not "invasive", we can always retrieve the original model capabilities and souce domain knowledge by turning off the detoxification factor $\beta=0$. In fact, on tasks that are most related to domain knowledge or require factuality, one should prioritize to minimize the divergence to reference distribution (the second term in our optimization objective) by applying a small $\beta$. To the extreme when $\beta=0$, SASA becomes the standard decoding strategy. This degree of freedom is similar to the decision between greedy decoding and top-p/K sampling with different temperature.
   Although it is not recommended to use top-p sampling with contrained decoding for common sense knowledge tasks, we show SASA with $\beta>0$ on mmlu in the following table for reviewer's reference.

2. SASA detoxification performance on different text lengths. In our experiments, we detoxify non-toxic RTP, challenging RTP, BOLD, and AttaQ datasets. Here, in the attached [figure] (https://imgur.com/a/VDhdcsX)(https://imgur.com/a/VDhdcsX), we analyze the lengths of these datasets and report their average max toxicity over the samples within the range. We observe no obvious increase in the toxicity level of the longer sequences, implying SASA's efficacy for longer prompts.

3. Baselines. RAD is published December 2023 and to our knowledge, is the state-of-the-art test-time LLM detoxicification method at the time of our submission. If the reviewer finds any specific better detoxification tools relevant to this paper, we are happy to include additional discussion.

4. Backbone models. Table 1-3 are drawn to match with baselines and we show the applicability of SASA to models by showing the discussion in Table 6. Per reviewer's request, we added RAD to the comparison for reviewer's reference. As can be seen from the figure (https://imgur.com/a/z3B5InV), while RAD still manages to detoxify challenging prompts, there is a notable gap from SASA (SASA yields Avg. Max Toxicity=0.283 at $\beta=100$ with perplexity 7.39 vs. RAD yields Avg. Max Toxicity= 0.408 at $\beta=300$ with perplexity 7.397).

We thank the reviewer for acknowledging SASA to be an interesting method that surpasses all the baselines in the experiments. We give our detailed response to your questions in the following:

1. Title. By Self-detoxifier, we mean we leverage the internal representations of LMs to guide attribute-conditioned generation instead of external LMs. To clarify this point, we offer to change the title to "Large Language Models can become Strong Self-Detoxifiers".

2. Contribution of the paper. SASA is a self-disciplined external-model-free test-time alignment method with theoretical justifications. With certain exposure to toxic content, it exploits the decoding LM with no structureal change and has no requirement for hosting another LM. This is a simple and neat solution towards more sustainable test-time alignment with small/little memory/time overhead. Specifically, as discussed in manuscript line 478-514, the original decoding on Llama-2-7b takes 2.9 hours on single V100 and consumes 15.5 GB of memory while SASA decoding takes 3.1 hours and uses 17.3 GB of memory. Finally, as the reviewer has noted in the review, our simple solution surpasses all the baselines in the experiments, which further strengthens our belief in simple yet effective approach other than sophisticated alternatives.

3. (Non-)linearly separable hyperspace. We understand your concern regarding our limitation section around the linear separability in the LM embedding space. We chose a linear approach on the initial decoding LM due to its simplicity, interpretability, and computational efficiency. Here are some more supports:

   • In the seminal work of self-supervised learning [1] and its followups, authors extensively use the linear evaluation protocol [2] on evaluating learned representatinos, by training a linear head with class labels. In this line of work, it has been argued that the power of linear evaluation in high-dimensional spaces can be understood by considering Reproducing Kernel Hilbert Spaces [3], and while non-linear heads can provide improvement over the linear evaluation, a linear model is adequate for evaluation [4]. Inspired by the linear evaluation propotol, in our framework, we use linear space modeling for toxicity modeling and detoxification on the inherent representations of language models.
   • Recent literature [5] has also discussed linear classifiers for each attention head across all layers and showed toxic/no-toxic examples can be well-separated. In [6], it has been studied that visualizations of LLM true/false statement representations also reveal clear linear structure.
   • Another benefit of linear space is that we can derive closed-form updates of constrained decoding for token-by-token generation, to steer the generation toward low-toxicity output, as specified in Sec. 4.6 runtime and memory usage.
   • Our experimental results on RTP, BOLD, and AttaQ also supported that this simple regime is sufficient to be more effective than baseline that relies on nonlinear external models.

   Going forward, if some particular LMs have more complicated embedding space or some attributes needs features beyond linear ones to be modeled, one can also leverage kernel methods or activations functions to introduce non-linearality. Follow-up works can also leverage embedding spaces induced from the earlier layers, which to the extreme will be building rules in the input space or RAD. Our work builds on the assumption and observation that the sentence embeddings of decoding LMs are informative and hence the linear strategy on their embedding space can already suffice good detoxification reuslts.

[1] Chen, Ting, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton, "A simple framework for contrastive learning of visual representations", In International conference on machine learning, 2020.

[2] Oord, Aaron van den, Yazhe Li, and Oriol Vinyals, "Representation learning with contrastive predictive coding", arXiv preprint, 2018.

[3] Bachman, Philip, R. Devon Hjelm, and William Buchwalter, "Learning representations by maximizing mutual information across views", Advances in neural information processing systems, 2019.

[4] Kolesnikov, Alexander, Xiaohua Zhai, and Lucas Beyer, "Revisiting self-supervised visual representation learning", In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019.

[5] Samuel Marks and Max Tegmark, "The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets", In First Conference on Language Modeling, 2024.

[6] Yu Li and Han Jiang and Chuanyang Gong and Zhihua Wei, "DeStein: Navigating Detoxification of Language Models via Universal Steering Pairs and Head-wise Activation Fusion", In First Conference on Language Modeling, 2024.