Risk-Aware Distributional Intervention Policies for Language Models

We thank the reviewer for the valuable feedback and appreciation of our work. Here are our replies to your questions.

The derivation of the intervention method in this paper is based on the assumption that distributions P and Q follow a normal distribution. Can the authors verify that the activation values sampled at each layer in LLMs indeed follow a normal distribution?

We have used two types of normality testing (univariate and multivariate) for the attention heads that are classified as undesirable ($\hat{P}$) and after applying intervention ($Q$) -- the results are not clear: the univariate returns normality most of the time with good p-values, but a more realistic test, the Henze-Zirkler test for multivariate normality, usually dismisses the null-hypothesis (means there is no conclusion that the distribution is normal).

If P and Q follow other distributions, can the method in this paper be correspondingly adapted?

If P and Q follow other distributions, we can still use the optimization to solve for the linear intervention. However, the probability guarantee will not be met all the time. We note that without putting the assumption on the high-dimensional distribution P and Q, the optimization problem will be ill-posed and, therefore, very hard to solve. For example, we might need to estimate P and Q via non-parametric methods. The linear intervention is also an optimal transport plan when using Gaussian assumptions.

When combining Few-shot prompting with the RADIANT method, will the accuracy of the classifier obtained in the first stage change? Is it necessary to retrain the classifier?

Indeed, we need to retrain the classifier, and combining few-shot prompting with the RADIANT method will result in a change in the accuracy of the classifier obtained in the first stage.

How does the accuracy of the first-stage classifier affect the effectiveness of the second-stage intervention method? The paper uses a Linear Logistic Classifier as the classifier in the first stage; if a more accurate classifier is used, can the intervention effectiveness be further improved?

The accuracy of the first-stage classifier significantly impacts the effectiveness of the second-stage intervention method. This is because a highly accurate classifier minimizes the false-negative rate, ensuring that undesirable text is effectively detected, and reduces the false-positive risk, preventing desirable text from being misclassified as undesirable. This is crucial to preserving the original semantics of desirable text while effectively intervening in undesirable cases.

In our work, we use a Logistic Classifier in the first stage with the training loss proposed in Section 2 and the subsequent distributional intervention that can be efficiently optimized using semidefinite programming. Moreover, our method demonstrates superior performance compared to other approaches, such as the Non-Linear Inference Time Intervention (NL-ITI) (Hoscilowicz et al., 2024), which employs an MLP classifier. We added this result to Table 1 of the updated PDF.

The method for determining the crucial hyper-parameter remains ambiguous. It's unclear whether the hyper-parameters $\alpha$ and $\Gamma$ are universally applicable or require case-by-case optimization. Considering the improvement provided by RADIANT, selecting an appropriate hyper-parameter is vital, as suggested by Table 6 and Table 7. However, the paper fails to explain how this selection should be done.

We apologize for the ambiguity in selecting $\alpha$ and $\Gamma$ in the main paper. We argue that similar to many of the machine learning frameworks and algorithms, having a fixed set of $\alpha$ and $\Gamma$ that can be universally applicable is highly improbable (think bias-variance tradeoff in model selection). Therefore, we suggest using the common practice in both academic and industrial settings, which is grid search combined with cross-validation.

RADIANT shows an increase in CE and KL divergence compared to ITI, suggesting that the method deviates from the initial distribution. This could indicate potential adverse effects on the model's performance on other tasks. The paper should address this concern and discuss how RADIANT impacts overall model performance.

While it is true that we observed a general increase in cross-entropy (CE) and Kullback-Leibler (KL) divergence when comparing RADIANT to ITI, the increases are typically not substantial, with differences ranging from 0.01 to 0.03. Furthermore, we found that higher True*Info and True metrics often correlate with more desirable qualitative outputs (as detailed in Appendix B), where our framework consistently outperforms ITI.

What are the cost differences between ITI and RADIANT as presented in Table 5?

From the theoretical aspect, it is obvious that a head intervention of ITI, which is just a vector addition, is faster than that of RADIANT, which comprises a matrix multiplication and a matrix addition operator. The table above reports the average percentage increase in inference time per answer of ITI and RADIANT across the base models. It is observed that the normal version of RADIANT imposes more additional time in inference than ITI does. However, it is worth noting that all interventions of RADIANT are conducted on the same layer, while those of ITI are on multiple pairs of layer heads. This attribute of RADIANT allows us to parallel the interventions, which is impossible for ITI. We denoted the parallel version of RADIANT as RADIANT-P. RADIANT-P offers the same decent results as RADIANT but imposes much less computation cost into base models than RADIANT and ITI. We also included this table in the updated PDF.

How is the difference between generation accuracy and probe accuracy used in the evaluation?

Generation accuracy refers to the performance on QA tasks (such as multiple-choice questions in TruthfulQA), while probe accuracy refers to the performance of the probing method described in Section 2 of our paper.

The equations and symbols presented in the paper can be challenging to follow without clear explanations of each component and its role in the method. Providing detailed descriptions and context would improve readability and comprehension.

Thank you for this constructive comment on our presentation. In our revised version, we added a paragraph outlining our intuition at the beginning of Section 3. We also revised the three beginning paragraphs of Section 3 to make it less notation-dense.

Reference

[1] https://huggingface.co/openai-community/gpt2-large

[2] Singh, Shashwat, et al. "Representation Surgery: Theory and Practice of Affine Steering." Forty-first International Conference on Machine Learning.

[3] Liu, Alisa, et al. "DExperts: Decoding-time controlled text generation with experts and anti-experts." arXiv preprint arXiv:2105.03023 (2021).

[4] Gururangan, Suchin, et al. "Don't stop pretraining: Adapt language models to domains and tasks." arXiv preprint arXiv:2004.10964 (2020).

[5] Yang, Zonghan, et al. "Unified detoxifying and debiasing in language generation via inference-time adaptive optimization." arXiv preprint arXiv:2210.04492 (2022).

[6] Dathathri, Sumanth, et al. "Plug and play language models: A simple approach to controlled text generation." arXiv preprint arXiv:1912.02164 (2019).

[7] Pozzobon, Luiza, et al. "Goodtriever: Adaptive toxicity mitigation with retrieval-augmented models." arXiv preprint arXiv:2310.07589 (2023).

The proposed method, RADIANT, appears to offer minimal modifications compared to the Inference-Time Intervention (ITI) method. The main difference lies in the selection of attention heads between RADIANT and ITI. This slight variation may not constitute a significant advancement over existing methods.

We thank the reviewer for raising the concern, but we argue that this is not the case. In fact, besides the difference in the selection of attention heads between RADIANT and ITI (as the reviewer has pointed out), we proposed a whole new way of finding the intervention value $\Delta$ for each of the selected attention heads. This is done by solving a novel optimization problem -- a stochastic program inspired by optimal transport theory. While involving technical details, the overall objective of this optimization problem is to be effective in converting the undesirable activations to the desirable regions, while minimizing the magnitude of the intervention to sustain the context of the input. For this fact, we firmly believe that this is a stark contrast with ITI in terms of finding each of the head's intervention value.

A major limitation is the narrow focus on a limited number of datasets. The paper does not sufficiently compare RADIANT with other related methods in the field. It seems that the method may be limited to the LLaMA family of models.

The paper lacks results from Supervised Fine-Tuning (SFT) as a baseline, which is important for benchmarking. Referring to the experimental settings used in ITI and including SFT results would allow for a more comprehensive comparison.

Thank you for the two valuable suggestions. We have conducted a comprehensive set of experiments, incorporating additional baselines and a new dataset focused on toxicity generation. For detailed information, we kindly direct the reviewer to our shared response. Below, we provide a brief summary of our efforts to address your concerns:

For the QA task, we included two recent baselines, NL-ITI (Hoscilowicz et al., 2024) and LITO (Bayat et al., 2024), which employ approaches similar to ITI. These comparisons can be found in our shared response or in Table 1 of the revised paper. Overall, RADIANT demonstrates superior performance compared to NL-ITI and LITO across three base models—Llama-7B, Llama3-8B, and Llama2-chat-13B—on nearly all metrics, except a few cases involving MC metrics.

As you suggested, we also expanded our experiments to evaluate RADIANT’s effectiveness on GPT and Gemma base models. The results show that RADIANT consistently outperforms finetuning-free methods in terms of truthfulness and informativeness. Further details on these results are available in our shared response or Appendix A.3 of the revised paper

We investigated RADIANT’s performance on a different task: toxicity mitigation. In this experiment, we compared RADIANT with various state-of-the-art methods, including MiMiC [2] (both Mean Matching and Mean+Covariance Matching versions), DEXPERTS [3], DAPT [4], UDDIA [5], PPLM [6], and GOODTRIEVER [7]. These baselines were categorized into two groups: methods requiring extensive fine-tuning or gradient-based computations during inference (e.g., DAPT, GeDI, PPLM, UDDIA, DEXPERTS, GOODTRIEVER) and lightweight inference-time methods (e.g., MiMiC, ITI, RADIANT). While the fine-tuning group generally performs better on toxicity metrics, these methods come with high computational costs due to the need for fine-tuning or gradient computations. In contrast, inference-time methods like MiMiC, ITI, and RADIANT offer comparable toxicity reduction but are significantly more resource-efficient. Among these, RADIANT achieves the best toxicity reduction while maintaining high fluency and diversity. Notably, RADIANT outperforms almost all methods in fluency (except UDDIA) and matches the diversity of top-performing methods like PPLM. This underscores RADIANT’s ability to balance efficiency and performance effectively. For a detailed discussion of the experimental setup, metrics, evaluation framework, and baseline results, we encourage the reviewer to consult our shared response or Appendix A.4 of the revised paper.

We thank the reviewer for the valuable feedback and appreciation of our work. Here are our replies to your questions.

I think there exists a big problem in paper writing. For example, some settings in Section 2 can be explained in the experimental settings in Section 4 or Appendix. I also don’t like the description in Section 3. The paper puts together too many symbolic definitions, which takes me a lot of time to understand.

Thank you for this constructive comment on our presentation. In our revised version, we added a paragraph outlining our intuition at the beginning of Section 3. We also revised the three beginning paragraphs of Section 3 to make it less notation-dense.

However, we argue that the placement of problem settings in Section 2 is intentional and crucial for clarity. Section 2 outlines the problem statement, providing the essential context for understanding the proposed method. This includes a description of the language model's architecture, the notation for activations at each layer, and the goal of detecting and modifying undesirable text. Moving this information would disrupt the logical flow of the paper and make it harder for readers to grasp the foundational concepts before diving into the specific methods and experimental details.

The reviewer criticized Section 3 for its use of symbolic definitions, finding them difficult to understand. While acknowledging the complexity, we argue that these definitions are necessary to present the novel method, RADIANT, with the required precision.

We aim to provide a rigorous justification and explanation for our method, so we make Section 3 technical. We also provide a clear definition of each technical term to avoid any confusion for the reader. Moreover, the notations we use are relatively standard and widely used in optimization and optimal transport, and many papers published at ICLR use similar notations.

We think that posing the approximation of the intervention as an optimization problem (more particularly a semi-definite program) is rigorous and necessary for a scientific ML venue such as ICLR. We think the current iteration of the section is as optimally simple as possible, as moving all the derivations to the appendix will make the paper less scientifically rigorous.

Moreover, to help alleviating the potential difficulty of some readers, in Section 3, we have included visual aids, such as Figure 2, which illustrates the effect of headwise intervention,

The presentations of the experimental results in table 1, table 2 and table 3 are redundant. The information can be summarized in one table instead.

The three tables describe three different tasks, separated by Subsections 4.2.1, 4.2.2, and 4.3. Each table showcases the effectiveness of RADIANT in a different aspect. Table 1 benchmarks the Llama 7-B model, which has always been used by previous works in the field, increasing our reliability and making a fair comparison to previous works. Table 2 benchmarks on the Llma3 8-B showcase the superiority of RADIANT on the newest model, unfolding broad applications of RADIANT on up-to-date models. Table 3 reported the results on a bigger base model, which shows that the effectiveness of RADIANT remains when scaling up the model size.

Dear Reviewer KLQW,

We have provided an extensive rebuttal that in our opinion has addressed your concerns. Could you at least please acknowledge that you have read it, and perhaps consider raising your evaluation of our work?

Best regards,

The authors.

Moreover, there can be more comparisons between experiments on different LLMs. For example, how does the performance of RADIANT change when the model size increases, and how do other baselines behave?

We appreciate your suggestion on improving the depth of our paper. To accommodate your concern, we add another analysis on how RADIANT and other baselines' performance changes when gradually increasing the model size. Particularly, the table shows the baselines' Truth(%) when increasing the model size.

The table shows that RADIANT’s performance consistently improves as model size increases, demonstrating its scalability and ability to make the most of larger models. For example, its Truth score jumps from 38.73% with GPT-2 Large (700M) to an impressive 74.20% with Llama2-chat-13B (13B). This upward trend highlights how well RADIANT adapts to the growing capacity of bigger models. In comparison, other baselines show mixed results. ITI improves slightly with larger models but remains behind RADIANT, while FSP performs decently on smaller models but doesn’t scale as effectively, sometimes even plateauing. Combining methods like FSP + ITI brings some improvement, but it still doesn’t match RADIANT’s performance. Interestingly, combining FSP with RADIANT consistently boosts results. For instance, with Llama2-chat-13B, FSP + RADIANT achieves 67.75%.. In summary, RADIANT stands out by scaling effectively with model size and outperforming other baselines across the board, especially as models grow larger.

However, I believe that RADIANT is not targeted at a specific model, so it should be valid for at least 2-3 different LLMs to verify that this method works.

Thank you for your constructive comment. Following your suggestion, we study the performance of finetuning-free techniques, including ITI, RADIANT, and FSP, on Gemma-2B and GPT-2 Large, which serve as alternative base models to the Llama model family. Table 3 below shows that RADIANT using few-shot prompting outperforms other methods by a large gap. Notably, FSP + RADIANT is superior to FSP + ITI in terms of both True * Info and True and MC1 scores. Concurrently, RADIANT, implemented separately, outperforms ITI and FSP in terms of True * Info and True scores while only slightly behind in MC1 and MC2.

Table 3: Quantitative results of different intervention methods on TruthfulQA dataset, across different Language Models Parameters of RADIANT: $\alpha = 2.5, \Gamma = 15$.

a. Gemma-2B

b. GPT-2 Large

I think experiments provided in this paper lacks depth. I really appreciate the substantial experimental results provided by authors. However, there lacks further analysis and discussions.

Thanks for your comment. We are sorry you are dissatisfied with our discussions and analysis. However, could you kindly be more specific on which further analysis and discussions we need to provide? The discussion period will last for another three days, and we are confident we can meet your further requests.

We note that, regarding your two previous concerns, we have already added them to our revised PDF version to deepen the discussion and analysis:

• Particularly, we incorporate our answers to your concern about our motivation, intuition, and what makes our framework stand out from other baselines in Section 3.
• Additional discussions on empirical results are included in the Appendix and Section 4. With the current version, we believe that our experimental analysis and discussions are extensive and cover almost all aspects. Our proposal performs better than various state-of-the-art baselines in the QA tasks across all model sizes that our limited resources allow us to test. Even in the toxicity mitigation task, our method is also promising compared to state-of-the-art methods like DExperts and MiMiC.
• We also show that our method not only works for the Llma model family but also be effective for other base models like GPT and Gemma. In the scenarios, base models that have been finetuned before like Vicuna, Alpaca, and Lofit ; our proposal also performs well, showing its broad applicability. In terms of a thorough analysis of the behavior of RADIANT, we write a special part in Appendix A to delve deeper into it.
• For each part, we all commented and analyzed the trend carefully.

With all the arguments, we kindly ask the reviewer to reconsider their scores.

Why uniformly weighting divergences in mean and variance in objective (3). Why this specific way of measuring the divergence between distributions? (Though I have a more general question: why define this objective in terms of probabilities)

We define this objective in terms of probabilities as an inspiration from the Bures-Wasserstein distance, an optimal transport problem that finds an optimal matching between two Gaussian distributions. Indeed, we re-emphasize that from the optimization objective (3), one can have the freedom to choose different types of divergence, different from the one we proposed in Theorem 1. This showcases the flexibility of our framework in solving the intervention steps.

Section 3 is a bit dense, and giving some intuitions up-front would make it a more enjoyable read.

Thank you for this constructive comment on our presentation. In our revised version, we added a paragraph outlining our intuition at the beginning of Section 3. We also revised the three beginning paragraphs of Section 3 to make it less notation-dense.

Reference

[1] https://huggingface.co/openai-community/gpt2-large

[2] Singh, Shashwat, et al. "Representation Surgery: Theory and Practice of Affine Steering." Forty-first International Conference on Machine Learning.

[3] Liu, Alisa, et al. "DExperts: Decoding-time controlled text generation with experts and anti-experts." arXiv preprint arXiv:2105.03023 (2021).

[4] Gururangan, Suchin, et al. "Don't stop pretraining: Adapt language models to domains and tasks." arXiv preprint arXiv:2004.10964 (2020).

[5] Yang, Zonghan, et al. "Unified detoxifying and debiasing in language generation via inference-time adaptive optimization." arXiv preprint arXiv:2210.04492 (2022).

[6] Dathathri, Sumanth, et al. "Plug and play language models: A simple approach to controlled text generation." arXiv preprint arXiv:1912.02164 (2019).

[7] Pozzobon, Luiza, et al. "Goodtriever: Adaptive toxicity mitigation with retrieval-augmented models." arXiv preprint arXiv:2310.07589 (2023).

Baselines are divided into two groups: those requiring extensive fine-tuning or gradient computations during inference(e.g., DAPT, GeDI, PPLM, UDDIA, DExperts, GOODTRIEVER) and light-weight inference-time methods (e.g., MIMIC, ITI, RADIANT). While the fine-tuning group generally performs better on toxicity metrics, these methods are computationally expensive due to the necessity of fine-tuning or gradient computations during inference.

Inference-time methods like MIMIC, ITI, and RADIANT achieve comparable toxicity reduction to fine-tuning methods but are significantly more resource-efficient. Among these, RADIANT stands out by delivering the best toxicity reduction while also maintaining strong fluency and diversity. In fact, RADIANT’s fluency surpasses nearly all methods in the fine-tuning group except UDDIA, and its diversity is on par with the best (PPLM). This highlights RADIANT’s balance of efficiency and performance.

The paper includes comparisons with ITI but lacks an explicit comparison with MiMiC. On the new Toxic Comments Classification Challenge benchmark, we compared RADIANT with MiMiC. While both achieve similar performance in Tox. Prob. and diversity metrics, RADIANT significantly outperforms MiMiC in Fluency and Exp. Max. Tox. This highlights RADIANT's effectiveness and superiority in the task.

MiMiC ensures that a broad class of classifiers cannot distinguish post-intervention activations from desirable ones, implicitly implying that further layers cannot see the difference. In contrast, this method optimizes protection against only specific classifiers trained in the first stage. Is this a limitation? It’s possible that a different classifier could still distinguish the activations. Could this affect the robustness of the intervention?

It is possible that a different classifier could still distinguish the activations. However, different from MiMiC, it ensures that a broad class of classifiers cannot distinguish post-intervention activations from desirable ones; in our RADIANT framework, we do not have any similar constraint or objective terms to ensure that no classifier could still distinguish the post-intervention activations and the human-labeled desired ones. Instead of that, we argue that this condition is not necessary to ensure truthfulness. This is because the empirical distributions of desired samples are approximated using a limited number of samples, while the actual distributions span a much broader space.

An intuitive example is as follows. We denoted A as our desired activations and B as activations of their paraphrase to passive expression. By common sense, we can use the classifier that distinguished activate and passive tone as a good candidate classifier for this case. But A and B still offer truthful answers.

We conclude by arguing that this is not our limitation. This is the difference between our approach to the problem and the MiMiC paper. In practice, although RADIANT does not consider this condition, the extra experimental benchmarks show that our framework still performs better than MiMiC, which ensures this condition. Our response to your questions above discusses the comparison between RADIANT and MiMiC.

We thank the reviewer for the valuable feedback and appreciation of our work. Here are our replies to your questions.

I’m less certain about the intuition behind the optimization objective itself. The part I find puzzling is the requirement to match as closely the covariance of the original 'undesirable' (non-intervened) activations with the 'corrected' activations. This objective seems to differ fundamentally from a similar regularization term in MiMiC by Singh et al. (2024), which aims to match intervened data with 'desirable' data.

Thank you for your thoughtful question. We think that there is likely a misunderstanding about this aspect of the paper. Indeed, we want to closely match the covariance of the post-intervention activations, not the original 'undesirable' ones, with the 'corrected' activations. We note that despite the two seemingly different learning objectives for the intervention, the high-level ideas of both MIMIC and our paper are the same.

In particular, our second criterion, in which we want the intervention to be effective in converting the undesirable activations to the desirable regions, is the same as the objective of MiMiC (aiming to match intervened data with empirically observed ‘desirable’ data distribution). This is reflected in the constraint of the optimization problem (3).

However, different from MiMiC, we argue that the intervention needs to match criterion (iii): minimize the magnitude of the intervention to sustain the context of the input – so that the output remains semantically correct.

I don’t quite understand why enforcing a distributional similarity between the undesirable activations and intervened steps is better than simply making the smallest step necessary for every activation (with the step size defined according to the probabilistic constraint). Could the authors clarify the rationale behind this assumption?

We argue that solving the optimization problem in Eq. (3) achieves exactly what the reviewer suggested. The goal is to identify an intervention that pushforwards the distribution of undesirable activations into the desirable region. However, this intervention must be small enough to avoid moving the distribution too far from its original state.

The evaluation primarily focuses on QA tasks (TrustfulQA in the main paper + two other QA datasets in appendix). Isn't the idea more broad than that?

Thank you for your thorough and constructive suggestion. We have conducted an additional experiment to verify the effectiveness of RADIANT on the toxicity mitigation task. The details of this experiment are available in Appendix 4.1, which is newly added in our revised PDF version. Here, we briefly present the experimental setting for the experiment and our results.

We use the Toxic Comments Classification Challenge dataset [1] with preprocessing from the MiMiC paper [2].
GPT-2 Large serves as the base model for our experiments in this task to have a fair comparison with previous works.

We report two metrics to measure the toxicity scores: Expected Maximum Toxicity, denoted as Exp. Max. Tox., and Toxic Completion Proportion abbreviated as Tox. Prob. We also include Fluency, which is evaluated by calculating the perplexity of the generated outputs using GPT-2 (XL) as a reference model. Moreover, we compute the diversity score assessed by examining the ratio of unique n-grams (1-gram, 2-gram, and 3-gram) to the total number of tokens in the generated text. It is worth noting that these metrics are main and common in previous papers.

We include several baselines that have the same goal of reducing the toxicity of LLMs, including MiMiC [2], DEXPERTS [3], DAPT [4], UDDIA [5], PPLM [6], GOODTRIEVER [7]. As for MIMIC, we consider two versions: Mean Matching (MM) and Mean+Covariance Matching (MCM). Both these versions are introduced in their original paper. The experimental result is presented in the below table.

We study the performance of finetuning-free techniques, including ITI, RADIANT, and FSP, on Gemma-2B and GPT-2 Large, which serve as alternative base models to the Llama model family. Table 3 below shows that RADIANT using few-shot prompting outperforms other methods by a large gap. Particularly, FSP + RADIANT enhances the True * Info score of Gemma-2B and GPT-2 Large by 25.14% and 16.16%, respectively. Notably, FSP + RADIANT is superior to FSP + ITI in terms of both True * Info and True and MC1 scores. Concurrently, RADIANT, implemented separately, outperforms ITI and FSP in terms of True * Info and True scores while only slightly behind in MC1 and MC2.

Table 3: Quantitative results of different intervention methods on TruthfulQA dataset, across different Language Models Parameters of RADIANT: $\alpha = 2.5, \Gamma = 15$.

1. Gemma-2B

b. GPT-2 Large

We benchmark the performance of RADIANT in mitigating toxicity in long-form text generation. In this task, large language models (LLMs) are required to complete an incomplete prefix of text. Typically, the prefix prompt is designed to elicit toxic content from LLMs. To ensure a fair comparison with previous works, we set up experiments following Singh et al (2024). and Pozzobon et al. (2023), as detailed below.

• Training Dataset We use the Toxic Comments Classification Challenge data, which comprises sentences and their human toxicity labels. The data preprocessing follows Singh et al., while the gathering of activations is identical to the procedure used in the QA task.

• Models Following existing works in the field, we adopt GPT-2 Large as the base model across all experiments in the toxicity mitigation task.

• Hyperparameters As mentioned in the QA task section, there are two pivotal hyperparameters in the RADIANT framework: $\alpha$ and $\Gamma = \Phi^{-1}(1-\gamma)$. These will be selected through a grid search procedure detailed in Appendix A.

• Baselines We include several baselines that aim to reduce the toxicity of LLMs, including MIMIC (Singh et al., 2024), DEXPERTS (Liu et al., 2021), DAPT (Gururangan et al., 2020), UDDIA (Yang et al., 2022), PPLM (Dathathri et al., 2019), GOODTRIEVER (Pozzobon et al., 2023). For MIMIC, we consider two versions: Mean Matching (MM) and Mean+Covariance Matching (MCM), as introduced in their original paper.

• Metrics We assess the performance of the models using three key metrics: toxicity, fluency, and diversity, outlined as follows:

  • Toxicity: We use the non-toxic split of RealToxicityPrompts and utilize the evaluation framework from Liu et al. and Singh et al. For each prompt in the dataset, the models generate 25 outputs, each capped at 20 tokens in length. The parameters of the shared decoding mechanism for all algorithms are presented in Table 1. These outputs are analyzed using the Perspective API, which estimates the likelihood that a human would perceive the text as toxic. Two metrics are derived:
    • Expected Maximum Toxicity (denoted as Exp. Max. Tox.): For every prompt, we identify the output with the highest toxicity score and compute the average of these maximum scores across all prompts.
    • Toxic Completion Proportion (abbreviated as Tox. Prob.): This metric tracks the fraction of outputs considered toxic, defined as having a score above 0.5 based on the Perspective API's threshold.
  • Fluency: We evaluate fluency by calculating the perplexity of the generated outputs using GPT-2 (XL) as a reference model. Lower perplexity values indicate more coherent and grammatically fluent text.
  • Diversity: We assess diversity by examining the ratio of unique n-grams (1-gram, 2-gram, and 3-gram) to the total number of tokens in the generated text. This metric captures variation in outputs, with higher values indicating more diverse language use.

References:

• Singh, Shashwat, et al. (2024) "Representation Surgery: Theory and Practice of Affine Steering." Forty-first International Conference on Machine Learning.
• Luiza Pozzobon, Beyza Ermis, Patrick Lewis, and Sara Hooker. Goodtriever: Adaptive toxicity mitigation with retrieval-augmented models. arXiv preprint arXiv:2310.07589, 2023.
• Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. Dexperts: Decoding-time controlled text generation with experts and anti-experts. arXiv preprint arXiv:2105.03023, 2021.
• Suchin Gururangan, Ana Marasovi´c, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don’t stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964, 2020
• Zonghan Yang, Xiaoyuan Yi, Peng Li, Yang Liu, and Xing Xie. Unified detoxifying and debiasing in language generation via inference-time adaptive optimization. arXiv preprint arXiv:2210.04492, 2022
• Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. arXiv preprint arXiv:1912.02164, 2019.

We have added two new baselines into Section 4.2.1, which are NL-ITI (Hoscilowicz et al., 2024) and LITO (Bayat et al., 2024). We notice that our framework RADIANT still consistently returned the highest True*Info and True metric, both when running alone and combining with FSP.

Table 2. Quantitative results of different intervention methods on TruthfulQA dataset, across different Language Models. Parameters of RADIANT: $\alpha = 2.5, \Gamma = 15$.

1. Llama-7B

b) Llama3-8B

c) Llama2-chat-13B

• Results The experimental results for baselines are shown in Table 1, where all methods utilize GPT-2 Large as their base model. The result for the original model is described in the first row. We categorize baselines into two groups:

1. The first group uses an extensive fine-tuning procedure and includes DAPT, GeDI, PPLM, UDDIA, DExperts, and GOODTRIEVER.
2. The second group contains inference-time fine-tuning-free methods like MIMIC, ITI, and RADIANT.

Baselines in the first group outperform those in the second group regarding toxicity metrics; however, they require either fine-tuning or computing gradients during inference time, which can be computationally intensive. MIMIC, ITI, and RADIANT achieve comparable toxicity reduction to many algorithms in the first group while consuming significantly fewer resources. Specifically, RADIANT outperforms PPLM and is equally competitive with DAPT. Notably, within the second group, RADIANT provides superior toxicity reduction compared to ITI and MIMIC while maintaining better fluency and diversity in generated sentences. The fluency of RADIANT is favored over nearly all algorithms in the first group except for UDDIA. Additionally, its diversity metric surpasses that of other baselines apart from PPLM.

Table 1: Quantitative results of different intervention methods on RealToxicityPrompts dataset. Parameters of RADIANT: $\alpha = 2 .5 , \Gamma =15$.