IF-Guide: Influence Function-Guided Detoxification of LLMs

We thank the reviewer for taking the time to read and evaluate our work. We also appreciate the recognition of its strengths. Below, we address the reviewer’s concerns and questions. We will incorporate this discussion into the final version of our paper for completeness.

(Weakness) IF-Guide’s Impact on Utility

We first clarify that, although fluency alone may suggest that IF-Guide has worse performance than the baselines, our method consistently achieves the best overall toxicity–fluency trade-off. To quantify this trade-off, we compare how much accuracy each method sacrifices to reduce a unit of toxicity, using the following metric:

$\frac{\text{Acc.}}{\text{mean}(\text{EMT}, \text{TP})};$

We did not use perplexity as it would disproportionately influence the denominator. We report the tradeoff using our results in Table 1. The best trade-off is bolded:

IF-Guide achieves the highest trade-off across all models, outperforming the baselines by 1.1–5.2$\times$. Given that IF-Guide introduces a small absolute change in fluency (0.93–5.18 for PPL and 0.01–0.06 for Acc.), this trade-off is favorable and highlights the effectiveness of our approach. We will include this comparison in the Appendix of the final version of our paper.

(Weakness 2) Contribution of IF-Guide’s Components

We thank the reviewer for this suggestion. To evaluate, we compare IF-Guide to three ablation baselines:

• Toxicity Filtering + Suppression: This tests whether off-the-shelf classifiers are sufficient for token selection. We use Detoxify to find the most toxic training sequences and apply our suppression objective to them.
• Naive Influence + Suppression: This tests the value of our attribution techniques over naive influence alone. We apply EK-FAC to select top-scoring tokens, without using our attribution or token selection method.
• IF-Guide + Filtering: This tests the importance of suppression-based training. We disable suppression by setting $\lambda = 0$, filtering toxic tokens instead.

The table below reports toxicity (on RTP) and fluency for each method when applied to Pythia-410M.

IF-Guide achieves 2–5.1$\times$ better toxicity reduction than the controlled baselines with similar utility (except for Detoxify; we find that the model selected far too many benign tokens, which substantially degraded utility). These results validate the effectiveness of our proposed contributions and help explain IF-Guide’s effectiveness.

(Weakness 3) Comparison to Korbak et al.

While Korbak et al. demonstrate some effectiveness in reducing toxicity, their approach is fundamentally different from ours in many ways:

• (1) Objective: Kobak et al. optimize for “general reward alignment” via human preference signals, whereas we target “safety-specific behavior” (toxicity suppression) via “data attribution”.

• (2) Method: Their approach uses preference-weighted reward modeling “at the sequence level”, while ours relies on “counterfactual influence estimate” to attribute and suppress toxic behavior “at the token level”.

• (3) Granularity: IF-Guide provides fine-grained, interpretable control over model behavior, in contrast to the broader alignment achieved through preference modeling.

These differences highlight the novelty of our contributions. While prior work has explored human preferences to reduce model harms [1, 2, 3], to our knowledge, we are the first to apply influence functions to this problem. Our approach also avoids dependence on expensive and potentially noisy human feedback [4, 5], and enables fine-grained, token-level attribution that can be directly coupled with our novel suppression objective for more effective toxicity reduction.

This positions our work as the first of its kind in this exciting and tangential direction.

[1] Ouyang et al., Training Language Models to Follow Instructions with Human Feedback, NeurIPS (2022)

[2] Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model, NeurIPS (2023)

[3] Ziegler et al., Fine-Tuning Language Models from Human Preferences, arXiv preprint (2019)

[4] Grattafiori et al., The Llama 3 Herd of Models, arXiv preprint (2024)

[5] Casper et al., Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback, TMLR (2023)

(Question 1) Why Do Standard Influence Functions Not Work?

We identify three fundamental reasons why standard influence functions are ineffective for reducing language model toxicity.

(1) Document-level scoring is ineffective. Standard influence functions only provide document-level influence scores. In our preliminary experiments, we attempted to utilize these scores for toxicity reduction; however, we found that suppressing entire documents destabilizes training, and filtering them proved ineffective (as shown in Figure 1). This led us to incorporate token-level attribution.

(2) Selected tokens lack necessary context. Even when adding token-level attribution, standard influence functions fail to capture the surrounding toxic context necessary for toxicity reduction. This is affirmed in Appendix D.4 where using a context window size of $w = 0$ reduces the effectiveness of IF-Guide by 2–4$\times$. We thus do not rely entirely on standard influence scores and add additional context.

(3) High-scoring tokens are not always influential. It is a known issue in the prior work that standard influence functions often assign high scores to irrelevant tokens [1, 2]. Our preliminary experiments revealed that many top-scoring tokens were erroneous, consisting of many non-toxic articles or repeated characters. We address this limitation with our differential attribution and document-based importance ranking.

[1] Grosse et al., Studying Large Language Model Generalization with Influence Functions, arXiv Preprint (2024)

[2] Choe et al., What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions, arXiv Preprint (2024)

(Question 2) DPO and RAD Implementations

We confirm that the reviewer’s understanding is correct: in Table 1, DPO and RAD are applied to the base model without any filtering. This design choice aligns our evaluation with prior work.

We thank the reviewer for the suggestion. As advised, we applied DPO and RAD to the Word and Toxicity-filtered Pythia-160M models. The table below reports toxicity (on RTP) and fluency for each setting:

IF-Guide remains the most effective base model. Applying DPO to the IF-Guide base model yields 3–6.5$\times$ greater reductions in EMT and TP compared to applying it to the filtering-based models; with RAD, the improvement ranges from 2.2–3.3$\times$. We will include this experiment in our revision for completeness.

(Questions 3 and 4) Impact of IF-Guide on Utility and Toxicity

We acknowledge the ambiguity in our statement in Section 3.2 and appreciate the opportunity to clarify. IF-Guide is designed to maximize toxicity reduction while minimizing utility degradation—not to eliminate utility loss entirely.

Do you try example filtering with your newly designed influence function? Is the accuracy better than word filtering or detoxify?

Yes, we show this comparison in Appendix D.4 by setting $\lambda = 0$, which applies filtering instead of suppression. Filtering achieves slightly improved utility—comparable to word and toxicity filtering—but comes at the cost of significantly worse toxicity reduction, performing 2–3$\times$ worse than suppression. This highlights the effectiveness of IF-Guide’s suppression strategy in balancing toxicity reduction with minimal utility degradation.

Does suppression improve utility compared to dropping tokens?

Suppression incurs a $\sim$5% fluency loss compared to dropping tokens, but achieves 2–3$\times$ more toxicity reduction, resulting in a substantially better overall trade-off. Please refer to our response to Weakness 1 for details.

Presentation Issues

We thank the reviewer for pointing these out. We will correct the errors in Table 5 and Figure 5, and carefully review the final version to ensure there are no remaining presentation issues.

We thank the reviewer for taking the time to read and evaluate our work. We address all the concerns and questions below and will incorporate this discussion into the final version of our paper.

(Weakness 1) Computational Complexity of the Inversion of the Hessian Matrix

We thank the reviewer for pointing out the complexity associated with influence functions. While influence functions traditionally involve inverting the Hessian matrix, which is indeed computationally expensive, IF-Guide avoids this bottleneck by using efficient approximations, EK-FAC [4].

EK-FAC approximates the Hessian using Kronecker products, which enables efficient inversion. Prior work has demonstrated the practicality of influence functions using EK-FAC even on large models with up to 50B parameters. For example, Ruis et al. [5] show that EK-FAC computation costs are comparable to performing approximately O(100k) gradient update steps.

We also note that IF-Guide is agnostic to the specific influence function approximation used. In scenarios where computational cost is a primary constraint, more lightweight alternatives such as LoGRA [6] can be adopted without modifying the overall framework.

Moreover, our influence scores are computed once and can be reused across model checkpoints or even transferred across models with similar architectures (as shown in our proxy experiments). This amortization further improves the overall efficiency of our approach.

We will include these clarifications and empirical cost details in the final version of the paper.

[4] George et al., Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis, NeurIPS (2018)

[5] Ruis et al., Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models, ICLR (2025)

[6] Choe et al., What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions, arXiv Preprint (2024)

(Weakness 2) Scalability of IF-Guide

We thank the reviewer for the suggestion to enhance the completeness of our evaluation. We note that full pretraining on a trillion-token dataset requires several hundred thousand GPU hours [1], which is infeasible at an academic scale. Instead, we maximized our available resources by pretraining billion-parameter models on a 1B-token dataset using 8$\times$ NVIDIA H100 GPUs, at an estimated cost of ~$10,000 USD.

[1] Touvron et al., Llama 2: Open Foundation and Fine-Tuned Chat Models, arXiv Preprint (2023)

Below, we provide our response along with additional evaluation conducted at this available scale.

IF-Guide scales to larger models. We demonstrate the scalability in the fine-tuning setting. We fine-tune Pythia-12B—which is 10$\times$ larger than our base models—using influence scores computed on the smaller proxy model, Pythia-1B. Below, we present fluency and toxicity metrics on the RTP benchmark across fine-tuning steps:

At just 520M fine-tuning tokens, IF-Guide reduces EMT and TP by 5.7–10.8$\times$ with negligible fluency loss; our method is effective for large-scale models even when the proxy is $>$10$\times$ smaller.

Beyond 12B parameter models, we refer to the standard influence function methods, which have been effectively scaled up to 50B parameter LLMs [2, 3]. IF-Guide adds negligible overhead to these methods, making it deployable within the same compute budgets used in these prior works. As we demonstrate, proxy models further reduce costs, further enabling scalability.

[2] Grosse et al., Studying Large Language Model Generalization with Influence Functions, arXiv Preprint (2024)

[3] Choe et al., What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions, arXiv Preprint (2024)

We will include these additional experiments and discussion in our final paper.

(Weakness 3) Overlap of Identified Toxic Tokens

This is a valuable suggestion, and we explore this by computing the overlap between toxic tokens identified by different models. Let $T_i$ denote the set of toxic tokens selected by model $i$. For model’s $i$ and $j$, we compute the Overlap Coefficient as follows:

$oc(i, j) = \frac{|T_i \cap T_j|}{\min(|T_i|, |T_j|)}.$

The table below reports the Overlap Coefficient values for all model pairs used in our evaluation.

The average overlap across all model pairs is 44.61% (≈8.9M out of 20M selected tokens). While this may seem modest at first glance, it is substantial when considering that the size of the training corpus is 1B tokens.

Overlap is higher within the same model family: Pythia model pairs average 47.22% overlap, while cross-family pairs average 42%. Overlap is also higher for similar scales: models within $\sim$500M parameters share 47.27% of selected tokens, compared to 43.61% for models differing by $>$500M.

Our results suggest that the model family is a stronger indicator of overlap than scale alone, providing a practical guideline for selecting proxy models. We will include this valuable experiment and discussion in the final version of our paper.

We thank the reviewer for taking the time to read and evaluate our work. We also appreciate the recognition of its strengths. Below, we address the reviewer’s concern about the scalability. We will incorporate this discussion into the final version of our paper.

(Weakness 1) Scalability of IF-Guide

We first emphasize that IF-Guide offers two complementary approaches, each tailored to address scalability challenges: (1) Pretraining: The method identifies toxic tokens and their linguistic context, and suppresses them during pretraining. (2) Fine-tuning: The toxic tokens identified by our method are used to guide suppression during fine-tuning. While our pretraining experiments are conducted on a 1B-token dataset, our fine-tuning evaluations are performed on models that have already been pretrained on trillion-token-scale datasets.

We note that full pretraining on a trillion-token dataset requires several hundred thousand GPU hours [1], which is infeasible at an academic scale. Instead, we maximized our available resources by pretraining billion-parameter models on a 1B-token dataset using 8$\times$ NVIDIA H100 GPUs, at an estimated cost of ~$10,000 USD.

Below, we provide our response along with additional evaluation conducted at this available scale.

[1] Touvron et al., Llama 2: Open Foundation and Fine-Tuned Chat Models, arXiv Preprint (2023)

How well will this method scale to trillion-token level pretraining?

We believe IF-Guide remains both computationally feasible and effective at larger scales:

(1) We target toxic content, not overall corpus size. We identify and suppress a small, representative set of toxic training examples (2% of the total dataset). Even in trillion-token scales, the fraction of suppressed tokens remains small, minimizing overhead and any downstream impact on model utility.

(2) The effectiveness saturates beyond 2%. In addition to (1), our evaluation in Appendix D.4 shows diminishing returns in toxicity reduction beyond 2% of the corpus. This implies that a small toxic subset can guide the suppression-based objective effectively, even in much larger training scales.

(3) Toxicity is likely to remain as a sparse signal at scale. Even in the trillion-token pre-training scenarios, toxic content will represent a small fraction, which will make our approach suitable. Our method will focus on the problematic data, not attempting exhaustive coverage. As long as IF-Guide identifies representative toxic instances, the method will propagate to reduce both explicit and implicit toxicity, as demonstrated in our evaluations.

Although the paper outlines a method for speed-up, IF computation is typically slow, especially at large scales?

We thank the reviewer for pointing out the complexity associated with influence functions. While influence functions are traditionally slow and impractical for LLMs, IF-Guide improves the scalability by using efficient approximations, EK-FAC [4].

EK-FAC approximates the Hessian using Kronecker products, which enables efficient inversion. Prior work has demonstrated the practicality of influence functions using EK-FAC even on large models with up to 50B parameters. For example, Ruis et al. [5] show that EK-FAC computation costs are comparable to performing approximately O(100k) gradient update steps.

We also note that IF-Guide is agnostic to the specific influence function approximation used. In scenarios where computational cost is a primary constraint, more lightweight alternatives such as LoGRA [6] can be adopted without modifying the overall framework. This includes retaining our batching and lower-precision speed-up techniques.

Moreover, our influence scores are computed once and can be reused across model checkpoints or even transferred across models with similar architectures (as shown in our proxy experiments). This amortization further improves the overall efficiency of our approach.

We will include these clarifications and empirical cost details in the final version of the paper.

[4] George et al., Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis, NeurIPS (2018)

[5] Ruis et al., Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models, ICLR (2025)

[6] Choe et al., What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions, arXiv Preprint (2024)

Will this method scale for larger models? Can a smaller proxy model be used for suppressed training on large models?

IF-Guide scales to larger models. We demonstrate the scalability in the fine-tuning setting. We fine-tune Pythia-12B—which is 10$\times$ larger than our base models—using influence scores computed on the smaller proxy model, Pythia-1B. Below, we present fluency and toxicity metrics on the RTP benchmark across fine-tuning steps:

At just 520M fine-tuning tokens, IF-Guide reduces EMT and TP by 5.7–10.8$\times$ with negligible fluency loss; our method is effective for large-scale models even when the proxy is $>10\times$ smaller.

Beyond 12B parameter models, we refer to the standard influence function methods, which have been effectively scaled up to 50B parameter LLMs [2, 3]. IF-Guide adds negligible overhead to these methods, making it deployable within the same compute budgets used in these prior works. As we demonstrate, proxy models further reduce costs, further enabling scalability.

[2] Grosse et al., Studying Large Language Model Generalization with Influence Functions, arXiv Preprint (2024)

[3] Choe et al., What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions, arXiv Preprint (2024)

Location of Our Limitation Section

We discuss the potential limitations of our work in Appendix B of the supplementary material.

We thank the reviewer for their time and valuable feedback. We have our answers to all the concerns and questions below, and the updates we will make to our manuscript.

(Weakness 1) Representativeness of the Toxic Query Set

We first clarify that a sample size of 10k is not a bottleneck. In Appendix D.4, we examine the effect of sample size on toxicity reduction by increasing it to 100k. Our results show that using more than 10k samples does not improve the effectiveness of our method.

We also clarify that our objective is not to remove all toxic tokens from the training data, but to identify a representative subset of toxic tokens that allows our suppression-based objective to effectively prevent the model from learning and exhibiting toxic behaviors. As noted as a strength in the review, our strong empirical results across both explicit and implicit toxicity types demonstrate the effectiveness of IF-Guide using 10k sample queries.

(Weakness 2) Comparison to Relevant Works

We reviewed the papers cited by the reviewer during the preparation of our manuscript. While they fall under the broad umbrella of “influence functions,” they address fundamentally different problems and therefore do not diminish the novelty of our work. We elaborate on the key differences below and will cite them in the Background and Related Work section for completeness.

Comparison to Wu et al. (2024)

(1) Wu et al. address a different problem.

Wu et al. propose a technique to address fitting errors in standard influence functions. Although we introduce techniques to improve attribution quality, our focus is not on addressing fitting errors. Instead, we leverage influence functions as a foundation for toxicity reduction (a different application), and subsequently introduce a substantially different methodology.

(2) Furthermore, our technical novelty is orthogonal.

Wu et al. propose to use a standard influence function. Applying their method directly to toxicity reduction would yield results comparable to those shown in Figure 1. In contrast, IF-Guide builds on top of standard influence functions, independently of their exact formulation. Although we use EK-FAC in our implementation, IF-Guide is compatible with alternative influence approximations such as LoGRA [1] or Wu et al.’s algorithm. Therefore, there is no overlap in novelty.

Comparison to Pang et al. (2025)

(1) From a research standpoint, our work offers clear novelty by demonstrating that a novel adaptation of influence functions can effectively address toxicity in language models.

Peng et al. address a fundamentally different problem: improving model performance during supervised fine-tuning (SFT), whereas our goal is to reduce toxicity. The only commonality is the use of influence functions, but the purposes and methodologies differ significantly, making direct comparison an apples-to-oranges scenario.

(2) In addition to that, there are two fundamental differences in terms of technical novelty.

Peng et al. use standard influence functions, which are not designed to specifically identify toxic tokens and cannot therefore target toxicity in the training data. Using such influence functions is the same as our result in Figure 1. Also, they remove the tokens identified by standard influence functions, whereas we suppress tokens within a context window: the identified token $t_i$, as well as its immediate neighbors $t_{i-1}$ and $t_{i+1}$ (i.e., $w=1$). In Appendix D.4 (Figure 8), we show that the approach of Peng et al. (comparable to $w=0$) only modestly reduces toxicity (from 0.58 to 0.49), whereas our method with $w=1$ reduces toxicity substantially to 0.09.

[1] Choe et al., What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions, arXiv preprint (2024).

(Weakness 3) Recognizing/Understanding Toxic Content

If a model is trained to suppress all toxic content, how does it recognize toxic content?

Although IF-Guide penalizes toxic tokens during training, models still recognize toxic content and leverage this signal to suppress harmful outputs. In Figure 5, we isolate the activation-space vector associated with toxicity and scale its magnitude. Unlike the base model—where increasing this component increases toxicity—in our model, increasing the component reduces toxicity, while decreasing it substantially increases toxicity. This indicates that IF-Guide learns a 1D direction in activation space that the model uses to recognize and suppress toxicity.

The paper does not empirically evaluate the model's ability to refuse harmful instructions.

Our evaluation demonstrates that, regardless of whether the model understands toxic content, it is unable to generate toxic outputs. Therefore, an explicit refusal mechanism is not required. Now, a possible follow-up question is: What if an adversarial user crafts queries to elicit toxic content? As shown in Figure 6, this scenario is less likely in our case. Compared to both pre-trained models (Base) and safety fine-tuned models (i.e., those trained with DPO), our method reduces the likelihood of generating toxic responses by at least 40%.

The paper does not empirically evaluate the model's ability to handle meta-level questions about toxicity.

No prior work on reducing toxicity answers this, but we take a step forward to answer the reviewer’s question to the best of our ability by prompting our models with:

I will give you a sentence. Determine if the sentence is toxic or non-toxic based on this definition: Toxic content includes insults, hate speech, threats, or harassment. If it is toxic, respond with: Toxic. If it is not toxic, respond with: Non-toxic. Sentence: '{sentence}'

To evaluate toxicity understanding, we fine-tuned two models on UltraChat-200k: the pre-trained Pythia-2.8B and its IF-Guide de-toxicified variant. We then prompted both models with 200 examples from RTP (100 toxic, 100 non-toxic) using the template above. We share the percentage of correctly classified examples below:

Both models recognize toxic content reasonably well but struggle with non-toxic cases. Given the comparable performance, IF-Guide does not appear to eliminate the model’s ability to reason about toxicity despite suppressing toxic tokens during training. We will include this experiment in the Appendix of our final paper.

(Weakness 4) Benefits of Suppression Over Filtering

We use suppression because it consistently achieves greater toxicity reduction than filtering. In Sections 4.2 and 4.4, IF-Guide outperforms filtering by 1.6–11$\times$ on explicit and implicit toxicity, and Appendix D.4 shows that using suppression over token filtering increases effectiveness by $\sim$2–3$\times$.

Our original remark on data imprecision and inefficiency was intended to critique document-level filtering and standard influence functions, not to imply that filtering is infeasible at scale. While filtering’s collateral damage may be negligible for multi-trillion token datasets, suppression remains more effective at reducing toxicity. We will revise the manuscript to make this clearer.

Expanding the Related Work

We acknowledge that our related work can benefit from including more works on toxicity reduction.

We also thank the reviewer for sharing the influence function references. While we already cite prior work applying influence functions to language models in our manuscript [2, 3], we note that they focus on different goals and do not address the attribution of toxic or harmful content. Nonetheless, we will ensure that these works are properly cited in the final version of our paper.

[2] Choe et al., What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions, arXiv Preprint (2024)

[3] Xia et al., LESS: Selecting Influential Data for Targeted Instruction Tuning, ICML (2024)

Many thanks for your detailed responses to my queries.

One query that I have still not been able to understand fully: given the related works I provided, can the authors provide a brief summary of how exactly they are going to position this work with respect to literature?

Question) if a substantial percentage of the training corpus is IF detoxified, how does it affect other downstream performance? Does it have other deleterious effects?

Q) In the response to reviewer 9ufk the authors say they can do fine-tuning for Pythia12B. At this point the model has already seen toxic tokens at pretraining. During post training, the model may not capture new toxic tokens, but how does it learn to get rid of existing toxicity? How much does toxicity reduce with regard to the base model?