Towards Minimal Targeted Updates of Language Models with Targeted Negative Training

Thank you for your review. We have responded to your concerns below:

1. [The experiments could be greatly strengthened. The authors study a single model T5 (220M). No purely autoregressive model is studied. Only two tasks are examined. This lack of breadth makes it hard for readers to access the generality of the claims in the paper. I believe this paper would benefit greatly from more experiments and ablations.]

Thank you for the feedback. First, we would like to emphasize that in order to generate Figure 2, we ran 7 * 9 = 64 finetuning jobs on two very different generation tasks (hallucination in summarization and toxicity in response generation), for 126 jobs in total. Figure 2 shows that TNT consistently outperforms or at least matches baseline methods across all rates of reduction in both tasks, demonstrating that its benefit is broad and robust to different definitions of unwanted. We’ve additionally added a set of ablations comparing methods on external data rather than model generations; TNT still outperforms baseline the results, illustrating that the proposed losses are better even under a different data context, and the main results are better than the ablations, confirming our hypothesis that preferencing more common token conditionals yields a more minimal update. Due to the extensive nature of each experiment on just one model and task, we were not able to complete such experiments for larger models (magnitudes more resource-intensive) or additional tasks (requires appropriate dataset), but we hope the mathematical analysis and updated experiments convinces the reviewer that there is merit in the proposed approach.

2. [There are very few comparisons to previous work, which in many cases tackle the same tasks presented in the experiments, e.g. avoiding toxic generations has been studied by [1,2]]

We chose as baselines other finetuning-based methods that take into account token-level information for learning. The works the reviewer mentioned do not fall into this setting, but we have instead included them as background or related work:

• Quark finetunes an existing language model on its generations by prepending inputs with sequence-level rewards and regularizing with a KL penalty on the token-level conditional distributions. Then, during inference time one conditions on high reward to sample from the model. TNT differs in that it utilizes token-level annotations and directly specifies the output token probabilities rather than conditioning the inputs on the rewards.
• Task Arithmetic subtracts the weights of pretrained and finetuned models to yield task vectors which are composed together to add or negate behaviors. As these task vectors are a function of vanilla finetuning, e.g. via filtering the dataset of toxic language, this approach, like finetuning, will not yield minimal targeted updates.

Do you have any additional questions or concerns that you would like to share? If not, would you kindly consider raising your score?

Thank you for your review! Addressing each comment below:

1. [The authors are missing connections to existing methods. For example, PPO would converge to the same closed-form presented in Eq. 2 when using the positivity/negativity classification as a reward function; this method would also avoid the drawbacks mentioned in the related work for inference time procedures.]

Thank you for bringing this up. We have added to the related work that, given a positivity/negativity classifier as a reward function, the objective PPO seeks to maximize (equivalent to minimizing reverse KL) has an optimal solution that is a soft version of the result TNT targets: namely, $p^*_{PPO}(x|c) = p_{o}(x|c)\exp(r(x)/\beta)$ while $p^*_{TNT}(x|c) = p_o(x|c)\mathbf{1}[r(x)]$, where the former is approximately equal to the latter when $\beta$ is small.

2. [It would be cooler to see benchmarks where the reward model may still be automated but captures some global property of the sentence that requires a learning-based technique such as yours to solve.]

We agree that the wordlist setup does not demonstrate the full practical utility of TNT and have replaced this experiment with one where we train a token-level classifier to identify toxic spans. In this updated experiment, the benefit of TNT over baselines is even more pronounced, with TNT methods overall (and each of TNRF, TNRR, and TNRLL individually) being strictly better than baselines across all ranges of similarity and reduction (see Figures 2(c) and (d)).

3. [(paraphrased for brevity) For the toxicity benchmark, learning is rather excessive for a base rate of 1.6% (e.g., natural baseline is to regenerate anytime output is toxic). Having a benchmark that requires a larger change will be a true test of this problem.]

We agree, and with our new broader definition of toxicity defined by a token-level classifier trained on human annotations, the level of toxicity of the original model is 8%. Moreover, this 8% rate is not constant across inputs; for instance, we see that for examples where the input contains toxicity (based on the same classifier definition), the model generations have toxic language 23.3% of the time. Thus in our updated experiment we have a setting where regeneration would be quite expensive for a certain subset of inputs, making finetuning a more practical alternative.

4. [I believe PPO is a more natural baseline for learning from a reward model.]

Thanks for the suggestion. We initially did not consider PPO as a baseline since it is an algorithm generally used to optimize sequence-level rewards. [1] recently introduced Fine-Grained RLHF which could be used to consider token-level rewards, but we did not have the chance to implement this baseline given time constraints (It is also worth noting that this baseline is much more complicated than the proposed approach, requiring sampling from the model during training as well as keeping track of four separate models, including a separate value model that is trained simultaneously to the policy model). Our experiments do hint at the opportunity for considering alternatives beyond such a baseline, though. Namely, whereas the objective considered by the PPO algorithm is equivalent to minimizing the reverse KL divergence between the current model $p_\theta(x|c)$ and the target distribution $p^*(x|c)$, regardless of whether we consider sequence- or token-level rewards, TNT encompasses a suite of objectives which consider minimizing not just the reverse KL between token-level conditional distributions but also other combinations of divergences, and our experiments show that other divergence combinations may be preferable in certain scenarios, even when the algorithm is fixed regardless of objective; in particular, at lower levels of reduction, TNT losses that use forward KL (i.e., TNRF and TNFF) tend to outperform TNRR at maintaining similarity with the original model generations.

5. [At the top of page 7, the paper mentions that all experiments were done with greedy decoding. Does that mean fine-tuning was also based on greedy generations? If only 1.6% of model generations were toxic, would the model get any gradient signal for 98.4% of the generations?]

Thanks for the question. The fine-tuning was also based on the greedy generations; however, the token-level KL divergence terms are computed using the full token-level output distributions, such that any deviation of any probability mass between the current model and the desired target output distributions will yield gradient signal. We have also confirmed empirically that the model gets gradient signal throughout training (via logging the gradient norm).

Thank you for your review. Thank you especially for highlighting a key strength of our approach in its ability to enable iterative model updates: “TNT allows for iterative updates to language models without needing all negative tokens to be specified upfront. This flexibility is advantageous for practical applications where updates may be continuous and ongoing.”

We first wish to clarify and address two points the reviewer made in the paper summary:

1. [The method operates by minimizing reverse KL-divergence, ensuring the updated model does not deviate significantly from its initial training.]

The target distributions of interest are the optimal distribution in terms of reverse KL divergence with the original under the constraint that negative tokens are avoided. Then, the suite of TNT algorithms optimize for these target distributions via different combinations of forward and reverse KL divergence (depending on whether the token conditional requires a negative update or not). This is different from “the method operates by minimizing reverse KL-divergence.”

2. [It requires access to the original model and detailed token-level annotations, presenting potential practical challenges.]

A finetuning method like TNT is specifically meant for the maintainers of a given language model (i.e., to iteratively update their model), so we do not foresee a problem in the method requiring access to the original model. We agree that the fact that the training algorithms require the output probabilities of the original model could present some issues around the amount of memory needed during training, but there are strategies to reduce memory consumption by trading off with compute time, e.g. relevant output probabilities distributions could be precomputed.

As for token-level annotations, we address the practicality of this information in Section 2.1. To summarize, while we agree that in some cases token-level annotations can be more expensive to collect than sequence level ones (e.g. it is easier to identify that a piece of code fails than to find the source of a bug), in many cases the two involve comparable effort, e.g. labeling an overall sequence with “has hallucination” or “has offensive language” generally requires identifying the hallucination or offensive language itself. In such a setting, we expect a method that takes this more fine-grained information into account to be more sample-efficient (a recent example being fine-grained RLHF vs sequence-level RLHF).

Thank you for your review. We address each of your points below:

1. [For the different proposed models, without an equivalent table similar to Table 1, it is hard to understand the effectiveness of the approach. Would it be possible to add rows for TNFF, TNRR and TNRF in Table 1?]

Thank you for the suggestion; we have updated Table 1 to include all TNT methods. We’d also like to clarify that Figure 2 (rather than Table 1) is meant to encapsulate the main result, as it considers all methods across all alpha values in one plot; Table 1 on the other hand only looks at a single fixed alpha value and is instead meant to focus on how changing the loss on negative tokens alone can reduce disfluencies. This is the reason why the original Table 1 focused on TNT methods that share the same loss as baselines on non-negative tokens (i.e., TNFLL and TNRLL). We’ve updated the text to draw attention to Figure 2 as the main result.

2. [From Table 1 TNFLL hallucination rate is substantially higher (than baselines), while TNRLL BLEU score is much lower (than baselines). Given this observation, I am hesitant to believe that the proposed approach is actually substantially better than the baseline approach.]

First, we wish to reiterate that Figure 2 gives the main result, that TNT methods yield a better trade-off of reducing unwanted behavior vs. maintaining similarity to the original. While we intended Table 1 to focus on disfluency results, based on your feedback we realize how the current presentation of Table 1 could be confusing, as we also report similarity and reduction metrics but only for a single fixed value of $\alpha$, which puts different methods at different points along their similarity vs. reduction curves. This fixed value of $\alpha$ in resulted in rows in Table 1 that are hard to compare along similarity and reduction metrics, e.g. if one row has more reduction but lower similarity than another, which is better? Figure 2 is meant to address this challenge. To make Figure 2 and Table 1 congruent, we have additionally updated Table 1 to choose the $\alpha$ that yields the best BLEU score among $\alpha$ values that achieve a given rate of reduction (75%). See the updated table for toxicity below. Now, the methods can be more easily compared across all metrics (similarity, reduction, and disfluency); for instance, one can see that, given a 75% rate of reduction is achieved, TNRF is strictly better than baselines on all metrics, all TNT methods except TNFLL yield better similarity results than baselines, and all TNT methods perform better than baselines on the number of total introduced disfluencies (sum of Repeats and Random ??).

3. [It is possible that this approach would steer the model towards not predicting certain words in certain contexts. I think this is a reasonably big limitation.]

We take the reviewer’s comment to mean that if the model is not context-aware when pushing down probability mass over certain tokens, then it is possible for the model to begin avoiding a given token even in contexts where it would be acceptable. Fortunately, this concern is not an issue for TNT because it is context-aware: all negative annotations are a function of the prefix, which means that the negative signal the model sees is of the form “X is bad when preceded by Y” rather than just “X is bad.” As the wordlist example does not showcase this context-aware property of the method since the annotations themselves do not take into account context, we replaced the wordlist experiment with one based on a token-level classifier that labels spans as toxic based on its context.

In response to your comments (reviewer in brackets), we have made the following changes:

1. Reproduced Figure 2 for other similarity metrics and updated Table 1 to include all TNT methods, to underscore the robustness of the results [DCRf]. The former shows that TNT outperforms baselines across a wide range of similarity metrics, and the latter shows that all TNT methods additionally outperform baselines in disfluencies introduced. Also added disfluency vs. rate reduction scatterplots (Figure 6) in the appendix to showcase disfluency results analogous to Table 1 for other reduction rates.
2. Replaced the current toxicity reduction experiment based on a wordlist with an experiment involving a token-level classifier for annotating toxic content, to emphasize the utility of the proposed method on a more interesting setting / definition of toxic [VP85]. In this updated task, the superiority of TNT over baselines is even more stark.
3. Added NADO to the related work and how TNT differs [fDnb]. Also included how the two can be combined in one algorithm that takes advantage of the contributions of both works.
4. Added an ablation which uses token-level annotations on external data instead of model generations, to increase the breadth of the experiments [eQid]. Under the additional experiments, TNT continues to outperform baselines.

Overall, our experiments analyze the results of over 190 finetuning runs, and we have a full page of related work. We've addressed each reviewer's specific concerns individually below.

We hope that reviewers find our response and updates satisfactory and are willing to consider raising their scores. Thank you!