Controlled Text Generation via Language Model Arithmetic

We thank the reviewer for their insightful questions, which we address below and are encouraged to hear that they find our work to be inventive and interesting.

Q: Does model arithmetic provide a more intuitive definition of magnitude of effect than prior work?

Please see Q4 of our main reply.

Q: How did you select the weights for model arithmetic in Table 5?

We selected these weights by gradually increasing the biasing strength until we found a degradation in the perplexity of the results. In an initial version of model arithmetic, we denoted these formulas using the equivalent notation $5M - 4M_\text{toxic}$, $25M - 24M_\text{toxic}$ and $100M - 99M_\text{toxic}$. We thus increased the first term from 5 to 25 to 100 because we noticed that increasing the strength remained fluent.

Q: Does toxicity reduction require the full complexity behind model arithmetic?

While toxicity reduction does not require the full depth in which we introduced model arithmetic, it can make use of all the parts we introduced. In particular it allows us to showcase that our newly derived operator, union, empirically outperforms prior work and further that model arithmetic allows for combining different methods, leading to even better results. Note that subtraction expresses the baselines, PreAdd, which we clearly outperform.

Q: Where can the full feature set of model arithmetic be best utilized to improve over prior work?

We use the full feature set of model arithmetic in several places to significantly outperform previous work. First, in Section 5.1, while the union operator by itself suffices to outperform existing work, the ability of model arithmetic to combine the union operator with a classifier allows us to widen the gap even further. The new sentiment control task presented in Appendix F displays this utility even further, as the combination widens the gap by a much bigger margin, especially as evaluated by GPT-4. This shows that the feature set of model arithmetic can be used to improve results, even on the simple tasks that can be expressed using prior work. Secondly, Section 5.2 discusses the application of conversational agents, with fine grained control over specific attributes. The examples discussed there cannot be expressed using prior CTG works because of the added features of model arithmetic. Since we show that model arithmetic indeed has fine grained control over the attributes at almost no cost in perplexity, it essentially solves this specific task and its improvement over prior work lies in the fact that we can express this.

Q: Do you have other examples of how model arithmetic might intuitively be used?

Model arithmetic can intuitively be used in several areas. Firstly, as shown in the example in Table 3, model arithmetic can be used to avoid detection of AI generated text, by biasing away from the classifier. This can in turn be used to train better detectors, since we can use this ability to generate adversarial examples to the current detector.

Furthermore, ensembling models or prompts to achieve better performance can be achieved by using linear operators and the union/intersection operators. Classifier-Free Guidance (Sanchez et al, 2023), a method we subsume, for example shows that performance of language models can be improved by biasing away from models that do not get any context (e.g. by omitting the input question).

Finally, by using the fact that one can simply optimize the constants over a certain corpus, model arithmetic can characterize texts. This makes it possible to identify whether a newspaper is politically more left- or right-leaning, or to replicate personal styles by analyzing personal messages.

We hope to have been able to address all the reviewers’ concerns, are happy to answer any follow-up questions they might have, and are looking forward to their reply.

We thank the reviewer for their insightful questions, which we address below and are encouraged to hear that they find our work to be clearly motivated and appreciate our results on toxicity reduction.

Q: Can you compare to more prior work in Table 5?

Please see Q3 of our main reply.

Q: Can you include human evaluation for the experiments in Table 5?

We included an evaluation by GPT-4 for both the toxicity reduction (see Table 6, Section 5.1) and sentiment control (see Table 10 and 12, Appendix F) tasks as a surrogate for human evaluation. We find that GPT-4 confirms our results and prefers our method over prior work. For further details, please see Q1 and Q2 of our main reply.

Q: Can you evaluate how well model arithmetic works for smaller models (e.g., GPT2, GPT2-large)?

We extended the toxicity results with results for the GPT-2 model family in Appendix I.1. Interestingly, we find that Fudge is more powerful for the smallest model GPT-2. This is caused by the smaller capacity of GPT-2 (with only 124M parameters, compared to the 355M parameters of the classifier) which makes it harder for a prompt-based method to generate non-toxic samples. For the larger models in the model family, our union operator performs better than prior work. Furthermore, we can leverage the flexibility of model arithmetic by applying a combination of a classifier and the union operator. This formula doubles the toxicity reduction with respect to previous work for the entire GPT-2 model family, thereby validating the results that model arithmetic is more effective at toxicity reduction.

Q: How do you think model averaging via output distributions could compare with weight averaging of the model parameters?

Weight averaging of model parameters, e.g., with task vectors (Ilharco et al., 2023), can be useful to create new model weights that exhibit the behavior and attributes of all used models. It is therefore similar to model arithmetic, but suffers from a couple drawbacks. First, it requires that the LLM is finetuned several times, which requires both data and computational power. Furthermore, it makes the assumption that the weights of the model behave very similarly to vectors, and that performance increases linearly as one goes from the starting weights to the fine-tuned weights. Finally, it requires that all models come from the same architecture and does not allow for the inclusion of classifiers. On the other hand, averaging the weights ensures that model inference speed does not increase, whereas for model arithmetic it does increase even with the proposed generalization of speculative sampling.

Q: How straight forward and performant would it be to swap in different models for each part here and capture the output distribution, if the vocabulary was shared?

Swapping different models is very easy, but the exact performance would depend on the use case.

A formula can use a mixture of several models under the mentioned condition that the vocabulary is shared. The current implementation is built on top of the popular transformers Python package from HuggingFace (Wolf et al., 2020) allowing us to use all models from its library and making the operation of swapping models as simple as changing a string from one name to another.

How performant the resulting formulas are depends on the underlying task and the models themselves. For example, we see that the same formula works very well for all tested models in Section 5.1 and Appendix F. In Appendix I.1, when applying the formulas on the smaller and less performant model family, the performance of the formulas is a bit different, with the Fudge method now being almost as good as our union operator.

We hope to have been able to address all the reviewers’ concerns, are happy to answer any follow-up questions they might have, and are looking forward to their reply.

We thank the reviewer for their insightful questions, which we address below and are encouraged to hear that they find model arithmetic to be a principled and theoretically founded CTG framework.

Q: How does model arithmetic differ from PreAdd?

Model Arithmetic improves upon PreAdd (Pei et al., 2023) in several significant ways: First, PreAdd only allows the use of 2 different LLMs with a negative coefficient between them. This restriction is a consequence of the limited theoretical framework which model arithmetic does provide, giving us the correct normalization and allowing us to compose more than 2 LLMs with positive and negative coefficients. Second, our method is able to use classifiers and the union operator. These expand the possible applications of model arithmetic by allowing attributes that cannot be expressed in natural language and provides a stronger biasing method. Specifically, the union operator has much better results than PreAdd on the task of toxicity reduction (see Section 5.1) and sentiment control (see Appendix F). Finally, our extension of speculative sampling allows us to increase the speed of inference and can also be applied on top of PreAdd.

Q: Can you run prior work for more parameters, such that the results in Table 5 become unbiased?

We extended the amount of parameters on which previous methods were evaluated and included updated values in Table 5. While we tried several parameters before, we now more thoroughly investigated the possible parameters and found that the parameters provided for SelfDebias and Fudge were already optimal, while the parameter for PreAdd could be increased slightly without sacrificing the perplexity of any model. We note that this does not change our conclusions and that our method still outperforms PreAdd.

Q: Can you compare to more prior work in Table 5?

Please see Q4 of our main reply.

Q: Can you broaden the scope of the evaluation further to tasks found in prior CTG work?

Please see Q2 of our main reply.

We hope to have been able to address all the reviewers’ concerns, are happy to answer any follow-up questions they might have, and are looking forward to their reply.

I increased the rating to 6 because, during the discussion period, the authors' answers and their revised version mitigated some of my concerns:

• The toxicity reduction comparison in Table 5 is limited, omitting works like Liu et al. 2021, despite its relevance mentioned on page 16. (In the discussion period, the authors pointed out that work of Liu et al. 2021 relies on expensive fine-tuning with training data, which is different from their zero-shot regime.)

• The evaluation is restricted to toxicity reduction, whereas other CTG studies often explore sentiment or topic control as well. It is recommended to broaden the evaluation scope to these areas as well. (In the discussion period, the revised version added an extra experiment evaluating model arithmetic on sentiment control in Appendix F, showing that their method also outperforms existing work.)

We thank the reviewer for their extensive review and insightful questions, which we address below. We are delighted to read that they find our work well-written, model arithmetic to be an elegant generalization of prior work and our experiments to be promising.

Q: Can you include GPT-4 or human evaluation?

We included an evaluation by GPT-4 validating our results for both the toxicity reduction (see Table 6, Section 5.1) and sentiment control (see Table 10 and 12, Appendix F) tasks as a surrogate for human evaluation. We find that GPT-4 confirms our results and prefers our method over prior work. For further details, please see Q1 and Q2 of our main reply.

Q: Can you further explore a more defined generation scenario, such as personalized translation?

We included an extra evaluation section in Appendix F on sentiment control, a task commonly found in prior work (Pei et al. 2023, Liu et al. 2021, Krause et al. 2020). We find that model arithmetic significantly outperforms prior approaches for this task, as discussed in more detail in Q2 of our main reply.

Q: Can you provide the perplexity graphs for the results in Section 5.2 to ensure that results remain sensible?

We included plots for the perplexity of the formulas in Section 5.2 in Appendix I.2. As expected, perplexity smoothly changes over different parameter values. Further, perplexity overall remains under the reasonable value of $6.2$ compared to $4.8$ by the highest prompted model.

Manual inspection of the data generated by the extreme points shows that the generated text is still fluent and the unprompted model $M$ associates a slightly higher perplexity to the text due to the high presence of an uncommon attribute that would normally not appear in the generated text by $M$. The only exception here is the educational topic, which uses a classifier with a very high weight ($12.0$) for which we found that a word sometimes gets replaced by a nonsensical one. We now also include several randomly selected examples of texts generated by the model under these high parameters in Appendix I.2. These examples show the various characters produced by the formulas and show that the generated output is still fluent.

Q: Is it possible that the proposed technique is brittle to the specific language model arithmetic expressions used?

We never encountered any brittle behavior while evaluating and testing on a wide variety of expressions, both in the examples presented in Figure 1, Table 2, Table 3, Table 4 and Table 17 and in the various expressions presented in Sections 5.1, 5.2, 5.3 and Appendix F. Only for the educational attribute in Section 5.2 we found a decreased fluency when using the classifier at very high strengths. We specifically evaluated the expressions from Sections 5.1, 5.2 and Appendix F for perplexity and attribute presence and always found that the formula acts as expected. Furthermore, manual inspection of data generated in all sections shows that generated results are still fluent and do not showcase any weird artifacts with the exception of the slight decrease in fluency for the classifier in Section 5.2.

Q: How much tweaking do the model arithmetic expressions require?

As discussed in Q4 of our main reply, we rely on Theorem 1 and the interpretation of coefficients as relative strengths to construct the formulas for all examples in the paper. This allowed us to easily choose the right ballpark for the examples shown in the paper without requiring any further tweaking.

Q: Does model arithmetic just shift the complexity of finding the right prompt to finding the right expression?

As noted in the previous answer, finding the right expression is relatively simple. More importantly, we note that formulas are more powerful than prompts and therefore do not just shift the complexity to finding the right expression, as shown by our results on toxicity reduction (see Section 5.1) and sentiment control (see Appendix F), and by examples showcasing interesting possibilities such as overstressing a specific attribute by biasing away from the standard model output (see e.g., Table 2) and using a classifier to generate more human-like text (see e.g., Table 3).

Q: Can the order of evaluation for speculative sampling be optimized, since terms with large coefficients will likely save more time if evaluated first?

Yes, terms with higher coefficients are evaluated more often with our speculative sampling method. Terms that have a higher influence on the output distribution, have a higher probability of rejecting a specific token after their evaluation, ensuring that the optimal speculative factor, according to the procedure outlined in Appendix E.1, is lower. This implies that these terms are evaluated more often in the sampling procedure.

We thank the reviewer for their insightful questions, which we address below and are encouraged to hear that they find model arithmetic to be interesting and the experiments to show its effectiveness.

Q: Can [model arithmetic] operators and their coefficients be determined in a theoretically-backed manner?

As discussed in Q4 of our main reply, we used Theorem 1 in our paper to choose parameters and terms in a theoretically grounded manner. This allowed us to quickly and intuitively construct formulas for all the examples presented in the entire paper, such as the ones in Figure 1, Table 2, Table 3, Table 4 and Table 17, without any finetuning of the coefficients being required. We note it is possible to determine the coefficients by optimizing them via gradient descent on a small number of samples. However, in the paper we focus on applications in the zero-shot setup and we found that a correct ballpark for all our examples was sufficient without requiring any finetuning.

Q: Can you clarify the contributions and novelty of the proposed method?

Sure! Model arithmetic improves in several ways over prior work:

First, previous works (using log probabilities) lack theoretical foundations when it comes to model biasing, which often prevents the introduction of multiple attributes. In contrast, our work addresses this by framing CTG as a minimization problem, finding the correct normalization required to bias towards more than one attribute and providing a natural extension to prior work. This allows us to express a lot of existing methods, provides a theoretical foundation for their constructions and naturally extends CTG – for the first time – to more complicated and relevant examples, such as for the conversational styles discussed in Section 5.2.

Second, our theoretical framework enables us to combine classifiers and linear operators and thereby allows for a higher flexibility with better results as shown in Section 5.1. Further, we introduce the novel operator, union, which has better results on the task of toxicity reduction and sentiment control and can be used on top of the other operators we use.

Third, our use of speculative sampling allows us to use model arithmetic at only a slightly increased cost compared to normal single-model generation. This approach can be used directly, also with prior work, and therefore speeds up any method that we can express.

We hope to have been able to address all the reviewers’ concerns, are happy to answer any follow-up questions they might have, and are looking forward to their reply.