Language Model Decoding as Direct Metrics Optimization

We thank the reviewer for the appreciation of our work!

Regarding the question raised by the reviewer:

Q1. Some evaluation metrics might contradict each other. For example, diversity and coherence are hard to achieve at the same time, as far as I know. I am wondering how the results will be if a different set of metrics is used.

1. We should clarify that our goal is not to maximize all these metrics simultaneously. Instead, our aim is to realize alignment with human texts, i.e., our method should generate texts that achieve the same level as human texts evaluated under different metrics. Compared with maximization, the goal of alignment is achievable in principle, as human texts themselves perform as a natural demonstration. And therefore, this principle is independent from what types of metrics are employed in our method.

2. Coherence and diversity do not necessarily contradict with each other, but are two important aspects in evaluating text generation models [1]. Merely optimizing towards one metric will result in highly biased models [2]. Due to the limitation of model capacity and expressiveness, it is challenging for existing models to maximize both coherence and diversity, so that it might leave us the impression that these two metrics contradict each other. However, the goal of aligning to the coherence and diversity of human texts is achievable (mentioned by statement 1). As demonstrated in our empirical evaluations, our method is able to achieve human-level coherence and diversity (Figure 2 of our original manuscript).

References:

[1] Montahaei et al., Jointly Measuring Diversity and Quality in Text Generation Models. NAACL 2019.

[2] Caccia et al., Language gans falling short. ICLR 2020.

We highly appreciate the reviewer’s timely response and the new question for us to clarify the purpose and design of our newly reported experiments!

As noted in the first point in our previous response: our optimization goal is not to maximize all these metrics, but instead align with the performance of human texts on these metrics, simultaneously. As a result, when the COH metric is optimized, it is supposed to be the closest to the COH of the human reference, which is confirmed in our experiment. Please feel free to let us know if this helps resolve your concern!

We are grateful to the reviewer for their feedback and suggestions, which will help improve our work.

Regarding the weakness and questions raised by the reviewer:

W1. Related works such as [1] are missing. It is worth discussing and comparing with these methods in the paper.

We appreciate the reviewer for pointing out this related work. And we have provided our discussion about it in the latest version of our submission in Appendix B.3.

Although the work suggested by the reviewer used evaluation metrics to improve generation, we have distinct differences from the following perspectives:

1. On the theoretical guarantees:

The reranking approach proposed in the suggested paper cannot guarantee improving the distribution of the base generation model to resemble the underlying text distribution from humans. Their objective was to optimize the final outputs towards some pre-defined metrics, but unfortunately they did not provide further theoretical analysis of the resulting performance.

While our method has the theoretical guarantee to obtain a better approximation to the text distribution from humans than the base language model. This is presented in Section 2.2 where we proved that the optimal decoding distribution has a lower perplexity on human texts than the perplexity of the base model distribution. In Section 3.5, we also empirically validated this perplexity improvement.

2. On the optimization objectives:

The reranking approach proposed in the suggested paper focused on improving generation quality, where they tune the weights of different metrics to maximize a given reference-based metric.

While we consider alignment with human texts in various aspects simultaneously, including but not limited to the quality of text. We believe that alignment with a wide range of aspects, such as coherence, diversity, and repetition, is necessary to achieve human-like generation.

The weights of different metrics are determined by exactly matching the level of aspects of human texts, instead of simply maximizing any particular metric. Therefore, we do not rely on any pre-defined reference-based metrics, as golden references with high quality and desired coverage are not always available in general text generation tasks.

W2. The paper only conducts experiments in two open-ended text generation tasks, where both automatic and human evaluations are hard. On the other hand, evaluations on other text generation tasks such as machine translation and text summarization are more accurate and can better indicate if their method is indeed effective.

We follow recent works on decoding methods [1,2] to adopt open-ended generation tasks in our experiment given that they encompass a broader range of linguistic complexity and diversity. These tasks also set challenges for current modern language models to align with humans with respect to a wide range of aspects. Moreover, the open-ended setting is more conformed with the trending and natural use case of large language models which is less constrained and more flexible compared with closed-ended generation tasks such as text summarization.

Nevertheless, to demonstrate the universability of our method, we conducted a new experiment on the standard summarization benchmark CNN/DailyMail using an off-the-shelf summarization model "pegasus-cnn_dailymail" from the official release in Appendix I.

Compared with the baseline methods, our method achieves better performance in alignment with various aspects like repetition, coherence, diversity, and information content (better when the score is closer to that of the reference), and ROUGE scores (better when the score is higher).

(continue in part 2)

We appreciate the comprehensive feedback and helpful recommendations provided by the reviewer that will enhance the quality of our work.

Regarding the weaknesses and questions raised by the reviewer:

W1. The mathematical motivations given by this paper are quite weakly supported/explained. For example, they motivate their use of the reverse KL for choosing the parameters of q by stating “KL(q || pθ) restricts the decoding distribution q to deviate minimally from the LM distribution pθ”, but the same argument could be made for the forward variant of the divergence.

Our motivation for using reverse KL instead of forward KL stems from the demand of high-quality texts when decoding from a language model. We should note that as KL divergency is not symmetric, forward KL and reverse KL measure the differences between two distributions quite differently (Section 1.6.1 in [1]). And we have explained this in Section 2.1 of our original submission. To make it clearer, we further elaborate the differences between these two types of KL divergency in details below and connect them with our motivation. We also added the discussions to Appendix C of our updated manuscript.

Minimizing the reverse KL, $KL(q || p_\theta)$ is known to encourage zero-forcing behavior [2], as it forces $q(x)=0$ when $p_\theta(x)=0$. This restricts $q$ to be mode-seeking and only explores the modes of the target distribution $p_\theta$ which contains samples with high likelihood under the ground-truth model's distribution and thus ignores the outliers.

Whereas, minimizing the forward KL, $KL(p_\theta || q)$ encourages zero-avoiding behavior, i.e., it avoids $q(x)=0$ when $p_\theta(x)>0$. This leads $q$ to be mean-seeking which spreads its support to cover all the non-zero density regions of $p_\theta$. As the distribution $p_\theta$ of language models is known to have unreliable long tails [3], the resulted $q$ can further overestimate those tails to produce low-quality samples [4].

We also added an intuitive visualization to illustrate the different behaviors of reverse and forward KL in Figure 4 of Appendix C of our updated manuscript.

Overall, comparing to the forward KL, minimizing the reverse KL leads to a more focused distribution and ignores the long tail in the given language model distribution, which is more suitable for our decoding scenario that demands generation quality.

W2. The method requires sampling in order to approximate the new parametric distribution (via the WIS algorithm). The runtime of this algorithm, and hence the additional computational complexity incurred by this decoding method, is not discussed. Further, there is an additional sampling + renormalization step that must happen during decoding (algorithm 2). The lack of discussion of these costs is especially pertinent given that the authors point to the tuning required by other decoding methods as one of their downsides “But unlike those previous decoding methods that require heavy manual hyper-parameter tuning for trade-off among different metrics”

We have updated the runtime analysis of Algorithm 1 (the WIS algorithm) in Appendix E of our updated manuscript, so as to justify our solution's benefit over previous decoding methods that count on manual hyper-parameter tuning.

First, we have noted in Section 2.3.1 of our original submission that Algorithm 1 is only used on the development set to determine the parameters $\\{\mu_i\\}^K_{i=1}$ defined in the energy function (prior to the decoding phase). Once settled, these parameters are fixed in the inference stage for decoding on the test set. As a result, Algorithm 1 does not incur additional computation at the decoding stage.

Second, the computational cost of Algorithm 1 is much lower than the manual hyper-parameter tuning methods employed in most of our decoding baselines. Specifically, the computational cost in Algorithm 1 consists of two parts.

• The first part takes one run of unconditional sampling from the base language model (line 2 of Algorithm 1). We typically set the number of samples to be the same as the development set, so that this cost is equal to one time sampling on the development set.

• The second part estimates the parameters $\\{\mu_i\\}^K_{i=1}$ via iterative gradient descent (line 3-6 of Algorithm 1). As the number of parameters is small (equals to the number of constraints which is usually in the scale of dozens in practice), and the metric scores $\\{f_i(x)\\}^K_{i=1}$ can be precomputed. The total computational overhead here is very small. In our experiments, it takes less than 1 minute to perform thousands of gradient descent steps, which can be ignored comparing to the typical cost consuming when sampling from a language model.

(continue in part 2)

We thank the reviewer for the detailed feedback and valuable suggestions to further improve our work!

Regarding the weakness and questions raised by the reviewer:

Please provide complexity analysis of the sampling approach and compare to competing approaches (Greedy, Top-k, CD, CS, etc). While the experimental results are strong it is important to compare the runtime of this method to determine practical efficacy given the authors claim it is "efficient" in the conclusion.

First of all, we have to clarify that we did not emphasize efficiency as an advantage of our method, although the decoding latency of our method is still on par with vanilla greedy decoding. In conclusion and future work discussions (Section 4), we only suggested finding more efficient sampling methods for our decoding framework as an important future work, rather than claiming we have already done so.

Second, in our submitted manuscript, we reported the decoding latency of our method with different $M$ in Figure 3 (the right-most subfigure). The inference latency is relative to greedy decoding. In our main experiments, we set $M=25$ and the latency of Algorithm 2 was only 1.35 times of that from greedy decoding, while our method achieved significantly better performance in overall metric alignment than greedy decoding and outperformed it in Mauve score by nearly 30 points.

We have updated the computational complexity and practical runtime analysis of different sampling approaches in Appendix F. To summarize, the main complexity of our method comes from the candidate sampling step (line 1-2 in Algorithm 2), which increases the computational complexity of vanilla greedy decoding by a factor of $M$ (the number of candidates). However, as this step can be done in parallel, which is highly optimized by modern GPU hardware, the actual inference latency grows much slower in terms of $M$. According to the right-most subfigure of Figure 3, the latency is still below 2 when $M$ is set to 100.

Analysis on the convergence of $\mu$ in Algorithm 1 and some sensitivity analysis to initialization would be helpful for practitioners.

We have updated the analysis about the convergence and sensitivity of $\mu$ in Appendix G. We consider three common initializations of $\mu$: (1) All-zero initialization. (2) Random initialization from a standard normal distribution. (3) Random initialization from a uniform distribution $U[0,1)$.

We took 5 runs with different random seeds in these three settings, and found that Algorithm 1 is quite stable across all these different initializations. To analyze the convergence of $\mu$, we plot the optimization trajectory of $\mu_i$ with different initializations in Figure 5 of Appendix G. The results suggest that the optimization of $\mu$ is quite stable under different initializations and they almost converge to the same optimal value, which demonstrates the practicality of Algorithm 1.

Given that different metrics perform stronger for different # of candidates $M$ and $\tau$, it would be helpful, in terms of robustness, to include some experimental results which show performance on different metrics if one optimizes for a single metric (i.e. coherence).

We appreciate the reviewer's suggestion. Firstly, we note that our choice of $M$ and $\tau$ is already able to achieve strong performance in universal alignment on different metrics than existing baselines in the main results (Table 1).

To demonstrate the robustness of the alignment results regarding to the choice of $M$ and $\tau$, we followed the reviewer's suggestion to include additional experiments in Appendix H of our updated manuscript, where we studied the performance of different metrics when one optimizes against a single metric. The results are presented in Table 5 of Appendix H. It shows that the performance on different metrics has a very small variance even when we optimize for a single metric, which indicates the robustness of our method.