LIRE: Listwise Reward Enhancement for Preference Alignment

We would like to thank the reviewer’s constructive comments and detailed review. We also thank the reviewer for pointing out some limitations. We answer all questions below and update the manuscript accordingly.

Q$_1$: "Narrow evaluation: Mainly tests dialogue and summarization. Could be extended to other LLM tasks."

A$_1$: Thanks for pointing out the limitation of our evaluation. For other LLM tasks, we consider the controlled sentiment generation task. Specifically, given a prefix (20 tokens) of a movie review in the IMDb dataset, the models are asked to produce positive reviews. We use GPT-2-large as the base model and evaluate the comparing methods. As can be seen from the results list below, LIRE achieves the best RM score, which better illustrates the generalization ability of the proposed method. We will update the manuscript and add discussion about this task in the paper.

Q$_2$: "Limited analysis: Does not perform ablation studies to isolate benefit of components."

A$_2$: Our goal is to maximize the expectation of the final rewards and this is achieved by combining two components: the probability distribution and the reward scores, which are actually under the framework of policy gradient. Specifically, we reformulate the probability distribution among the multiple model generations through softmax normalization to incorporate information beyond a single pair, but a whole list of responses. So these two components are an integral whole, and can not be separated for optimization.

Q$_3$: "Hyperparameter sensitivity is unclear."

A$_3$: The main hyperparameter is the temperature T in Equation 4, and we demonstrate the performance under different T for the dialogue task, which is also included in Appendix Table 11. Larger T makes all the samples more uniformly weighted, while smaller T shifts the probability mass to the best sample (more like SFT training), therefore, T should be within a suitable range to boost performance.

Q$_4$: "Lacks user studies: Human evaluations could provide more insight beyond automatic metrics."

A$_4$: Originally, we chose the evaluation metrics referring to other papers [1][2]. However, we do realize that this may not be a fair indicator. To ensure fairness for other methods and a more reliable comparison, aside from the automatic metrics, we also conduct human evaluation. The results are as follows. LIRE gains the highest win rate against human-written baselines as well as other comparing methods, which is in line with the metric results in general. We do realize the importance of conducting user study and we truly thank the reviewer for pointing out this crucial point. We update the manuscript accordingly to cover human evaluation results.

[1].Yuan, Zheng, et al. "Rrhf: Rank responses to align language models with human feedback without tears." arXiv preprint arXiv:2304.05302 (2023).

[2].Song, Feifan, et al. "Preference ranking optimization for human alignment." arXiv preprint arXiv:2306.17492 (2023).

We thank the reviewer for the detailed and constructive comments. Below are our responses to all the questions and comments, and we hope that the responses are reasonable and satisfactory enough to address the reviewer’s concerns and we welcome further discussion.

Q$_1$: "The LIRE objective is essentially the original policy gradient objective..."

$A_1$: As illustrated in Section 3, our objective is derived within the general policy gradient framework, and our focus is on how to better formulate the probability distribution over the multiple model generations during training. We construct P in Equation 4 by taking a softmax over all model generations. This operation makes the $P_{\pi(\theta)}(y|x, A)$ $**not**$ proportional to $\exp(\frac{1}{T})\sum_{k}logP(y_k |x)$. Consequently, when T=1, the LIRE objective still incorporates information from other responses(the denominator), which corresponds to the concept of “listwise”, and this makes it different from the policy gradient framework as the reviewer illustrated. This practice also brings a demeaned reward score $R(x^{(i)}, y^{(i)})-E_{(y′^{(i)}∼\pi_\theta(y^{(i)}|x^{(i)}))}R(x^{(i)}, y′^{(i)})$ in Equation 6, which brings advantages that responses with higher rewards contribute a larger part to the gradient descent.

Q$_2$: "varying T introduces slight fluctuation in performance..."

A$_2$: Essentially, the temperature parameter T is introduced to modify the probability distribution of the sampled model completions for a given query. Larger T makes all the samples more uniformly weighted, while smaller T shifts the probability mass to the best sample (more like SFT training), therefore, varying T introduces slight fluctuation in performance and T should be within a suitable range to boost performance.

$Q_3$: "The performance improvement is marginal..."

A$_3$: In original Table 3 and 4, the (number) in green indicates performance improvement of the model trained with the current training set (the column name) compared to the model trained with a dataset of fewer responses ( the previous column). So as the size of candidate responses increases, the performance continues to improve. We are sorry for the confusing illustration and will remove the (number) for clarity. The performance improvement (RM score) for summarization and dialogue tasks are summarized in the following. In Table 7, we also show that the performance boost is more evident if we utilize Algorithm 1 (iteratively updating the training samples) to increase the quality of the training samples. Generally, the pattern is a growing RM score if we increase the candidate numbers, while other pairwise methods do not display this pattern.

$Q_4$: "Could you provide a possible explanation of why PRO and RRHF perform significantly worse..."

A$_4$: This is an interesting topic. The RRHF objective tries to rank the probabilities of model generations according to the rank given by reward scores, the model relies on long sequence length to incorporate more preference information; the PRO objective leverages (n-1) comparisons in a list of n responses, and for each comparison, the best example is contrasted against the "negative" examples. According to the Minimum Bayes Risk(MBR) framework [1], choosing the outputs of a machine learning system based not on the output with the highest probability, but the output with the lowest risk (largest reward) among multiple candidates can be more powerful. Both RRHF and PRO do not directly optimize for reward scores, while constructing the loss with sophisticated modeling of probability distribution in the sampled space, and thus may, in our opinion, underperform when the number of candidates is small.

[1]. Bertsch, Amanda, et al. "It's MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Risk." arXiv preprint arXiv:2310.01387 (2023).

Thanks for your reply. The policy gradient under $m$ trajectories takes the form:

$J_\theta=\frac{1}{m}\sum_{j=1}^{m} \log\pi_\theta(a_j|s_j)R(\tau_j)$

so the log-probability of resposnes $\log\pi_\theta$ is multipled by $R$ to achieve optimization. On the other hand, our objective takes the form:

$J_\theta=\sum_{j=1}^{m}P_{\pi_\theta}(a_j|s_j)R(\tau_j)$

where $P_{\pi_\theta}$ represents the soft-max normalized log probabilities of responses as in Equation 4, which is essentially still a probability distribution itself. Even when T=1, we cancel out log and exp, resulting in $P_{\pi_\theta}=\frac{\pi_\theta(a_i|s_i)}{\sum_{j=1}^{m}\pi_\theta(a_j|s_j)}$

and this is not equalivent to $\frac{1}{m}\log\pi_\theta(a_j|s_j)$.

In softmax policy gradient [2][3], the policy is first parametrized using $\pi_\theta(.|s)=softmax(\theta(s,.))$, where $\theta(s,a)$ is referred as logits for an action (a single token in our settings). However, this practice is implemented in the token space for constructing the distribution for a single sentence, and during the policy optimization, actions(tokens) are actively sampled according to this constructed sampling distribution, with no softmax applied to multiple sentences to formulate a softmax-normalized probability distribution. Differently, in LIRE, given several candidate responses, we take a softmax over the sampled model completions(sentences) to construct a probability distribution $P$ for policy optimization.

According to the results in Table 11, T=1 actually produces good results, and we do not specifically choose T $\neq$1 since this is actually a good practice. We also implemented the policy gradient equation above, leading to worse results of -1.12(RM) and 18.89 (PPL).

[2]. Mei, Jincheng, et al. "On the global convergence rates of softmax policy gradient methods." International Conference on Machine Learning. PMLR, 2020.

[3]. Li, Gen, et al. "Softmax policy gradient methods can take exponential time to converge." Conference on Learning Theory. PMLR, 2021.

We thank the reviewer for the detailed and constructive comments. We respond to the comments below and update the manuscript accordingly.

Q$_1$: "How to constrast LIRE and listwise version of DPO introduced in DPO paper appendix? What are the main conceptual differences and performance differences?"

A$_1$: Thanks for bringing up this constructive topic. The DPO objective is extended from binary preferences to multiple responses according to the Plackett-Luce model, and it resembles the top-k probability defined in [1], which gives a permutation probability distribution that relies on the position of a response in the permutation. However, in LIRE, we do not explicitly model an ordinal ranking, instead, the ranking (position) information is given by the reward scores. For an explicit and clear performance comparison, we also provide the results for summarization (TL;DR-3) and dialogue (HH-6) tasks. DPO performs slightly worse than LIRE on both tasks, which suggests that utilizing rewards explicitly helps align preference better. We update the manuscript accordingly to include the discussion of listwise DPO. Thanks for your suggestion!

Q$_2$: "...cannot clearly attribute performance improvements to the proposed components. More ablation study may help provide more insights..."

A$_2$: Our goal is to maximize the expectation of the final rewards. This is achieved by combining two components: the probability distribution and the reward scores, which are actually under the framework of policy gradient. Specifically, we reformulate the probability distribution among the multiple model generations through softmax normalization to incorporate information beyond a single pair, but rather a whole list of responses. So these two components are an integral whole, and can not be separated for optimization.

Q$_3$: "...reward model is trained in optimization and may not be a good evaluation metric..."

A$_3$: Thanks for bringing this up. Initially, we chose the evaluation metrics referring to other papers [2][3]. However, we do realize that this may not be a fair indicator. To ensure fairness for other methods and a more reliable comparison, we also conduct human evaluation (47 feedback from volunteers, each consisting of 50 question samples) aside from the automatic metrics. The results are as follows. LIRE gains the highest win rate against both human-written baselines as well as other comparing methods, which is in line with the metric results in general. We do realize the importance of conducting human evaluation and we truly thank the reviewer for pointing out this crucial point. We update the manuscript accordingly to cover human evaluation results. Besides, we improve the presentation (e.g. PPL) in the paper and give a better discussion of the metrics.

[1].Cao, Zhe, et al. "Learning to rank: from pairwise approach to listwise approach." Proceedings of the 24th international conference on Machine learning. 2007.

[2].Yuan, Zheng, et al. "Rrhf: Rank responses to align language models with human feedback without tears." arXiv preprint arXiv:2304.05302 (2023).

[3].Song, Feifan, et al. "Preference ranking optimization for human alignment." arXiv preprint arXiv:2306.17492 (2023).

We thank the reviewer for the very detailed and constructive comments. Below are our responses to all the questions and comments, and we have updated the manuscript accordingly. We hope that the responses are reasonable and satisfactory enough to address the reviewer’s concerns and we welcome further discussion.

Q$_1$: "...claim that their objective implicitly includes the SFT objective?..."

A$_1$: According to the derivative of the LIRE objective in Equation 6, the gradient of each sampled response is weighted according to the reward scores. For queries that include human-annotated responses in the candidate list, LIRE includes the human-annotation during loss calculation, and thus the human-preferred response also contributes to the gradient decent. This can be perceived as an implicit SFT loss component in a sense. Additionally, we can enforce an explicit SFT loss to better adhere to human preferences. We will modify the relevant script and tone down the narrative to make the paper more rigorously presented. Thanks for the advice!

Q$_2$: "...adding just the SFT loss would lead to the model collapse. This is exactly the reason regularization objectives like KL-div are including in alignment objectives..."

A$_2$: Thanks for pointing out the different effects of SFT loss and KL divergence and we really appreciate that. We conduct experiments adding KL divergence to the LIRE objective and compare the averaged rewards under different KL penalties. We report averaged sequence-level KL and the corresponding averaged reward scores. The results demonstrate that just adding SFT loss will, as the reviewer suggested, not help preserve knowledge from the pretraining steps. Increasing the level of KL penalty helps mitigate the divergence between the training policy and the anchor policy, at the sacrifice of some reward losses. We update the manuscript in Section 5.5 accordingly and will cover the discussion of KL divergences.

Q$_3$: "Can the authors provide evidence in terms of divergence metrics/regularization metrics between a policy tuned with LIRE and the anchor policy to prove that the LIRE objective.."

A$_3$: As suggested in the above table, LIRE has a KL divergence metric of 9.75, which is slightly larger than 9.07 (0.01KL penalty). Adding just SFT loss to LIRE actually leads to a larger divergence of 11.73, despite achieving the best reward score of -0.817, which validates the reviewer's point made in Q$_2$. We also observe a tendency for reward decrease when increasing the level of KL penalty, and LIRE objective achieves a good balance between reward score and KL divergence against the anchor policy. We thank the reviewer for pointing out this constructive topic and will include more discussion on this in the manuscript.

Q$_4$: "...Another line of work that is similar and involves sampling of multiple responses is Sequence Likelihood Calibration SLiC..."

A$_4$: Thanks for pointing out this omission. We try to compare with the SLiC objective, specifically, we choose the rank calibration loss and cross-entropy regularization loss for the pairwise human feedback as introduced in [1] for the dialogue and summarization task. The comparison results are as follows. SLiC achieves good performance on TL;DR while performing worse on the HH task, and LIRE still achieves better performance. We will include the discussion of SLiC[2] in the paper for a more complete comparison.

Q$_5$: "Finally the paper does NOT consider "list-wise" approaches at all."

A$_5$: This is a good point. Essentially, LIRE does not rely on an ordinal ranking, instead, the ranking (position) information is given by the reward scores. This is different from the top-k probability defined in ListNet[3], which gives a permutation probability distribution that relies on the position of a response in the permutation. In LIRE, the sampled model generations are normalized with softmax, which means information of the whole list is incorporated into a single response in the list, with different reward scores weighing each response differently in loss calculation according to Equation 6. The reward scores bring a more quantified differentiation for each sample compared to ordinal ranking. This is different from standard "listwise" paradigms and we are sorry for the confusion. We have updated the manuscript accordingly to give a clearer illustration of this idea.

[1].Zhao, Yao, et al. "Slic-hf: Sequence likelihood calibration with human feedback." arXiv preprint arXiv:2305.10425 (2023).

[2].Zhao, Yao, et al. "Calibrating sequence likelihood improves conditional language generation." arXiv preprint arXiv:2210.00045 (2022).

[3].Cao, Zhe, et al. "Learning to rank: from pairwise approach to listwise approach." Proceedings of the 24th international conference on Machine learning. 2007.

Q$_6$: "The authors claim that their proposed approach does not require a KL constraint..."

A$_6$: This is a good point. According to Equation 6, the gradient of each sampled response is weighted according to the reward scores. For queries that include human-annotated responses in the candidate list, LIRE naturally includes the human-annotation contributing to the gradient descent. This can be perceived as an implicit SFT loss. Additionally, we can enforce an explicit SFT loss to better adhere to human preferences. According to [6], Cross entropy (the same as SFT loss under the MLE framework) and KL divergence regularization perform similarly and prevent models from deviating significantly from their fine-tuned objective. However, we do realize that without a KL constraint, alignment training would lead to a distributional collapse. We have modified the presentation and thanks for bringing this out.

Q$_7$: "The reward itself is computed from a model trained on pairwise comparison data. This seems somewhat opposed to the core problem.."

A$_7$: The reward model is trained on pairwise preference, and the comparison between pairs of data is transitive to the whole list of data, given any input data. This partial order relationship modeled by conditional probabilities can be used to score an entire sequence comparably, which means this pairwise preference training for reward model is actually a surrogate for achieving list-wise comparison. So this is in line with our list-wise approach.

Q$_8$: "Having additional win rate metrics, especially comparing two algorithms directly..."

A$_8$: Thanks for pointing out a better way for performance comparison. To ensure fairness for other methods and a more reliable comparison, aside from the automatic metrics, we also conduct human evaluation. The results are as follows. LIRE gains the highest win rate against human-written baselines as well as other comparing methods, which is in line with the metric results in general. We do realize the importance of conducting user study and we truly thank the reviewer for pointing out this crucial point. We update the manuscript accordingly to cover human evaluation results.

[1] Gulcehre, Caglar, et al. "Reinforced self-training (rest) for language modeling." arXiv preprint arXiv:2308.08998 (2023).

[2] Cao, Zhe, et al. "Learning to rank: from pairwise approach to listwise approach." Proceedings of the 24th international conference on Machine learning. 2007.

[3] https://github.com/tatsu-lab/stanford_alpaca

[4].Song, Feifan, et al. "Preference ranking optimization for human alignment." arXiv preprint arXiv:2306.17492 (2023).

[5]. Rafailov, Rafael, et al. "Direct preference optimization: Your language model is secretly a reward model." arXiv preprint arXiv:2305.18290 (2023).

[6].Zhao, Yao, et al. "Calibrating sequence likelihood improves conditional language generation." arXiv preprint arXiv:2210.00045 (2022).

We appreciate the thorough and constructive feedback from the reviewer. Our responses to all questions and comments are provided below. We hope that the responses are reasonable and satisfactory enough to address the reviewer’s concerns and we welcome further discussion.

Q$_1$: "...how the generative sampling is done under the proposed distribution over top-k generations."

$A_1$: Line 2 in Algorithm 1 is the offline dataset generation process where we gather $**offline**$ training data. In the experiment, the preference data was originally gathered from both human preferences as well as from Alpaca-7B using diverse beam search (when $e$=1); for $e$>1, we can additionally leverage the self-enhancement strategy proposed in Algorithm 1 and use the trained policy after $I$ epoch to generate new offline data to refresh the previous candidate pool (this practice is also adopted in ResT[1]). During training, the generative distribution defined in Equation 4 actually takes a softmax over the $m$ sampled model generations. This is different from the top-k probability defined in ListNet[2], which gives a permutation probability distribution that relies on the $**position**$ of a response in the permutation. In LIRE, the sampled model generations are normalized with softmax, with different reward scores weighing each response differently in loss calculation according to Equation 6, and no explicit ranking is required during this process.

Q$_2$: "Would it be possible to differentiate between the policy distribution and the log-probability under the model for a generation“

$A_2$: We are sorry for the confusion and have modified the manuscript. Now $\pi_{\theta}$ in Equation 4 refers to the model sampling distribution which aligns with other papers. We also fixed Equation 3 for autoregressive model generation $P_{\pi_{\theta}}(y_{j,k}^{(i)}\|x^{(i)},y^{(i)}_{j,<k})$. Thanks for the reviewer's detailed review!

Q$_3$: "...it seems like the PPO model achieves worse reward compared to a baseline Alpaca-7B model, which seems very counter-intuitive..."

A$_3$: Alpaca-7B is finetuned using the Alpaca dataset [3], in which the root verb of "summarize" makes up a very small fraction of the whole dataset, while other root verbs of "write", "suggest", "explain", etc., consist a large fraction, and these topics share more similarity of the HH dataset. This may account for Alpaca being a strong performer for questions in HH, however, in the summarization task, PPO performs significantly better than Alpaca-7B.

Q$_4$: "How many samples from the test split are considered for the GPT-4 evaluation ?"

A$_4$: For the GPT-4 evaluation, we use 100 samples from the test split for each method for comparison. Since it is verified in [4][5], that GPT-4 judgment has a high agreement with human evaluation, we think 100 questions should be reasonable to obtain a GPT-4-based estimation that reflects human preference. Besides, the MT and Vicuna Bench both consist of 80 questions for evaluation using GPT-3.5 and GPT-4. For the above reasons, we chose 100 questions for GPT-4 evaluation.

Q$_5$: "It seems like after LIRE, there are a fair number of samples for which the RM scores actually reduced..."

A$_5$: This is quite an interesting point. We visualize all the instance-level RM variation for each comparing method and find that all the methods display a fair large amount of reward reduction after training, which seems to be a common issue for different methods. However, compared to other methods, LIRE has a smaller ratio of instance-level reward drop across the test samples and better RM improvement on average. We list the percentage of the samples with declining scores in the whole test set for each method and LIRE has the smallest ratio of 27% of score decrease. We speculate that it may be due to factors such as bias during the data sampling process, which will be left as our future work. Thanks for the reviewer's question and we have added more visualization results in Appendix A.9.