Q1

We provide the training hyperparameter settings of Mistral-7B-base in Table 7 and our dataset is a combination of UltraFeedback and SimPO, which is not equivalent to papers like DPO and LiPO. So it is normal to have a different result on the MT-Bench evaluation benchmark. The performance order of baselines generally aligns with previous alignment papers.

Q2

The ablation results on discount functions are provided below:

It shows that $\log_2(x+1)$ achieves the best performance. A potential reason is that the log function provides a smoother discount.

Q3

There is no significant difference between BPR and All Pairs. If increased training data can enhance model performance, All Pairs should consistently achieve superior performance over the BPR method. We also add a variant that compares multiple positive responses against the worst response:

Qwen2-0.5B

Mistral-7B

Thanks for your constructive and valuable feedback which improves our work. Please see our detailed response below:

w1&w2&w3&w4

Inspired by the Listwise Preference Optimization (LiPO) framework, we are the first to propose aligning human preferences directly with the NDCG metric to the best of our knowledge. To additionally analyze why OPO outperforms other methods, we counted their correct ranking pairs and successful ranking flips (i.e., incorrect to correct) during training. OPO exhibited significantly higher efficiency in flipping incorrect pairs compared to DPO and LiPO. We can also theoretically prove that the optimal ranking accuracy of NDCG-based methods is bounded by the ranking accuracy of the reference model. But to achieve such an optimal policy, OPO doesn’t necessarily maximize the reward margin like DPO and LiPO, which leads to higher efficiency in increasing the ranking accuracy metric. The detailed results are provided in Appendix A.9.

w5

“the gain function should be proportional to the label” means $G(\psi_i)>G(\psi_j)$ if $\psi_i>\psi_j$. The gain function should correctly reflect the order of ground truth labels. The ablation study on the gain function also reveals that there is no significant performance gap between $G(\psi)=2^\psi-1$ and $G(\psi)=\psi$. We also supplement the ablation study on discount functions, which will be provided below.

w6

Thanks for your supplements on baselines. NeuralSort and PiRank lacks column normalization compared with OPO, which is already shown in our ablation studies. Now we summarize the model performance on NeuralSort, PiRank, and OPO:

We appreciate your thorough response, which has clarified some of our initial concerns. However, we share the perspective of other reviewers regarding the incremental nature of OPO's technical contributions. Additionally, it is worth noting that [1] demonstrates that LambdaRank loss (used in LiPO) inherently optimizes NDCG.

[1] The LambdaLoss Framework for Ranking Metric Optimization.

w3

It is popular to employ offline algorithms with several iteration rounds in the industry[1], and it is straightforward to apply OPO in an iterative paradigm. Traditional iterative pairwise methods sample best-of-n responses and assign them reward scores with the separated reward model, then they choose preferred response $y_w$ and non-preferred response $y_l$ based on assigned rewards and apply alignment algorithms like DPO on these response pairs. For OPO, it shares the same sampling-rewarding stage and the only difference is that we input a list of K responses with their assigned reward scores and employ listwise algorithms on them.

Regarding other additional values, we conduct comparisons of flipping incorrect pairs during training across DPO, LiPO, and OPO, which is provided in Appendix A.9.

[1] Dubey, Abhimanyu, et al. "The llama 3 herd of models." arXiv preprint arXiv:2407.21783 (2024).

w2

The paper does not have a convincing argument on why the deployed objective is a better choice, so the experiments are at the risk of cherry-picking

To comprehensively highlight the differences in performance among various methods, we counted their correct ranking pairs and successful ranking flips (i.e., incorrect to correct) during training. OPO exhibited significantly higher efficiency in flipping incorrect pairs compared to DPO and LiPO. The detailed results are provided in Appendix A.9.

Note that even if there's an argument specific to the objective, the contribution is still likely limited since the formulation and general objective (e.g., optimizing NDCG) are more important and the loss alternatives are not that different (as studied intensively in the IR literature).

Experiments have shown that there is a performance gap among different learning to rank objectives. We added several LTR baselines to demonstrate the reliability of the experimental results. The results indicate that OPO consistently outperforms these methods.

Qwen2-0.5b

Mistral-7b

Thanks for your constructive and valuable feedback which improves our work. Please see our detailed response below:

w1

This is because the weights use listwise information so they are no longer pairwise.

LambdaRank uses the DCG difference to re-weight the pair error compared to the original DPO formula, which can be interpreted as a Pairwise Logistic Loss with lambda-NDCG weights. This is the major reason why we classified it as a pairwise method. But we’ll modify the claims to a listwise approach in our revised paper to keep the alignment with LTR literature.

It is even more confusing that LiPO pretty much has the exact same idea (while being public almost a year ago) to optimize NDCG.

Thanks for pointing out this potential confusion. The LiPO framework proposed the concept of Listwise Preference Optimization (LiPO) and introduced LambdaRank. Although LambdaRank incorporates DCG information, its theoretical relationship with NDCG has not been thoroughly analyzed. In fact, based on their gradient formulations, LambdaRank is more closely related to DPO (pairwise logistic loss). When $\delta_{w,l}$ is large, the gradient of LambdaRank will be strengthened and vice versa. However, it still falls short of the struggle of pairwise logistic losses, which maximizes the reward margin instead of ranking accuracy or win rates.[3] We can show that OPO has a higher speed in correcting incorrect ranking accuracy pairs (i.e., make $\frac{\pi\_\theta(y_w|x)}{\pi\_\theta(y_l|x)}>1$ given $\frac{\pi\_{\text{ref}}(y_w|x)}{\pi\_{\text{ref}}(y_l|x)}<1$) than DPO and LiPO in Appendix A.9.

$$\nabla\_{\theta}\mathcal{L}\_{LiPO}=-\Delta\_{w,l}\cdot\beta\cdot\sigma(s(x,y\_l)-s(x,y\_w))\left[\nabla_\theta \log\pi(y\_w|x)-\nabla\_\theta\log\pi(y\_l|x)\right]=\Delta\_{w,l}\cdot\nabla\_{\theta}\mathcal{L}\_{DPO}$$

[1] Burges, Christopher, Robert Ragno, and Quoc Le. "Learning to rank with nonsmooth cost functions." Advances in neural information processing systems 19 (2006).

[2] Wang, Xuanhui, et al. "The lambdaloss framework for ranking metric optimization." Proceedings of the 27th ACM international conference on information and knowledge management. 2018.

[3] Chen, Angelica, et al. "Preference Learning Algorithms Do Not Learn Preference Rankings." arXiv preprint arXiv:2405.19534 (2024).

Q1

There is a gap between using NDCG as an evaluation metric and a training objective, since the NDCG metric is non-differentiable with respect to reward scores $\mathbf{s}$, which prevents the utilization of gradient descent to optimize models. So we have to employ a differentiable surrogate approximation of NDCG to optimize it.

Q2

The win rate metric is the standard evaluation metric in LLM alignment literature, such as DPO and SimPO. To overcome the sensitivity of a single metric, we employ four different win rate metrics across different reward models and evaluation benchmarks. As shown in the literature, a 5% improvement in win rates is considered significant, but achieving a significant p-value can be difficult as the test set contains only about 2k prompts.

We thank the reviewer for their constructive and valuable feedback which improves our work. Please see our detailed response below:

w1&w3

The logistic-based methods like RLHF, DPO, and LiPO maximize the reward margin of the preferred response over the non-preferred one, but experiments[1] have shown that they struggle to increase ranking accuracy $\pi_\theta(y_w|x)>\pi_\theta(y_l|x)$, which is a more practical metric in generation scenarios. We counted their correct ranking pairs and successful ranking flips (i.e., incorrect to correct) during training. OPO exhibited significantly higher efficiency in flipping incorrect pairs compared to DPO and LiPO. The detailed results are provided in Appendix A.9.

w2

LTR methods can be applied to the offline alignment problem, but their performance differs. To comprehensively investigate the LTR literature, we added additional baselines of BCE, RankNet, ListNet, and PiRank. The results indicate that OPO consistently outperforms these methods.

Qwen2-0.5b

Mistral-7b

[1] Chen, Angelica, et al. "Preference Learning Algorithms Do Not Learn Preference Rankings." arXiv preprint arXiv:2405.19534 (2024).

We would like to thank you for the detailed comments and suggestions. We have responded to each of the concerns below.

Q1

1. The OPO is an offline listwise alignment algorithm.

2. The performance differences between online and offline RLHF approaches have been well discussed in the alignment literature. [1][2] Online methods integrate on-policy sampling during the training stage and demonstrate a superior performance with a smaller KL divergence, which means the RLHF policy drifts less from the SFT policy. Offline algorithms also face the challenges of fitting on sub-optimal pre-generated training data. A possible solution is to employ the offline methods in multiple iteration rounds, where new preference data and annotations are generated from the latest policy and used for the next training round.[3]

3. Online algorithms can be viewed as offline algorithms running on $D_{online}$, which is more on-policy than $D_{offline}$. Under specific sampling methods such as rejection sampling or best-of-n [2], offline algorithms like DPO and our OPO can be naturally applied in online scenarios. Please also note that employing offline algorithms with several iteration rounds in the industry is more popular, which is discussed in (2).

4. Like other offline methods, applying OPO in an iterative paradigm is straightforward. Traditional iterative pairwise methods sample best-of-n responses and assign them reward scores with the separated reward model, then they choose preferred response $y_w$ and non-preferred response $y_l$ based on assigned rewards and apply alignment algorithms like DPO on these response pairs. For OPO, it shares the same sampling-rewarding stage and the only difference is that we input a list of K responses with their assigned reward scores and employ listwise algorithms on them.

[1] Tang, Yunhao, et al. "Understanding the performance gap between online and offline alignment algorithms." arXiv preprint arXiv:2405.08448 (2024).

[2] Gao, Leo, John Schulman, and Jacob Hilton. "Scaling laws for reward model overoptimization." International Conference on Machine Learning. PMLR, 2023.

[3] Dubey, Abhimanyu, et al. "The llama 3 herd of models." arXiv preprint arXiv:2407.21783 (2024).

Q2

1. We have now provided the step-by-step example in Appendix A.8

2. The original NDCG metric is deterministic, while it is non-differentiable with respect to reward scores, which prevents the utilization of gradient descent to optimize models. In order to apply backpropagation algorithms, we need to replace ranking operators like $\tau(j)$ with differentiable and continuous operators like $\hat\tau(j)$ in ApproxNDCG, and such operators are probabilistic.

3. The hyperparameter $\tau$ in OPO controls approximation precision. As shown in Table 6, when $\tau$ is set to a high value, the probabilistic permutation matrix P_sort could not correctly reflect the true permutation, which is harmful to model performance.

For example, having the same [0,1,0,0] rows in the entire matrix

No, $P_{\text{sort}(\mathbf{s})}[i,j]$ denotes the probability that response $y_j$ is ranked in the $i$-th position after re-sorting. If scores=[1,1,1,1], the rows of P_sort will be [0.25,0.25,0.25,0.25].

4. NeuralSort and its variant PiRank are both differentiable probabilistic sorting algorithms and can be employed for NDCG approximation. However, their permutation matrix $\widehat{P}$ may not be column-stochastic, meaning each column may not sum to one. This can cause some $G(\psi_i)$ to contribute to the overall loss objective disproportionately and adversely affect model performance. The ablation study is shown below.