MallowsPO: Fine-Tune Your LLM with Preference Dispersions

Thank you for providing suggestions to better improve the paper. We address your questions below:

Assuming the probability distribution of ranking, rather than the pairwise preference, loses information about reward difference.

We understand the reviewer's concern and agree with the simple example. However, we’d like to point out that this concern, in the setting of language modeling, does not matter for several reasons.

• Firstly, the reviewer’s concern raised about no difference in the probability of the rankings by swapping any two items will be resolved as the total number of items to rank are sufficiently large—a scenario that is common in language modeling tasks. Here we illustrate the effect of additional items for the relative probability of observing two rankings which differ only in the rankings of 3 items. If there are no additional items, then observing a relative order of items of value 100,2,1 in 100,1,2 in Mallows-$\theta$ is (i.e. observe (1,3,2) for central ranking (1,2,3)):

which will be the same as observing (2,100,1) since:

However, if there are additional 97 items of value (100 items in total of value from 1 to 100), things will be quite different. Consider the central ranking is (100,99,98,97,...,2,1) for the items of value from 1 to 100. If two observed ranking are $\mu_1$ and $\mu_2$, and $\mu_1(i)=\mu_2(i)=100-i$ holds for all other $i$'s from 3 to 99, we have (100,1,2) becomes (100,99,98,...,3,1,2) and (2,100,1) becomes (2,99,98,...,3,100,1). Thus:

while

which means two observations $p([2,100,1])<<p([100,1,2])$ since $\phi\in (0,1)$. We also included such a discussion in the updated version of our draft, please see Appendix C.3.

• Secondly, in DPO’s formulation, there is no explicit step for reward modeling, which is a core strength as well as the beauty of the approach – its simplicity and directness are preserved by excluding the reward modeling step. Therefore, although for MallowsPO in theory there is an underlying ranking function, in practice the final objective of MallowsPO similarly does not involve its modeling either. Moreover, from this perspective, our MallowsPO framework embraces DPO as a special instance of MallowsPO-\theta, by letting $\phi(x) = 1$. Thus, in the context of preference optimization, MallowsPO provides a more comprehensive representation of information compared to DPO, ensuring that valuable insights are preserved rather than diminished.

• Lastly, from the perspective of human feedback, it is more appropriate and natural to model the rank of an object rather than the reward. Consider the following toy example: given two articles A and B, it is generally easier for a person to tell their preference between the two rather than assign precise scores to each article. In the original technical report of InstructGPT [2], human annotators provided preference data by ranking sets of 4 to 9 responses instead of assigning exact reward scores, illustrating this principle.

Reference

[1] Mallows, C. L. (1957). Non-null ranking models. I. Biometrika, 44(1/2), 114-130.

[2] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., ... & Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35, 27730-27744.

Insufficient ablations: lacking baseline of <DPO+disparity>

Long story short, the proposed <DPO+disparity> is exactly MallowsPO-$\theta$, one of the MallowsPO instances, please see Corollary 2 of Section 3.2. The proposed method is simple yet novel while offering valuable insights.

• Firstly, we’d like to again emphasize that this method is not proposed as an isolated and ungrounded method but from the pairwise probability of the Mallows-$\theta$ model. This foundational link ensures the robustness and theoretical basis of our method.

• Moreover, to recall one of our contributions, we address the issue of how to efficiently estimate the dispersion or disparity parameter using the entropy associated with winning and losing items. This idea builds on the interpretation of the dispersion parameter in the Mallows model, which quantifies distribution spread and is conceptually related to entropy.

• Thirdly, DPO + disparity—also referred to as MallowsPO-$\theta$ in our work—represents just one instance within the broader MallowsPO framework, which is theoretically rooted in the Mallows model. As demonstrated in Theorem 1, we establish a one-to-one mapping between the distance function in the Mallows model and the MallowsPO loss objective. This highlights that numerous other instances of MallowsPO can also be derived and explored.

• Lastly, the idea of contextual scaling also originates from the MallowsPO framework. Specifically, it corresponds to the term $-2\log\phi(x)$ in Corollary 2 (multiplied by regularization parameter $\beta$ and logits difference), motivating us to use MallowsPO as a modular component for enhancing other preference optimization methods, such as IPO and simPO, to achieve further performance boosting.

For Figure 9, how does the training curve differ for different prompts, i.e., $x$?

As stated in line 404, there is no prompt or contextual information $x$ for the bandit experiment, as it involves selecting from a predetermined set of arms. Without contextual information $x$ to estimate the dispersion, the dispersion parameter is fixed, in our case at 0.05, for illustration.

The label of Figure 8 (b) is maybe a mistake: I think it should be # of epochs rather than KL.

Thank for your question of clarification and we want to say that this is not a mistake. We meant to plot KL vs reward because it demonstrates the dynamics of the reward of the training policy and its KL with respect to the reference policy for every 100 epochs during training. KL regularization has already been extensively employed in existing literature, either as a practical computational constraint [3][4] or as a mechanism to balance the trade-off between accuracy and diversity [5].

To interpret the plot in Figure 8(b), at the beginning, the training policies of both methods are identical to the reference policy, resulting in 0 KL and around 12.9 reward. As the training progresses, the KLs of the training policies for both methods will become larger and larger. As a result, the training policy of DPO converges to the one with larger KL and smaller reward.

References

[3] Tang, Y., Guo, Z. D., Zheng, Z., Calandriello, D., Munos, R., Rowland, M., ... & Piot, B. (2024). Generalized preference optimization: A unified approach to offline alignment. ICML 2024

[4] Munos, R., Valko, M., Calandriello, D., Azar, M. G., Rowland, M., Guo, Z. D., ... & Piot, B. (2023). Nash learning from human feedback. ICML 2024

[5] Wang, C., Jiang, Y., Yang, C., Liu, H., & Chen, Y. (2023). Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints. ICLR 2024.

Much thanks for your reply and for asking further clarification questions.

How do <DPO+disparity> and MallowsPO-$\theta$ become equivalent to each other?

We would like to showcase the equivalence of these two in terms of their resulting loss objectives. For the same preference dataset, both DPO and MallowsPO-$\theta$ end up with a loss objective to optimize, thus we could take their loss objectives as our first class citizens. DPO loss (as in Equation (4)) is derived by combining the cross-entropy loss and the change of variable technique (as you mentioned):

where the expectation is with respect to the preference dataset $\mathcal{D}=(x,y^w,y^l)$ and $r^{\theta}(x,y) = \beta \log \frac{\pi_{\theta}(y \mid x)}{\pi_{\text {ref }}(y \mid x)}$ is known as the implicit reward model of DPO. Compared to DPO loss, MallowsPO-$\theta$ loss objective differs only in an extra contextual term which corresponds to our dispersion estimators (we write as $f(x)$ for simplicity):

This is exactly <DPO+disparity> (if calling $f(x)$ as the disparity term, while in our paper we refer as dispersion term), that's what we mean by the equivalence of MallowsPO-$\theta$ and DPO. We also want to emphasize that ultimate DPO "training" does not involve any exact reward as in the loss objective, and only relative order/rank matters. Similarly the exact rank does not appear in the objective function of MallowsPO. So whether the true underlying reward is (100, 1, 2) or (2024,1,2) will not change the DPO or MallowsPO loss objective. To capture the significance of DPO preference pairs to improve DPO has been explored in papers like ODPO [1], but this is beyond the scope of our work.

Also, can you elaborate the derivation of Eq. (8)?

Thanks for your suggestion. We provide a more detailed derivation of the Equation 8 in Corollary 2 in the Appendix B and mark it in blue for your reference. We also include the concrete objective of MallowsPO-$\theta$ and MallowsPO-$\phi$ in Appendix B and Appendix G to showcase a more clear connection between MallowsPO-$\theta$ and <DPO+disparity>. Briefly speaking, Equation (8) illustrates that pairwise preference is indeed Bradley-Terry with each item’s reward $y$ being $\log(\phi(x)) \mu_0(y\mid x)$, with $\mu_0$ denoting the central ranking.This further justifies what we claim about why MallowsPO-$\theta$ can be understood as DPO+disparity.

References.

[1] Direct Preference Optimization with an Offset (ODPO) https://arxiv.org/pdf/2402.10571

We sincerely appreciate your kind words on the novelty and soundness of our work. We also thank you for providing insightful questions, here we address them below:

The discussion and estimation of the dispersion index is not fully clear to me. ablate the sensitivity of MallowsPO to other possible estimation approaches

The dispersion index $\phi(x)$ basically indicates how spread the completion distribution $P(y|x)$ is for a specific prompt $x$. To estimate $\phi(x) \in (0, 1]$, we consider using the entropy of $P(y|x)$. However, computing this exactly is difficult. Therefore, we come up with the following way to estimate it:

$f(x) = \frac{1}{2}\cdot \frac{1}{\log (n)} \cdot (H_{\text{win}}+ H_{\text{loss}})$

The intuition of this estimation comes from two parts:

• Entropy estimation. Note that to the entropy of a generated sequence drawn from $P(Y|X)$ can be expressed as the sum of conditional entropy through the chain rule: $H (Y_1,Y_2,\cdots,Y_n\mid X) = H(Y_1|X) + H(Y_2|X, Y_1) + H(Y_3|X, Y_1, Y_2) + …..$ Therefore, we estimate above using the average of two realizations: winning and losing samples (which yields a non-biased estimator), which is $H(\pi(\cdot\mid X))\approx \frac{1}{2} \sum_{i=1}^{N-1}\left[H(Y_{i+1}\mid Y_i=Y^w_i)+H(Y_{i+1}\mid Y_i=Y^l_i)\right].$

• Upper bound. In addition, because the dispersion $\phi(x) \in (0, 1]$, we normalize such a quantity by the upper bound of the entropy, which is $N\cdot log(|V|) = log(|V|^N)$, where $N$ is the max length and $|V|$ is the size of vocabulary (possible tokens), so that $f(x) \in (0,1]$. $|V|^N$ can also be understood as the total number of possible sequence of length N.

We then estimate $-2\log(\phi(x))$ with $\phi\cdot \log(f(x))$. We empirically verified that such an estimator aligns well with the true $-2\log(\phi(x))$, shown in Figure 3, supposing we know the $\phi^*$. An immediate question is: How do we know determine $\phi^*$ in practice? Fortunately, a connection exists within the penalization of DPO. Specifically, while the penalty coefficient in DPO is represented by $\beta$ while the penalty coefficient in MallowsPO-$\theta$ is $-2log(\phi(x))\beta$. To ensure consistency, we impose $\phi$ such that

$E_{x\sim X}\left[\phi\cdot \log(f(x))\right]= 1.$

For other dispersion estimators, to the best of our knowledge, we don’t think there are other scalable and efficient estimating approaches for the dispersion index, comparing to our method which induces minimum extra computational burden. We agree that developing other dispersion estimation technique could possibly enhance the improvement brought by MallowsPO, and would like to leave the pursuit of efficient and more accurate dispersion estimation techniques in the future works.

on how many generation? does this require drawing more samples for an accurate estimate?

Our proposed method does not require generating answers or sampling multiple times. Please see the above explanation of our way to estimate the dispersion.

the dispersion of the unknown preference model (i.e. humans)? If so, I do not see why one should use the entropy of the SFT model (which we would still like to align to human preferences) to estimate such index.

Theoretically, the base model is typically trained on human-generated data such as news articles, textbooks or Wikipedia, which inherently reflect human preferences. Assuming that we may get a multimodal distribution for answers to a fixed prompt, SFT is known to guide the format of response which helps the language model determine the response format we desire and specify a subset of modals in human preferences. Therefore, even though the SFT model has not fully been aligned to given preferences, it should still, to some extent, reflect the broader spectrum of human preferences embedded within its pre-training data. Our proposed dispersion estimators based on this assumption indeed have shown the capability to capture the dispersion of the prompts empirically, as shown by Figure 1.

We sincerely appreciate your kind words about the clarity and interest of our work. We are also grateful for your valuable comments, which will help us enhance the quality of our paper. Below, we address your questions and suggestions in detail:

additional clarity in Section 3.1

Thank you for your suggestion on providing more background on Mallows Ranking Model. We have revised Section 3.1 to enhance the intuitive understanding of the Mallows model. Concretely, we add a 3-items illustrative example and compute their discrepancy function values under Mallows-$\theta$ and Mallows-$\phi$ ranking models separately in Section 3.1 (we mark the added paragraph in blue). We also include more details about this example, including computations, visual plots of probability density function, and their changes with respect to the dispersion parameter $\phi$ in the Appendix C. For transition from traditional DPO to MallowsPO under Mallows Ranking Model, please see the two paragraphs after Corollary 2.

further details on how practitioners could leverage these insights in real-world applications could enhance the paper’s practical relevance.

For practical relevance, we consider the following directions on further leveraging these insights in real-world applications, focusing on curriculum learning and personalized alignment through contextual scaling. We outline these future directions in the conclusion section.

Curriculum learning. Curriculum learning is a training strategy inspired by human learning processes, where models are trained on tasks or data organized from simpler to more complex examples, instead of presenting all examples randomly or simultaneously. This approach has been shown to guarantee faster training convergence [1], better generalization [2] and improved model robustness [3]. Within our framework, our proposed dispersion index can be utilized to indicate the determinism or diversity of the response to different questions, offering guidance on the complexity of the prompts to achieve curriculum learning.

Personalized Alignment. Furthermore, by considering $\phi(x, p)$, where $p$ is a personal feature, our method can be adjusted to consider a personalized preference ranking model. Note that how to achieve this with DPO is not straightforward. However, it is more natural in MallowsPO because of the existence of dispersion parameter $\phi(x)$ in the Mallows model, which controls how spread out the distribution should be. In this context, dispersion can be extended beyond the prompt level to encompass different users or user groups $p$. In terms of the preference optimization objective, different contextual scaling on the penalization coefficient can be considered to accommodate varying user or user group preferences for responses generated by the Supervised Fine-Tuning (SFT) model. Contextual scaling, tailored to different user preferences, enables a customized user experience.

Both curriculum learning and personalization through contextual scaling present promising directions for further development and can be expanded based on our foundational research. However, these ideas merit dedicated presentation and discussion in separate works and are thus reserved for future exploration. We have updated these discussions in Section 6 (Conclusion), which are marked in blue, and in Appendix F.

additional insights into the MallowsPO model’s performance under varying preference dispersion scenarios (e.g., low vs. high dispersion prompts) could strengthen the evaluation

The key insight behind MallowsPO's performance under varying preferences, which is a natural phenomenon in reality, is its explicit incorporation of an inductive bias that captures dispersion information for the preference model, which leads to improved evaluation performance. This approach contrasts with the original DPO, which does not account for this aspect. While it is possible to rediscover the Mallows model using the Bradley-Terry (BT) model by redefining a new reward function as $\tilde{r}(\cdot; x)=\phi(x)⋅r(\cdot; x)$, MallowsPO directly builds a structure that effectively channels dispersion information for improved modeling. Leveraging this structural insight of the preferences has already shown clear empirical gain for LLM preference optimization in our work. We would like to investigate theoretically provable benefits of MallowsPO over vanilla DPO without dispersion in future work.

Reference.

[1] Bengio, Y. et.al. (2009, June). Curriculum learning. In Proceedings of the 26th annual international conference on machine learning (pp. 41-48).

[2] Graves, A. et.al. (2017, July). Automated curriculum learning for neural networks. In international conference on machine learning (pp. 1311-1320). PMLR.

[3] Weinshall, D. et.al.. Curriculum learning by transfer learning: Theory and experiments with deep networks. In International conference on machine learning (pp. 5238-5246). PMLR.

Dear Reviewer Nu5X,

We hope this message finds you well. We kindly want to check if you had a chance to review our rebuttal, and if you have any further questions or comments we can address to help with your evaluation. Thanks again for your efforts and suggestions in improving our manuscript.

Sincerely,

The authors

Dear Reviewer Nu5X,

We hope this message finds you well. As the deadline for revising the manuscript approaches on November 27th at 11:59 PM AoE, we are writing to follow up again to check if we have addressed your concerns to help with your evaluation. Your feedback has been invaluable in refining our work, and we sincerely appreciate the time and effort you’ve dedicated to improving our manuscript.

Sincerely,

The authors

Dear Reviewer Nu5X,

As today is exactly the revision deadline, we kindly request your feedback on our rebuttal. We would greatly appreciate it if you could let us know whether we have successfully addressed your concerns. Thank again for the time and effort you’ve dedicated to improving our manuscript.

Sincerely,

The authors

We sincerely thank you for further questions and comments to improve our manuscript. We have updated our manuscript per your suggestions, and below we also address or answer your concerns one by one.

Questions in Point 1 raised by reviewer

What is the function $g$ in Thm 1 (Line 203)? It was not introduced in previous context nor in the proof. Is it meant to be a link function? If it is, it is assumed to be a specific link function?

Yes, it is meant to be the link function, which implicitly appeared in the proof of Theorem 1. We have edited the descriptions of Theorem 1 (also the paragraph before it) to have a clearer explanation of the assumption and role of this link function. We have also marked our changes in orange for your reference. Thanks for pointing this out!

Questions in Point 2 raised by reviewer

Kindly refer to the added intuition in Section 3.1 of the revised manuscript, as well as our responses to the reviewer’s questions:

Can we safely draw consistent conclusions with different Mallows model

The answer is NO. Different discrepancy functions yield different Mallows Ranking Model, and we can easily find examples of inconsistency between different Mallows Ranking Models. Take a four item example, if the true ranking is $\mu_0=(1,2,3,4)$, consider two ranks $\mu_1= (2,1,4,3)$, $\mu_2=(1,4,2,3)$, then we can compute that for Spearman’s rho discrepancy (utilized in Mallows-$\theta$) $d(\mu_0,\mu_1)=4<d(\mu_0,\mu_2)=6$, however the Kendall’s tau (utilized in Mallows-$\phi$) can be computed as $d(\mu_0,\mu_1)=2=d(\mu_0,\mu_2)$. This showcases that different Mallows Ranking Models are not consistent because of their difference in discrepancy function. We have updated more details on this computation in Appendix C.4 for your reference. We have also included this conclusion in the introduction of Mallows Ranking Models near the end of Sc3.1, providing more insights per your suggestion.

As a result, in reality, how do we choose which Mallows model to use for preference learning? Can authors comment under which situations do one model excel the other? Is there any guideline?

Thanks for raising this point. We have to admit that we haven’t found clear guidelines on choosing among Mallows Ranking Models, namely Mallows-$\theta$ and Mallows-$\phi$. For simplicity of implementation, we would recommend to always first try the Mallows-$\theta$ model, which is easy to modify upon codes of DPO and implement, and we have also demonstrated the effectiveness and clear advantages of MallowsPO over DPO baseline empirically. If one does not have a concern of computational budget, he/she can always try both MallowsPO-$\theta$ and MallowsPO-$\phi$ provided in our paper.

Do Mallows models always excel BT models in better capturing human preference? Can we prove it theoretically (may not be easy)? Or authors can comment whether Mallows models are more general in the sense that pairwise preference data is not necessary as in BT model etc.

For this, if we focus on the application of aligning language models from human preferences and the perspective of designing loss objectives, we are pretty confident that the answer is yes. Essentially, MallowsPO incorporates an extra freedom, “dispersion” in the loss objective, which is the central concept of our paper and allows a more expressive characterization of human preferences. One can always choose a constant 1 as dispersion, which will then yield DPO since DPO does not consider dispersion. This shows that DPO is a special instance of MallowsPO-$\theta$, thus MallowsPO indeed can yield better performance, both theoretically and empirically.

Questions in Point 3 raised by reviewer

Whether the proposed method can be easily implemented as DPO?

Yes, the implementation is as easy as DPO and we have also already included the code in the supplementary materials. MallowsPO is pretty straightforward to implement, thanks to our choice of dispersion estimator. Long story short, our estimator needs to compute the entropy from the logits, which is a direct product in the computation of DPO objective, thus there is minimum extra computational burden.

Can authors comment whether this computational benefit can be always achieved?

Thanks for observing this and pointing it out. Throughout this paper, we have been focusing on comparing the final policy DPO and MallowsPO converge to, and we indeed did not compare the rate of convergence to the policy. We are thus not sure about your question since that’s not our paper’s primary claim or focus, but we would like to pursue this in future works.

does MallowsPO bring significantly extra computation overhead in order to compute the dispersion?

Similar to the question earlier, there is a minimum extra computational burden introduced by MallowsPO over the DPO.

I thank the authors for their responses. After going through the responses and the revised draft again, here are some questions that I still have:

1. What is the function $g$ in Thm 1 (Line 203)? It was not introduced in previous context nor in the proof. Is it meant to be a link function? If it is, it is assumed to be a specific link function? What properties it should satisfy for Thm 1 to hold? It needs to be clearly articulate before introducing the extra notations.

2. I appreciate the authors' effort in providing examples in the main text and appendices. However, further insights (not necessarily to be long) should be provided. The added example highlighted in blue (Line 169 - 177) is purely calculation. Audience will benefit further if authors can add a sentence or two for the key takeaways.

   • In particular, given the same central ranking and the same observation, different Mallows model lead to different dispersion. How do we compare these results? Can we safely draw consistent conclusions with different Mallows model? e.g. If a completion $y_1$ is better than $y_2$ under one Mallows model, does it suggest $y_1$ is always better than $y_2$ under other Mallows model?
   • From my understanding, any Mallows model cannot perfectly capture / model the human preference, BT model likewise. As a result, in reality, how do we choose which Mallows model to used for preference learning? Can authors comments under which situations do one model excel the other? Is there any guidline?
   • Do Mallows models always excel BT model in better capturing human preference? Can we prove it theoretically (may not be easy)? Or authors can comment whether Mallows models are more general in the sense that pairwise preference data is not necessary as in BT model etc.

   Such above insights will be more informative.

3. I appreciate the authors' responses in discussing the possible extensions regarding the practical applicability. However, extending to curriculum learning and personalized alignment will be far beyond the scope of the current manuscript. What I wonder is that, in practice,

   • Whether the proposed method can be easily implemented as DPO? From authors' response to other reviewers, if MallowsPO is equivalent to DPO+disparity, does it suggest that only small modifications are expected?
   • From Fig. 9, it seems MallowsPO can converge faster than DPO. Can authors comment whether this computational benefit can be always achieved? If yes, can we prove the faster convergence can be indeed achieved?
   • Compared to DPO, does MallowsPO brings significantly extra computation overhead in order to compute the dispersion? Empirical comparisons in terms of computation will be benefitial.

We sincerely thank you for the further question and discussions.

We have shown in Corollary 2 that when the discrepancy function $d(\cdot,\cdot)$ is chosen as Spearman’s rho (i.e. Mallows-$\theta$ model), the link function is $g(y)=\sigma(-2\log\phi(x)\cdot y),$ which is a scaled sigmoid function also defined in Equation (8). In Corollary 3, when $d(\cdot,\cdot)$ is chosen as Kendall’s tau (i.e. Mallows-$\phi$ model), the link function $g(\cdot)$ is a bit more complicated, which is defined after Equation (9).

Back to reviewer's general question about Thm1, as far as we know, there is not a simple closed-form mapping from any discrepancy function $d(\cdot,\cdot)$ to a link function in the Mallows setting. For some discrepancy functions $d(\cdot,\cdot)$, it's possible that corresponding link function that satisfies the condition we propose in Thm 1 may not exist. Fortunately, for Mallows-$\theta$ and Mallows-$\phi$, such link functions do exist and yield a closed form. Thus we do not required extra conditions on link functions like convexity.

Out of the scope of the project, in general any increasing link function that takes value in (0,1) should work for practical optimization, which is satisfied by sigmoid (DPO), scaled sigmoid (Mallows-$\theta$) and the link function of Mallows-$\phi$.

Though our Thm1 does not need extra properties of the link function, we also would like to mention that if we need the model to have additional "nice" properties, it may require log-concavity, see e.g. Tang 2020 which discussed the MLE estimation of the binary response/preference models. This indeed follows your intuition that better properties of link functions like convexity could yield theoretical benefits (also possibly in practice for optimization). We will add this comment in a future version of the paper. Thanks for the enlightening question.

We sincerely thank you for your active engagement during the rebuttal period, and appreciate for your emphasizing that our paper is interesting and of practical interest to apply Mallow’s theory in addressing the problem of preference dispersion.

Our work is the first to consider the ranking model beyond BT setting which already shows clear empirical gain, and we believe that can also entrigger a line of future research to further utilize the concept of dispersion (including directions like personal alignment and curriculum learning we mentioned earlier). We agree with the reviewer that deeper insights can benefit our paper. Given that our rebuttal answers are rather long (we try to be as detailed as our best), we want to summarize here (also highlight) as a friendly reminder that in our revision:

• We have included the insights on whether different Mallows model are consistent (raised by the reviewer's first response) detailedly with a concrete example in Appendix C.4.

• For insights on difference between BT and Mallows, we emphasized in our earlier response and discussed throughout our paper that DPO can be treated as a special instance for MallowsPO-$\theta$ by treating dispersion index as a constant, which is the primary interest of this paper for LLM alignment/preference optimization. Thus Mallows model (which yields MallowsPO) indeed `exceeds' BT (which yields DPO) as it effectively channels dispersion information to improve modeling. As a remark, however, we tend not to make claims on Mallows exceeding BT from a pure ranking model perspective, as they represent fundamentally different approaches to modeling rankings, and their comparative merits have been debated for decades. Additionally, the complexity of human preferences further complicates such comparisons. Moreover, such a comparison lies outside the primary focus of our work. Nevertheless, we are willing to include more theoretically rigorous arguments and discussions of comparing ranking models that already appear in the literature in the camera ready version if accepted.

We sincerely hope that our summarization of answers and revision could help for your final assessment, and thanks again for your efforts on improving our manuscript.

Dear Reviewer Nu5X,

We sincerely appreciate your efforts in improving our manuscript and active discussion with us. Since the discussion period will end within hours, we would sincerely appreciate it if you could kindly consider adjusting your scores if we have addressed your concerns. Thanks again!

We sincerely thank you for your thoughtful feedback. We are glad you found our work clear, well-structured, and a meaningful contribution to LLM fine-tuning. Your recognition of the MallowsPO framework, its theoretical and empirical advancements is greatly appreciated.