Thank you for taking the time to review and provide feedback on our work! We are glad to address your questions and provide further clarification on our research:

---

Weakness 1: LD-DPO is just a code-level heuristic method.

Thank you very much for recognizing our analysis of the DPO length sensitivity issue and the questions raised about the design of the LD-DPO method, which we will explain below.

• Our theoretical analysis indicates that the length sensitivity of DPO stems from the issue of the $\log\sigma(r_w-r_l)$ function, which is utilized in DPO as a means of fitting preference distribution. Theoretically, we could significantly modify the fitting function to avert the length sensitivity. In fact, there are many methods available using other functions such as RRHF, KTO, IPO. However, their overall performance is less satisfactory compared to that of DPO (refer to [1]). Hence, we assert that the $\log\sigma(r_w-r_l)$ function is still the optimal fitting function at present.
• Early in our experimental process, we contemplated a formal fine-tuning of the $\log\sigma(r_w-r_l)$ function with the aim of alleviating the length sensitivity of the DPO, but the outcomes did not prove to be satisfactory. In fact, both R-DPO[2] and SimPO[1] hope to alleviate this problem by fine-tuning the form. However, through a theoretical derivation similar to Eq.5-Eq.7, it can be observed that incorporating a constant term related to the length of the preferred data pair has no impact on the relationship between the gradients. Since the loss function will continue to assume the form of the following formula, the conclusion drawn from Equation 7 still remains valid. Therefore, we have abandoned this idea. $\mathcal{L}=-log\sigma(\beta\log\frac{\pi(y_w|x)}{K_1}-\beta\log\frac{\pi(y_l|x)}{K_2})$
• In fact, LD-DPO is a concise and efficient approach, which addresses the root cause underlying DPO's length sensitivity, corrects the flaws present in the original DPO algorithm, and attains good outcomes.

---

Weakness 2: The description for Eq.10 is not rigorous.

Thank you very much for your valuable question and we will explain Eq.10 further.

• The aim of preference optimization is to enable the model to learn human preferences within the data. Whether the preferences are labeled manually or by an LLM, length preference is an aspect of these preferences. What this paper intends to mitigate is the significant sensitivity issue of DPOs with respect to length preferences. Consequently, it is desirable to separate the length factor from the modeling process.
• After decoupling the length preference, the likelihood can then be naturally split into two parts, namely the length preference part and the other part (which, within the context of this paper, represents the human-like preferences part). Viewed from another angle, both the variation in token content and the quantity of tokens will exert an influence on the likelihood. Consequently, Eq.10 serves as an intuitive mathematical model for us.

---

Overall, LD-DPO can be seen as an enhancement of the DPO algorithm. Currently, the DPO algorithm has gained extensive acceptance among researchers and is widely utilized in the alignment phase of Large Language Models (LLMs). Nevertheless, numerous researchers[2-4] have noticed that LLMs after undergoing DPO processing tend to have the issue of output redundancy, and several solutions have been put forward[1-2]. However, the current research on this particular problem remains unclear, which has led to these proposed methods failing to achieve satisfactory outcomes.

Our work alleviates this problem through conducting a profound theoretical analysis of DPO. We contend that DPO is afflicted with a length sensitivity issue and, based on this, we have devised a concise and efficient LD-DPO algorithm. As shown in the paper, LD-DPO demonstrates outstanding results across multiple benchmarks. It mitigates the problem of redundant replies in post-DPO LLMs and attains a better alignment with human preferences.

---

Thank you again for taking the time to review and provide valuable feedback on our work, and we hope that our exlanation could address your concern!

[1] SimPO: Simple Preference Optimization with a Reference-Free Reward

[2] Disentangling length from quality in direct preference optimization

[3] Rethinking LLM-based Preference Evaluation

[4] Post-hoc reward calibration: A case study on length bias

Thanks for the response. I still have the following concerns:

About response to weakness 2

This still does not answer why the decoupling defined in Eq.10 can effectively decouple the "human-like" part and the "verbosity" part. Why could it not be an addition instead of multiplication? My point is that the formulation and "human-like""verbosity" descriptions are not rigorous, especially when you need prior knowledge for $\alpha$ to determine how much of "human-like" part preference there is.

About response to weakness 1

It seems that your point is that the methods with better theoretical guarantees often result in unsatisfactory empirical performance. That's why you propose this simple and better-performed approach.

I have an idea, to desensitize length, why not just average the likelihood of response by the length like the formulation below? $\hat{\pi}=\pi^{-l}(y|x)$ Just taking the power of negative length $-l$ could result in a strictly length-desensitized likelihood, which is much more concise and intuitive than your proposed formulation (Eq.11). And unlike your formulation, this formulation will not result in information loss for the tokens exceeding the public length.

Thank you very much for your response, we consider that a further explanation is necessary.

• In fact, the method in [1] takes the power of $\frac{1}{l}$, just using a different form like this: $\log \pi^{\frac{1}{l}}(y|x) = \frac{1}{l}\log \pi(y|x)$
• Our scenario is to improve the comprehensive conversational ability of LLMs, and the preference optimization dataset we chose is UltraFeedback[2], which is designed to align with human preferences and improve the helpfulness of the model. This dataset is representative of the alignment phase and matches the actual scenario, which is enough to prove the effectiveness of the method.
• We cite the theoretical approach in [3] with partial corrections. However, our contribution is to give a complete proof of the DPO length sensitivity, to explain theoretically this problem recognized by researchers, and to propose a efficient method LD-DPO. These are not mentioned in [3].

---

Thank you very much for taking the time to engage in a discussion with us, and we hope that our response will give you further insight into the motivations and contributions for our work.

[1] The Hitchhiker’s Guide to Human Alignment with *PO

[2] UltraFeedback: Boosting Language Models with Scaled AI Feedback

[3] Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective

Thank you for your reply, we did have a problem with the last reply.

We reread the paper [1], and its methodology is actually as in Table 2 like this: $\log \pi^{\frac{1}{l}}(y|x) = \frac{1}{l}\log \pi(y|x)$ As for your pointing out that Eq.1 is actually the author comparing it to the $\gamma$ in SimPO[2].

This does differ from your idea, but there may be some similarities. Since both $-l$ and $\frac{1}{l}$ are exponential weightings that tend to have a trend opposite to the length of the data. However, $-l$ may lead to a change in the sign of the optimization objective, and we will conduct experiments related to your idea to verify its validity.

We truly appreciate you taking the time to discuss our work with us and provide many valuable comments！

[1] The Hitchhiker’s Guide to Human Alignment with *PO

[2] SimPO: Simple Preference Optimization with a Reference-Free Reward

Thanks for your time in reviewing and providing feedback for our work! We are eager to further elaborate on our motivations and address your questions:

---

Weakness 1: Redundant symbol definitions.

Response: Thank you for pointing out our problem with the definition of symbols, which we will explain further：

• In fact, by defining $\mathcal{X}_1, \mathcal{X}_2, \mathcal{K}_1, \mathcal{K}_2$, we hope to enhance the simplicity of the subsequent formulas and thus improve the readers’ reading experience. Otherwise, the formulas will look very cluttered as shown below: $\mathcal{L}=-log(\frac{(\pi_r(y_l|x)\pi_t(y_w|x))^\beta}{(\pi_r(y_l|x)\pi_t(y_w|x))^\beta+(\pi_r(y_w|x)\pi_t(y_l|x))^\beta}).$

---

Weakness 2: The color scheme and meaning of Fig.3 is unclear.

Response: Thank you very much for pointing out the problems we had with drawing the image. It was indeed our mistake, we will redraw Fig.3 and the content description in the following link: http://gxwhy.net/ads/Fig3.pdf, and subsequently update them in the paper.

---

Weakness 3: Some spelling and grammatical mistakes.

Response: Thank you for pointing out some of the spelling and grammatical issues in the article. We will review the entire article to correct the original issues and apologize for any reading troubles.

---

Question 1: Generalization of analytical methods to the RLHF-base approach.

Response: We are grateful for your valuable questions regarding our analytical methods! We will provide a more detailed elaboration based on your questions.

• As you mentioned, our analysis is applicable to most DPO-based methods, like SimPO, R-DPO, etc. This is because these methods need $y_w$ and $y_l$ as preference data pair, and fitting implicit rewards based on model-predicted probabilities can result in length sensitivity.
• For RLHF-based methods, we review the unified optimization paradigm for RLHF, where $\pi_a$ is actor model and $\pi_r$ is reference model: $J_r(\pi_a)=\mathbb{E}_D[r(x,y)-\beta\log\frac{\pi_a(y|x)}{\pi_r(y|x)}]$ Intuitively, the gradient of the RLHF-based methods depends on the reward $r(x,y)$ and the KL divergence term, compared with the former, the latter has a negligible correlation with the data length.
• As described in numerous papers[1-3], the length bias issue of the RLHF-based approach arises from the length preference of the reward model itself, which is different from the DPO-based methods described above. Regarding this problem, we believe that a correction to the training process can be achieved by adding a length-related penalty term to the reward part. (There is already relevant work concerning this approach [3-4])

Overall, our method is indeed more suitable for achieving length desensitization in DPO-based methods. For RLHF-based methods, penalizing the length of the reward component might be simpler and more effective.

---

Question 2: Comparison of LD-DPO and DPO performance.

Response: We are grateful for your careful reading of the experimental analysis section in the article as well as your meaningful questions! We will explain the performance issues of LD-DPO in detail.

• In Section 3.1 of the article, we conduct a detailed analysis of the length sensitivity of the DPO. It seems that DPO has no information loss. However, the severe length sensitivity of the optimization objective causes its mathematical modeling to fail to effectively represent the real information.
• By remodelling the likelihood, LD - DPO alleviates the impact of length. It seemingly discards certain information, thus enabling the optimization process to proceed in a more accurate direction.
• It is an undeniable fact that the performance of LD-DPO deteriorates when the value of $\alpha$ is chosen close to 0. This is because, in such a case, it actually discards some of the information within the data.

Overall, LD-DPO can be regarded as a trade-off between an "inaccurate optimization direction(DPO)" and the "loss of some information($\alpha=0$)", neither of which is advantageous. This explains the inverted U-shape of the performance curves shown in Fig.4. Consequently, LD-DPO surpasses DPO on numerous benchmarks. Moreover, you can also refer to the case study in Appendix D.

---

Question 3: Hyperparameter sensitivity of LD-DPO.

Response: Thank you very much for your valuable question and we will explain further about the hyperparameter sensitivity of the LD-DPO.

• Undeniably, the performance of LD-DPO is indeed influenced by the hyperparameter $\alpha$, presenting an overall inverted - U shape across many benchmarks..
• To be honest, our experiments revealed that for the same model, making an arbitrart selection of $\alpha$ across a wide span(for instance, in the case of Llama3-8B-Instruct, within in range of [0.3, 0.7]) almost invariably leads to better performance than that of DPO. This indicates that LD-DPO has good robustness in performance with respect to the $\alpha$
• Our analysis of the various capability models in Section 5.2 shows that the length sensitivity of DPO is relevant to the model's capabilities. Researchers can select the appropriate $\alpha$ based on the model’s capabilities. For example, $\alpha$ near 0.6 can be chosen for a recent 7B LLM, and $\alpha$ near 0.8 can be chosen for a recent 70B LLM.

Overall, LD-DPO is indeed sensitive to $\alpha$. However, it also provides a wider range of options at the same time. Moveover, we will provide the recommended intervals for the different capability models currently available in a subsequent version of the paper.

---

Thank you again for your valuable comments on our work, and we hope that our exlanation could address your concern!

---

[1] Disentangling length from quality in direct preference optimization

[2] Rethinking LLM-based Preference Evaluation

[3] Post-hoc reward calibration: A case study on length bias

[4] Improving alignment of dialogue agents via targeted human judgements

We want to sincerely thank the reviewers for their time and effort in evaluating our paper. We would appreciate it if you could kindly confirm that the rebuttal was received and let us know if any additional steps or clarifications are required from our side. Your feedback is highly important to us, and we remain available to address any further concerns or questions.

Please let us know. Thanks.

Weakness 6: Hyperparameter sensitivity of LD-DPO.

Response: Thank you very much for your valuable question and we will explain further about the hyperparameter sensitivity of the LD-DPO.

• Undeniably, the performance of LD-DPO is indeed influenced by the hyperparameter $\alpha$, presenting an overall inverted - U shape across many benchmarks..
• To be honest, our experiments revealed that for the same model, making an arbitrart selection of $\alpha$ across a wide span(for instance, in the case of Llama3-8B-Instruct, within in range of [0.3, 0.7]) almost invariably leads to better performance than that of DPO. This indicates that LD-DPO has good robustness in performance with respect to the $\alpha$
• Our analysis of the various capability models in Section 5.2 shows that the length sensitivity of DPO is relevant to the model's capabilities. Researchers can select the appropriate $\alpha$ based on the model’s capabilities. For example, $\alpha$ near 0.6 can be chosen for a recent 7B LLM, and $\alpha$ near 0.8 can be chosen for a recent 70B LLM.

Overall, LD-DPO is indeed sensitive to $\alpha$. However, it also provides a wider range of options at the same time. Moveover, we will provide the recommended intervals for the different capability models currently available in a subsequent version of the paper.

---

Weakness 7: The color scheme of Figure 3 is inappropriate.

Response: Thank you very much for pointing out the problems we had with drawing the image. It was indeed our mistake, we will redraw Figure 3 and the content description in the following link: http://gxwhy.net/ads/Fig3.pdf, and subsequently update them in the paper.

---

Question 1: Derivation problem for lines 808 to 810.

Response: Thank you very much for carefully reading our Appendix section and asking valuable questions! We will provide further explanations.

In fact, we are solving for the partial derivatives in Eq.18-Eq.21 for the actual values, while lines 808-810 are analyzed for the absolute value of the gradient. Since the result in Eq.15 is negative, the increasing or decreasing trend may change when the absolute value is taken. You can re-check our content and please keep correcting us if you still have doubts.

---

Thank you again for your valuable comments on our work, and we hope that our exlanation could address your concern!

---

[1] Towards a unified view of preference learning for large language models: A survey

[2] SimPO: Simple Preference Optimization with a Reference-Free Reward

[3] Towards analyzing and understanding the limitations of dpo: A theoretical perspective

[4] Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators

Thank you for taking the time to review and provide feedback on our work! We are glad to address your questions and provide further clarification on our research:

---

Weakness 1: Eq.4 to Eq.5 is not rigorous.

Response: Thank you very much for pointing out our problems with formula writing, we will explain further:

• We apologize for lacking a more detailed explanation of the process from Eq.4 to Eq.5. We will add an explanation similar to “Assuming that only $y_w$ and $y_l$ are used to approximate the above expectation in the case of identically distributed data, gives us the following empirical estimation, ...” in the next version of the article to clarify why the expectation sign was omitted. Indeed, numerous related papers, such as [1-3], adopt this approach as it contributes to a more concise presentation of the paper.

---

Weakness 2: Lines 211 to 215 is vague and overly intuitive.

Response: Thank you very much for your valuable suggestions on our rationale section. We will re-explain it.

1. Through the analysis within the paper, we have reached the conclusion in Eq.7 that the absolute magnitude of the gradient in the two optimization directions of the DPO depend on the predicted probabilities of $y_w$ and $y_l$ by the actor model.
2. The DPO algorithm uses sentence-level predicted probability, calculated as: $\pi(y|x)=\prod_{i=1}^{len(y)}p(y_i|x,y_{<i})$ where $y_i$ is the i-th token in $y$ and $y \in \{y_w, y_l\}$. Furthermore, since $p(y_i|x,y_{<i})\in[0,1]$, the large probability that $\pi(y|x)$ is smaller when $y$ is longer, which is indeed intuitive, and we give a sideways proof of this in Figure 2.
3. Combining the analysis in 2 with Figure 2, we can conclude that in the general case:
   1. When $y_w$ is longer than $y_l$, the DPO optimization objective has a larger gradient in the $y_w$ direction.
   2. When $y_l$ is longer than $y_w$, the DPO optimization objective has a larger gradient in the $y_l$ direction.
4. In lines 214 to 215, we hope to explain why this problem causes lengthy output from the post-DPO model:
   1. When DPO goes to increase the probability of a longer $y_w$, which is a directed optimization, the length increase is obvious.
   2. When DPO goes to decrease the probability of a longer $y_l$, which is not a directed optimization. We don't know if the output will be shorter, but it is a fact that there is a missed opportunity to directionally optimize for shorter $y_w$.

The above content is what the text intends to convey in lines 211 to 215, with an emphasis on the causes of length sensitivity of DPO and its resulting impact. We hope that our explanation can address your concern!

---

Weakness 3: Motivation of Eq.7 is unclear.

Response: Thank you for your valuable question and we will explain the motivation for Eq.7.

Obviously, to reduce the loss of DPO, the model can either choose to increase $\pi(y_w|x)$ or decrease $\pi(y_l|x)$. Therefore, the ratio of the gradient values of these two can reflect the tendency of the model in these two optimization directions as well as the influencing factors. It is worth noting that for this analytical process, we refer to [3], which is a paper analyzing DPO theory, although it does not address length sensitivity.

---

Weakness 4: "Probability bias" in line 416 is unclear.

Response: Thank you very much for pointing out our clerical error in line 416. We intended to use “probability difference” instead of what was written there. We apologize for any ambiguity this may have caused.

---

Weakness 5: The effect of the hyperparameter $\alpha$ remains unclear.

Response: Thank you for your valuable questions and we will explain further about Figure 4.

In fact, the reason for the different performance of LD-DPO on MT-Bench and AlpacaEval 2 when $\alpha=0$ comes from the different evaluation metrics.

• For MT-Bench, the evaluation metric is "score", i.e., the responses are evaluated using the judge model(such as GPT4). In this situation, when excessive information is lost during training and the output is overly short, the score is lower than that of DPO. Moreover, the judge model itself has a preference for long responses, which is one of the reasons for the significant drop in the score.
• For AlpacaEval 2, the evaluation metric is "length-controlled winrate". The existence of a length-related penalty factor within the computational equation will, to a certain extent, counterbalance the length preference of the judge model. Consequently, when $\alpha=0$(i.e., when the answer length is at its shortest), LD-DPO will also exhibit good performance on AlpacaEval 2. You can refer to [4] for more detailed information about AlpacaEval 2.

We want to sincerely thank the reviewers for their time and effort in evaluating our paper. We would appreciate it if you could kindly confirm that the rebuttal was received and let us know if any additional steps or clarifications are required from our side. Your feedback is highly important to us, and we remain available to address any further concerns or questions.

Please let us know. Thanks.

Thanks for the response,

I've read them all. I encourage authors to revise their manuscripts directly. For me, the most interesting part of this submission is the analysis of equations 5-7. Reviewer Jjwf pointed out that these analyses heavily rely on the paper [1]. This raises concerns about the novelty of the submission. I decided to keep the current rating.

[1] Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective