Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints

We deeply appreciate the reviewer for the thoughtful comments! You can find our responses to each question below.

1. Re weakness 1 (novelty):

Thank you for the comments. In our work, we build upon the formulation introduced in the DPO paper, which analytically links the reward function to the policy. A critical aspect to note is that the DPO's derivation, while comprehensive, only facilitates a closed-form mapping between the policy and the reward model for reverse KL divergence. This limitation highlights the need for a more nuanced analysis, particularly when considering other types of divergences. Extending from reverse KL to other divergences is not merely a matter of substituting the divergence constraint, as the original derivation of DPO will not lead to a closed form mapping for other divergences.

Our work steps in to fill this gap by meticulously examining the KKT conditions, rather than directly following the DPO's derivation. This approach has enabled us to demonstrate that a closed-form mapping between the policy and reward model is indeed possible for a variety of commonly used divergences. This finding represents an important and non-trivial advancement beyond the original scope of the DPO paper.

To further address the reviewer's concern about the novelty of our method: While it is true that our work extends the concept of reverse KL regularization to include a wider array of divergences, the implications of this extension are profound. The inclusion of multiple divergences not only broadens the applicability of the model but also introduces extra flexibility and adaptability in policy optimization. This improvement is useful in scenarios where different divergences may yield more optimal results, depending on the specific characteristics and requirements of the task at hand. Therefore, our contribution should be viewed not merely as an extension, but as an enrichment of the existing DPO framework.

2. Re weakness 2 (theoretical findings):

Thank you for the suggestion on improving the theoretical findings! We recognize the importance of emphasizing the novel aspects of our theoretical contributions. While our derivations are built on the foundation laid by DPO, our approach differs in several aspects. Primarily, our method extends the applicability of closed-form mappings to a broader range of divergences, not just the reverse KL divergence. This extension is non-trivial and addresses a significant gap in the DPO paper. Furthermore, our work introduces a detailed exploration of the relationship between various f-divergences and the model's calibration error as well as a theoretical bound. This aspect of our research explains why divergence efficient algorithms should be favored.

Regarding the transition from RKL to f-divergences and the use of the Lagrange multiplier in our derivations, we would like to note that these changes are not simply replacements of divergences. Merely applying the derivation from the original DPO paper does not yield a closed-form solution for the functional mapping between policy and reward. Therefore, we respectfully disagree with the reviewer that the derivations are mainly from the DPO paper.

3. Re Q1 (assumptions and limitations):

Thank you for the good question. In our f-DPO framework, the key assumptions are relatively minimal and generally satisfied in standard language models. The reviewer can also find those assumptions in section 4, page 6. We restate the assumptions below. Firstly, we assume that the base language model assigns a non-zero prediction probability to all tokens, a condition typically met by models employing softmax functions. This ensures that our framework is applicable to a wide range of existing language models.

Regarding the divergence function $f(\cdot)$ used in $f$-DPO, we require three main conditions:

• 1. $0 \notin \text{dom}(f')$ (zero is not in the domain of the derivative of $f$ );
• 2. $f(1) = 0$ (the function evaluates to zero at 1), and that
• 3. $f$ is strictly convex around 1.

These conditions are met by several commonly used divergences, including $\alpha$-divergence, Reverse KL Divergence (RKL), Forward KL Divergence (FKL), and Jensen-Shannon Divergence (JSD). Thus, our model is adaptable to a variety of divergence choices.

However, it's crucial to also consider the limitations and their implications for $f$-DPO's applicability. The constraints on the divergence function $f$ may limit the selection of divergences to those that conform to these specific conditions, potentially excluding some divergences (e.g., total variation distance, Chi-squared distance) that might offer unique benefits in certain applications.

4. Re Q2 (tuning $\beta$):

This is a really good observation! Adjusting the $\beta$ hyperparameter can influence the diversity, but it will also influences the regularization of divergence from the base model! A higher value of beta can enhance generation diversity, but may lead to poor performance on the downstream task. On the other hand, using different divergence measures allows for a more nuanced control over this balance. Different divergences can offer alternative ways of quantifying and constraining the deviation, potentially allowing for increased diversity in outputs while still maintaining closer alignment with the original model's accuracy and reliability. This approach can provide a more flexible and targeted method to balance accuracy and diversity, compared to solely adjusting the beta hyperparameter.

In essence, the advantage of using different divergences lies in their ability to provide a more tailored approach to balancing model accuracy with output diversity, potentially leading to more effective and controlled modifications than simply tuning the beta value.

Once again, we appreciate your time and constructive feedback. If further questions or concerns arise, please don't hesitate to reach out. Thank you!

We deeply appreciate the reviewer for the thoughtful comments! You can find our responses to each question below.

1. Re weakness and questions:

Thank you to the reviewer for bringing up an important topic!

In our work, both $f$-PPO and $f$-DPO optimize the policy under the same objective, as stated in Section 4, Page 5. The primary difference lies in their methodologies.

• PPO adopts a two-stage method: it initially learns a reward function from offline data and then optimizes the policy using policy gradient techniques under the learned reward model.
• In contrast, DPO establishes a closed-form functional relationship between the policy and the reward function, allowing us to optimize the policy in a supervised manner using offline data.

Theoretically, both methods should converge to the same solution. However, in practice, due to the inherent instability of RL training – as also pointed out in the original DPO paper – PPO tends to underperform compared to DPO. In our experiments, we observe that incorporating the divergence penalty into the reward function exacerbates this instability. The varying value of the divergence, particularly for forward KL divergence, complicates the learning of the value function and makes gradient estimation more error-prone. Thus, the enhanced performance of f-DPO is largely attributable to its more stable training strategy.

Additionally, as noted by Gao et al. (2023), reward hacking presents a significant challenge in RLHF, often due to imperfect proxies or limited offline data. While offline RL shows promise in addressing reward hacking in policy optimization, both PPO and $f$-DPO, despite utilizing offline data, do not effectively handle this issue. We typically resort to early stopping as a mitigation strategy. Therefore, we believe that incorporating design principles from offline RL algorithms, such as CQL or IQL, could improve the DPO framework by preventing reward hacking in a more principled way. In this context, offline RL algorithms could be seen as complementary to both PPO and $f$-DPO. However, we do agree with the reviewer that adding more baseline will further strengthen the findings of our work. In the future, we will consider incorporating design principles from offline RL (e.g., pessimism) into the $f$-DPO framework to further enhance its robustness against reward hacking.

Once again, we appreciate your time and constructive feedback. If further questions or concerns arise, please don't hesitate to reach out. Thank you!

References:

Gao, Leo, John Schulman, and Jacob Hilton. "Scaling laws for reward model overoptimization." In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.

Many thanks to the reviewer for the thoughtful observations and comments! Our detailed responses to the questions can be found below.

1. Re Q1 and Weakness:

Thanks for the great question. In response to the question on selecting the appropriate f-divergence for f-dpo, we suggest initially considering the Jensen-Shannon divergence. Our recommendation is based on its ability to yield more diverse responses compared to the commonly used reverse KL divergence. This preference is supported by our findings, where JS divergence can result in a more diversified response as measured by different diversity metrics. Additionally, despite underperforming in reverse KL in reward metrics, it was favored in the GPT-4 evaluation, as illustrated in Figure 7 of the appendix. This observation also aligns with recent studies, such as Sun et al., 2023.

Furthermore, to accommodate diverse application needs, another strategy could involve training multiple models, each employing a different divergence, such as JS divergence, alpha divergence (with alpha values ranging from 0.1 to 0.9), and forward KL divergence. This approach allows users to experiment and choose the divergence that best aligns with their specific requirements or preferences.

Additionally, while our current framework does not optimize f-divergence simultaneously with the policy, this poses an intriguing area for future research. Incorporating simultaneous optimization could potentially further enhance the adaptability and effectiveness of the framework in various real-world scenarios.

To address your concerns, we’ve also added the above discussions in our revised paper. You can find them in the Section C of Appendix. The revisions are marked in red color.

2. Re Q2 (RKL vs FKL):

The distinction between Reverse KL Divergence (RKL) and Forward KL Divergence (FKL) primarily lies in their inherent properties: RKL is mode-seeking, while FKL is mass-covering. This can be discerned from their mathematical formulations. RKL is defined as $E_q[\log(q(x)/p(x))]$, where $q(x)$ is the distribution being optimized, typically representing the language model undergoing fine-tuning. In the RKL scenario, $q(x)$ may assign zero probability to values where $p(x) > 0$. In contrast, FKL is defined as $E_p[\log(p(x)/q(x))]$, which necessitates that $q(x)$ assigns a non-zero probability to all values where $p(x) > 0$. This requirement inherently encourages FKL to promote a distribution $q(x)$ that covers the entire range of $p(x)$, thereby enhancing diversity.

As for the reviewer's inquiry regarding the impact of RKL and FKL on diversity, the observed differences can be attributed to these underlying properties. RKL's mode-seeking behavior tends to focus on specific peaks in the probability distribution, potentially leading to less diverse outcomes. Conversely, FKL's mass-covering nature ensures a broader coverage of the probability landscape, contributing to greater diversity in the results. This theoretical perspective, supplemented by empirical evidence as shown in our experiments, underscores the importance of carefully selecting the appropriate f-divergence based on the desired characteristics of the language model.

The above discussions are included in Section D of appendix in our revised version, please feel free to check them out!

Once again, we appreciate your time and constructive feedback. If further questions or concerns arise, please don't hesitate to reach out. Thank you!

References:

Sun, Simeng, Dhawal Gupta, and Mohit Iyyer. "Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHF." arXiv preprint arXiv:2309.09055(2023).

We appreciate the reviewer for the thoughtful comments!

1. Re weakness and questions:

We agree that incorporating experiments with larger models, such as LLaMA 7B, would indeed strengthen our work. We chose the Pythia 2.8B model in alignment with the DPO paper. Should resources allow, we plan to rerun the experiments using the LLaMA 7B model and include the results in the final version!

Thank you again for your valuable time and consideration. Should you have any additional concerns or questions, kindly let us know. Thank you!