Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language Models

Thank you for your review and feedback on our paper. We address your points below.

1. Performance concerns: The reviewer expressed concern that DDRO does not demonstrate compelling advantages over existing models, potentially limiting its practical applicability. Regarding the magnitude of improvement, it's important to evaluate this within the context of single-iteration preference optimization. We believe that achieving dramatic performance gains from just one optimization step is challenging. In fact, this tendency for incremental progress is common in the field (see works such as ORPO[1], CPO[2], and SPPO[3], which often report similar levels of improvement after one optimization step).

   Considering this background – where substantial leaps are not always expected from a single iteration – we believe DDRO's results are noteworthy. We highlight three key points demonstrating its effectiveness:

   • Broad Consistency: DDRO achieves the best performance on the majority of benchmarks and model sizes tested. This result could indicate that explicitly modeling preferences causes a failure to model the preferences necessary for many benchmarks, limiting good performance only to those benchmarks where the model luckily matches the required preferences.

   • Significant Gains on Specific Tasks: On specific benchmarks like GSM8K with Llama-7b, DDRO shows substantial improvements over KTO (over 50% relative gain) and BCO (over 20% relative gain), which we believe are non-trivial, especially in the context where large gaps are not always expected after one iteration of alignment.

   • Effectiveness in Paired Settings: DDRO achieves performance better than DPO in the paired setting, despite discarding the comparison information that DPO utilizes. This underscores the effectiveness of DDRO.

   References:

   • [1] Hong, J., Lee, N., & Thorne, J. (2024). ORPO: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691.
   • [2] Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Van Durme, B., ... & Kim, Y. J. (2024). Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. arXiv preprint arXiv:2401.08417.
   • [3] Wu, Y., Sun, Z., Yuan, H., Ji, K., Yang, Y., & Gu, Q. (2024). Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675.

2. Proofreading errors: Thank you for pointing out the specific typo ("he" -> "the" in Line 165) and the broken MMLU citation (Line 358). We apologize for these oversights. We will meticulously proofread the manuscript, correcting these and any other errors, before submitting the camera-ready version.

We appreciate your feedback and hope these clarifications address your concerns about DDRO's performance and practical relevance.

Thank you for reviewing our paper and providing valuable feedback. We address your points below.

1. Limited improvement and average results: The reviewer observed that the empirical improvement of DDRO over KTO seems limited and suggested providing average results across datasets. Regarding the magnitude of improvement, we would like to place this in the context of one-iteration preference optimization. It is generally understood that achieving dramatically large performance gains from a single iteration of alignment can be challenging. This characteristic of incremental progress is commonly observed in the field, as evidenced by recent works like ORPO, CPO, and SPPO, which also often report performance improvements of a similar nature after one optimization step. Viewed within this context – where substantial leaps are not always expected from one iteration – we believe DDRO demonstrates notable effectiveness. We wish to highlight three key points:

   Consistency: DDRO achieves the best performance on the majority of benchmarks and model sizes tested.

   Significant Gains: On specific benchmarks like GSM8K with Llama-7b, DDRO shows substantial improvements over KTO (over 50% relative gain) and BCO (over 20% relative gain), which we believe are non-trivial, especially in the context where large gaps are not always expected after one iteration of alignment.

   Paired Setting: DDRO achieves performance better than DPO in the paired setting, despite discarding the comparison information that DPO utilizes. This underscores the effectiveness of DDRO.

   Regarding averaging results: While technically feasible, averaging scores across different metrics and scales can be misleading. Metrics with smaller variance or lower chance rates might be unjustly de-emphasized. We believe presenting the full suite of results, as in Figure 1, provides a more transparent and informative picture of DDRO's broad effectiveness, rather than potentially obscuring performance variations through averaging.

2. Comparison with GSIL [1]: Thank you for your comment regarding GSIL. While both GSIL and our method use density ratios, their objectives and approaches are fundamentally different. First, construction of the objective is different. GSIL aims to minimize $\mathrm{KL}(\pi_{\text{data}}) \| (\pi_\theta)$, effectively making the density ratio $(\pi_{data} / \pi_\theta)$ approach 1. Our method, however, aims to match the policy density ratio $(p_\theta / p_{ref})$ to the true density ratio $(p_+ / p_{ref})$. Thus, our core objectives differ significantly. Second, GSIL requires multiple iterations to estimate $\pi_{\text{data}}$, and its theoretical convergence is not guaranteed. In contrast, our method runs a single iteration, offering advantages in efficiency and theoretical tractability. We hope this clarifies the distinction.

3. Comparison with state-of-the-art methods like SPPO: We appreciate the suggestion to compare against state-of-the-art methods. However, our work primarily focuses on the setting where offline, binary feedback labels are available (i.e., unpaired preference data). Methods like SPPO operate under a different, richer setting. SPPO typically requires access to a pre-trained reward model, allowing it to use continuous reward scores, and often involves online interaction or scoring during training. Comparing DDRO directly with methods designed for such richer information settings might not be entirely appropriate, as the available data and feedback mechanisms are fundamentally different. Our goal was to develop a theoretically sound and practically effective method specifically for the common scenario of offline binary feedback, where methods like SPPO might not be directly applicable or would require significant adaptation.

Thank you again for your comments, which encourage us to clarify the positioning and strengths of DDRO.

Thank you for your insightful comments and questions regarding our work on DDRO. We appreciate the careful reading and valuable feedback.

1. Handling neutral data: The reviewer asked if DDRO can handle "neutral" labels, which might arise when annotators evaluate single responses without direct comparison. Currently, DDRO is formulated to utilize binary preference signals (preferred/unpreferred) and it cannot directly incorporate a distinct "neutral" category. However, we believe that binary feedback collection is a prevalent and practical scenario. Major LLM services like Gemini, Claude, and ChatGPT often employ simple thumbs-up/thumbs-down mechanisms to collect data, which correspond directly to the binary setting DDRO addresses. Therefore, we consider the setting DDRO operates in to be sufficiently realistic and widely applicable.

2. Determining $p(+|x)$ and robustness: The reviewer inquired about determining the constant $p(+|x)$ and DDRO's robustness to its value. We assumed $p(+|x)$ to be constant (denoted as $t$) for simplicity in our analysis. This constant $t$ can be empirically estimated by sampling responses from the reference policy $p_{\text{ref}}$ for a set of prompts $x$ and calculating the proportion of responses deemed "preferred". However, our empirical investigations (which we can add details of in the final version) showed that the training dynamics and final performance of DDRO were not highly sensitive to the specific value chosen for $t$. Using a standard default value like $t = 0.5$ generally works well in practice, suggesting that precise estimation of $p(+|x)$ is often unnecessary.

3. Scenarios for $g_\theta$ misspecification and mitigation: The reviewer asked for intuitive examples where the learned density ratio model $g_\theta$ might be misspecified, even when initialized from the reference policy. Misspecification can occur if the true preferred distribution $p^+(y|x)$ is too complex to be accurately represented by the chosen model architecture, given its parameterization (e.g., the LLM has insufficient capacity). In that case, the model might not be able to capture the true function even with infinite data. The bias term in our analysis (Section 3.2) relates directly to this discrepancy between the best possible model within the chosen hypothesis space and the true function; it depends on the model class, not the data distribution. Strategies to mitigate this bias primarily involve using model architectures known for their expressive power and ensuring the model has sufficient capacity (e.g., using larger models or architectures proven effective). Therefore, processing data (including prompt selection strategies) do not reduce the model bias, although they influence the data distribution which affects the variance term.

4. Inconsistency between Figure 1 and Section 5.2: Thank you for pointing out this discrepancy. As you mentioned, the "GSM8K" in the text describing Figure 1 should have referred to "AlpacaEval LC Winrate". We apologize for this mistake and will correct it in the camera-ready version.

5. DDRO underperformance on AlpacaEval: Following the correction above, the reviewer asked for insights into why DDRO might specifically underperform KTO on AlpacaEval LC Winrate. We hypothesize that KTO's specific preference model assumptions might align particularly well with the nature of the AlpacaEval benchmark. This could lead to strong performance on this specific benchmark, potentially at the expense of broader generalization. DDRO, lacking such model-specific biases, results in strong results across a wider array of benchmarks.

6. Error in Equation (6) and MMLU reference: Thank you for identifying the potential error in the KL term in Equation (6) and the broken MMLU reference in Section 5.1. We will carefully review Equation (6) to ensure correctness regarding the preferred/unpreferred KL terms and fix the MMLU citation. We will perform a thorough proofread to catch any similar issues.

We appreciate the reviewer's constructive feedback, which will help enhance the clarity and correctness of our paper.

Thank you for your thoughtful review and constructive comments on our paper. We appreciate the opportunity to address your concerns and clarify aspects of our work.

1. Is the density ratio $g$ an implicit preference model?

   We thank the reviewer for this insightful question. The density ratio $g = p^- / p^+$ reflects inherent preferences derived directly from the true distributions ($p^+, p^-$) without assuming a specific parametric structure (like the Bradley-Terry model used implicitly by methods like DPO or KTO). Therefore, $g$ represents the data's preference information itself, not a constructed model, allowing DDRO to avoid explicit preference modeling.

2. Performance comparison and statistical consistency: We appreciate your suggestion. However, our primary theoretical claim regarding statistical consistency focuses on the convergence target – ensuring the learned model converges to the true optimal distribution under certain conditions – rather than the rate of convergence. Our experiments aim to show DDRO achieves competitive or superior performance on various benchmarks at a fixed data size, implicitly supporting the benefit of a sound convergence target.

3. Justification for avoiding explicit preference models: We note that the very fact that our method demonstrates advantages on the majority of benchmarks, despite underperforming on a single benchmark, suggests that preference-modeling-methods may have become overly specialized to the particular types of preferences required only by certain benchmarks. Also, we believe that demonstrating the limitation of specific models (like the Bradley-Terry model assumed in DPO) does not necessarily require datasets with highly complex preferences. It is sufficient to show scenarios where preferences, even simple ones, deviate from the assumed model structure. For example, Bradley-Terry model does not capture a non-transitive relatoinship such as in rock-paper-scissors. We acknowledge that including demonstration of such cases would strengthen our argument and therefore plan to add a discussion or a simple illustrative example to the camera-ready version.

   • [1] Pan, Y., Tsang, I. W., Chen, W., Niu, G., & Sugiyama, M. (2022). Fast and robust rank aggregation against model misspecification. Journal of Machine Learning Research, 23(23), 1-35.

4. Proof for Proposition 3.3: We apologize for the omission. We had intended the paragraph preceding Proposition 3.3 to serve as an justification. We will provide a formal proof for Proposition 3.3 in the appendix of the camera-ready version.

5. Missing citations for related work: Thank you for bringing these relevant papers to our attention. We have reviewed them carefully. We acknowledge these papers implicitly involve density-ratio through regularization (like odds ratios) or KL constraints. However, these are primarily for stabilization or performance enhancement, and they do not appear to directly estimate or optimize the density ratio itself as the main target. In contrast, the central contribution of our paper is the introduction of a consistent theoretical framework built directly upon the perspective of direct density ratio matching. Therefore, while valuable contributions, we believe their direct relevance to our paper's central novelty is limited.

6. Typos and missing references: We will correct these errors and conduct a thorough proofreading using automated tools and careful manual checks to identify and fix any other potential issues in the final version.

7. Generation of negative samples in unpaired Ultrafeedback: Unpaired datasets like Ultrafeedback typically consist of triplets: a prompt, a generated response, and a binary label (e.g., "good" or "bad") assigned by human annotators. Therefore, responses labeled as "bad" constitute the set $D^-$.

8. DDRO vs. KTO on AlpacaEval LC Winrate: We interpret this result as potentially stemming from KTO's underlying preference model (based on spectrum theory) aligning particularly well with AlpacaEval benchmark. This focused alignment might come at the cost of performance on other, broader benchmarks. In contrast, DDRO, by directly learning from the preference signals without an intermediate model, avoids such model-induced biases, leading to more robust and generally strong performance across a wider range of benchmarks, as observed in our experiments (e.g., BBH, GSM8K, MMLU, TruthfulQA).

9. Potential for zero probability mass in Eq. (6): As mentioned in the paragraph just before Eq. (6), this can indeed occur and potentially lead to gradient spikes. To mitigate this, we introduced the function $S(\cdot)$ which shapes the loss landscape to be more benign.

Thank you again for your detailed feedback, which will help us improve the final version of our paper.