Data-Centric Human Preference Optimization with Rationales

Dear Reviewer EsKw,

We want to thank you for your elaborate comments and suggestions, which led to valuable extension of our work. We have carefully addressed your concerns and welcome any additional feedback. Please let us know if you have any further questions, we will be happy to address them.

Kind Regards,
Paper8553 Authors

We appreciate the references provided by the reviewer, as they represent significant contributions to the field. Our work primarily focuses on the DPO extension, highlighting its adaptability to other pairwise preference learning methods, such as ORPO, due to our formulation's ability to incorporate a rationale loss. Since RLCD shares similarities with DPO, it could potentially support contrastive prompts with the appropriate rationales. The same reasoning applies to the West-of-N method, which selects the best and worst responses from a pool for pairwise preference training. Our framework could extend this approach by including rationales for each response pair. Self-Taught Evaluators also share some similarities with our approach, as they create judgments to identify better responses. However, their judgments are typically limited to selection without providing detailed rationales. Moreover, they consider their method in the SFT setting, flattening all data into instructions, which differs from our preference learning objective. Additionally, in the synthesis generation process, they generate one general response and one clearly orthogonal response, which is not assumed in our work.

Joint probability decomposition

We do not assume the conditional independence of preference and rationale given the prompt x. Therefore, if we were to apply Bayes' rule to the joint probability in the reverse order, we would obtain: $p^*(y_w \\succ y_l, r | x) = p^*(y_w \\succ y_l |x, r) \\cdot p^*(r | x)$, which demonstrates that the preference also depends on the rationale. We choose to decompose the probability in the original way because it allows us to separate the common preference term from the rationale term, which can be easily integrated into most pairwise preference learning methods.

Information-theoretic results

Due to space constraints, we have moved our theoretical results to the Appendix and kept a general overview of our results in the main text. We apologize for any inconvenience or potential confusion this may cause. Our first result (Theorem 1) establishes the connection between the mutual information of the true preferences and rationale given the prompt, I(Z;R∣S), and the informativeness of the rationale. Specifically, this dependency indicates that as the informativeness of the rationale about the preference increases, the mutual information also increases. This suggests that higher-quality rationales can enhance the understanding of the true preferences. Our second result (Theorem 2) shows that training with rationales can reduce generalization error, especially when the rationale is useful for predicting the preference, which can eventually boost learning efficiency.

How are draws measured in Figure 2?

We use the LLM as a judge to determine which response is preferred between models. However, in cases where the judge cannot decide on a preferred response, we consider it a draw. Additionally, we shuffle the responses to prevent any ordering bias from influencing the judge’s decision.

Clarification of the drawing rate

We apologize for the confusing phrasing of our claim. In the experiment, we observed that as the data size increases, the RDPO winrate improves relative to DPO, while the draw rate remains constant. This indicates that the increase in RDPO's winrate over DPO is due to converting previously losing cases into winning ones. We hope this clarifies our claim.

Typo

Thank you for catching that.

Explanation of Figures 2 and 3

In Figure 2, we compare DPO and RDPO against the SFT model, but both DPO and RDPO involve varying data sizes. As the reviewer pointed out, DPO outperforms RDPO compared to SFT at 1K data points. In Figure 3, we fixed the DPO model at 12K data points for Ultrafeedback and varied the RDPO training data size, which may have contributed to the potential confusion. We apologize for this oversight. For the plots, we conducted the winrate evaluation three times and observed consistent scores with negligible standard deviation. We apologize for any misunderstanding.

Why focus on a small dataset size?

While training on larger datasets is possible, our primary goal is to enhance data quality for preference learning by augmenting the dataset, as an alternative to simply increasing annotated data. We demonstrate data efficiency by reducing the number of annotated pairs needed—by 2 to 4 times compared to DPO—while achieving similar performance. Although increasing annotated data could potentially improve performance, this is not guaranteed. Many popular pairwise preference datasets are similar in scale to ours, such as the OpenHermes2.5-DPO dataset (https://huggingface.co/datasets/argilla/OpenHermes2.5-dpo-binarized-alpha), the multi-turn preference dataset Capybara Preferences (https://huggingface.co/datasets/argilla/Capybara-Preferences-Filtered), and the code preference dataset CodeUltraFeedback (https://huggingface.co/datasets/coseal/CodeUltraFeedback_binarized). Regarding the Ultrafeedback dataset (original version), it appears that DPO does not significantly improve performance over SFT, although DPO still achieves a majority winrate against the SFT model. However, for the ORCA dataset, we observe a performance increase with the DPO model, as shown in Figure 2.

Choices of DPO model.

We selected the model that had converged on the winrate later in training, as it had been exposed to more preference data.

RewardBench/AlpacaEval Benchmarks

We evaluated our results on the AlpacalEval 2.0 dataset as a benchmark. In our experiments, we focused on the direct winrates of DPO and RDPO against their own responses to assess performance, rather than comparing against a third-party model response (GPT-4 in the case of AlpacalEval). This approach allowed us to directly observe the impact of adding rationales to the training, providing a clearer comparison with models trained without them.

Poor quality rationales

When RDPO is trained with low-quality rationales, we observe a degradation in performance, with standard DPO (without rationales) performing better. To improve the quality of the rationales, we recommend using stronger models for generation (in our case, Llama-3-8B-Instruct is sufficient, though more powerful models are emerging with ongoing developments). Additionally, after generating the rationales, we suggest verifying their correctness by leveraging a mixture of verifiers for robust assessment. For high-stakes domains, we envision having experts supervise the rationale generation process to ensure correctness and maximize the informativeness of the rationales. Additionally, we can leverage multiple models to improve the quality of the rationale, which can boost the valuable cues for the model to learn.

RDPO performance in Figure 5

In Figure 5, we examine the effect of the source model on the generation of rationales and their impact on performance. We observe that RDPO still achieves a higher winrate over DPO when using Llama-3-8B-Instruct. However, when the rationales are generated using the weaker source model, Phi3-Mini-4K, the winrate gap narrows. This suggests that for the Llama-3-8B-Instruct model, the rationale should be generated by a stronger model than Phi3-Mini-4K, highlighting the importance of the source model. This finding aligns with our theoretical result, which indicates that more informative rationales lead to improved preference prediction.

Dear Reviewer EsKw,

We want to thank You for Your helpful comments, which led to a number of interesting discussions. We have responded to each of Your concerns and questions. Hopefully, You will find that they adequately address Your concerns. Before the rebuttal phase is over, please let us know if You have any more questions or need any clarification. We would be happy to address them.

Best Wishes,
Paper8553 Authors

We thank the reviewer for their thoughtful attention to the Llama-3.1-8B-Instruct experiment presented in Figure 5. While this case exhibits a marginal performance gap, with a win rate of 52–45% (normalized to 54–46%), we respectfully emphasize that our method consistently delivers significant improvements across multiple tested settings. Specifically:

• Across datasets: Mistral-7B-Instruct-v0.2 trained with rationale-enhanced preference optimization surpasses its counterpart trained without rationales on both the Orca and Ultrafeedback datasets, achieving win rates exceeding 60%.
• Across models: RDPO consistently outperforms DPO on Mistral-7B-Instruct-v0.1, Mistral-7B-Instruct-v0.2, and Zephyr-beta models, achieving win rates consistently above 55%.
• Across methods: Both RDPO and RORPO demonstrate improvements over DPO and ORPO, respectively, as shown in evaluations on Mistral-7B-Instruct-v0.2 and Llama-3.1-8B-Instruct using the AlpacaEval 2.0 benchmark.

These findings reinforce RDPO’s value as a broadly applicable preference-learning framework that robustly improves existing approaches, such as DPO and ORPO, rather than being tailored to a specific model or dataset. Regarding the Llama-3.1-8B-Instruct experiment, the modest improvements can be attributed to two key factors. First, the inherent capability of Llama-3.1-8B-Instruct surpasses Mistral-7B-Instruct-v0.2, making substantial gains more challenging to achieve on this stronger baseline. Second, the general preference datasets like Orca and Ultrafeedback, which include pre-existing responses, may not be fully optimized for Llama-3.1-8B-Instruct. For example, prior works [1-5] generate new response pairs dynamically during training, producing synthetic datasets that differ substantially from our setting that improves pre-existing preference data in an offline manner. While their online generation strategy may explain their larger improvements, this observation reveals an exciting opportunity to extend our method: creating preference pairs and incorporating rationales in an online manner. We appreciate the reviewer's insight in highlighting this direction and have added it to our discussion of future work.

This broad pattern of improvement aligns with established practices in machine learning research, where techniques are evaluated based on their overall effectiveness across multiple scenarios rather than performance in any single setting. [6-9] We have expanded our discussion of the Llama-3.1-8B-Instruct experiment in the paper to provide deeper analysis. However, we respectfully suggest that this single case be viewed in the context of RDPO’s broader demonstrated effectiveness, which remains its key contribution to the field. We believe this comprehensive evaluation provides strong evidence for our method's contribution to preference learning.

[1] RLCD: Reinforcement Learning from Contrastive - Distillation for Language Model Alignment. Yang et al., 2024.
[2] West-of-N: Synthetic Preference Generation for Improved Reward Modeling. Pace et al., 2024.
[3] Self-Taught Evaluators. Wang et al., 2024.
[4] Meng, Yu, Mengzhou Xia, and Danqi Chen. "Simpo: Simple preference optimization with a reference-free reward.", 2024
[5] Wu, Yue, et al. "Self-play preference optimization for language model alignment.", 2024
[6] Ethayarajh, Kawin, et al. "Kto: Model alignment as prospect theoretic optimization.", 2024
[7] Park, Ryan, et al. "Disentangling length from quality in direct preference optimization.”, 2024
[8] Pal, Arka, et al. "Smaug: Fixing failure modes of preference optimisation with dpo-positive.", 2024
[9] Guo, Yiju, et al. "Controllable preference optimization: Toward controllable multi-objective alignment.", 2024

Dear Reviewer EsKw,

Thank you for your insightful feedback, which sparked valuable discussions about our work. We have thoroughly addressed your questions. As the deadline approaches, please do not hesitate to reach out with further questions, as we would be happy to address them.

Warm Regards,
Paper8553 Authors

Dear Reviewer EsKw,

As the discussion period wraps up today, we want to emphasize how much we value your feedback. Having addressed your questions, we now kindly ask for your insights. We sincerely appreciate you taking the time to share your thoughts.

Warm Regards, Paper8553 Authors

The method is very intuitive and simple.

We present a novel, data-centric approach to advancing the field of preference learning. Our method’s simplicity enables seamless integration with existing techniques, such as DPO, ORPO, and potentially SimPO or SPPO, as discussed in our paper. By incorporating rationales, we demonstrate how models can more effectively and efficiently learn from preference datasets by explicitly understanding the reasoning behind human choices. This rationale-driven approach shifts the focus from algorithm-centric improvements to uncovering the underlying logic of preferences. Our findings reveal that providing sufficient explanations significantly enhances model performance, offering a fresh and previously underexplored perspective on preference learning.

Difficulty of collecting data

We appreciate the reviewer's perspective on data collection. As illustrated in Figure 2, simply increasing the number of preference pairs does not always lead to better performance. Our method prioritizes extracting deeper insights from existing preference data. In specialized domains, generating additional valid preference pairs can be particularly challenging due to limited domain expertise. Our approach seeks to maximize learning from expert-curated data while addressing these limitations. Additionally, obtaining high-quality human preference data is often expensive. By leveraging rationales, our method enhances the utility of existing data without requiring further collection efforts. We believe rationales help guide the model to adopt correct reasoning patterns, avoiding pitfalls like reliance on superficial features (e.g., response length), which are common in traditional preference learning approaches [1,2]. Although generating rationales introduces additional LLM inference costs, the expense for a 500K dataset can remain within $100. This cost is offset by the significant gains in learning efficiency, making it especially valuable when high-quality preference data is scarce or costly. Moreover, in cases where human annotators themselves struggle to determine a preferred answer due to unclear reasoning, it is inherently challenging for a model to discern the rationale behind a choice. These scenarios demonstrate the inherent difficulty of the task, regardless of the approach used. Finally, when addressing model misalignment, we recommend employing multiple diverse models to mitigate biases and outliers in the generated rationales. This strategy can improve the quality of the rationale signals provided to the model, further enhancing its learning outcomes.

[1] Azar, Mohammad Gheshlaghi, et al. "A general theoretical paradigm to understand learning from human preferences." International Conference on Artificial Intelligence and Statistics. PMLR, 2024.
[2] Park, Ryan, et al. "Disentangling length from quality in direct preference optimization." arXiv preprint arXiv:2403.19159 (2024).

Figure 2 and 3 explanation

While Figure 2 shows that RDPO outperforms DPO by approximately 5-8 percentage points, our approach also achieves the same win rate against SFT as DPO, but with significantly fewer pairwise data points. By enhancing the data with rationales, we reduce the annotation effort by a factor of three. Additionally, Figure 2 highlights the win rate against the SFT model, serving as an intermediate comparison between DPO and RDPO. In the direct comparison, depicted in Figure 3, RDPO achieves a win rate of 60-65% against DPO across various training data sizes for both the Orca and Ultrafeedback datasets.

Runtime and average length

We appreciate the reviewer's suggestion. We report the runtime for RDPO and DPO for one epoch on Llama-3.1-8B-Instruct, using 12,000 Orca examples, as follows:

• RDPO General: 6770 seconds
• RDPO Specific: 6950 seconds
• DPO: 3583 seconds

While processing additional tokens nearly doubles the runtime, RDPO compensates for this by requiring fewer annotations while achieving comparable or superior performance to DPO. Additionally, we report the average response lengths for the Orca dataset:

• Chosen responses: 786
• Rejected responses: 981
• Rationale responses: 411

Winrate computation

To compute the winrates against DPO and SFT, we generated responses from each model for 512 fixed test samples from a given dataset. For each comparison (RDPO vs. DPO and RDPO vs. SFT), we used an LLM judge to select the better response. If the judge could not determine a preferred response, the comparison was marked as a draw. The win rate is calculated as the number of RDPO responses preferred, divided by the total of 512 samples.

Dear Reviewer 5bwS,

We want to thank you for your helpful comments, which led to valuable discussion about our work. We have carefully addressed your concerns and welcome any additional feedback. Please let us know if you have any further questions, we will be happy to address them.

Kind Regards,
Paper8553 Authors

Dear Reviewer 5bwS,

We would like to thank You for Your comments and suggestions, we tried our best to address every question raised. We hope that our answers could resolve Your concerns. We are happy to address additional suggestions. Since the rebuttal period is closing soon, we would love to be able to respond any further questions.

Best Wishes,
Paper8553 Authors

Dear Reviewer 5bwS,

Thank you for your thoughtful feedback, which contributed to meaningful discussions about our work. We have thoroughly addressed your questions and updated our work to reflect the changes. With the deadline approaching, we welcome any additional input and are more than happy to address any further questions.

Best Regards,
Paper8553 Authors

Dear Reviewer 5bwS,

As the discussion period wraps up today, we want to emphasize how much we value your feedback. Having addressed your questions, we now kindly ask for your insights. We sincerely appreciate you taking the time to share your thoughts.

Warm Regards,
Paper8553 Authors

Dear Reviewer jgVq,

We want to thank you for your helpful feedback, which led to valuable discussion about our work. We have carefully addressed your questions and welcome any additional feedback. Please let us know if you have any further questions, we will be happy to address them.

Kind Regards,
Paper8553 Authors

Exploration of components

While we agree with the reviewer that conducting more in-depth experiments to evaluate the interaction between the preference training objective and the rationale objective would be valuable, we have undertaken several analyses to explore this relationship. Specifically, we have studied the impact of incorporating rationales into the original preference learning objectives. Our investigations include assessing performance improvements by varying dataset sizes, evaluating rationales of different quality, conducting parameter ablation studies, and integrating rationales into various preference learning methods (e.g., DPO, ORPO) across different models and datasets. Notably, we observed that adding rationales designed to promote specific properties (e.g., conciseness) consistently imparts those traits across models and datasets—a behavior not evident in the original DPO approach. This suggests an intriguing avenue for future research: exploring whether rationales can transfer certain meta-properties to models.

Scale of gamma

As gamma decreases towards 0, we observe performance degrading to that of vanilla DPO/ORPO. Starting around a value of 0.2, we see the benefit of incorporating rationales into the preference dataset, which enhances the performance of the rationale-enhanced model compared to the vanilla preference-trained model. Performance then stabilizes as gamma approaches 10, with a slight decline in performance occurring as gamma increases beyond 10, up to 100. However, it is important to note that for different preference objectives, the optimal value of gamma may vary, highlighting the significance of gamma in achieving the best performance.

Rationale without DPO

We appreciate the reviewer’s suggestion. While our framework was introduced to complement existing preference learning objectives for more efficient data learning, we have not yet explored the use of rationales independently, without the preference objective. This suggestion is akin to the concept of Self-Taught Evaluators, where the preference learning objective is removed and models are supervised fine-tuned. We value this input and see it as a promising direction for future work.

Wang, Tianlu, et al. "Self-taught evaluators." arXiv preprint arXiv:2408.02666 (2024).

Gamma in theory and in practice

The gamma introduced in the theoretical formulation ($\\beta = 0.5 + \\gamma$) represents the informativeness or quality of the rationale, while the gamma used in our experiments refers to the influence of the rationale loss on the preference training, which is independent of the rationale's quality. We apologize for any confusion caused by the notation.

Dear Reviewer jgVq,

We want to thank You for Your helpful comments, which led to a number of important extensions of our work. We have addressed each of Your questions. Please let us know if You have any more questions, we would be happy to address them within our allowed period.

Best Wishes,
Paper8553 Authors

Dear Reviewer jgVq,

Thank you for your feedback! We have addressed your concerns and would greatly appreciate any additional feedback you may have. If there are further suggestions to improve our work, we would be happy to address them.

With Appreciation,
Paper8553 Authors

Dear Reviewer jgVq,

As the discussion period wraps up today, we want to emphasize how much we value your feedback. Having addressed your questions, we now kindly ask for your insights. We sincerely appreciate you taking the time to share your thoughts.

Warm Regards, Paper8553 Authors

Dear Reviewer ZabL,

We want to thank you for your positive assessment. We appreciate the economic perspective on the utility of the dataset which aligns with our motivation to improve the effectiveness of data. We appreciate your suggestion on guiding the project owners with limited annotation budget to show the practicality of the method. Please let us know if you have further questions, we will be happy to address them.

Kind Regards,
Paper8553 Authors

As highlighted by the reviewer, our goal is to enhance the utility of the dataset by improving the information efficiency of the annotated pairs. By providing rationales alongside the annotations, we aim to not only enable more efficient learning from preferences—thus reducing the need for excessive annotation—but also to potentially improve the interpretability and understanding of these preferences.

We appreciate the reviewer for the suggestion. To assist project owners in evaluating the trade-offs, we present a cost-benefit analysis of the approach. The table outlines the cost of using the API to generate rationales for a given number of annotations. It also highlights the RDPO win rate compared to the SFT model for each data budget and estimates the number of annotations that could potentially be saved from using DPO to achieve the same level of performance as RDPO.

While we used open-weight models to generate the rationales in our study, the table illustrates the associated costs when utilizing an API model, specifically gpt-4o-mini. This model is referenced with pricing of \$0.150 per 1M input tokens and \\$0.600 per 1M output tokens. We show the results on the Mistral-7B-v0.2-Instruct trained on the Orca dataset.

Dear Reviewer ZabL,

We want to once again thank you for your helpful comments. We have addressed your questions. Please let us know if you have any more questions, we would be happy to address them before the rebuttal period ends.

Kind Regards, Paper8553 Authors