Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts

We thank Reviewer i7yC for providing detailed feedback and valuable suggestions. We provide further clarifications below.

W1: Thank you for your question. In the context of paired data, we provide an analysis of a random mini-batch from the weight matrix with a per-GPU batch size of 8 and a distance temperature $\tau = 0.5$, as shown in Q1. The diagonal entries (corresponding to response pairs from the same prompt) exhibit weights fluctuating around 0.5, dominating the loss contribution. In contrast, weights for other samples fluctuate around 0.1, based on their similarity to the corresponding prompt. This indicates that even in a random mini-batch, the model effectively learns additional information from rejected responses to other prompts.

Moreover, we have demonstrated the effectiveness of RPO on general-purpose datasets, such as ultrafeedback, through results presented in Table 1 and Table 2 (refer to the general rebuttal section). Additionally, in Table 4 of the manuscript, we showcase results under a completely random unpaired setting on a general chat task. Even in this setting, RPO outperforms KTO. While KTO fundamentally focuses on increasing the reward of chosen responses and reducing the reward of rejected responses, RPO further enhances this process by leveraging prompt similarity to assign differentiated loss weights to various pairs.

In summary, we have validated the effectiveness of RPO across both general-purpose datasets (ultrafeedback/capybara) and specific-task datasets (HH/Summarization). The mini-batch similarity issue does not detract from RPO’s ability to effectively utilize rejected responses and outperform baseline methods, even in diverse datasets.

W2: We strongly disagree with the dismissive tone in your assessment of this paper's contributions. We submitted our work to ICLR because we are genuinely excited about sharing our insights with the community and believe in their value. We respectfully request that you also review the comments from other reviewers, which highlight the significance of our methodological contributions. Taking our response into account, we kindly ask you to consider adjusting your position.

Our work addresses a critical gap in the original DPO framework, specifically the lack of comparisons across prompts. Inspired by human learning processes, we propose constructing a contrastive matrix to incorporate contrastive learning with relative prompt-answer pairs. Through extensive experiments, we have validated the effectiveness of this approach.

Furthermore, the concept of Cross-Prompt Comparisons introduced in RPO extends beyond its current scope. For instance, it can be applied to reward model training. This idea also opens up opportunities for exploration in other areas of RLHF. We believe that simple yet effective methods like RPO are both valuable and capable of driving meaningful advancements.

Q1:

1. Proportion of weights > 0.5:

The proportion of weights greater than 0.5 under different per-GPU batch sizes (for the Anthropic HH dataset with $\tau = 0.5$) is summarized as follows:

In our final experiments (with a per-GPU batch size of 8), the proportion of weights greater than 0.5 is only 0.55%.

2. Examples from specific mini-batches:
   • Per-GPU batch size = 2:

     [[0.8516, 0.1484],
      [0.1484, 0.8516]]
     

   • Per-GPU batch size = 4:

     [[0.6602, 0.1147, 0.0913, 0.1328],
      [0.1206, 0.6953, 0.0845, 0.1001],
      [0.0991, 0.0869, 0.7148, 0.0972],
      [0.1357, 0.0972, 0.0913, 0.6758]]
     

   • Per-GPU batch size = 6:

     [[0.5156, 0.0898, 0.0713, 0.1040, 0.0986, 0.1206],
      [0.1006, 0.5781, 0.0703, 0.0835, 0.0830, 0.0835],
      [0.0815, 0.0713, 0.5859, 0.0796, 0.0879, 0.0928],
      [0.1094, 0.0781, 0.0732, 0.5430, 0.1069, 0.0894],
      [0.1030, 0.0771, 0.0811, 0.1064, 0.5391, 0.0933],
      [0.1250, 0.0771, 0.0845, 0.0879, 0.0923, 0.5352]]
     

   • Per-GPU batch size = 8:

     [[0.4102, 0.0713, 0.0569, 0.0830, 0.0786, 0.0962, 0.1177, 0.0859],
      [0.0864, 0.4980, 0.0605, 0.0718, 0.0713, 0.0718, 0.0767, 0.0635],
      [0.0698, 0.0613, 0.5039, 0.0684, 0.0757, 0.0801, 0.0645, 0.0747],
      [0.0845, 0.0605, 0.0569, 0.4199, 0.0825, 0.0693, 0.1455, 0.0806],
      [0.0845, 0.0635, 0.0664, 0.0874, 0.4434, 0.0762, 0.0845, 0.0952],
      [0.1060, 0.0654, 0.0718, 0.0747, 0.0781, 0.4531, 0.0820, 0.0684],
      [0.1138, 0.0610, 0.0505, 0.1377, 0.0757, 0.0718, 0.3965, 0.0923],
      [0.0923, 0.0559, 0.0649, 0.0845, 0.0942, 0.0664, 0.1021, 0.4395]]
     

Q2: Thanks for your question. The approach you suggest can indeed be regarded as an offline variant of RPO. In our experiments, we first constructed an 8-nearest HH dataset based on prompt similarity and then applied RPO to expand the original data with additional contrastive pairs for training. However, we found that the results of this offline approach were comparable to those achieved by our current online version.

From a statistical perspective, both offline RPO, which uses a global similarity matrix, and online RPO, which leverages a batch-wise similarity matrix, are designed to achieve similar objectives. For instance, stochastic gradient descent often matches or even surpasses global gradient descent in performance. Furthermore, our results consistently show that RPO outperforms DPO, further validating our claim that introducing additional contrastive pairs enhances performance.

Thanks for your detailed response. Through careful consideration regarding the useful information that can still be mined among samples within the training batch, I acknowledge that although the method described in the paper is not an excellent solution, I recognize the value in your observation of this phenomenon. Therefore, I have decided to adjust the score given.

I share a similar view with Reviewer hL8X, believing that the main benefit of RPO comes from the runtime data augmentation, where the augmentation involves creating new pairs among similar prompts. Due to the high sparsity of text sequences, any method that increases data density in the text sequence space will be effective. This is also one of the important research directions for LLM at present. Let's consider a thought experiment: Suppose I have a dataset $S=\\{x_i,y_i\\}$, and construct a dataset $D=\\{x_i,y_w=y_i,y_l=y_j\\}, j \ne i$, where. Assuming the method of constructing the dataset is unknown, can we achieve an improvement in model performance through RPO?

I apologize for an incorrect symbol reference in W1 and Q1. My intention was to express that "the probability of high similarity among prompts in a small batch is low." Therefore, I should have referred to the symbol $\tilde{\omega}$ at Eq. 10 instead of $\omega$ at Eq. 11. Based on this change and considering your response regarding Q1, I am concerned about the phenomenon when "Per-GPU batch size = 2," where a high-weight (high $\omega$) but potentially low-value (low $\tilde{\omega}$) training pair is always created. This implies that effective sample pairs can be derived from any two unrelated prompts. On the other hand, in the case of large batches, even if all prompts are highly related, the normalization effect of Eq. 11 causes originally high-value (high $\tilde{\omega}$) samples that could be formed through pairwise combinations to become low-weight (low $\omega$). These two examples illustrate the areas where RPO as a training optimization method may be underconsidered.

So, to my personal opinion, it might be a better choice for this paper to be organized from the perspective of data augmentation rather than training optimization. I believe that the contribution level of a chosen-reject sample pair to the training task should be an objective constant, meaning it should be more related to $\tilde{\omega}$ rather than to batch size or other samples within the batch. Therefore, if we consider the extraction of effective sample pairs from a the whole dataset, it may indeed be attractive.

We sincerely thank Reviewer i7yC for your thoughtful feedback on the benefits of RPO compared to existing methods, your alternative explanation from the runtime data augmentation perspective, and your suggestion for additional ablation studies.

1. First and foremost, we agree that runtime data augmentation could be a key factor in RPO’s effectiveness. It enables more efficient utilization of data within each GPU batch while incurring negligible computational and memory overhead. We also greatly appreciate your insightful comment: “Due to the high sparsity of text sequences, any method that increases data density in the text sequence space will be effective. This is also one of the important research directions for LLM at present.”

   • While we fully agree with this perspective, our investigation suggests that increasing data density alone does not fully account for the effectiveness of RPO. This can be verified through experiments, as you suggested, which align with the ablation study presented in Table 1 of our manuscript. In this study, we analyzed Diagonal Weighting and Uniform Weighting under RPO. As shown in Table 1 of the manuscript, these methods perform reasonably well but fail to outperform DPO, let alone RPO with embedding-distance-based reweighting.

   • Regarding the specific example you hypothesized, we believe you are referring to a scenario where data is augmented by consistently treating the winning response of one prompt as the losing response of a different prompt. While this setting has not been explored by any prior methods, RPO can indeed be applied directly using diagonal, uniform, or embedding-based reweighting.

   • Although we have not conducted experiments specifically for this scenario, it aligns closely with the unpaired setting where we contrast the winning response of a prompt with losing responses from other prompts. In this context, our results clearly outperform KTO, a method specifically designed to address this unpaired setting.

   • These findings indicate that while runtime data augmentation and increased data density in the text sequence space are valuable and worth emphasizing, they are not sufficient to surpass DPO in paired settings. The key lies in combining these concepts—an integral part of RPO’s innovation—with the proposed embedding-distance-based reweighting, which is critical to RPO’s success.

   • We greatly appreciate your insights and hope this response effectively addresses your concerns.

2. Thank you for acknowledging the previous misunderstanding and clarifying your concern that RPO as a training algorithm may be underexplored.

   • We would like to emphasize that the current RPO solution is limited by the memory constraints of GPUs, restricting us to a per-GPU batch size of 8. Alleviating this constraint through more advanced computing platforms, improved implementations of distributed computing, and additional code-level optimizations could further enhance RPO's performance. However, we argue that the potential for further improvement does not diminish the value of our current solution. On the contrary, it is an encouraging signal that further efforts should be dedicated to the RPO framework to develop future solutions.

   • Second, retrieval-augmented techniques could potentially be used to identify contrastive pairs. While these techniques are interesting and promising, they are beyond the scope of the current paper.

   • Third, even under extreme constraints, such as a per-GPU batch size of 2, we retain the flexibility to adjust $\tau$. It is worth noting that we fixed $\tau = 0.5$ when responding with "Examples from specific mini-batches." However, $\tau$ can be tuned to control the level of contrast between related and unrelated pairs, further enhancing the adaptability of RPO.

   • We appreciate your thoughtful feedback and hope this addresses your concerns effectively.

3. Thank you for suggesting that we organize our paper from the perspective of data augmentation rather than training optimization. We appreciate your insight and how it highlights ways to better explain and illustrate our contributions. In our next revision, we will make an effort to emphasize the runtime data augmentation perspective more prominently. However, we hope the reviewer considers this suggestion as a matter of personal preference regarding the paper’s style rather than a fundamental weakness. If RPO can inspire future work to explore the data augmentation perspective and develop improved solutions, we hope you will agree that it is a valuable contribution worthy of publication in a top-tier conference like ICLR.

We thank Reviewer hL8X for providing detailed feedback and valuable suggestions. We provide further clarifications below.

W1: Thanks for pointing this out. RPO's flexibility lies in its ability to adjust the temperature parameter $\tau$, which allows us to control the proportion of additional relative prompt-response pairs included. This adaptability makes it possible to fine-tune $\tau$ for different task types and datasets. For example, in domains like mathematics and code, where even minor token-level differences in prompts can lead to entirely different or contradictory answers, we can set $\tau$ to a very small value. In such cases, RPO would effectively degenerate into DPO, restricting comparisons to responses originating from the same prompt.

For the tasks covered in this paper, such as general dialogue and summarization tasks, we have thoroughly validated the effectiveness of RPO through extensive experiments across various base models. Nevertheless, for future applications in domains like code or mathematics, incorporating a token-level-aware embedding model could complement our current framework. However, our current experiments focus on general dialogue tasks, and the results strongly support the effectiveness of using sentence similarity to compute weights in this context.

W2: Thank you for pointing this out. The motivation for RPO stems from human learning processes, where individuals transfer knowledge from similar questions or mistakes they have encountered. By building contrastive relationships across similar prompts, RPO is designed to emulate this aspect of human learning. While it can be interpreted as a form of data augmentation or data strengthening, this interpretation does not diminish its significance as an effective and impactful method.

We would like to emphasize that RPO, compared to DPO, does not introduce any additional data. Both RPO and DPO iterate through the same training dataset—consisting of 160,000 data samples (or 320,000 prompt-response pairs)—exactly once. RPO instead enhances the learning process by leveraging additional relationships derived from the existing data, making it a straightforward yet powerful approach.

W3-1 (Prompt Encoding Model): Thank you for your question! One of the key advantages of RPO is its use of a very lightweight model, MiniLM-L6-v2 (only 22M in size), to encode prompts, which introduces almost no additional computational overhead. In contrast, fine-tuning models typically have sizes of 7B or larger, and using these larger models to encode prompts could result in significant computational costs. However, exploring the use of the fine-tuning model for encoding prompts could be an interesting direction for future experiments.

W3-2 (Stronger Models): As shown in Table 3 of the manuscript, we provide a comparison of RPO with baseline methods on Llama-2-13B, demonstrating its effectiveness on larger models. Additionally, in Table 2 (refer to the general response section), we present results of fine-tuning Gemma-2-9B-it (a highly capable RLHF model) with RPO integrated into SimPO. These results further highlight the effectiveness of RPO.

We thank Reviewer 7wmZ for the positive feedback and valuable suggestions. Below, we provide further clarifications:

W1: We conduct thorough experiments to validate the effectiveness of RPO. The strong benchmark performance supports our claim that the newly introduced relative comparison is effective. Contrastive learning within win-lose pairs derived from related prompts aligns well with natural human reasoning, as demonstrated in Figure 1.

Q1: Since the model used for encoding prompts, all-MiniLM-L6-v2, is only 22M in size, the encoding time for each batch of prompts is extremely short, averaging just 0.3 seconds. Additionally, as shown in Figure 4, the training time of RPO is nearly identical to that of baseline methods such as DPO/IPO, indicating that the additional computational cost introduced by prompt encoding is negligible throughout the training process.

Q2: As shown in Table 2 (kindly refer to the general response section), we present the results of integrating RPO with SimPO. The outcomes on gemma-2-9b-it and Llama-3-8B-Instruct demonstrate that RPO_SimPO consistently outperforms SimPO, achieving superior results.

We thank Reviewer 2Go9 for providing detailed feedback and valuable suggestions. We provide further clarifications below.

W1: Unlike DPO, which only considers win-lose contrasts within the same pair, RPO enables the construction of additional contrast lines for every prompt pair. As the reviewer pointed out, if two prompts $i$ and $j$ are significantly different, our reweighting technique ensures that the contrast line is heavily downweighted. This is because the answers to prompts $i$ and $j$ cannot be meaningfully extended to one another, as $i$ and $j$ are semantically unrelated.

W2: Thank you for pointing this out. However, we would like to clarify that the optimal value of beta may vary depending on the base model and dataset. For the tasks covered in this paper—dialogue tasks on the HH dataset and summarization tasks on the summarization dataset—the baseline hyperparameters, such as beta, follow the default experimental settings provided by KTO. Please refer to the configuration at: https://github.com/ContextualAI/HALOs/tree/main/config/loss. Specifically, the default value for beta is set to 0.1. Regarding the batch size, all experiments default to a per-GPU batch size of 8, which also aligns with KTO's default settings. Please see: https://github.com/ContextualAI/HALOs/blob/main/config/model/base_model.yaml#L40.

In our updated experiments (fine-tuning Meta-Llama-3-8B-Instruct on a general chat task with on-policy ultrafeedback dataset), we followed the settings of SimPO. In these experiments, all methods were tuned to their optimal hyperparameters, and our approach still achieved superior results. Please refer to Table 1(refer to the general response section) for details.

W3-1 (scores for other models): As mentioned in Line 471, the Mistral-7B model emerged as the top performer in our preliminary experiments, which led us to select it for detailed studies in the Summarization task. Since the OpenLLM leaderboard primarily evaluates models based on knowledge reasoning capabilities, and the HH dataset and Summarization dataset focus on assessing models' abilities to generate helpful and harmless responses and produce high-quality summaries, respectively, we did not evaluate our approach on the OpenLLM leaderboard.

However, in our updated experiments (fine-tuning Meta-Llama-3-8B-Instruct on a general chat task with an on-policy ultra-feedback dataset), we conducted a detailed comparison with baselines, including results from the OpenLLM leaderboard. Please refer to Table 1 (kindly refer to the general response section) for further details.

W3-2 (Compare to DRO): Since DRO conducted experiments using the T5 model, their paper did not provide directly comparable results. Additionally, the method is not open-sourced, making it difficult for us to reproduce their results. Furthermore, DRO represents a framework that is independent of DPO/SimPO and similar approaches. As such, we believe it is appropriate to exclude DRO as a baseline for our work.

W3-3 (Comparison with KTO using unpaired Capybara and Ultrafeedback datasets): Thanks for pointing this out. Here, we aim to investigate the performance of our method in the unpaired dataset setting compared to KTO. The Ultrafeedback dataset mentioned remains a paired setting (each item is a quadruple: $(x, y_1, y_2, \text{label})$. Additionally, the argilla/distilabel-capybara-kto-15k-binarized dataset is a high-quality multi-turn dialogue dataset, making it particularly suitable for evaluating our method in multi-turn scenarios.

To address the reviewer's concern, we also updated our experiments by fine-tuning meta-llama/Meta-Llama-3-8B-Instruct on the unpaired argilla/ultrafeedback-binarized-preferences-cleaned-kto dataset (where each item is a triplet: $(x, y, 0/1)$. The results, presented in Table 3, demonstrate that RPO continues to outperform KTO in this unpaired setting.

Table 3

W4 and W5: Please see Table 1.