Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?

Contradictions

We have conducted an analysis to measure the contradictions between explicit and implicit margin metrics ($M_r$ and $M_\pi$) by comparing the Jaccard similarity of subsets selected by each metric:

• Subset selection: Using $M_r$ and $M_\pi$, we select the top-rated k% subsets from existing datasets, denoted as $D_r$ and $D_\pi$.
• Jaccard similarity: We calculate the similarity between the selected subsets using:

• Comparison: We compare the resultant similarity $J(D_r, D_\pi)$ with two uniformly selected subsets $J(D_{u1},D_{u2})$. A lower similarity $J(D_r, D_\pi) < J(D_{u1},D_{u2})$ would indicate contradictions between $M_r$ and $M_\pi$.

We measure the similarities on all three datasets in our paper:

As shown in the table, the similarities $J(D_r, D_\pi)$ are consistently smaller than those of uniformly selected subsets across varying top-k ratios and datasets. This observation validates the claim that explicit and implicit margin metrics can indeed provide contradictory evaluations for preference selection.

Smaller Implicit Margins

Explanation of implicit reward margins:

• Computation: The implicit margins, $M_\pi = |\hat r_\theta(x,y_w) - \hat r_\theta(x,y_l)|$, are indeed computed based on the initial model (i.e., the SFT model $\pi_\text{SFT}$).
• Motivation: The implicit margin shows the LLM's ability to discern preferences. A small implicit margin indicates the model initially lacks the ability to distinguish such preference, making the data suitable for alignment training.

Empirical evidence: We measure the distributions of implicit margins before and after the alignment training on the Llama v2 dataset. The results, as shown in Figure I in this anonymous link, demonstrate a notable increase in $M_r$ after alignment training. Specifically, the average $M_r$ rises from 0.072 to 0.742, indicating improved preference recognition ability of the model.

Given these insights, data with relatively smaller initial implicit margins generally indicates that the LLM cannot tell the preference, thereby requiring further training to align on such data.

Method Implementation

Our SimPO-based implementation of the implicit reward is because:

1. The implicit reward $\hat r_\theta^{Sim}(x,y) = \beta\log\frac{\pi_\theta(y|x)}{|y|}$, has been demonstrated to be effective in SimPO's paper.
2. The formulation does not require additional conditions (e.g. reference model), allowing for its direct application in different optimization methods, e.g., DPO, IPO.

Throughout our paper, the implicit reward in our $M_{AP}$ metric is consistently calculated using the SimPO formulation $\hat r_\theta^{Sim}(x,y),$ regardless of whether the training method is SimPO or DPO. And the empirical results also validate its effectiveness across different training settings.

More Baselines

In response to your suggestions, we conduct additional experiments to include uniform and $M_\pi$ baselines under Table 1's setting, and here are the results of Llama models on Alpaca Eval:

As shown in the tables, our $M_{AP}$ metric consistently outperforms existing baselines for both optimization methods.

Hyperparameters

The value of hyperparameter $\alpha$ can be found in Line 991 and Line 1008 of Appendix D.1, where we generally set the value within {0.5, 1, 2.5, 5.0}.

Explain Margins & Motivation

We would like to clarify that our paper is not centered on the notion that "large explicit reward margin" and "small implicit reward margin" individually imply high-quality data. Instead, one key motivation for our work is that relying solely on either of these margins is insufficient for assessing data quality. As demonstrated in Figure 1, the two instances with high explicit reward margins or small implicit margins are evaluated as low-quality data by our metric (Line 71-74).

The core idea of our metric is to quantify the data quality by measuring the discrepancy between the current model's preference and the aligned optimum on the data. The alignment optimum is captured through explicit reward margins (Equation 8), and the model's preference is indicated by implicit reward margins. So the proposed metric $M_{AP} = |r(x,y_w) - r(x,y_l)|-|\hat r_\theta(x,y_w) - \hat r_\theta(x,y_l)|$ quantifies the gap from the current implicit margin to the target explicit margin, thereby indicating the potential for alignment on the given data.

Due to space limits, we put all tables in this anonymous link: https://anonymous.4open.science/r/tables-11915/ZHMc.md.

Large/Small Margins

To reiterate, the primary claim of our paper is not that "large explicit reward margin" and "small implicit reward margin" individually imply high-quality data. Instead, the key motivation for our work is that relying solely on either of these margins is insufficient for assessing data quality.

As demonstrated in Figure 1, we evaluate instances with high explicit reward margins or small implicit margins as low-quality data (Line 71-74). The main idea of our proposed metric is to consider the gap between the model's current implicit margin and the target explicit margin to measure the potential for alignment learning.

However, if assessing data quality using a single margin is required, we acknowledge that a higher explicit margin or smaller implicit margin can be beneficial. This notion is supported by recent works (Line 40) and our experiments (Figures 4, 5, and 8). In response to your suggestions, we've conducted additional experiments by following the setup of Figure 4, but reversely selecting data via these metrics. As shown in Table I of the linked page, reversely selecting training data indeed results in poorer performance, thus verifying these metrics' efficacy from an opposite standpoint.

Too Large Gaps

While a larger gap might require more intensive alignment training, the direction of the update remains correct. Moreover, the common practice in preference dataset construction for alignment training, such as SimPO, involves sampling responses $y_w$ and $y_l$ using the model that is being trained. This practice ensures that both the chosen and rejected responses are within the model's generation capabilities, thus mitigating the label exposure bias problem.

Different Reward Models

In our study, we have indeed considered the impact of different reward models by incorporating two RMs: PairRM (0.4B) and ArmoRM (8B), for preference annotation and metrics evaluation. Figures 4 and 8a in the paper illustrate the results based on datasets annotated using ArmoRM and PairRM, respectively. As shown in the figures, our proposed $M_{AP}$ metric outperforms existing baselines under both RMs, attesting to the effectiveness of $M_{AP}$ metric under different reward models.

Reward Noise & Ranking

The ranking disagreement might not be the key determining the data quality, due to the reward noise issue. As suggested, we select 40% data subsets in which the ranking of explicit and implicit rewards are opposite. We compare this strategy with existing metrics on both Llama datasets as described above.

As shown in Table II of the linked page, although selecting data with contradictory ranking between explicit and implicit rewards can improve performance to some extent, it cannot outperform any of the existing baselines.

Such an observation is consistent with our motivation of "Reward noise regularization". As discussed in Line 212-215, given the LLM's ability to discern certain preferences, the opposite ranking between explicit and implicit rewards—suggesting contradictory preference judgments between the RM and the LLM—may indicate a higher risk of noisy reward model annotations, and thus resulting inferior model performance.

Other Directions

We sincerely appreciate your suggestions for exploring the non-linear mapping, LLM-as-Judge, weighted loss and curriculum learning settings. While we believe they hold promise for future research, the limited timeframe for the current study does not permit us to explore all these directions, so we plan to investigate them in future work.

Novelty

While existing methods focus solely on either explicit or implicit margins and often lack in-depth explanations, our work introduces a key innovation by measuring the data quality with the gap between the model's aligned optimum and the current model's preferences. This gap serves as a theoretical basis for evaluating data quality in alignment training, distinguishing our approach from prior work.

Although the final form of $M_{AP}$ incorporates the two existing metrics ($M_r$ and $M_\pi$), its core idea of gap quantification and the logical derivation of our approach are novel aspects not explored in existing metrics., which contributes significantly to the advancement and originality of this work.

Training Details & Significance Testing

We report the training time, standard deviations and confidence intervals of current evaluations in Table III & IV of the linked page.

References & Presentation

Thanks for your suggestions, we will make the following revisions:

1. Include relevant references of data selection in general LLM tasks
2. Merge repetitive equations and figures
3. Revise sections by splitting the Discussion section into separate Conclusion and Limitations sections

Additional Alignment Approaches

Thanks for your suggestion! Aside from the DPO and SimPO methods, we conduct additional experiments using the IPO method for alignment training.

Like the setting in Section 3.2, we utilize different metrics to select top-40% subsets from existing preference datasets for subsequent IPO training. We utilize the two Llama-based datasets for experiment, benchmark the trained models on Alpaca Eval 2.0, and report both Length-Controlled (LC) win rates and raw Win Rates (WR):

On the default Llama dataset of SimPO (based on the PairRM reward model):

On the Llama v2 dataset of SimPO (based on the ArmoRM reward model):

As shown in the table, our proposed $M_{AP}$ metrics still outperform existing baselines under this new alignment method, further certifying its effectiveness.

Reference

• IPO: Azar, Mohammad Gheshlaghi et al. “A General Theoretical Paradigm to Understand Learning from Human Preferences.” ICML 2024.

The $M_+$ Metric

The $M_+$ metric is a modified version of the proposed $M_1$ metric designed to adapt to the unidirectional update nature of existing preference optimization methods (e.g. DPO, SimPO).

As described in section 3.1, we first quantify the discrepancy between the current model and the aligned optimum on the preference data $(x,y_w,y_l)$ via the $M_1$ metric: $M_1(x,y_w,y_l;\theta,r^*) = |(r^*(x,y_w) - r^*(x,y_l)) - (\hat r_\theta(x,y_w) - \hat r_\theta(x,y_l))|.$ To ensure practicality, we then introduce two key adaptations to the $M_1$ metric, and the first one is termed unidirectional calibration (Lines 182-202). The rationale behind this adaptation is explained as follows:

In the optimization process of DPO, the loss function $-\log\sigma(\hat r_\theta(x,y_w) - \hat r_\theta(x,y_l))$ tends to unidirectionally increase the implicit margin term $\hat r_\theta(x,y_w) - \hat r_\theta(x,y_l)$ during optimization. So we would like to select data with a low implicit margin term requiring to be increased through training.

However, the original $M_1$ metric measures a bidirectional gap between $\hat r_\theta(x,y_w) - \hat r_\theta(x,y_l)$ and $r^*(x,y_w) - r^*(x,y_l)$, which can lead to selection of data with either a very high margin or a very low margin. While data with a low margin term, $\hat r_\theta(x,y_w) - \hat r_\theta(x,y_l)$, is desirable, data with a high margin term will be further increased by the DPO training process, which is counterproductive.

To address this, we opted to remove the absolute value from $M_1$, implementing what we refer to as unidirectional calibration, and formulated a revised metric, $M_+$: $M_+(x,y_w,y_l;\theta,r^*) = (r^*(x,y_w) - r^*(x,y_l)) - (\hat r_\theta(x,y_w) - \hat r_\theta(x,y_l)).$

This new metric effectively selects data with a minimal implicit margin term $\hat r_\theta(x,y_w) - \hat r_\theta(x,y_l)$ to increase, aligning with the unidirectional nature of preference optimization methods.

We hope this elucidation clarifies the intent and application of the $M_+$ metric within our study.

Readability

Thank you for highlighting this issue. We will strive to enhance the readability in our revised version. Specifically, we plan to: (1) merge the two $M_{AP}$ equations (Eq. 11 and Line 236) to reduce redundancy and enhance clarity; (2) include a concise overview of our metric formulation at the beginning of Section 3 to guide readers through our methodology with greater ease. These changes aim to streamline the presentation and make the content more coherent to readers.

Ablation I: absolute values

Although Eq.11 incorporates two additional absolute values compared with Eq.10: $|r(x,y_w) - r(x,y_l)|$ and $|\hat r_\theta(x,y_w) - \hat r_\theta(x,y_l)|$, only the latter absolute values (on implicit rewards) actually changes the metric's value. The reason is that, when constructing the preference dataset $\{(x,y_w,y_l)\}$, two responses $y_1,y_2$ are annotated by the reward model $r$ to determine the winning/losing responses: $y_w,y_l \in \{y_1,y_2\}$, such that $r(x,y_w) > r(x,y_l)$ (line 182-185). Therefore, by definitation we have $r(x,y_w) - r(x,y_l) \ge 0$ and $|r(x,y_w)-r(x,y_l)| = r(x,y_w) - r(x,y_l)$. Given that the absolute value applied to the external reward gap does not alter the outcome, there will be no need to conduct additional ablations.

We will explicitly clarify this point in our revised manuscript to prevent potential confusion. Thank you for pointing out this problem!

Ablation II: Varying-k & Bottom-40%

Thanks for your insightful suggestions! We have conducted additional experiments to address your feedback on varying selection ratios and the impact of selecting bottom-40% samples.

For varying k values, we expanded our experiments by selecting different top-k on the default SimPO's Llama v1 dataset using various metrics, and evaluated the resulting models using Alpaca Eval 2.0. We reported both Length-Controlled (LC) win rates, which are more preferable to reduce length bias, and raw Win Rates (WR):

As shown in the table, our $M_{AP}$ metric continuously outperforms other methods across different proportions of selected data.

For the bottom-40% selection, data was reversely chosen using various metrics from the Llama v2 dataset for SimPO training:

Across all metrics—when selecting the bottom 40% based on $M_r, -M_\pi, M_{AP}$—the performance of resultant models was poorer than the uniform baseline, indicating lower data quality through reverse selection. Notably, the lowest performance observed was with our $M_{AP}$ metric, verifying its efficacy in identifying high-quality datasets from an opposite standpoint.

These experiments affirm the effectiveness of our proposed method, and we will incorporate the findings in our revised version.

More Iterative Baselines & References

Thank you for your valuable suggestions!

In the iterative preference learning experiments, we augment the default iterative preference learning pipeline by introducing an additional data selection procedure guided by the proposed metric $M_{AP}$. The role of our proposed metric in this process is to select high-quality preference data for subsequent training. Therefore, it serves as an orthogonal strategy alongside existing iterative preference optimization techniques [1,2,3]. While integrating our data selection process with these optimization methods poses interesting prospects, we're afraid the current timeframe does not allow us to explore all potential directions.

We sincerely appreciate you for highlighting these relevant works. We will include them in our references and discuss potential integrations of our $M_{AP}$ metric with various iterative preference learning methods in the future work section of our revised version.

Suggestions & Questions Regarding Figure 1

Thanks for your valuable suggestion!

In Figure 1, we present two examples to compare the existing metrics:

• Explicit reward margin: $M_r = |r(x,y_w) - r(x,y_l)|$, which quantifies how much $y_w$ is more preferable than $y_l$.
• Implicit reward margin: $M_\pi = |\hat r_\theta(x,y_w) - \hat r_\theta(x,y_l)|$. With $\hat r_\theta(x,y)$ indicating the current model $\pi_\theta$'s preference evaluation on $y|x$, this measures how well the model discerns the preference between $y_w$ and $y_l$.

While existing works select data with larger explicit reward margins or smaller implicit reward margins for training, we illustrate how these metrics can yield contradictory evaluations for the same data in Figure 1—demonstrating inaccuracies since at least one evaluation must be incorrect.

1. In Example I, the chosen response is clearly correct and more preferable, resulting in a substantial explicit reward margin (indicated by the blue bar), and thus deemed "high-quality" by $M_r$. However, the large implicit reward margin (shown by the orange bar) suggests that the current model $\pi_\theta$ can already distinguish the preferences between $y_w$ and $y_l$, thus being "low-quality" by $M_\pi$. Consequently, despite what $M_r$ indicates, this data cannot further improve preference learning as the model is already well-aligned on this sample—a scenario where relying on $M_r$ is misleading.
2. Example II presents two nearly identical responses $y_w$ and $y_l$, resulting in a very small explicit reward margin, and thus being "low-quality" by $M_r$. Similarly, the model cannot tell the preference between such similar responses and produces a small implicit reward margin, which will be evaluated as "high-quality" by $M_\pi$. Given the negligible preference between the responses, this data should be considered as low-quality—highlighting how reliance on $M_\pi$ can also be fallacious.

These examples underscore the shortcomings of relying solely on a single margin, and motivate us to derive a more propoerly designed metric:

• Since $M_r$ meaures the difference between $y_w$ and $y_l$, it serves as an alignment target indicating how the preference should be on the specific data.
• In contrast, $M_\pi$ corresponds to the model's current state, indicating how the preference of the current model is on this data.
• As shown in the two examples, evidenced by the examples, data quality cannot be determined merely by a large target value ($M_r$) or a small current value ($M_\pi$); rather, it is the gap between the current and target preferences that holds significance.
• From this insight, our proposed $M_{AP}$ metric evaluates preference data quality by quantifying the gap from the current implicit margin to the target reward margin, thereby measuring the potential for further alignment training.

We acknowledge that the connection between the contradictions presented in Figure 1 and the core concept of the proposed $M_{AP}$ metric is not sufficiently clear in the current figure. To address this, we will enhance Figure 1 by including more informative text within the figure or title to explicitly highlight that existing metrics fail on these two examples because they solely focus on either the current model or the target value, neglecting the crucial gap between them. Thank you once again for your insightful suggestion!