Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Model

Concern#1: Does this imply that the proposed training objective relies on a perfectly clean dataset??

Thank you for raising this interesting question. No, it is not a requirement to have a perfectly clean dataset. The reason for the weak negative samples is to provide high variance. Even if some $y_w^j (j \neq i)$ is accidentally better than $y_w^i$ in the training dataset, it should not be a problem. However, if all or too many such $y_w^j$ becomes better than $y_w^i$, we might end up with lower variance as a result, which is not desired according to the theorems (see our response to concern#2 for more information about weak negatives and the variance of negatives).

Concern#2: What is the rationale behind computing the reward for irrelevant x and y pairs? Could the authors provide a more solid theoretical justification for introducing negatives in Eq. (11) and Eq. (12)?

The rationale, as mentioned in the response to Concern#1, is that the irrelevant x and y pairs can provide high variance. The premise of Theorem 3.2 and Theorem 3.3 only states that the Var[Y|Z] has to be positive. However, from the proof of Theorem 3.2, we can also see that the value of Var[Y|Z] is positively associated with the 2nd-order derivative of the energy discrepancy. Therefore, the higher variance will lead to a more convex Energy Discrepancy functional/loss.

We also discuss their intuitive usefulness as a regularizer in section 4.3. The negative gradients in DPO might push the policy parameters towards unexpected directions because they are likely not pointing to the RLHF minimizer. Therefore, the additional negative gradients introduced by the weak negatives can alleviate this issue.

Concern#3: does the performance improvement over DPO depend on the use of a large batch size?

No, because we use 64 as the global batch size for all the models and methods (including DPO) reported in the paper. As for why 64 in particular, this is because it is the highest possible number we can set to prevent our most memory-consuming setting (i.e., EPA's 1:3:10 setting in Table 3) from OOM.

Concern#4: sampling more mismatched responses from the training data instead of the minibatch.

We believe an irrelevant response from a mini-batch is not meaningfully different from an irrelevant response elsewhere, because the mini-batches are randomly sampled from the dataset in the first place. Also, as we mentioned in the response to Concern#2, the value of weak negatives is from the high variance as opposed to where they are sampled from.

Concern#5: EPA and DPO under the same training time?

The x axis in Figure 2.(b) is proportional to the number of prompts (ie., x in the datasets). However, since EPA (1:1:2) has exactly twice the amount of the responses (ie, y in the datasets) used in DPO, we think scaling the dynamics curve of EPA rightward by a coefficient of 2 is close to what you ask for. In such a graph, we do not think EPA achieves faster convergence compared to DPO. However, we also do not intend to make such a claim in the paper. Instead, we do agree the computation cost of EPA is its disadvantage compared to DPO.

Concern#6: more benchmark evaluations, such as Arena-Hard

Thank you for the suggestion. We quickly ran Arena-Hard for EPA against the two most important baselines: DPO and DPO-PL, using the same checkpoints we use in Table 2. The results are as follows. And, we can see that EPA is still better. We will add them to the final version of the paper. However, we believe the widely used MT-Bench and Alpaca-Eval 2.0 are enough for most of the experiments in the paper.

Concern #1: generalizability to more datasets and more base models

(Please refer to our response to reviewer xiBU for our additional experiments using a different dataset and a different model)

Concern #2: limited discussion on computational trade-offs

Thank you for pointing out the limitation of this paper. As the main focus of this paper is on the theoretical benefit of using our EBM instead of BTM to model human preference, we have not included many possible further analyses regarding the computational cost of EPA (which is one among many other possible ways to fit the EBM). However, there is an explicit question raised by reviewer RQg9 (Concern#5) about a computational comparison between DPO and EPA. We hope our response there can provide additional insights.

Concern #3: "Could you clarify how robust that uniqueness advantage remains if your negative sampling is incomplete or unrepresentative (e.g., limited coverage in real-world data)? Specifically, do suboptimal negative samples risk undermining uniqueness—leading to the same drawbacks that can occur under BTM—and, if so, how severe is the impact?"

Thank you for raising these very interesting questions that we think suffice for many future research directions.

From a general point of view, we agree that suboptimal negative samples do risk undermining the uniqueness. Although there are no designated experiments in the paper to show this, we do think there are some related observations in the paper.

• Observation 1. In Figure 2.(b), EPA also degenerates (slightly) after some epochs, which means the dataset might still be not enough.
• Observation 2. In Table 5, as we stated in the paper, the fact that there are two tricks (adding a margin and using a dynamic weight on the loss) that can further boost the performance of EPA means it is empirically not perfect.
• Observation 3. In Table 1, the slope-1 linear regression error is non-zero.

However, in all these cases, we see EPA is still better than DPO. Therefore, although EPA is not robust enough, it is safe to say that it is more robust than DPO. In other words, the error caused by using EPA to estimate the unique MLE of the EBM is milder than the intrinsic flaw of BTM of not being guaranteed to have a unique MLE in the first place.

Concern #1 (also raised by reviewer ZCWk): generalizability to more datasets and more base models

As this is a point mentioned by multiple reviewers, we managed to run quick experiments on another relatively small dataset and on another base model. Although the experiments can never be exhaustive, we believe the results (as follows) should be additional evidence for the generality of our findings. We will include them in the final version of the appendix.

• dataset:

  • source: https://huggingface.co/datasets/argilla/distilabel-intel-orca-dpo-pairs
  • reference: Álvaro Bartolomé Del Canto et al., 2024; https://github.com/argilla-io/distilabel
  • description: a cleansed version of the widely used dataset (Intel/orca_dpo_pairs)

• results: (MT-Bench scores for epoch#1/#2/#3) | base model | EPA | DPO | |------------|--------------------|----------------| | Llama3-8B | 6.84/6.98/7.04 | 6.94/6.83/6.71 |

Concern #2: A.5 requires an assumption of the non-negativity of $\pi_{\text{ref}}(x, y)$

Thank you for the suggestion. Yes, the assumption is required, which is true for both DPO (as explicitly stated in (Rafailov et al., 2023)) and EPA. Although this is a mild assumption that does not affect the main theories of our paper, we will add this to the final version of the appendix for comprehensiveness.

Concern #3: Does DPO-PL suffer from the same issue as DPO, particularly in terms of the non-uniqueness of solutions?

Yes, we think so. The proof will be analogous to that of DPO. It is not difficult to show that proposition B.5 is also true for the Plackett-Luce Model by only checking that the reconstructed reward $\tilde{r}_{\hat{\theta}}$ also leads to the same expected likelihood of data. We will add the exact proof to the final version of the appendix.

Besides the proof per se, there is an intuitive understanding of why this is the case. As we have mentioned in the first paragraph of subsection 3.1., the Plackett-Luce Model also just models human preference among a finite number of candidates. This is problematic when the actual space of candidates is infinite.

Concern #4 (also raised by reviewer PHXo): detailed differences between EPA and other multi-response losses such as DPO-PL, infoNCA, etc.

We are thankful that some reviewers point out the similarity shared by EPA and other multi-response losses. This is exactly the reason why we include many such losses as baselines for comparison in Table 2. However, for a clearer description of the differences, we hope the following table can suffice. We will add it to the final version of the paper.

Concern #1: The implication of Proposition B.5: the issue of the dataset or the issue of the DPO algorithm?

We believe a reasonable definition of a good algorithm is one that works for datasets "easy" to collect. If there are too many constraints to collect the datasets, it is indeed an issue of the algorithm.

B.5 gives a necessary constraint which is itself not easy to meet, i.e., we can hardly sample all $y$s from an infinite space given a finite data point budget.

What makes the datasets "even harder" to collect is that the B.5 constraint is not sufficient. This means even if we somehow are able to sample all $y$s in an infinite space, we still do not guarantee such a dataset will allow the DPO algorithm to work. As mentioned in the paragraphs before B.5, one can refer to (Ford, 1957; Bong & Rinaldo, 2022) for more situations for BTM to have non-unique MLEs.

Concern #2: "why EBM still have unique minimizers when some y* never appear in their training data?" & "what bout EPA"

As shown in Theorem 3.1 (also B.4), EBM having a unique minimizer to its maximum likelihood estimation is not dependent on any requirements on the structure of $p(y|x)$. Even when some $p(y^*|x)=0$, the proof we provide to Theorem 3.1 still holds.

For EPA, we acknowledge that there will be an error in the approximation (illustrated as the fuzzy dashed curve in Figure 1). This is why we need to check empirically whether or not the error introduced in EPA is smaller than that of DPO. For example, Table 1 explicitly shows that the slope-1 linear regression error is smaller for EPA, although the error is non-zero.

Concern #3: Difference between EPA and RPO, infoNCA, etc.

We noticed that the RPO paper was posted to arxiv.org after the submission of our paper. So, we were not aware of this concurrent work. Although EPA itself may or may not be a special case of RPO, we argue that our paper is more about the underlying energy-based preference model as opposed to EPA (which is only one of many possible ways to fit the EBM).

(see our response to xiBU for EPA vs other losses)

Concern #4: Why setting $\beta=0.01$ for Table 2?

As mentioned in the paper's subsection 5.1.4, the philosophy to set $\beta=0.01$ in Table 2. is to treat it as a given part of the RLHF objective (Eq.1). As for the reason why we choose 0.01 in particular, it is two-fold.

• Firstly, we did tune the $\beta$ for multiple values ranging from 0.01 to 0.5 (see Figure 2.(a)). $\beta$ close to 0.01 (low-KL region) will make both EPA and DPO achieve higher rewards.
• Secondly, $\beta=0.01$ is also the default setting for NCA and infoNCA (Chen et al., 2024a) and also a recommended setting for Mistral-based DPO (Tunstall et al., 2023) and KTO (Ethayarajh et al. (2024) ). For IPO's best $\beta$, we did preliminary experiments with $\beta=0.1$ and $\beta =0.01$, and found $\beta=0.01$ works better. Therefore, we believe setting $\beta =0.01$ is a reasonable choice. We will add these results (MT-Bench scores) in the final version of the appendix: | $\beta$ | IPO (epoch#1/#2/#3) | |--------|----------------------------------------| | 0.1 | 6.73/6.88/6.87 | | 0.01 | 7.20/7.31/7.23 |

Concern #5: "how are the weak negatives added to DPO or DPO-PL training?" & "what about InfoNCA?"

In the caption of Table 4., we have briefly stated how the weak negatives are added. The following are some examples for "+UF-WEAKx1".

suppose the original pairwise dataset has 3 data points: $(y_w^1, y_l^1), (y_w^2, y_l^2), (y_w^3, y_l^3)$

• For DPO, weak negatives are added "as additional $y_l$ to be paired with the original $y_w$". This means we add something like these to the original dataset: $(y_w^1, y_l^2), (y_w^2, y_w^4), (y_w^3, y_l^1)$.

• For DPO-PL, "as additional negatives ranked after the $y_w$ and $y_l$". This means we replace the original dataset with something like ${(y_w^1, y_l^1, y_w^2), (y_w^2, y_l^2, y_l^1), (y_w^3, y_l^3, y_l^2)}$. The K-wise ranking information required by DPO-PL can be implicitly inferred here. For example, for $(y_w^1, y_l^1, y_w^2)$, the ranking is $y_w^1 > y_l^1 > y_w^2$.

We do not consider InfoNCA because the purpose of Table 4. is to study if EPA's main advantage comes from introducing more computation alone or being the product of a better preference model. However, InfoNCA is not based on a "preference model". Therefore, we do not think it is relevant for the purpose here.

Concern #6: more data points regarding the computational cost.

We provide our actual training time to support our linear-time-complexity argument in sec 5.2.3:

Concern #7: the benefit of weak negatives

(please refer to our response to reviewer RQg9's concern#2)