Official Response to Reviewer xAZC

We thank reviewer xAZC for the valuable feedback. It really encourages us to see that the reviewer finds our work to be a good contribution to the field, regarding pointing out the value of suboptimal responses and countering the likelihood decline trend in alignment. We answer the reviewer's question below:

Q1: How do you compare your work with the paper titled Direct Preference Optimization with an Offset (https://arxiv.org/pdf/2402.10571).

A1: We thank the reviewer for mentioning this paper. ODPO also wants to leverage the reward gap information for better optimization. We notice that ODPO is followed by some recent work like SimPO due to its effectiveness, which reviewer UYN1 has mentioned.

In simple words, ODPO modifies the DPO loss function to be $L_\theta^\text{ODPO} = - \log \sigma(r_\theta(x,y_w) - r_\theta(x,y_l) - \gamma)$, where $\gamma$ is determined by the reward gap in datasets.

Connection with our methods:

1. InfoNCA and ODPO share similar motivations but have distinctive approaches. In our understanding, ODPO can only guarantee the learned reward gap is larger than $\gamma$, while InfoNCA converges if and only if the learned reward gap is equal to $\gamma$.
2. InfoNCA can be directly used for multiple responses while ODPO is designed for pairwise responses.
3. Based on our experience, the above two distinctions won't cause very much empirical difference in practice regarding language quality. We personally feel InfoNCA loss is more mathematically elegant, though. After all, it is our own method :)
4. Compared with NCA, ODPO is optimizing relative rewards just like DPO, so we guess the problem of logp decline still exists. NCA may perform better in preventing the logp decline trend.

Official Response to Reviewer WFsu

We appreciate the reviewer WFsu for the very detailed comments on our paper and are glad that the reviewer finds our paper to be helpful. Below we address the reviewer's questions and hope this can help the reviewer increase the confidence of our work.

Q1: The paper assumes a model with unlimited capacity. Why is this important and what are the implications otherwise?

A1: We assume unlimited model capacity mainly because we want our proof to be strict. There's nothing special here.

Imagine we want to fix a function $f(x)=x^2$ using neural networks. We must first assume our model is expressive enough (unlimited capacity) to fully achieve convergence. Otherwise, if our model $f_\theta$ is simply a linear MLP (limited capacity), we cannot achieve $f_\theta(x)=x^2$ no matter how hard we try.

Q2: It is not obvious to me how equation 11 leads to predicting the model source of a single response in Figure 3

A2: Eq 11 is the NCA loss function. For convenience, we suppose there are only $K=2$ responses $y_1$ and $y_2$, with their respective reward $r_1 > r_2$.

$L_\theta^\text{NCA} =- \sum_{i=1}^2 \left[\frac{e^{r_i}}{e^{r_1}+e^{r_2}} \log \sigma (r_\theta(x,y_i)) + \frac{1}{2} \log \sigma (-r_\theta(x,y_i))\right]$

We compare with $L^\text{InfoNCA}_\theta$ for reference:

$L_\theta^\text{InfoNCA} = - \sum_{i=1}^2 \left[\frac{e^{r_i}}{e^{r_1}+e^{r_2}} \log \frac{e^{r_\theta(x,y_i)}}{e^{r_\theta(x,y_1)}+e^{r_\theta(x,y_2)}}\right]$

According to our proofs in the paper, one optimal solution for the above training loss is that

$r_\theta(x,y_1) = r_1$ and $r_\theta(x,y_2) = r_2$.

However, if we let $r_\theta(x,y_1) = r_1 -12345.0$ and $r_\theta(x,y_2) = r_2 -12345.0$, we will find this is still the optimal solution for InfoNCA.

$- \sum_{i=1}^2 \left[\frac{e^{r_i}}{e^{r_1}+e^{r_2}} \log \frac{e^{r_\theta(x,y_i) - 12345.0}}{e^{r_\theta(x,y_1) - 12345.0}+e^{r_\theta(x,y_2) - 12345.0}}\right]=- \sum_{i=1}^2 \left[\frac{e^{r_i}}{e^{r_1}+e^{r_2}} \log \frac{e^{r_\theta(x,y_i)}}{e^{r_\theta(x,y_1)}+e^{r_\theta(x,y_2)}}\right]$

This is exactly why we say InfoNCA only targets relative rewards. The loss function is happy as long as $r_\theta(x,y_1) - r_\theta(x,y_2)$>0. However, $r_\theta(x,y_1)$ can be a very negative number, which we do not want.

For the NCA loss, it is otherwise, because inside $\sigma$ there is only a single reward $\sigma(r_\theta(x,y_i))$ instead of many relative model rewards $\log \frac{e^{r_\theta(x,y_i)}}{e^{r_\theta(x,y_1)}+e^{r_\theta(x,y_2)}}$.

$L_\theta^\text{NCA} = L_\theta^\text{NCA}(x,y_1) +L^\text{NCA}_\theta(x,y_2)$

$L_\theta^\text{NCA}(x,y_1) =- \left[\frac{e^{r_1}}{e^{r_1}+e^{r_2}} \log \sigma (r_\theta(x,y_1)) + \frac{1}{2} \log \sigma (-r_\theta(x,y_1))\right]$

$L_\theta^\text{NCA}(x,y_2) =- \left[\frac{e^{r_2}}{e^{r_1}+e^{r_2}} \log \sigma (r_\theta(x,y_2)) + \frac{1}{2} \log \sigma (-r_\theta(x,y_2))\right]$

$(r_1>r_2, \frac{e^{r_1}}{e^{r_1}+e^{r_2}} > \frac{1}{2})$

You can see that $L_\theta^\text{NCA}(x,y_1)$ is only influenced by $r_\theta(y_1)$ and is unaffected by $r_\theta(y_2)$. For NCA, if $r_\theta(x,y_1) = r_1 -12345.0$ is a very negative number, $L_\theta^\text{NCA}(x,y_1)$ would surely be very large and contradicts the training loss minimization. In other words, NCA forces $r_\theta(x,y_1)$ to be positive and $r_\theta(x,y_2)$ to be negative respectively and independently. This is why we say Eq 11 leads to predicting the model source (optimize absolute reward) of a single response.

We'd also like to refer the reviewer to Appendix H, where there is a more detailed example.

Official Response to Reviewer UYN1

We thank reviewer UYN1 for the praise and suggestions regarding our work. We are pleased to see the reviewer's acknowledgment of our theoretical contribution in bridging the gap between DPO and classic contrastive learning theories. Below we address the reviewer's questions and hope our responses can help increase the reviewer's confidence in our work.

Q1: The proposed method looks trivial by integrating classic contrastive learning theories. Different methods coming out at the same time. They look similar and not easy to find which is better, such as the concurrent work: SimPO.

A1: There are indeed numerous recent approaches for solving alignment tasks, such as DPO/KTO/IPO/ORPO/SimPO, and it is challenging to definitively determine which is superior. Still, we would like to highlight some key features of InfoNCA/NCA that we find to be unique.

1. Unlike most previous methods, InfoNCA/NCA is not based on any kind of preference model assumptions, such as Bradley Terry or Plackett-Luce models. It is directly derived from existing contrastive learning Bayes theories and is deeply connected with NCE methods that have been widely validated. The theoretical cleanness should provide more confidence for the users.
2. Our methods, unlike SimPO, can handle continuous explicit rewards instead of just preference data. To the best of our knowledge, there are very few work/algorithms that are directly motivated by this task.
3. Our method guarantee strictly convering to the target optimal policy. The theoretical proofs are strict and purely Bayes (Appendix ). This offers a unique theoretical perspective for understanding existing alignment approaches, as reviewers EyJu and EZsK have also pointed out.

Q2: It would be better to list more training details such as hyper-parameter search space.

A2: We'd like to refer the reviewer to Appendix C of the paper, where we have listed all important training details (we think). Source code is also provided to ensure reproducibility.

We ablate $\beta \in \{3e-4, 1e-3, 3e-3, 1e-2, 3e-2, 1e-1, 3e-1, 1.0\}$ and $\alpha \in \{0.01, 0.1, 0.33, 1.0, 3.33\}$ and $K \in {2,4}$ for ablation studies. For the main experiment Table, we all use consistent hyperparameters same as the Zephyr DPO baseline for a fair comparison and to avoid hyperparameter overfitting.

We will be glad to provide any additional experiment details if the reviewer is interested in any specific aspects.

Official Response to Reviewer EZsK (1/2)

Q1: The motivation is weak. The description that DPO can only be used for pairwise preference data is wrong (such as Line 31 and 43). In fact, the appendix of DPO says that it can be applied to multiple preference data based on the Plackett-Luce model.

A1: We thank the reviewer for the valuable feedback on the PL model. PL can indeed be combined with the DPO algorithm to handle multiple responses. Since this method is only mentioned in the appendix and does not have any accompanying experiments in DPO's paper, we did not notice this part originally.

However, we respectfully disagree with the reviewer regarding "weak motivation". The PL+DPO method does not hurt the core contribution of our work. Our paper is titled "Alignment of LMs with Explicit Rewards". NCA is strongly motivated to handle explicit reward data that naturally can be applied to responses of arbitrary numbers. In contrast, like the BT model, PL model is still a preference/ranking-based method and cannot handle reward datasets.

We have rephrased all related descriptions in our paper to avoid previous unstrict claims. (We cannot update our official manuscript during rebuttal due to review policy, though)

For instance, (Line 31)

DPO is only tailored for preference data with K = 2 responses per instruction x.

(Line 43)

However, 43 unlike DPO which is built upon assumptions of Bradley-Terry models preference models like Bradley-Terry or Plackett-Luce model, InfoNCA is strictly derived from InfoNCE

We have also conducted an additional experiment to compare with PL-based DPO methods:

**Q2: The proposed method InfoNCA uses both preference data and reward data for training, .....more computational overhead compared with PPO and DPO.....Is it a fair comparison if the methods use different magnitudes of data? **

A2: We respectfully disagree with the reviewer's comment and think there is clearly some misunderstanding of our experiments, which we hope to clarify:

1. Our method can handle both preference and reward data, but it only requires one data type for training. As a matter of fact, we use either 100% of preference data or 100% of reward data throughout our experiments. There is no mixed dataset used. (Refer also to A3).
2. PPO is at least 10 times more computationally expensive than our method. Our experiment finishes in a bit more than 1 hour using H100 GPU sever, while PPO takes about a day because it requires online data sampling and learning separate reward model/value model/policy model.
3. Our method is as efficient as DPO, becuase DPO is simply a special case of our method. If DPO uses only pairwise responses, and InfoNCA uses 4 responses, then InfoCNA requires about 2 times the computational resource. This is reasonable because there are two times of data. Besides, using the same amount of data, our methods outperform DPO (Table 2, Line 212 of our paper).
4. We have compared with previous work that uses the same amount of data with our method (Line 208, Table 2, Table 3), so we believe the comparison is fair. Our results show that even previous work is given the same data magnitude, InfoNCA still outperforms them.

Results with the same data:

Q3: What is the amount and percentage of reward and preference data in each dataset

A3: Please refer to the first point in A2.

Q4: Since InfoNCA and NCA are reward-based methods, why not compare them with PPO? Table 2 should also include PPO.

A4: We thank the reviewer for this suggestion, and present experimental results below.

Initial results show that PPO underperforms DPO or InfoNCA methods, perhaps due to PPO's inherent instability or our insufficient hyperparameter tunning.

Why not previously directly compare NCA with PPO? PPO and InfoNCA target different kinds of "reward". InfoNCA targets REAL dataset reward and features direct optimization. In contrast, PPO targets a Reward Model and requires online data sampling. We prioritize comparing with DPO-like methods since DPO is actually our special case so we can keep exactly the same hyperparameters.

Reminder

QA5: We refer the reviewer to the global rebuttal posted at the top of the webpage for the rest (EZsK part 2/2) of our response. This is due to the severe page limit.

Dear reviewer,

We copy the second part of our rebuttal response here to ease reading. In our responses, we have compared with more baselines and clarified the details of the data we are using based on your suggestions. We hope we can address your concerns and are glad to answer any further questions you might have.

Thank you again for your valuable feedback!

Official Response to Reviewer EZsK (2/2)

Q5: suggest comparing it with PRO (https://arxiv.org/pdf/2306.17492).

A5: PRO is a quite related work of ours because it can also handle response data of arbitrary length. It is also highly related to the PL method as we have discussed in A1. Still, PRO is essentially a ranking/preference-based method, while (Info)NCA are reward-based methods.

Theory comparison:

We have added a new section as Appendix G.2 in our paper to compare with PRO theoretically: (cannot update the manuscript during rebuttal, though)

The main difference between PRO and the PL+DPO method is that

1. PRO has different reward formulations. It formulates $r_\theta$ as the average log-probability of a sentence $\frac{1}{\|y\|}\Sigma\log \pi_\theta(y|X)$ instead of the log-probability ratio $\log \frac{\pi_\theta(y|x)}{\mu(y|x)}$ as is required by DPO.
2. Because there is no $\mu$ in reward models. PRO needs to be additionally regularized by an SFT loss in order to stay close to the pretrained LLM $\mu$.

Additional Experiments:

Overall we find PRO (PL+SFT) method helps, but slightly underperforms the PL+DPO method. We are not confident about this conclusion because we did not faithfully replicate PRO according to its source code, but only modified our trl implementation of DPO+PL to test PRO methods. We mainly search sft_weight $\in$ [0.01, 0.05, 0.1 0.5], and keep other hyperparameters consistent with train_hh.sh of PRO repo, or the trl/DPO settings.

We appreciate the reviewer for the prompt and positive feedback and are glad we can address the reviewer's concerns.

Official Response to Reviewer EyJu (1/2)

Q1: (Major Concern) Fundamental Assumption is we can sample from the optimal policy $\pi^\*$, which is practically impossible because we can only access non-optimal LLM policy $\mu$.

A1: We'd like to clarify that the availability of $\pi^\*$ is NOT a concern of our proposed (Info)NCA algorithm, both theoretically and practically.

We are glad the reviewer is interested in this question, though, because it is a core contribution/difficulty in our algorithm's derivation.

Take InfoNCA loss for instance (Eq.6, Line 110 in the paper):

$L_\theta^\text{InfoNCA} = - E_{p(x) \prod \mu(y_i|x)} \sum_{i=1}^K [\frac{e^{r(x,y_i)/\alpha}}{Z(x)} \log \frac{e^{r_\theta(x,y_i)}}{\sum_{j=1}^K e^{r_\theta(x,y_j)}}]$

Here $\mu$ is the pretrained LLM distribution. We can see the optimal policy $\pi^*$ does not show up in the loss function, which means $\pi^*$ is NOT required. From $E_{p(x) \prod \mu(y_i|x)}$ we can also see that, ideally all responses should be sampled from suboptimal $\mu$ instead of $\pi^*$.

Why $\pi^\*$ does not show up in the loss function? After all $\pi^\*$ is required/used for (Info)NCA algorithm derivation (Figure 3, Line 98 and 146, etc.):

Recall that $\pi^\*:= \mu(y|x)\frac{e^{r(x,y)/\alpha}}{Z(x)}$ is actually defined by the pretrained LLM $\mu$ and a reward function $r$. Thus $\pi^\*$ can actually replaced by $r$ and $\mu$ in practice. This is exactly why we need dataset "rewards". Specifically, this is accomplished by the "rejection sampling" technique. Image we can first sample $K >> 1$ responses $y_{1:K}$ from $\mu(\cdot|x)$, score each response with $r_{1:K}$, then we can resample a single response $y$ from $y_{1:K}$ with the probabilty ratio $p_k \propto e^{r_k/\alpha}$. In this way, the distribution of $p(y) = \mu(y|x)\frac{e^{r(x,y)/\alpha}}{Z(x)}$ can be proved if K is infinitely large.

This idea can be applied to elimitate the requirement of $\pi^\*$ by importance sampling in algorithm derivation: $E_{\pi^*(y|x)} f(x,y) = E_{\mu(y|x)} \frac{e^{r(x,y)/\alpha}}{Z(x)}f(x,y)$

so that

$L_\theta^\text{InfoNCA} = - E_{p(x) \prod \pi^\*(y_i|x)} \sum_{i=1}^K \left[\log \frac{e^{r_\theta(x,y_i)}}{\sum_{j=1}^K e^{r_\theta(x,y_j)}}\right]$

$= - E_{p(x) \prod \mu(y_i|x)} \sum_{i=1}^K \left[\frac{e^{r(x,y_i)/\alpha}}{Z(x)} \log \frac{e^{r_\theta(x,y_i)}}{\sum_{j=1}^K e^{r_\theta(x,y_j)}}\right]$ (proof Line 463 in Appendix)

Q2: In some extreme situations, all the generated responses can be harmful, while our ideal optimal policy should output harmless responses. (Contradict with Fundamental Assumption). If all the scored responses are harmful, the proposed methods can never achieve the ideal alignment optimum ...

A2: We think the above problem arises fundamentally because of the problem definition of alignment tasks instead of our proposed (Info)NCA algorithm.

Recall that most alignment problems define the desired policy distribution as: $\pi^*\propto \mu(y|x)e^{r(x,y)/\alpha}$.

By definition, we not only want $\pi^\*$ to maximize reward $r(x,y)$, but we want $\pi^\*$ to stay close to pretrained policy $\mu$ as well. If all responses are harmful, then $\mu$ must be a very low-quality distributional prior for $\pi^\*$ (All responses should be sampled from $\mu$ as we have explained in A1). Consequently, it is natural that $\pi^*$ may struggle to get high performance because it needs to stay close to a low-quality LLM $\mu$.

Q3: Mixing the human written (golden) response into the preference data, then applying NCE to the prompt and the golden response( combining InfoNCA with some prior work such as APO [1] and SPIN [2].)

A3: We agree with the reviewer. Mixing golden responses into datasets would increase data quality because these responses do not come from the suboptimal policy $\mu$. Instead, they are from a presumably superior policy (human). This strategy is illuminating and would surely be very helpful if we can somehow access high-quality data beyond just a suboptimal policy $\mu$.

The basic idea of APO/SPIN is to perform some GAN-like training and contrast STF/golden data with model-generated responses. Their contrastive training style can be readily combined with (Info)NCA training methods. This is a very promising research topic, though we suppose it is beyond the discussion scope of our current article because we focus on the preference/reward training paradigm. We believe this does not hurt or even support the core contribution of our method.

Q4: Comparison with another baseline: first train an RM with the multi-response preference, and next sample two responses with scores from the learned RM. Then DPO can be conducted on the RM-scored pairwise preference data.

A4: We conduct additional experiments (online DPO setting and PPO settings) and present the results below. RM is pretrained on the UltraFeedback datasets, named UltraRM.

Overall, results that are trained by applying RM underperforms directly finetuned from dataset rewards. We believe this is mainly because of the learned RM is not robust enough compared with the GPT-4 annotator.

Reminder

QA5-QA6: We refer the reviewer to the global rebuttal posted at the top of the webpage for the rest (EyJu part 2/2) of our response. This is due to the severe page limit.

Dear reviewer,

We copy the second part of our rebuttal response here to ease reading. In our responses, we have conducted additional experiments and compared with more baselines based on your suggestions. We hope we can address your concerns and are glad to answer any further questions you might have.

Thank you again for your valuable feedback!

Official Response to Reviewer EyJu (2/2)

Q5: Lack of two-response alignment: compare InfoNCA, NCA, and DPO on a binary-response preference set.

A5: Perhaps we did not fully understand the reviewer's comment. As a matter of fact, the whole section 5.2 aims to compare these algorithms on binary-response preference sets. This includes Table 3, Figure 4 (left), Figure 5, and Figure 6(left).

Note that in pairwise-preference settings, InfoNCA becomes exactly the DPO algorithm, so we mainly compare NCA and DPO (InfoNCA).

We copy the main results below: UltraFeedback (binary)

UltraInteract (binary)

Q6: The real human evaluation results are not reported, which should have made the paper's claims more persuasive.

A6: We have actually reported human evaluation results in the original paper and would like to refer the reviewer to Table 2 (win vs DPO), and Figure 7in the paper.

Still, GPT-4 based evaluation results are most indicative to ourselves because datasets are annotated by GPT-4. After all, you want your training rewards and test rewards to be aligned to make sure the algorithm itself is valid given that our work is somehow more theoretical rather than product-oriented.

Dear reviewer,

Thank you for your valuable comments regarding our submission. We have posted our responses and hope to address your concerns about the fundamental assumption of our method (availability of the optimal policy), as well as the comparison with several baselines such as APO and SPIN.

Could you please take a look at our responses and give us some further feedback so that we can continually improve our work? Thank you so much in advance!

Best regards,

The Authors