CREAM: Consistency Regularized Self-Rewarding Language Models

Thank you for your constructive comments! We respond to your questions below and would appreciate it if you could let us know if our response helps address your concerns.

Weakness 1: It's not clear to me why one would want to use self-rewarding LMs at the 7B scale, as it would not be too difficult to get preference annotations from a larger model and train on them instead.

Response 1:

• First, we believe it's meaningful to democratize the self-rewarding paradigm to LLMs of community affordable size, such as 7B models. Advanced and larger LLMs (e.g., ChatGPT) are not always accessible, especially in specific application scenarios like those involving medical or privacy-sensitive data.
• We want to emphasize that our method is theoretically applicable to LLMs of any size (not limited to 7B). However, we present empirical results with 7B LLMs due to limited computational resources.
• To validate our method with larger LLMs, we rented h100 gpus to test the CREAM on Llama-2-13B model without any hyperparameter tuning. The results confirm that our method remains effective for the 13B LLM.

Weakness 2: The algorithm appears to treat consistency as a property of the model \theta_t, but I wonder whether it would be better to treat it as a property of the data. That is, for any model, there will be some examples for which preference can be assessed confidently, and others for which it cannot. It would be ideal to use the full DPO loss for the first type of examples, and regularize strongly on the second type. Instead, the proposed approach seems to set the regularization parameter based on some property of the optimization path: as long as the LM has not changed its rankings from time t-1, it will use the standard DPO objective.

Response 2: Thank you for your suggestion. The introduced (in)consistency serves as the uncertainty quantification for the model to prevent the model itself being overconfident due to the stochastic training noise or incorrectly labeled prefernece data. Such a uncertainty, usually referred to as epistemic uncertainty, is a property for each model.

We further add two variants to explore treating the consistency as a poperty of the data instead of the model:

• Threshold = x: This variant uses the normal DPO for samples (confident dataset) whose consistency rate $C > x$, and uses ours C*DPO + (1-C)*RDPO to regularize the training for other samples (unconfident dataset).
• CREAM w/ Dynamic Consistency: Instead of original CREAM using the average consistency rate across the dataset, this method uses dynamic consistency rate for each sample, using each sample's own consistency rate to regularize the training.

The results are shown as below.

• We observe that as the threshold increases beyond a certain value (including more samples for regularization), the performance gains converge. Actually, CREAM is compatible with using a threshold to fine-grainedly select the data to regularize, however, this inevitably introduces an additional hyperparameter. Besides, CREAM already achieves sufficiently good performance, without this added complexity.
• Compared to the "CREAM w/ Dynamic Consistency" variant, our method, which utilizes the average consistency rate, significantly reduces the variance in estimating dataset uncertainty, resulting in improved performance.

Weakness 3: I would like to see more analysis of the sum of DPO losses. In particular, what is the effect to the policy of applying the update in step 14 of the algorithm when C=1/2? My guess is that the probability for both y+ and y- decreases, but it would be great to have some formal analysis because this situation could be relevant, e.g., at the beginning of training.

Response 3: Thank you for your question! Here we take the gradient analysis for the sum of the DPO losses.

• Recall that CREAM uses the $L = C * D + (1-C) * RD$ for training, where $C$ is the consistency rate, $D$ and $RD$ are DPO loss and ReverseDPO loss, respectively.
• We can simplify the DPO loss by substituting the rewarding gap $[\log [\pi_\theta(y+) / \pi_\text{ref}(y+)] - \log [\pi_\theta(y-) / \pi_\text{ref}(y-)]]$ as $x_\theta$, then, the $D$ loss is written as $\log \sigma (x_\theta)$, and $RD$ loss is written as $\log \sigma (-x_\theta)$.
• Recall that the sigmoid function $\sigma$ has the poperty that $\sigma(x) = 1 - \sigma(-x)$. Then, $RD$ can be converted to $\log (1-\sigma(x_\theta))$. Our regularized loss $L$ is $C \log \sigma (x_\theta) + (1-C)\log (1-\sigma(x_\theta))$.
• The gradient of $L$ with respect to model params $\theta$ is $\frac{\partial L}{\partial \theta} = (C-\sigma(x_\theta)) \frac{\partial x_\theta}{\partial \theta}$.
• When $C = 1/2$, it is pushing the model to learn $\sigma (x_\theta) = 1/2$, i.e., encouraging the model to have $[\pi_\theta(y+) / \pi_\text{ref}(y+)] = [\pi_\theta(y-) / \pi_\text{ref}(y-)]$. And so far there is no theoretical evidence that the likelihoods of both y+ and y- will decrease. However, based on empirical results of existing works[1,2,3], the $\frac{\partial x_\theta}{\partial \theta}$ part will decrease both likelihood of y+ and y- but increase their gap. Thus, the combined optimization will potentially lead to:
  • decrease both likelihood of y+ and y-
  • increase the gap between y+ and y- but stay close to $[\pi_\theta(y+) / \pi_\text{ref}(y+)] = [\pi_\theta(y-) / \pi_\text{ref}(y-)]$

Additionally, we supply experiment results of CREAM with $C=1/2$ as follows. $C=1/2$ is generally better than normal SRLM, which confirms the effectiveness of the proposed regularization.

[1] Rafailov, Rafael, et al. "Direct preference optimization: Your language model is secretly a reward model." Advances in Neural Information Processing Systems 36 (2024).

[2] Azar, Mohammad Gheshlaghi, et al. "A general theoretical paradigm to understand learning from human preferences." International Conference on Artificial Intelligence and Statistics. PMLR, 2024.

[3] Park, Ryan, et al. "Disentangling length from quality in direct preference optimization." arXiv preprint arXiv:2403.19159 (2024).

Question 1: The method seems to be applied to LMs that have already been post-trained (LLAMA 2 & 3), which is perhaps why the impact is less than in Yuan et al. Would it be possible to apply the method to a model immediately after pretraining or instruction tuning?

Response 4:

• Our method does require the model to have some initial alignment capability, as the adopted DPO rewarding relies on the model being aligned, otherwise the rewards would not be meaningful. The relatively weaker results of Llama-2 compared to Llama-3 can support this point.
• Note that in the self-rewarding scenario, the original paper SRLM[4] also requires the model to be "post-trained" version, where they adopt Llama-2-70B-Chat as the initial model (M0).
• We conduct additional experiments to test the effectiveness of CREAM with LLMs that are not post-trained. Specifically, we use Llama-3-8B-NO-Chat-Version as the base model. The results, shown below, reveal that:
  • M0 performs poorly due to its lack of instruction following ability. However, after training on a small amount of seed SFT data, M1 demonstrates improved performance, especially on the reasoning task GSM8K.
  • Both SRLM and CREAM can effectively boost the performance, while CREAM often provides greater gains.
  • We find that the performance gains of CREAM in the 3rd iteration (M2 -> M3) exceed those in the 2nd iteration (M1 -> M2). This may be because M2 has better alignment than M1, and the effectiveness of our method relies on the model's alignment capability.

Based on these experimental results, we conclude that SRLM-like methods can still be applied to pre-trained models, provided they have undergone slight alignment (e.g., through SFT training).

[4] Yuan, Weizhe, et al. "Self-Rewarding Language Models." Forty-first International Conference on Machine Learning.

Thank you for your time and efforts reviewing our paper. We sincerely value your insightful suggestions and feedback. We would like to address your concerns by point-to-point response as follows.

Weakness 1: The biggest weakness of this paper is that it is not contextualized well with prior work on reward hacking and using ensembles to mitigate this. Given that proposing variants of DPO and related algorithms is increasingly crowded, this is especially important. Beyond discussing and comparing with this prior work, it is also important to understand how pessimism-based approaches compare, both theoretically and empirically, with the proposed methods based on agreement metrics. Similarly to this paper, there is a growing body of work that aims to mitigate reward hacking through reward model ensembles (e.g. https://arxiv.org/abs/2310.02743, https://arxiv.org/abs/2312.09244, etc.). These papers provide analysis showing various correlation metrics during RLHF. “Disagreement” between reward models can be penalized by considering pessimistic aggregations ......

Response 1: Thank you for pointing out those related works. We will add a paragraph in Sec. 2 Related Work in the updated version, discussing the relationship between these works and CREAM. In particular, the major difference is that both reward hacking and ensemble methods are dealing with the reward model. In this case, these works add the conservative value estimation for the estimated reward. However, in CREAM, we are only collecting the preference data using LLM's self-assessment and do not use the exact value of the reward. In this context, adding the conservative value function to the reward might not be effective because we are comparing the estimated rewards for preference data instead of maximizing the estimated reward function.

To tackle this problem, CREAM employs regularization to mitigate the over-estimation of the reward. We would like to point out that in the offline RL literature[1,2,3], both regularization and conservative value estimation (such as taking the minimum value over a set of estimation) are well-known methods for implementing the 'pessimism' in offline RL. Specifically, CREAM considers the regularization as the KL divergence between the preference model $P_{\theta}$ and the uniform Bernouli distribution so that the regularization would prevent the LLM from being overconfidence to one response $y$ compared with the other response $y'$. We would also like to highlight that this regularization is extremely important in SRLMs, where both $y$ and $y'$ are generated by the language model itself during the training process.

[1] Kumar, Aviral, et al. "Conservative q-learning for offline reinforcement learning." Advances in Neural Information Processing Systems 33 (2020): 1179-1191.

[2] Farahmand, Amir, et al. "Regularized policy iteration." Advances in Neural Information Processing Systems 21 (2008).

[3] Schulman, John. "Trust Region Policy Optimization." arXiv preprint arXiv:1502.05477 (2015).

In addition to this dissusion, we also include an ablation study on using the aforementioned conservative value estimation methods (Ensemble-Worst) and Ensemble-Mean in SRLMs. Specifically, ensemble method trains 3 different $\pi_\theta$ models with 3 different learning rates [7e-7, 1e-6, 3e-6] to serve as the ensemble models in each iteration, since no external models can be involved in the self-rewarding scenario. Then, these three models would independently reward and rank the responses:

• Ensemble-Worst selects the minimum reward for ranking the response.
• Ensemble-Mean selects the average reward for ranking the response.

The results below suggest that CREAM has advantage against Ensemble methods in the alignment performance a cross iterations. Additionally, compared with the ensemble model which requires a batch of reward models, CREAM only requires the model from the last one iteration and thus would be more efficient than the ensemble methods.

Question 1: Equation 3.4 - My understanding is that typically SFT is performed first, and then DPO, rather than optimizing both objectives jointly, as this equation implies.

Response 4: Yes, in our framework, we combine these two objectives for the convenience of unifying those existing iterative preference fine-tuning works. However, this does not impact the subsequent theory analysis of the rewarding bias issue. Besides, as indicated in Algorithm 1, we first apply SFT training and then DPO training in our implementations, which aligns with the approach used in original SRLM[7].

[7] Yuan, Weizhe, et al. "Self-Rewarding Language Models." Forty-first International Conference on Machine Learning.

---

Question 2: grammar mistakes & other nits

Response 5: Thank you for pointing out those grammar mistakes and helpful suggestions on polishing this paper. We will integrate all of them into the revised paper. We believe this revision will enhance both the readability and clarity of the paper. Additionally, we want to clairfy that $[k] = \{1, 2, \dots, k\}$ represents for the set of positive integer values, while sets denoted as calligraphic uppercase letters (e.g., $\mathcal{D}$) refer to a general sets whose elements and size are context-dependent. We have revised the descriptions to avoid potential confusion.

Weakness 2: The motivation for introducing the framework in 3.1 was not clear to me. How is it different from prior work, e.g. Yuan et al., 2024? Why is it necessary to introduce this new framework to evaluate the key contribution of the paper, i.e. the impact of the consistency reward? This made the experiments more difficult to analyze, because it was unclear whether the only change between "SRLM" and "CREAM" was the consistency reward or whether other factors contributed to the difference.

Response 2: We would like to highlight our contribution by proposing a universal framework including several existing iterative preference fine-tuning works, which not only includes the work of Yuan et al., 2024 (SRLM)[4], but also covers other iterative eference fine-tuning works[5,6]. This universal framework enables us to better design some theoretical-grounded methods tackling with the limitations of the existing algorithms, such as the overconfident in SRLMs. As in CREAM, we propose adding consistency-based regularization as presented in Eq. 3.5 for SRLM. Another contribution of the proposed framework is that we can convert the regularzation into a easier-to-implement label smoothing DPO method.

In conclusion, although the difference between the CREAM and SRLM mainly lies in calculating the consistency rate and using it in DPO training stage, the CREAM delivers more theoretical insights on introducing implicit regularization through this consistency rate and offering a universal understanding of the iterative preference-based fine-tuning works.

[4] Yuan, Weizhe, et al. "Self-Rewarding Language Models." Forty-first International Conference on Machine Learning.

[5] Lee, Harrison, et al. "RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback." arXiv e-prints (2023): arXiv-2309.

[6] Wu, Yue, et al. "Self-play preference optimization for language model alignment." arXiv preprint arXiv:2405.00675 (2024).

---

Weakness 3: The overly formal exposition of the proposed method makes the intuition difficult to follow. For example, the definition of the regularization loss, introduced in equation 3.5, does not include any intuition for why this term was chosen.

Response 3: The intuition behind the regularization loss in Eq. (3.5) is explained as follows: Consider two response $y$ and $y'$ generated by the same language model using the same input prompt $x$. When the language model is required to self-judge the preference order between $y$ and $y'$, intuitively, the Bayesian prior for $y \succ y'$ or $y \prec y'$ should be equal. The KL regularization proposed in Eq. (3.5) is suggesting that the preference model $P_{\theta}(y \succ y')$ should approximate a uniform distribution. Therefore, we use the KL divergence between the preference model and the uniform distribution to regularize this distance. We will include this explanation into the paper for better readability in the updated version.

This indicates that if we use the same, single model for reward distillation, the model can hardly be improved.

Thanks, I acknowledge that since your rewards are coming from the intrinsic reward model of the policy, this is different from the setting of prior work, and much of it does not apply directly. This also appears to be different from the original SRLM paper referenced as a baseline, which proposes using a special prompt for generating rewards ("LLM-as-a-Judge").

Thanks for the discussion, and I apologize for my confusion. I now have a revised opinion of the paper:

• Within the proposed scope of using the intrinsic reward model of the policy to generate rewards for self-improvement, the methods appear to be novel, as I am not aware of prior work that studies this setting. From a presentation perspective, this difference from prior work does not seem to be emphasized as significantly as it should be. It doesn't appear to be mentioned until Line 167 in section 3.1. If this is indeed the first paper proposing to use intrinsic rewards from the policy for self-improvement (and if this is why methods from prior work are not applicable), that seems significant and worth clarifying for the reader earlier and more prominently.

• The idea of using the intrinsic reward model from the policy as a source of rewards for improving the policy does not seem intuitively sound, but the authors demonstrate that their method works empirically, at least to the extent that it can nearly match using a reasonably strong external RM as a source of rewards. (Calling this baseline "oracle" seems misleading)

• If the method were instead using a source of rewards different from the policy (i.e. the SRLM setting using "LLM-as-a-judge", or a different RM), then there are many methods from prior work that are relevant and could potentially serve as effective regularization.

It's not clear to me how interesting the proposed scope is to the community (why not just get preference judgements from a stronger RM), but I am raising my score because some of the weaknesses in my original review have been resolved.

Thank you for your valuable feedback to help us improve this paper! We would like to answer your question by point-to-point response. We greatly appreciate it if you could let us know if our responses address your concerns.

Weakness 1: In the line #244, there is a lack of information about the reason for using reversed preference order.

Response 1: Since our training is based on the preference data, it is intuitive to adopt a reversed preference order if there is evidence that the annotated preference data is reversed. In our implementation, we actually apply the label smoothing to the DPO training using $C * DPO + (1-C) * RDPO$, where $C$ acts as the label smoothing factor. Here, Label smoothing is a well-known technique to help regularize the model training, enhancing the generalization and stabilizing the training process[1,2,3]. We will revise this paragraph to make the explanation more intuitive in the updated version.

[1] Müller, Rafael, Simon Kornblith, and Geoffrey E. Hinton. "When does label smoothing help?." Advances in neural information processing systems 32 (2019).

[2] Zhang, Chang-Bin, et al. "Delving deep into label smoothing." IEEE Transactions on Image Processing 30 (2021): 5984-5996.

[3] Lukasik, Michal, et al. "Does label smoothing mitigate label noise?." International Conference on Machine Learning. PMLR, 2020.

---

Weakness 2: Once again, in the line #328, The reasons for preparing a reverse DPO dataset and swapping the best response with the worst response are somewhat unclear.

Response 2: As explained in the above Response 1, we use the label smoothing for regularization. The RDPO requires to swap the selected and the rejected response. However, it is important to note that we do not need to prepare a separate reverse DPO dataset. Instead, we simply multiply $-1$ when calculating the DPO reward gap during training, which adds no additional computation overhead.

Question 1: In the line #445 - 447, It describes as “SRLM with prompt rewarding is not effective”, but the performances are increasing on the GSM8K dataset in the Table 3. Can you explain why this could be the case?

Response 3:

While the performance of P-SRLM (SRLM with prompt rewarding) shows a slight improvement on the GSM8K dataset during the 2nd and 3rd iterations, it significantly drops in accuracy across all other four datasets. This may be because GSM8K is a reasoning task with a clear correct/wrong criterion, making it easier for the LLM to judge and thus obtain more reliable preference data for training. However, its improvement on the GSM8K dataset (less than 1%) is very marginal compared to CREAM, and this slight gain could also be potentially attributed to randomness.

---

Question 2: In Table 4, the performance of CREAM dis not improve when we compared to Llama2 M1 on the Arc-E, Arc-C, OBQA, and SIQA datasets. What are your thoughts on why it didn’t improve, even though CREAM showed better performance compared to Llama3 M1?

Response 4:

As mentioned in Line 375, the Llama2 model has relatively weaker foundational performance, indicating that its alignment capability is not very strong. Since CREAM relies on DPO rewarding to annotate training data (preference pairs), its effectiveness depends heavily on the model's alignment. Llama2's poor alignment (even after SFT, its performance still drops on 2 datasets) will inevitably lead to degeneration especially for the beginning iteration, i.e., M1 -> M2. However, after the initial iteration, we find that CREAM contributes to stabilizing its rewarding, which finally helps it improve for the next iteration, i.e., M2 -> M3.

---

Question 3: Regarding the reversal of the DPO dataset, if ( C = 0 ) indicates an inverse correlation between $\theta_{t-1}$ and $\theta_{t}$, isn’t learning from this strategy placing too much confidence in the assumption that reverse preferences are meaningful? This concern arises particularly when the initial inconsistency between $\theta_{t-1}$ and $\theta_{t}$ might be due to stochasticity rather than an actual flaw in the self-rewarding model being trained. As such, did the authors conduct experiments to analyze how the trend of inconsistency evolves between consecutive iterations, especially at later stages of training (e.g., around t = 5 or t = 6), to see if this inconsistency stabilizes later in training?

Response 5: We would like to highlight that the consistence between $\theta_{t-1}$ and $\theta_t$ can rarely be as low as $C \approx 0$. This is because $C = 0$ means for any preference pair $y, y'$, two consecutive model $\theta$ and $\theta_{t-1}$ give totally reversed prefernence. For the scaled Kendal Tau, that means if the ranking for 5 generated responses is $A > B > C > D > E$, to reach $C=0$, we need to change the ranking to $E > D > C > B > A$, which is very rare in SRLM. Since $\theta_t$ is trained using the prefernece data given by $\theta_{t-1}$, they tend to have similar behavior and thus such a circumstance making $C = 0$ can hardly happen in practice.

Furthermore, in response to your question, we also analyze the distribution of the consistency and supply the results for 4th, 5th and 6th iterations as follows. The results below also show the consistency trend for CREAM. We can find that the initial consistency rate 0.39 is acceptable, which will not result in heavily relying on reverse preferences. Besides, the regularization of CREAM is shown to encourage the consistency of rankings and maintain stability in the later rounds of training.

---

Question 4: presentation/typos

Response 6: Thank you for pointing out those grammar errors and typos. We have carefully revised and polished the paper accordingly.

Dear reviewer Nh1y,

Thank you for your time and efforts reviewing our paper and responses. As the discussion DDL is approaching, we would like to follow up to see if the response addresses your concerns or if you have any further questions. We would really appreciate the opportunity to discuss with you.

Thank you again for your time and valuable reviews.

Sincerely,

Authors

Thank you for your time and efforts reviewing our paper and responses. We are glad to hear that all your concerns have been addressed in our previous response. We would like to answer your question as follows:

First, we would like to point out that unlike the existing preference dataset (e.g., Ultrafeedback[1]), in the SRLM setting, the preference data are generated by the model itself. Specifically, we randomly sample 5 responses for each question. After rewarding and ranking, we select the top-1 scored response (A) and bottom-1 scored response (B) to form the preference pair (A, B), ensuring the gap between A and B is as large as possible. That means it can hardly make A $\approx$ B. This preference data construction process inherently helps avoid cases where A is approximately equal to B.

Second, we note that CREAM is able to handle such extreme case that $A \approx B$ by dynamically setting $C = 1/2$. This is because when the reward of the top-1 and bottom-1 responses among $N = 5$ responses are similar, then the remaining three responses, which fall in the middle of the rankings, would also tend to have rewards similar to both A and B (i.e., A $\approx$ others $\approx$ B). This situation suggests that the ranking can be highly uncertain (random). In such cases, the expectation of Kendall Tau coefficient is 0: $\mathbb{E}(\tau) = \frac{\mathbb{E}(V-D)}{\binom{N}{2}} = \frac{2E[V] - \binom{N}{2}}{\binom{N}{2}} = \frac{2 \left( \binom{N}{2} \times 0.5 \right) - \binom{N}{2}}{\binom{N}{2}} = \frac{\binom{N}{2} - \binom{N}{2}}{\binom{N}{2}} = 0$ where V and D are concordant pairs and discordant pairs in rankings. That means, our consistency rate C will naturally be 1/2, suggesting that CREAM already adopts a more optimal strategy to learn from such similar preference pairs.

Additionally, we supply experiment results of CREAM with C=1/2 as follows. Though universally setting C=1/2 is better than normal SRLM, it is inferior to CREAM's dynamically calculated C.

In summary, we would like to clarify that:

• The design of SRLM suggests that the $C$ is usually close to 1 especially for later iterations, as we demonstrated in Appendix C.
• In the extreme case where $C = 0.5$, CREAM can indeed correctly calculate this $C$ and thus eliminate the gradient step on this situation.

We hope our response could address your additional question. Thank you!

[1] Cui, Ganqu, et al. "ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback." Forty-first International Conference on Machine Learning. 2024.

Thank you for your insightful and helpful reviews! We would like to address your concerns by point-to-point response. We would appreciate it if you could let us know whether your concerns are addressed by the responses.

Weakness 1: I think first the author should add some baselines such as the original form of the method which just uses the KL constraint toward the Bernoulli distribution. It would be good to break down where the improvement comes from and how much the ranking correlation helps the methods.

Response 1: Thank you for your suggestion. We add two ablation models/baselines:

• SRLM+KL: SRLM with KL constraint towards the Bernoulli distribution, which introduces a simple regularization term $\lambda [\frac{\pi_\theta(y|x) }{\pi_\text{ref}(y|x)} - \frac{\pi_\theta(y'|x)}{\pi_\text{ref}(y'|x)}]^2$ to the DPO training loss.
• CREAM w/o Ranking Correlation + C: This is our method CREAM with a manually set fixed Consistency Rate ($C$). We perform a hyperparameter search to set $C$ to validate the contribution of our automatically determined $C$ via ranking correlation.

The results of these two baselines are shown below. We can find that

• Although SRLM+KL introduces regularization and improves performance compared to SRLM, its overall performance is weaker than CREAM. Further, CREAM maintains its advantage against SRLM+KL over multiple iterations, demonstrating that CREAM performs better than directly restricting the KL divergence.
• Comparing CREAM w/o Ranking Correlation + C with the original CREAM, we find that the original CREAM consistently outperforms the variant with manually set $C$ values. This indicates that a fixed consistency rate is often inferior to the dynamically calculated consistency rate derived from ranking correlations.

Weakness 2: Then another very important experiment I think is that previous self-rewarding methods can not maintain the improvement beyond 3 or 4 iterations, how will your method help this? Will there still be significant improvement for the 5th or 6th iteration?

Response 2: To explore continuous improvements, we conducted additional experiments with SRLM (baseline method) and CREAM (ours) using Llama-3 for the 4th, 5th, and 6th iterations. The results are reported as follows. According to the results, we have the following findings:

• CREAM converges at the 4th iteration (M4) while the baseline method SRLM started performance degradation much earlier at the 2nd iteration (M2). This shows that CREAM helps stabilize the self-improvement training by mitigating the rewarding bias issue.
• CREAM does not seriously harm the performance after convergence (i.e., during M5 and M6), while SRLM drastically drops the performance. This suggests that adding this consistency-based regularization is beneficial to preventing model from degeneration in the long term.

Table ↑ and ↓ indicate the performance improvement and degradation compared to the method’s last iteration.

Weakness 3: Another thing I am thinking is that given the final form, in the training you use a normal DPO and a reversed DPO, and weight the loss functions. Then this is basically the same as just weight the normal DPO with a weight maybe ranging from -1<->1. Why don't the author try this for example shifting the C weight and only keeping DPO loss? This would be much more efficient in implementation I assume?

Response 3: We would like to emphasize that the final form of CREAM, which combines a normal DPO and a reversed DPO, cannot simply be replaced by reducing the weight in a normal DPO. Specifically, the CREAM loss is $C \log \sigma(r(y_+) - r(y_-)) + (1 - C) \log \sigma(r(y_-) - r(y_+))$ differs fundamentally from a weighted DPO loss $C \log \sigma(r(y_+) - r(y_-))$. From the optimization perceptive, the latter weighted DPO loss behaves similar with a regular DPO loss with a smaller learning rate. In addition, since the preference probability $P(y_+ \prec y_-) = \sigma(r(y_+) - r(y_-))$ and $P(y_+ \succ y_-) = \sigma(r(y_-) - r(y_+))$, we would highlight that the CREAM loss can be kind of viewed as a cross-entropy loss $C\log P(y_+ \succ y_-) + (1 - C) \log P(y_- \succ y_+)$ with label-smoothing factor $C$. Thus, The weighted DPO loss cannot deliver such a observation.

The detailed derivation is as follows:

• First, the proposed loss can be expressed as $C * D + (1-C) * RD$, where $C$ is the consistency rate, $D$ and $RD$ are DPO loss and ReverseDPO loss, respectively.
• For the DPO loss, $D = \log \sigma (a -b)$, where the sigmoid function $\sigma$ converts the reward gap $(a-b)$ into a probability $P$. For $RD$, the reward gap is $(b - a)$. Recall that the sigmoid function has a property that $\sigma (x) = 1 - \sigma(-x)$. Using this property, we can convert the sigmoid(gap part) of $RD$ to $(1 - P)$. Thus, the $RD$ loss is $\log (1-P)$ and $D$ loss is $\log P$.
• Combining these, we can get $C * D + (1-C) * RD = C * \log P + (1-C) * \log (1-P)$, which corresponds to a cross entropy loss for a binary classification task (binary preference judgment).
• The proposed loss acts as a label smoothing to regularize the training. This regularization enhances the model's generalization ability and performance.

We also take additional experiments to validate the SRLM using $\text{weight} * \text{DPO}$, where $\text{weight} \in [-1.0, 1.0]$. For SRLM + Weight method, M3 is trained based on the best checkpoint of M2 across different weights. From the results below, "NA" means the negative of the DPO loss leads to catastrophic forgetting, where the LLM fails to generate fluent sentences. According to the results, we observe that CREAM outperforms the SRLM+weight method, indicating its effectiveness.

Dear reviewer 1A7Q,

Thank you for your time and efforts reviewing our paper and responses. As the discussion DDL is approaching, we would like to follow up to see if the response addresses your concerns or if you have any further questions. We would really appreciate the opportunity to discuss with you.

Thank you again for your time and valuable reviews!

Sincerely,

Authors