The Trickle-down Impact of Reward Inconsistency on RLHF

Q1:The way that the authors constructed the dataset, is filtering by the cosine similarity between SimCSE embeddings that are in the range of [0.75, 0.9]. This seems convenient but I wonder how reliable is this method. Have you done a manual inspection to measure the agreement rate between the method and human evaluators? Or maybe you could try using a model-based approach like prompting GPT-4?

A1: Regarding cosine range, we have consideration for the following factors:

• Upper Bound: We set the upper limit (e.g., 0.9) to avoid retrieving identical instructions. This is crucial to ensure that $I_{A} \circ r_{A}$ is better than $I_{B} \circ r_{A}$.

• Lower Bound: We avoided setting the lower bound too low (e.g., 0.5) to maintain the challenge for the RM. A lower range would make it too easy for the RM to judge, failing to test whether RM has a nuanced understanding of human preferences.

• Range Interval: We didn’t opt for a narrow range (e.g., [0.85, 0.9]) as it would greatly limit the size of our Contrast Set.

To satisfy all of three factors, we can only handcraft a proper range, so we set an upper limit (e.g. 0.9) to avoid retrieving identical instructions, and the lower limit (e.g. 0.75) is adjusted mostly to make sure we can sample enough test examples in each human preference dataset.

We provide an automatic evaluation for the quality of our benchmarks, which uses GPT4 to evaluate our contrast instructions. Specifically, we use In-context learning (appending four demonstrations where each has a better and a worse response) to help GPT4 distinguish better or worse responses in our benchmarks. The results are shown in Appendix G. We can observe that GPT4 achieves around 95% accuracy on our benchmarks, indicating a high matching degree between our benchmark and GPT4. This guarantees the quality of our Contrast Instructions to a certain extent.

Please note that our human performance (avg. 80% accuracy) on our contrast instruction does not indicate flaws in our Contrast Instructions. Since this performance is calculated under scenarios where the annotators are forbidden to use networks and search engines, the annotators’ performance would be limited to their knowledge. With the help of search engines and other tools, our annotators achieve an average performance beyond 95%.

Q2: As I am more interested in the benchmark dataset itself, the evaluation for dataset validation seems limited. The author should focus on providing a reliable benchmark as the major scientific contribution; I believe the finetuning methods discussed in the manuscript are not as significant. Therefore, the authors should provide more evaluation results on more LLMs (open-source + closed-source) to validate your benchmark dataset. The evaluation results on popular models like GPT-4 would be very helpful. Although you can't get the weights, test by prompting would be sufficient. I am also curious to know the difference in consistency between the pre-trained model vs their SFT-ed version (i.e. llama2-7b vs llama2-7b-chat).

A2: For GPT4’s results on our Contrast Instruction, please refer to our answer to your Q1.

RM consistency does not equal the consistency of the final DPO-trained LLM, that’s why we use the ‘trickle-down effect’ in our title. However, exploration on SFT/RLHF LLMs would be interesting, we have explored potential methods for assessing preference consistency in LLMs trained via SFT/RLHF, each with its own limitations:

• Prompting DPO-trained LLMs to choose the better response often leads to them not following the instruction correctly, such as repeating the instruction instead.
• Calculating the likelihood of instruction-response pairs is hindered by length bias, with DPO-trained LLMs tending to favor longer responses, resulting in poor performance on preference benchmarks.

Q3:The benchmark dataset should be submitted along with the paper, as I believe this is the core contribution of the paper.

A3: We’ve just uploaded the data in the system, feel free to check.

Hi,

I have taken a look at the newly added Appendix G and H. I praised the efforts for running these experiments during this short amount of time. The results are interesting to see - a human with a search engine performs nearly perfectly and GPT-4 performs on par with humans, while other smaller models perform nearly as random-picking. I wouldn't be able to draw any significant conclusion from the scaling tests, but I understand that it would be too much to ask you here to finetune larger models.

Overall I like the idea of this work, though the rigorousness of the methods could be improved. My opinion on this paper remains the same - the authors should focus on providing a reliable benchmark supported by lots of validations instead of focusing on bringing more things into the basket of "novel contributions".

Most of my questions have been resolved in this rebuttal. I agree to raise my review score. I hope the authors can continue working on this study in the future.

Q5: ContrastDataset provides is challenging dataset, how are humans able to do good on it? Is it mainly hard for the LLMs since the representations are sub-optimal?

A5: While Reward Models (RMs) currently struggle to differentiate patterns within Contrast Instructions, this challenge is not mirrored in human performance. Humans are adept at discerning subtle semantic differences, an ability that RMs have yet to develop fully.

This disparity highlights the difficulty RMs face in learning representations that effectively separate human-preferred responses from non-preferred ones. At least, our results show the limitation of the main-stream protocol for modeling RM, thus motivating more sophisticated designs for RM training.

Q6: Recent work on Direct Pref Optimization (DPO) shows that it can learn with learning reward models, will such an issue happen there as well?

A6: Given that DPO algorithms do not incorporate an explicit RM, evaluating their consistency in the same manner as RMs is not feasible.

Moreover, RM consistency does not equal the consistency of the final DPO-trained LLM. That’s why we use ‘trickle-down effect’ in our title. However, we have explored potential methods for assessing preference consistency in LLMs trained via DPO, each with its limitations:

• Prompting DPO-trained LLMs to choose the better response often leads to them not following the instruction correctly, such as repeating the instruction instead.
• Calculating the likelihood of instruction-response pairs is hindered by length bias, with DPO-trained LLMs tending to favor longer responses, resulting in unfair evaluation on preference benchmarks.

Q7: Reward ensemble seems to work well in RLHF to mitigate over-optimization as also followed in [1] for Robotics. Will be interesting to see if that helps and have a discussion around the same.

A7: We tested the efficacy of ensembling various Reward Models (RMs) using checkpoints from Stack-exchange available on Huggingface, including

• https://huggingface.co/trl-lib/llama-7b-se-rm-peft
• https://huggingface.co/mnoukhov/llama-7b-se-rm-peft
• Our trained LLaMa-7B reward model (On Stack-exchange)
• Our trained LLaMa2-7B reward model (On Stack-exchange)

Our experiments incorporated three ensembling techniques referenced in the paper [5]: mean, worst, and uncertainty-based ensembling. However, as shown in our results below, this approach did not yield significant improvements over our baseline model. Overall, there is no statistical significance between performance of three ensembling methods, based on a pair-wise t-test where p value is not smaller than 0.05. This outcome indicates that simply ensembling different RMs can not address the core issues we are investigating.

[5]: Reward Model Ensembles Help Mitigate Overoptimization

If you have any other questions, please feel free to ask.

Thanks you for your comments and questions. We hope our responses can resolve your concerns:

Q1: The paper claims the issue in reward inconsistency is due to the reward over-optimization issue (citing Gao et, al). Still, the reason for such reward over-optimization and how that causes the inconsistencies in the particular Contrast Dataset is unclear. More specifically the author claims that "From the RMs’ perspective, correctly distinguishing between a clearly good vs. bad response is easy". But the contrast dataset for example shown in Figure 1, they are very different questions although textually there are common words. Does that mean the current LLMs are not able to produce representations that can separate the two is not very clear and needs further clarification. For ex: "A is a good student" and "A is a bad student", they have a lot of words in common but in representation space they should be extremely different and should be trivially separated if the representations are reasonable. Thus it's not very clear how Contrast Dataset is providing a challenging dataset to test RM model inconsistency.

A1:

Please note that we are NOT claiming the specific benchmark itself is challenging for all reward models or LLMs. What we are claiming here is the Contrast Instructions benchmarking strategy, when applied to a specific RM training set to create an evaluation set, can reveal inconsistencies of the resulting RMs.

First, we want to clarify that the (short) example shown in Figure 1 is mostly for illustration purposes due to space constraints. Most examples in contrast instructions involve much longer responses that are difficult for the models to distinguish – some of the examples can be found in Table 1.

But even with the “A is a good student” vs. “A is a bad student” example, the representation would be “extremely different” only when your model is properly trained to generalize to the task that it’s solving. e.g. sentiment classification. Let’s say if the task prompt asks “how topically relevant are the two sentences”, the two would have similar representations. Here you are making the assumption that the model generalizes well from training, and so it would be sensitive to semantic changes even when there’s a lot of lexical overlap between responses. What we intend to show with our $C_{res}$ and $C_{ins}$ metric is exactly that RMs do NOT generalize well with the standard training objective. In NLP, similar generalization issues and insensitivity to examples with high lexical overlap has been observed across different settings [e.g. 1, 2]. We are observing similar patterns with RMs as well.

Reward Over-optimization – For example, within the open-source community (e.g. w/ Llama1 and 2), examples from StackExchange make up for majority of the RM training data. StackExchange follows a specific “proxy” way of defining human preferences, e.g. within the same question, each human preferences example are constructed from responses to same question/thread, and responses with the highest votes are considered the preferred responses. What we intend to show with our $C_{res}$ metric is that RMs trained this way only fits the training distribution, and don’t generalize to what we think RM should be doing – i.e. being able to distinguish between good/bad responses consistently. With contrast instructions, we observe that RMs fail to do so even when the questions come from the domain (e.g. StackExchange) as seen in training.

[1] Gardner, Matt, et al. "Evaluating Models’ Local Decision Boundaries via Contrast Sets." Findings of the Association for Computational Linguistics: EMNLP 2020. 2020.

[2] McCoy, Tom, Ellie Pavlick, and Tal Linzen. "Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference." Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.

Thanks for your efforts in reviewing our paper. Here are the responses that we hope to resolve your concerns.

Q1: I was not sure of whether the contrast instructions generated automatically do mean something different where one answer is strictly better than the other -- for instance . The authors say they restricted this based on a particular cosine distance range, but some more rigorous evaluation around this could have made the impact of the dataset a bit stronger. For instance, one can query a bigger open LLMs in order to validate that the contrast instructions are fit for purpose. The other option could have been to conduct human eval on a randomly selected subset of the contrast instructions dataset.

A1: Thank you for the comments. We have human evaluation results on $C_{res}$ on each dataset in table 8 of Appendix B. Due to space constraints, we only reported human performance averaged over 4 datasets in Table 2.

After reading the reviews, we conducted a more thorough human evaluation on each of the dataset. On average over 4 datasets , we see that humans rank responses correctly ~82% of the time. If we allow for Google search to access relevant knowledge, the accuracy goes up to 96%. We also conducted the same evaluation with GPT4 prompting with 4-shot demonstrations, from which we observed 95% accuracy in terms of $C_{res}$. We think the gap between RM vs. human performance helps justify our observations on RM inconsistency.

Regarding cosine range, we have consideration for the following factors:

• Upper Bound: We set the upper limit (e.g., 0.9) to avoid retrieving identical instructions. This is crucial to ensure that $I_{A} \circ r_{A}$ is better than $I_{B} \circ r_{A}$.

• Lower Bound: We avoided setting the lower bound too low (e.g., 0.5) to maintain the challenge for the RM. A lower range would make it too easy for the RM to judge, failing to test whether RM has a nuanced understanding of human preferences.

• Range Interval: We didn’t opt for a narrow range (e.g., [0.85, 0.9]) as it would greatly limit the size of our Contrast Set.

To satisfy all of three factors, we can only handcraft a proper range, so we set an upper limit (e.g. 0.9) to avoid retrieving identical instructions, and the lower limit (e.g. 0.75) is adjusted mostly to make sure we can sample enough test examples in each human preference dataset.

We have updated the human eval and GPT4 results in Appendix G, where both human and GPT4 obtain over 90% accuracy on our Contrast Instruction. We believe these results can serve as a quality check for the data built by our Contrast Instruction benchmarking strategy. Moreover, we’ll try to allot more space to show human performance in the main body as well.

Q2: The authors could have just chosen one single baseline LLAMA 7B and use that to validate their claims. The behaviour of LLAMA 7B could be ground in the kinds of datasets and training procedures that it was subject to. Having few other RM baselines trained with similar RM training loss could have made the claims even more stronger.

A2: Thanks for you this suggestion! – In Appendix H, we've added experiments with RMs trained with backbone models of different sizes. We observe that scaling model size leads to (1) increased test performance on the original human preference dataset, but (2) the $C_{res}$ and $C_{ins}$ performance remains flat. This echoes with our hypothesis that – standard RM training fits the only the training distribution, but do not lead to a generalized model that can distinguish good/bad responses consistently. Just like what our general response said, this RM consistency issue is model-agnostic.

Q3: The human evaluation was done on 100 randomly selected data points and no standard errors and variance numbers were reported around the results. Since some of the results are close to each other, having the error bands around the result should help clarify the significance of improvements. One other concern about human evaluation that I have is about the possibilities of potential bias as the authors themselves serve as human annotators for human eval of the results. There are several important statements made in this paper based on human evaluation. I would have preferred at least a mixture of external annotators (who are not aware of the work or how the responses were generated) to de-bias the evaluations.

A3: Thank you for your comments.

Regarding your suggestion, we recruit two emergency Computer Science expert annotators (as understanding StackExchange responses requires domain knowledge in CS). The new results are shown in Appendix J, which aligns with our human evaluation. Besides, we provide the Cohen kappa correlation to show consistency between human annotation, which show that these two human evaluations share a certain agreement.

Thank you for your comments and questions! Here are the response that we hope to resolve your concerns.

Q1: It's a little unclear why CONVEXDA is used as a way to robustify the RM. It seems that it might work because it specifically targets the proposed benchmark, which mainly tests inconsistent instruction-response pairs with lexically similar words, but I'm not sure if the method can work well in scenarios where inconsistent responses/instructions are beyond just lexically similar words with different meanings.

Q2:I'm also less clear about why REWARDFUSION is needed and how it contributes to fixing inconsistency. Again, it seems to mainly target the type of inconsistency due to lexically similar words with different meanings, which may not be general enough.

A1+A2: One can embed lexical variations in training when training RMs, but that is not the point. Our goal was to introduce methodologies to improve RM consistency in a manner that is agnostic to the choice of evaluation (i.e., not presupposing any knowledge of the patterns inherent in Contrast Instructions), thereby ensuring a fair evaluation.

Our proposal approaches (ConvexDA and RewardFusion) are not informed of the pattern Contrast Instructions, and serve as general augmentation methods. Moreover, they both contribute to performance enhancements on the original test sets, as evidenced in Table 7 (RMEval) and Table 8 (Original test set), indicating that they are beneficial beyond the Contrast set pattern. Therefore, our approaches are designed not to be informed by the patterns specific to Contrast Instructions. They are crafted to address the general generalization of RM, like other data augmentation tricks. This overarching goal explains why the improvements (on the Contrast set) attributed to these methods are incremental.

Q3: Moreover, I wonder if it would work equally well to simply use a different LLM to generate those similar pairs for augmentation.

A3: We use different LLM paraphrases to generate augmented data for ConvexDA, and the results are updated in the Appendix I. We apply three different paraphrasers, including T5-based, Falcon-based, and Parrot-based paraphrases, as shown in Table 13, the choice of paraphraser in ConvexDA does not substantially influence the effectiveness of ConvexDA.

Q4: Please compare to methods that simply use LLMs for data augmentation.

A4: In Appendix F, we present a comparison between ConvexDA and standard data augmentation (DA) techniques, such as paraphrasing. ConvexDA is not aimed at outperforming traditional data augmentation methods in terms of raw performance. Figure 8 shows that ConvexDA achieves the desired augmentation effect with greater efficiency, making it a superior choice when considering the trade-offs inherent in the data augmentation process.

If you have any other questions, please feel free to ask.

Q5: Please clarify what “finetuning with contrast instructions format” consists of.

A5: The Contrast Instruction format is identical to the original benchmark, where each pair has an "Instruction" and "response" joined together. We've corrected this writing to clear up any potential confusion.

If you have any other questions, please feel free to ask.

Thank you for your insightful comments and questions! We’ve made a revision to the draft and updated results that could address some of your comments.

Q1: The authors only sample instruction pairs that lie within the similar range of 0.75 and 0.9. It would be useful to explain the sensitivity of this hyperparameter and study how well this prevents sampling semantically similar instructions.

A1: We hand-crafted our cosine similarity range for the following reasons:

• Upper Bound: We set the upper limit (e.g., 0.9) to avoid retrieving identical instructions. This is crucial to ensure that $I_{A} \circ r_{A}$ is better than $I_{B} \circ r_{A}$.

• Lower Bound: We avoided setting the lower bound too low (e.g., 0.5) to maintain the challenge for the RM. A lower range would make it too easy for the RM to judge, failing to test whether RM has a nuanced understanding of human preferences.

• Range Interval: We didn’t opt for a narrow range (e.g., [0.85, 0.9]) as it would greatly limit the size of our Contrast Set. To satisfy all of three factors, we can only handcraft a proper range.

In Appendix G, we include a human analysis on each of the dataset, to see how well humans can do in terms of $C_{res}$ and $C_{ins}$. On average over 4 datasets , we see that humans rank responses correctly ~82% of the time. If we allow for Google search to access relevant knowledge, the accuracy goes up to 96%. We think the gap between RM vs. human performance helps justify our observations on RM inconsistency, as well as contrast instructions as a benchmarking strategy.

Q2: When evaluating existing RM on this benchmark the reward models get a low C_res score of 53.6, even though C_res conceptually resembles the RM learning objective. This is surprising given the benchmark was constructed using the same datasets used for training. It would be useful to verify if appropriate hyper-parameter tuning was performed and if the model is able to overfit and get a high C_res score.

A2: We want to clarify that – $C_{res}$ only conceptually resembles the RM training objective, as in, both involve distinguishing between two responses given an instruction. In practice, however, RM training data is usually constructed using “proxies” of human preference labels. For example, within the open-source community (e.g. w/ Llama), examples from StackExchange make up for the majority of the RM training data. StackExchange follows a specific “proxy” way of defining human preferences, e.g. within the same question, each human preferences example is constructed from responses to the same question/thread, and responses with the highest votes are considered the preferred responses. What we intend to show with our $C_{res}$ metric is that RMs trained this way only fits the training distribution, and don’t generalize to what we think RM should be doing – i.e. being able to distinguish between good/bad responses consistently. With contrast instructions, we observe that RMs fail to do so even when the questions come from the domain (e.g. StackExchange) as seen in training.

To your concern about our implementation – In our experiments, we actually tested on an off-the-shelf open-source RM trained on StackExchange data– Stack-LLaMa from the Hugging Face RLHF team (https://huggingface.co/trl-lib/llama-7b-se-rm-peft). The results are shown in Table 3. We see close to random performance from the off-the-shelf RM under $C_{res}$ as well.

Q3: Limited models and scales

A3: Thanks for your suggestion – In Appendix H, we've added experiments with models of different sizes. We observe that scaling model size leads to (1) increased test performance on the original human preference dataset, but (2) the $C_{res}$ and $C_{ins}$ performance remains flat. This echoes our hypothesis that – standard RM training fits only the training distribution, but does not lead to a generalized model that can distinguish good/bad responses consistently.

Q4: The two methods ConvexDA and RewardFusion that are proposed are not used to build more consistent reward models that are used during the RLHF stage. Instead fine tuning on contrast instructions format is used. However, in appendix C, it is mentioned the “more consistent” reward model is equipped with ConvexDA.

A4: We apologize for a typo in Appendix C related to our experimental setup. In our research, we fine-tuned the consistent RM with our contrast set without ConvexDA. Using ConvexDA and RewardFusion to improve the RM for RLHF didn't significantly improve the chatbot's performance due to their limited improvements in enhancing RM consistency. Since we focus on the trickle-down effect of RM consistency, we select the most effective method (training on Contrast Instruction) in RLHF to better observe such effects.