How to Evaluate Reward Models for RLHF

We thank the reviewer for their comments. We appreciate that the reviewer recognizes the importance of PPE, especially our in-depth analysis of downstream performance. Please let us know if there are any questions.

We thank the reviewer for their helpful suggestions. We address comments below:

“Some of the results seem pretty surprising (e.g. the low correlation with reward bench), for me it’s a bit unclear why we observe them. It would be great if the authors could try to reason more about these observations (as outlined below)”

We agree that the discovered discrepancy between RewardBench metrics and post-RLHF preference performance warrants further study. However, our work's focus was not to disprove RewardBench. We can provide the following hypothesis at the requests of the reviewer: we hypothesize that these reward models may have over-optimized for RewardBench's specific preference distribution rather than capturing broader human preferences, potentially exceeding RewardBench's measurement capabilities. However, we note that initial improvements in RewardBench score may still correlate well to real post RLHF human preference outcomes. Ultimately, these insights were only possible through our end-to-end experiments, which enables the research community to further investigate and discuss the true correlations between benchmark metrics and downstream performance. We believe this highlights the value of comprehensive evaluation approaches like ours in understanding real-world model behaviors. We will include these explanations in the final version to give readers better context on Figure 4 and its implications.

“‘Reward Model Accuracy’ as the best metric is not a novel contribution in itself, you may argue that it’s good to have it validated towards human preferences”

We assert that no previous work has explicitly modeled the correlation between reward model behavior and downstream LLM performance. In particular, we find it surprising, and a significant contribution, that correlative measures, such as Spearman correlation, are not predictive of downstream outcomes. Moreover, it was counter intuitive that best of K metrics were overall less predictive than accuracy metrics. We believe there is inherent value in validating in-depth how reward models should behave for optimal downstream performance. We hope the review will consider the novelty of our end-to-end experiment.

“Overall, i find the findings a bit weak, i.e. i don't find it easy to draw clear lessons for reward model design (larger vs. smaller models), i think it would have been very helpful to discuss these results more”

The main goal of our research is to understand how we should measure reward models. Only once we know how to best measure reward models can we then iterate quickly to understand how we should design them. Nevertheless, our research comes with several lessons on how good reward models need to behave. Reward models should be accurate on an individual response level, while overall biases (eg. favoring OpenAI models over Anthropic models) are not particularly important as long as they are within reason. Reward models should show good performance in every subdomain, that is the lower bound performance must be high. This suggests that reward models should be trained to be exceptionally robust. We feel reward model training aimed at robustness may be an interesting future research direction based on our work in this paper. This phenomenon has been observed in theory ([1], [2]) but had not been investigated in empirical studies before our work. We hope the reviewer sees the value in these contributions, and recognizes the potential for PPE to provide a foundation for iterating and understanding reward model design moving forward.

We thank the reviewer for their comments. We are glad to hear that the reviewer feels the proposed changes will improve quality.

We have since uploaded a revision to paper where proposed changes from the rebuttal have been fully integrated. A detailed list of all changes is in the official comment above.

With regards to changes specific to reviewer PDs3:

1. We've added a new figure detailing model accuracies on page 4 (Figure 2).
2. Figure 3 (best of k curves) caption was re-written.
3. Figure 4 showing model performance is now a bar graph instead of a table for clearer visualization.
4. Appendix A1 contains a table of the datasets in PPE (page 15).
5. Appendix A.2.1 and A.3.4 contain the various summary statistics for different benchmarks and metrics. (page 25, page 32)
6. Appendix A.6.1 (page 34) details the comments and context regarding RewardBench correlations as an extension to Figure 6.
7. Minor changes for clarity are spread throughout.

In light of these new revisions, we hope the reviewer considers reevaluating their score.

Once again, we thank the reviewer for their detailed feedback.

We thank the reviewer for their insightful suggestions and questions. We address this feedback below:

“Is there a predictive accuracy plateau after a certain K-value? Is it possible to include a graph demonstrating diminishing returns/values of predictive accuracy with K?”

We thank the reviewer for suggesting this additional analysis on K. Note that our paper finds that best of K metrics are overall less predictive than accuracy metrics which are inherently K=2 (Figure 3, left side). As suggested by the reviewer, we create an additional graph showing how as K increases, the predictive power (correlation) decreases slight.. We provide an anonymous imgur link showing the graph here: https://imgur.com/a/EaX9OcX. We will attach this into the appendix along with analysis of its implications. We believe this is an excellent extension of our study which aims to understand which metrics are and are not predictive. That being said, we still release all 32 responses per prompt in PPE such that users can choose to run at whatever K they wish.

We will include recommendations on which metrics to run for maximum efficiency and predictive power in section 7 for the final version. For reference, we copy this new content below:

“Based on our end-to-end study, we recommend those seeking the most predictive power from PPE run the human preference set as well as the MATH accuracy metric. We suggest that users pay particular attention to the lower bound accuracy across the main human preference set categories (easy, hard, instruction following, coding, math, and similar). Considering our findings, this configuration likely maintains full predictive power of PPE with less than half of the runtime.”

“Underdiscusses the possibility that the RM evals are computationally very expensive to run in the the PPE framework. Best-of-K performance curves and pairwise accuracy have a huge computational burden, especially as K increases. Please provide runtime estimates for different K values or discuss strategies for making the evaluations more computationally efficient.”

We detail the runtime estimate below for an 8B reward model inference on a single standard Nvidia A100 80GB GPU:

Following the findings of our results section, we find the most predictive power can be achieved by running the human preference set and accuracy metrics on MATH. This runtime is under 42 minutes and achieves 77% Pearson correlation with downstream performance.

For comparison, the time required to run the end-to-end validation experiment to observe ground truth reward model rankings would be over a week between model training and collecting enough real human votes.

We document more runtime details below:

Human Preference Set (most correlated to downstream) runtime: 32 minutes Each Correctness Preference Set: 1 minute when K=2. Increases in K increase runtime linearly. K=8 would take around 4 minutes, and, if desired, the full K=32 would take about 16 minutes. There are a total of 5 correctness sets. Expected runtime if running ALL metrics at full K=32: < 2 hrs

We thank the reviewer for this suggestion and will detail our benchmark runtimes and highlight our improvement from a full end-to-end validation. We construct a table containing this information below the next comment, which we will add to our final paper version.

We thank the reviewer for their insightful feedback. We address their comments below:

“The study employs DPO as the RLHF algorithm to evaluate post-RLHF performance. However, the offline DPO algorithm may face generalization issues. Utilizing an online PPO algorithm to obtain post-RLHF models and assess their performance would offer a more comprehensive evaluation of various reward models. This additional experimentation is essential to support the authors’ core claim of being “the first reward model benchmark explicitly linked to post-RLHF performance.”

We fundamentally agree with the reviewer: PPO would be the preferred algorithm for this experiment, cost and implementation aside. Unfortunately, the compute requirements of PPO are much more difficult to manage. We would love to explore PPO in future research, but costs and time make it infeasible to do so in this rebuttal period. We ended up choosing DPO for cost, complexity, and stability. A nice feature of DPO is that the experiment is very controlled: the supplied responses are identical, exactly only the reward is different with no randomness from sampling. Ultimately, we still believe our end-to-end experiment is informative and helpful to the research community. We are happy to alter any specific claims the reviewer feels are too strong given the inherent constraints of the RLHF process. However, we understand that this direction is fundamentally an important area of exploration. Therefore, we will add additional comments in the limitation section to address this, as well as soften claims made in the results section that may be read as too broad.

Thanks for the detailed responses.

If possible in the future, I suggest that the authors include experiments with various types and scales of base models, along with PPO experiments, to better support the benchmarks and arguments presented in this paper. I am inclined to maintain my current score.