RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style

We are very glad that you recognize the originality of our benchmark as the first to evaluate reward models while controlling for style. We are grateful for your positive feedback on the significance of our results. We appreciate your detailed feedback and suggestions for improvement. We treasure the opportunity to address your concerns and improve our work.

---

Weaknesses 1 and Question 1: The open-source plan of RM-Bench.

We definitely plan to release the data used in our experiments on HuggingFace along with the corresponding evaluation code on GitHub.

As devoted supporters of the open-source community, we will include not only the data but also the model weights files used in Sections 4 and 5 to facilitate reproducibility and further research.

Due to the anonymity policy, we cannot include the link to the data in the submission version. However, we will ensure that the link to the data is included in the camera-ready version of the paper upon acceptance.

---

Weakness 2: Ablation study to show why RM-Bench is more accurate.

We appreciate the suggestion. We have included an ablation study in Appendix O of the revised version of our paper.

In short, we evaluated the correlations with policy model performance based on Easy, Normal, and Hard Accuracy, respectively:

• Easy Accuracy: Represents the scenario without substance or style control, with a correlation of 0.07.
• Normal Accuracy: Represents the scenario with substance control applied, with a correlation of 0.30.
• Hard Accuracy: Represents the scenario with both substance and style control applied, with a correlation of 0.45.

These results show a consistent increase in correlation with the application of substance and style control. This testifies that the reason RM-Bench is more accurate is because of the substance control and style control design.

We will ensure this information is included in the camera-ready version of the paper if it is accepted.

---

Weakness 3: Correlation results for each of the benchmark types (math, code, safety).

We have included the correlation results for each benchmark type (math, code, safety) in Appendix P. The results demonstrate consistent correlation across all benchmark types.

We will ensure this information is included in the camera-ready version of the paper if it is accepted

We appreciate that the reviewer recognizes the originality and significance of our work. We thank the reviewer for their detailed feedback and constructive suggestions. Here, we treasure the opportunity to address your concerns and improve the quality of our work.

Before addressing the specific points raised by the reviewer, we would like to point out that the contribution of RM-Bench is not limited to style-control (robustness) but also to substance-control (sensitivity to substance change, like the comparison between $y_c^{L,M}$ and $y_r^{L,M}$) and a more fine-grained evaluation method (style-substance evaluation matrix) compared to existing benchmarks like RewardBench. Thus, the insights we found in Table 4, Section 5 are not solely based on style-control but also on substance-control.

---

Weaknesses 1: In Table 4, Section 5, the RM-Bench performance should be compared based on $y^{L, M}$ only.

We further provide the results of RM-Bench performance based on $y^{L, M}$ only in Table 4, Section 5 in Appendix N in the revised submission.

In short, for Table 4 (DPO vs. Sequential Classifier), the conclusion still holds. The DPO model outperforms the Sequential Classifier in RM-Bench. When the reference model is not available, the DPO model performance is degraded, indicating the importance of the reference model in DPO.

For Section 5, the correlation with the policy model performance based on $y^{L, M}$ only is also provided in Appendix N.2. The correlation is weaker than the correlation based on hard accuracy, which indicates the importance of style-control in the evaluation of reward models.

Additionally, we would like to kindly refer you to Appendix O, where we conduct a comprehensive ablation study to investigate the impact of style-control and substance-control on the correlation with the policy model. The results show that both style-control and substance-control are important to provide a better correlation with the policy model performance.

---

Weaknesses 2: What is the behavior of the model when you use the accuracy $r(y_c) > r(y_r)$?

Here we provide the results of Reward Model Accuracy by comparing $y_c^\varnothing$ and $y_r^\text{L}$ solely based on the correctness score in the following table. When the correctness score of $y_c^\varnothing$ is higher than $y_r^\text{L}$, the data point is considered correct, as $y_c^\varnothing$ is the correct response while $y_r^\text{L}$ is the response with errors.

These results support the same conclusion we made in Section 4.3: the correctness score provided by the Nemotron-4-340B-Reward model is still far from satisfactory, except for the Safety domain. This indicates that the model still fails to separate correctness from other factors like verbosity when assigning correctness scores.

Weakness 4: Clarification on length control in Table 2.

We would like to clarify that there is no specific length target for each type of response. The difference between a detailed response and a concise response is that the detailed response contains more information, explanation, and details, while the concise response only contains the key information to answer the query.

It is natural that different queries may result in concise responses of varying lengths. For example, a concise response to "How to print hello world with Python?" is shorter than a concise response to "Give me a bubble sort code snippet."

This definition aligns with the common practices in communities like Chatbot Arena [1], where they define "concise" means clear but not verbose or excessive.

For quality ensurement, please refer to the response to Weakness 3.

[1] Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference: https://arxiv.org/abs/2403.04132

---

Question 1: Overlap with RewardBench.

The user queries in RM-Bench are sourced from RewardBench, but all responses are freshly generated using GPT4o to ensure high-quality, controlled response pairs.

These new-generated responses allow us to conduct substance and style control evaluations, which are not available in RewardBench.

---

Question 2: Clarifying token numbers of short and long responses in Math and Code domains.

We would like to clarify that the average token of short and long responses in Math and Code are far from identical as Table 2 shows. :

• For Math: Short responses average 319 tokens, while long responses average 500 tokens (a 50% increase).
• For Code: Short responses average 503 tokens, while long responses average 628 tokens (a 20% increase).

Besides, considering the token number of response is conditional on the query (different query requires different length response, for example, answer 1+1=? requires less token number than solve x^2 -2x + 3 = 0).

Below, we provide the average ratio of token counts between long and short responses of each query in Math and Code tasks:

This ratio further highlights the substantial difference between long and short responses for each query.

Additionally, as we originally stated in Lines 429–430: "Nemotron-340B-Reward struggles to disentangle correctness from other factors," here we mainly want to emphasize the overlap in correctness between the chosen and rejected. We would further refine this part in the revised version to avoid confusion.

---

Question 3: Data points in Figure 4.

We would like to clarify that the four data points signify stable increases in reward model performance, resulting in a rank Spearman correlation of 1.0 with p-value p<0.05.

And Yes, the four points correspond to models trained on different preference datasets as stated in Lines 438–444. From left to right, the datasets are:

1. Chatbot Arena 2023
2. Nectar
3. StackExchange
4. HH-RLHF

This information has been added to Figure 4 of Section 5 in the revised version.

---

Question 4: Request for additional graphs based on Code Hard or Code Normal accuracy.

As we have mentioned above, the preference data is not only StackExchange, but also includes other datasets from general domains like Chatbot Arena 2023, HH-RLHF, and Nectar. Considering the corresponding policy model benchmark here is Auto Arena Hard, a general domain benchmark. Thus, we would like to clarify that it may not be appropriate to use Code Normal accuracy as the x-axis.

For additional model results, please refer to the response to Weakness 2.

---

Question 5: Clarification on the y-axis in Figure 8 (Appendix H).

The y-axis represents the relative improvement of the best-of-n results over naive greedy search decoding.

---

Typos:

We have corrected the typographical errors and appreciate you bringing them to our attention. For the omitted sentence on Page 4 Lines 182–183, we have completed the sentence for clarity. Here we mean "Wary LLM is used to generate refusal to benign queries, and Unc LLM is used to generate harmful responses to dangerous queries."

Thank you for the detailed and constructive feedback! We treasure the opportunity to address your concerns and improve our work.

---

Weakness 1: RM-Bench should include responses from more SOTA models like Claude, Gemini, etc.

Following your suggestion, we developed a new version of RM-Bench using the Gemini-1.5-Pro model. This version follows the exact same data curation pipeline outlined in our paper. We have reevaluated the top-10 models in Table 3 using the Gemini-based RM-Bench. Here are the results:

The Spearman correlation for the rank of these models between the GPT4o-based RM-Bench and Gemini-based RM-Bench is 0.833 with p-value p<0.05, indicating consistent results.

This consistency reflects that the inherent bias introduced by GPT4o in RM-Bench is not significant. This may be due to the following:

• First: the evaluation bias concern of using GPT4o mainly occurs when the GPT4o is used for both generating test content and subsequent evaluation [1,2] but in RM-Bench, the GPT4o only serves as the test content generator, the evaluation is done by the objective metrics (e.g., the correctness of the response), which is not affected by the bias of GPT4o.
• Second: so far (2024-11-20), GPT4o is the most powerful LLM (expecting the reasoning specific model like o1) we have, and it achieves both top 1 in Chatbot Arena [3] under w/o style control setting. This indicates that the style and substance are well balanced in GPT4o, thus the bias introduced by GPT4o is not significant.

We are actively testing more reward models with Gemini-based RM-Bench and will provide updated results in the camera-ready version upon acceptance.

[1] Large Language Models are not Fair Evaluators: https://arxiv.org/abs/2305.17926

[2] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena: https://arxiv.org/abs/2306.05685

[3] Leaderboard of Chatbot Arena: https://lmarena.ai/

---

Weakness 2: Requiring one more model (maybe from Qwen or LLaMA family) for the correlation analysis.

To address this, we incorporated the LLaMA-3-8B model into our analysis. Specifically:

• LLaMA-3-8B was trained on the Tulu-V2 dataset to obtain an SFT variant. Then, it was trained on the UltraFeedback dataset with the DPO method and Bradley-Terry method to create a DPO variant and a Sequence Classifier Reward Model variant. Finally, with the Sequence Classifier Reward Model variant, we trained a PPO variant of LLaMA-3-8B.

The following table presents the updated results in Table 4:

As shown, the DPO variant of LLaMA-3-8B outperforms the sequence classifier, and performance decreases when the reference model is not used. This observation aligns with the results in the original Table 4.

• For Section 5 results, we also added LLaMA-3-8B results to the correlation figures. These updated figures are included in Appendix M.
• In summary, after incorporating LLaMA-3-8B results, the positive correlation between RM-BENCH and policy model performance remains intact, further supporting the conclusions in Section 5.

---

Weakness 3: Missing details about prompts and quality checks in the appendix.

Thank you for this suggestion! To enhance reproducibility, we have included the prompts used in Appendix L.

For quality assurance:

• Tasks with Golden Answers (Math & Code): We ensure compact responses still contain the correct answers. For Math, this means providing the golden answer; for Code, ensuring the correct solution snippet is included.
• Chat Domain Tasks: Human annotators were ask to
  1. Verify that error information is successfully injected into responses.
  2. Confirm that the original correct information is completely replaced by error information.
  3. Ensure all chosen/rejected responses convey the same meanings.
• Safety Domain Tasks: Since the key in safety is about rejecting or not rejecting queries from users, we use a keyword table that includes terms like "Sorry", "not engage", "can not assistant" and so on to filter out some data points, then we finally manually scan out the dataset for doubt check the correctness of benchmark