Thank you for your notes and the great questions. We address each question individually below.

Q1: Why do High-Agreement Prefs require no rejection of the majority vote, but the High-Agreement Ties allow it?

• We define High-Agreement preferences as those where no annotators rejected the majority-preferred response to distinguish them from Diverging Preferences where at least one annotator disagrees with the majority-preferred response. Our reward modeling experiments and evaluation metrics in Tables 2 and 3 are based around testing reward models on their ability to distinguish between these two cases.

• In Table 2, we include High-Agreement Ties as a reference point to demonstrate that reward models predict significantly smaller differences in reward to instances where the majority of annotators labeled the instance as a tie compared to examples with High-Agreement or Diverging Preferences. While we could compute the mean difference in reward on only examples where that annotators unanimously agreed were tied to get a similar point of reference, we chose to only consider instances where the majority due to the scarcity of such examples (only occurs in 21 HelpSteer2 and 16 MultiPref test examples total).

Q2: In lines 319-323, why is the mapping interval set like this? I believe the intervals could have a great influence on the reward model.

• We map intervals this way following the intuition that when responses where the difference in reward falls within some fixed range around 0 represent “ties” or “slight preference”, and the “significant preference” label constitutes. We selected the specific range such that “ties” and each “slight preference” label each contain an equal sized range of reward-difference values. We do not experiment with multiple interval mappings; however, we agree further tuning of these intervals may indeed yield greater performance, particularly if the dataset contains more preference strength labels (e.g., slight/moderate/significant rather than simply slight/significant). We leave further investigation into the impact of setting these intervals and such additional preference labeling settings to future work.

Q3: The CDF estimation is an important detail for training and evaluating the reward model, which I think should be discussed in the main text.

• Thank you for these notes. We incorporate these additional details into our main text.

Q4: In line 348, I don't fully understand what you mean by "use the predicted joint probability of annotators labeling the response as a 1 or 5".

• Here, we compute the product of the probability assigned to the “1” and “5” labels. We will clarify this in our revisions.

Q5: In line 361, "using smaller differences as a predictor" is not informative. What do "smaller differences" mean exactly?

• Following the equation in earlier in the sentence, we compute the difference in rewards for each response $| r_A − r_B |$. When evaluating AUROC, we are evaluating the binary classification performance of using this value as a binary classifier over multiple threshold values ($t$), where if $| r_A − r_B | < t$ the example is classified as having diverging preference and it is classified as a high-agreement preference instance otherwise. We will further clarify this in our revisions.

Thank you for your response. We address each point individually below.

1. High-Agreement Ties:

We would like to clarify that the primary purpose of our experiments in Section 3 is to demonstrate that models are assigning similar rewards to examples with High-Agreement and Diverging Preferences. Only providing the results for High-Agreement Preferences and Diverging Preferences, however, leaves us with 2 natural followup questions: (1) While the predicted rewards for examples with High-Agreement and Diverging preferences appear similar, what differences constitute a “similar” versus a “different” predicted reward? (2) What are the predicted rewards for examples that have neither Diverging nor High-Agreement preferences.

We include High-Agreement Ties and define them in this manner to answer both these questions. (2) is answered as our “high-agreement ties” category comprises the bulk examples that have neither Diverging nor High-Agreement preferences. Our results from this category also answers (1) by demonstrating that the differences between model predictions on examples with High-Agreement Preferences vs Diverging Preferences are minimal in comparison to model predictions on examples with “High-Agreement ties”. While we expect that changing this category to one with stricter criteria (e.g., redefining it as “unanimous ties”) would further exaggerate the difference between this category and High-Agreement + Diverging Preferences, this does not directly serve either of the two goals highlighted above.

2. Mapping Interval

Regarding the comment that “we set this by intuition”, we would like to clarify the ideas of mapping of preference strength (e.g., “significant” vs “slight”) to different reward difference thresholds has been established in prior works exploring margin-based losses (Dubey et al., 2024b). When it comes to selecting the particular threshold hyperparameter values, we note in our response above that, just like with any other hyperparameter, we expect performance to improve with further tuning. Furthermore, these threshold hyperparameters will also affect additional hyperparameters (similar to how manipulating/normalizing output scale will affect learning rate, max epochs, etc). Overall, as we note in Appendix A, we do minimal hyperparameter tuning for all systems, ensuring fair comparisons between systems. We will make note of these discussions in our revisions and leave further investigations on tuning these intervals to future work.

3. Smaller difference

We do not set a single threshold $t$. The definition of the AUROC (Area Under the Receiver Operating Characteristic curve) is computed by measuring the true positive rate versus the false discovery rate of the classifier over all possible thresholds. For a more thorough description of this metric, see [1] and [2].

[1] https://developers.google.com/machine-learning/crash-course/classification/roc-and-auc

[2] Plex: Towards Reliability using Pretrained Large Model Extensions

Dustin Tran, Jeremiah Liu, Michael W. Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, Neil Band, Tim G. J. Rudner, Karan Singhal, Zachary Nado, Joost van Amersfoort, Andreas Kirsch, Rodolphe Jenatton, Nithum Thain, Honglin Yuan, Kelly Buchanan, Kevin Murphy, D. Sculley, Yarin Gal, Zoubin Ghahramani, Jasper Snoek, Balaji Lakshminarayanan

Arxiv 2022

Thank you for your thoughtful and thorough review! Due to space constraints, we split up our discussion of each individual comment and include answers to questions in the following response.

W1: Evidence for the claim that "reward models predict differences in rewards that resemble high-agreement preferences, even when trained on all annotator labels" (lines 265-266)

• Our claims are and analysis are based primarily on the results in Table 2, where we perform the suggested experiments by training Bradely-Terry and MSE-Regression reward models on both the Helpsteer2 and MultiPref datasets in two settings: (1) training with all annotator labels and (2) training with only the aggregated annotator label. In this table, we report the average difference in rewards predicted by each system on test instances with High annotator agreement and with diverging preferences. In Figure 2, for illustrative purposes, we provide the full histogram of difference in rewards for a single trained reward model; however, our claims and analysis are based on comparing the mean reward differences in Table 2 rather than the visual similarity of the histograms in Figure 2. That said, we agree that having the full histograms for all reward models may be of interest to some readers and plan to include in the appendix in our revisions.

W2: Missing Citations of Related Work

• In our Related Works section under the “Annotator Disagreement in NLP” heading, we cite and discuss both of the suggested references listed: Uma et al., 2021 and Jiang and de Marneffe, 2021. Regarding the earlier works mentioned, thank you for pointing these out! In our revisions, we have added references to both de Marneffe et al., 2012 and Poesio and Artstein, 2005 to our Related Works discussion.

W3: Table 1: Do the frequencies in the two datasets sum up to 1?

• The frequencies of each category do not sum to one, and nor do the frequencies in each of the 4 high-level classes. As noted in Section 2.1, there are often multiple possible causes for diverging preferences for a given example, and each example is labeled with all categories that apply. As such, the frequencies in Section 2 sum to greater than 1 and we evaluate both Cohen's kappa (comparing full label set equivalence), as well as Krippendorff’s alpha with MASI distance when evaluating annotator agreement.

W4: Code release

• We will release all code upon acceptance. The to release the Helpsteer2-Disagreements so all datasets used in this work are also made publicly available.

W5: Concerns with extrapolation to recent, larger reward models

• In our common reviewer response above, we supplement our findings by adding two SOTA reward models (the 2 best performing systems on RewardBench) to our reward modeling experiments. In these experiments, we demonstrate that our observations and findings are consistent across these additional SOTA reward models. For the full experiments and results, see our common response above.

Dear authors,

I response to one block at a time.

I see that W3, W4 and W5 have been addressed. Thank you for the response.

W1: The full histograms will strengthen the paper.

W2: There is a discrepancy in what you wrote "we cite and discuss both" and the submission. Correct, Jiang and de Marneffe, 2021 is already cited -- I was wrong. However, Uma et al. 2021 is not cited.

Thank you for your thoughtful review and comments! We address each of your questions individually below.

Q1: Downstream Impact:

• In Section 5.1, we discuss how practitioners may desire different pluralistically-aligned behaviors from their LLMs. For example, some practitioners may want their language model to refuse to respond to prompts a minority of annotators belive the system should refuse. Likewise, practitioners may use their LLM to ask users clarifying questions in cases of diverging preferences due to task ambiguity. In this section, we then demonstrate how existing LLM-as-Judge methods are biased against systems that exhibit such pluralistically-aligned behaviors.

• To improve LLM evaluations, in Section 5.3 we suggest using our distributional reward models to improve existing LLM-as-Judge benchmarks by identifying and removing divisive prompts from their evaluation sets. The resulting benchmarks are, therefore, less influenced by such pluralistically-aligned design decisions from practitioners. In Section 5.3, we also apply this to one existing benchmark, WildBench, and demonstrate that our distributional reward models are able to effectively identify such divisive examples and systems generating pluralistically-aligned responses (e.g., asking to clarify an underspecified prompt, refusing to respond to a borderline unsafe/toxic prompt) are being punished by receiving low scores for their responses.

Q2: How does scale impact our findings?

• In our common reviewer response above, we supplement our findings by adding two SOTA reward models (the 2 best performing systems on RewardBench) to our reward modeling experiments. In these experiments, we demonstrate that our observations and findings are consistent across these additional SOTA reward models.

Q3: Alternative prompting methods for LLM-as-Judges for recognizing diverging preferences.

• We do not experiment with novel prompting approaches for creating LLM-as-Judges that can recognize and appropriately evaluate examples with diverging preferences; however, we do provide experiments with multiple prompting methods from two different LLM-as-Judge benchmarks. MT-Bench/Arena-Hard (which uses a single, static prompt for all examples) and WildBench (where the authors curated unique prompts for each example) and found that both methods identify clear winners in cases where preferences diverge. While we propose that removing such divisive comparisons from these benchmarks is one way to improve these evaluations, improved prompting methods represent an exciting alternative approach for future work.

Thank you for thoughtful comments! We agree that our explanations and discussions of limitations and future work could be expanded on, and we address each comment below and will include these additions in our revisions. Below, we also address “Outdated Model Concerns” by including results from current SOTA reward models into our analysis.

Q1: “Discussion on Limitations: What specific limitations of the proposed distributional reward model should be addressed in future research?”

• A1: In this work, we propose a novel method for training and evaluating distributional reward models, and we demonstrate a use-case by using them to identify divisive examples in LLM-as-Judge benchmarks. Future research might explore alternative methods for using distributional reward models to not only detect divisive responses, but also train LLMs to generate pluralistically-aligned responses in such cases (discussed in Section 5). In our revisions, we will add such further discussions of future works and limitations.

Q2: “Complexity of Technical Details: Which technical details were particularly complex, and how might they be simplified for better understanding?”

• A2: Several other reviewers have noted areas in which our writing can be simplified and improved. In particular, reviewers have suggested numerous ways we can improve our explanations of our distributional reward models and baseline methods in Section 4. We will incorporate such changes and more in our revisions. Furthermore, to support future efforts building upon our work, we plan to release our code upon acceptance.

Q3: “Practical Applicability: What are the potential real-world applications of the proposed approach, and how could its limitations affect these applications?”

• A3: In Section 5.3, we demonstrate one method in which distributional reward models can be utilized to improve LLM-as-Judge benchmarks by identifying divisive prompts from these evaluation sets. Under the “Results and Recommendations” heading of Section 5.3, we suggest that such examples can be removed from these benchmarks to diminish the bias LLM-as-Judge exhibit against pluralistically-aligned responses to divisive prompts.

Q4: “Outdated Model Concerns: How does the findings with Llama-3-8B Instruction model impact recent research?”

• A4: We address these concerns by adding two SOTA reward models (the 2 best performing systems on RewardBench) to these experiments in our common reviewer response above. In these experiments, we find that our findings remain consistent across these additional SOTA reward models. For the full results and descriptions of our experiments, see our general response above.