Reward Modeling with Ordinal Feedback: Wisdom of the Crowd

We thank the reviewer for all the comments. Here are our responses to the questions.

1. “I think more experimental results is needed to prove the superiority of the proposed method. For example, the authors could leverage the method in [1], setting up a seperated RM as the oracle and test the performance of the proposed method by evaluating the accuracy.”

Thanks for the reviewer’s kind reminder. Actually, that is exactly what we have done in our work: setting up a trained RM as the gold-standard model to simulate human preference data. We apologize for the confusion caused, and we’ll improve our presentation by emphasizing that point in our future versions.

2. “The theory seems to be fascinating. However, the empirical superiority of the proposed method against the binary baseline seems not to be obvious due to Table 2 and Figure 2.”

We’re happy that the reviewer appreciates our theory. For the empirical effectiveness of our method, we want to make some clarifications. The motivation behind our work is that (some of) the current practice of RM training simply discards all those fine-grained labeled samples (e.g., “slightly better”), even though many companies have already launched fine-grained feedback systems for human annotators. This waste of precious human-annotated data motivates us to think about whether any better ways can be applied to use that part of the data than simply discarding it. As for Figure 2, our comparisons of different levels of feedback (binary, 3-level, 5-level, and oracle) are made under the same amount of data. However, we can use more data compared to the current throwing-away solutions. For example, if there are 80% of binary labels and 20% of fine-grained labels (e.g., “tied”), the current practice uses only those 80% of data, while our approach uses all of them. We further show that incorporating some “tied” samples doesn’t harm (and even benefits) the RM learning via Table 2. In a word, our approach makes improvements in two ways: decreasing the noise and increasing the effective data sample size.

We thank the reviewer again for the time, and we’re glad to refine our draft to those suggestions. Please let us know if there is any further questions arise in the following week; we will get back to you timely.

Thanks for the valuable questions. Before we dive into the rebuttal, we’d like to summarize the reviewer’s major concerns.

1. The construction of ordinal feedback via Theorem 3.2 is not justified. There are possible reasonable options (e.g., the Luce-Shepard model) that have not been considered.

2. Assumption 3.1 alone is insufficient to guarantee an impactful expansion on the feedback set $\mathcal{Z}$ by introducing some vague options.

3. Theorem 4.6 and Corollary 4.7 prove that ordinal feedback may bring up some benefits, but they do not fully characterize how large the benefits are.

4. The linear system of $\beta$’s is underdetermined: there are $m \times n$ variables yet only $2m + n - 2$ independent linear constraints, which makes Proposition 4.3 seemingly trivial.

For Point 1, the answer is that the Luce-Shepard model is not considered because it cannot induce a natural reward function (RM). Algorithms like RLHF need a reward function for every prompt-response pair for the following policy gradient steps. That reward function is obtained via pair-wise comparison in the BT model: two reward scores, two probabilities. But if multiple levels of feedback are considered, those probabilities do not naturally correspond to two reward scores. For example, the Luce-Shepard model formulates those probabilities with multiple scores: $p_i = \frac{\exp(s_i)}{\sum_j \exp(s_j)}$. How to obtain two scores with three $s_i$’s remains unclear, as the additional “tied” term cannot be neglected. Our work is to construct a simple plug-in solution to those multiple-level feedback systems while keeping only two score functions. Due to practical limitations, we have to turn to RMs rather than real human annotators to simulate the labeling process; Theorem 3.2 is only to show how this simulation should be conducted rather than modeling a single annotator’s behavior (e.g., whether s/he chooses the closest $z_i$ or not). We admit that the simulation process may not fully prove the effectiveness of our method in the real world, but that’s the best one can do without really hiring annotators.

Points 2 and 3 are about the same question: How large is the benefit of ordinal feedback? We thank the reviewer for encouraging us to think deeper. The reduction of the Rademacher complexity (Theorem 4.6 and Corollary 4.7) is a direct result of Jensen’s inequality. The answer to the question is: the reduced quantity is approximately of the same order as the conditional variance in the hierarchical coupling. For example, suppose there are two ordinal feedbacks $Z$ and $Z^\prime$, $Z$ is a hierarchical expectation of $Z^\prime$, and the coupling is $(W, W^\prime)$. Then Jensen’s gap would be of the same order of $\mathbb{E}[\mathrm{Var}(W^\prime | W)]$ (see https://arxiv.org/pdf/1707.08644 for example), by assuming the supremum in the Rademacher complexity as a function of feedback $Z$ (or $Z^\prime$) is twice differentiable, $c_1$-strongly convex, and $c_2$-smooth (which can be satisfied in some non-restrictive cases). We are glad to include more in our future versions. The quantity $\mathbb{E}[\mathrm{Var}(W^\prime | W)]$ plays a critical role in the reduction of noises: if the hierarchical expectation is constructed by introducing some options that would never be chosen, then this conditional variance would be zero (since $W^\prime = W$ almost surely). But it’s very unlikely that all the annotators would avoid a “tied” option for all the samples. And hence the conditional variance would always be non-zero, indicating a strict benefit of ordinal feedback.

The last question’s answer is: there are limitations on the range of $\beta$’s. The feasible region of each $\beta$ is $[0, 1]$, as it stands for the conditional probability. The underdetermined linear system’s solutions’ existence may be trivial in $\mathbb{R}^{m \times n}$, but it is possible to be infeasible in $[0, 1]^{m \times n}$. For example, a binary feedback with $\mu(0) = \mu(1) = 0.5$ can never be a hierarchical expectation of a 3-level feedback with $\mu(0.5) = 1$.

For the minor point about the significance of the improvement, an important point is that we are comparing different levels of feedback in the same amount of data. However, many practices are still throwing away all the “tied” samples (e.g., Llama 2/Llama3). We believe those thrown-away “tied” samples would benefit the learning process even within the same amount of data (Table 2), not to mention that including them would further increase the data volume.

We thank the reviewer again for the suggestions and for carefully reading our manuscripts. We look forward to further discussions in the following week.

We thank the reviewer for appreciating our work and the interesting questions.

1. Despite in the domain of RLHF, what is the difference between your framework and training upon soft labels in knowledge distillation?

The major difference between our work and the knowledge distillation of trained RMs is the character of those RMs. In our work, we use the trained RMs’ soft labels to simulate the underlying belief of a population of annotators. Ideally, we would like the data to be collected from real human annotators, but current open-sourced human preference datasets do not include multiple levels of feedback simultaneously, which makes it hard to show the benefits of more fine-grained feedback numerically. We have to resort to the oracle RMs to simulate those multiple levels of feedback; in practice, for LLM companies, such ordinal feedback should be collected from real human annotators. Using RMs to simulate human annotators’ feedback, perhaps, isn’t the best way to convince the audience, yet it is a probably the only feasible method and has been applied in many papers (e.g., “Scaling Laws for Reward Model Overoptimization”). We are not hoping to improve over those trained RMs via knowledge distillation. Instead, we view those RMs as the gold-standard models (“oracle”) to value the benefits of fine-grained feedback. On the contrary, the knowledge distillation hopes to obtain a better student model via the soft labels from the teacher model. Such a process, although closely related to our theoretical results (Theorem 4.9), has a different goal from our work. Our work aims to show that (1) fine-grained ordinal feedback can be taken a good advantage of by endowing it a numerical meaning rather than just throwing it away, and (2) by providing the population with ordinal feedback options, there are some provable benefits (under the assumption of the “wisdom of the crowd”).

2. How does upsampling the conflicting labels in preference dataset relate to this ordinal method?

We thank the reviewer for this question. In our opinion, the samples with conflicting labels are those with stronger noises or larger variances (that is, the preference probability close to 0.5 in our framework). If we treat the preference learning problem as a binary classification problem (to decide which one is preferred), the noisy parts are those close to the underlying decision boundary. Upsampling that part is a method to improve the learning on those “hard cases”, reducing the noise by averaging. In that sense, our ordinal feedback method shares the same spirit of decreasing the noise but through a different approach. Our goal is to prevent the noises caused by the annotation system itself. For example, if the annotation system only contains binary options, and annotators are faced with a sample $(x, y_1, y_2)$ with very similar $y_1$ and $y_2$. Then, the probability of each response being chosen is one-half, leading to possible conflicting labels in the dataset. But many of the conflicts can be avoided if a third option, “tied”, were provided. Instead of upsampling those “hard cases” under the same feedback system, we try to point out that the learning could also be improved by designing a better feedback system.

We thank the reviewer again for the questions and suggestions. Also, we have corrected those typos. We are happy to have further discussions in the following week.

We thank the reviewer for the kind comments on our work. Here are our responses to the reviewer’s concerns and questions.

1. Experiments only contain 2 base models, which appear not to be very solid for the LLM community.

Thanks for pointing out that issue. We have conducted some further experiments on Qwen 2.5 base models, and the updated results can be found in this anonymous link: https://anonymous.4open.science/r/icml25-anonymous-4r32. We apologize for not including further foundation models due to the limitations of time and computational resources. We are glad to add more results to our future versions.

2. The literature on the “intransitivity” risk of the BT model during RLHF is not mentioned in this paper.

Thanks for the reviewer’s notice. The term “intransitivity” describes cases with cyclic relations in the pairwise preference comparisons. The literature focusing on the intransitivity risk of RLHF (e.g., the paper the reviewer mentioned) mainly focuses on providing alternative ways to model the preference probabilities (e.g., the Blade-Chest model) rather than a scalar reward score in the BT model. However, our paper aims to provide a plug-in solution to fully use the fine-grained feedback under the same framework of the scalar reward function. In that sense, the intransitivity of RLHF is somewhat “orthogonal” to our work. But we are glad to include more discussions about the intransitivity case of preference learning in our future versions, and thank the reviewer for pointing them out.

3. Consider RM is a core composite in RLHF, what is the ultimate goal for generalizing the RMs to cover “hard cases” and improving the expressiveness of “rewards”?

Thanks for the question. As we understand, we guess the “hard cases” represent those samples where $(x, y_1, y_2)$ has a preference probability close to 0.5. From our interpretation, those “hard cases” are discarded even though options such as “tied” are introduced by (some of) the current practice, while those cases do have a positive chance to appear in the following RLHF step. For example, LLMs’ responses to some prompts may be intrinsically hard to distinguish, making almost all the pairwise comparisons for those prompts impossible to learn. Discarding those “hard cases” might cause the reward function landscape to be unpredictable and undesirable on those prompts, deteriorating the RLHF quality (e.g., over-optimization issue). For the second part (improving the expressiveness of “rewards”), we believe it’s a promising direction for future LLM alignment studies, considering the subtlety and diversity of human preferences. Researchers have made several attempts to handle the preference beyond a scalar reward function, including multi-dimensional scores (e.g., HelpSteer2 includes “helpfulness”, “correctness”, and so on) and the intransitive model mentioned by the reviewer. We believe the ultimate goal of such research is to train LLMs to better align with subtle and diverse human values.

4. How does the theoretical work make wider impact to the optimization algorithms and human-in-the-loop in operation, e.g., in an active learning setting?

It’s an interesting question to see how this work is linked to the active learning algorithms. For example, the uncertainty sampling algorithm originates from the spirit that “those samples close to the decision boundary are more valuable in learning the decision boundary”. In that sense, when training RMs, uncertainty sampling would require the learner to query more samples “near the boundary”, that is, preference probabilities close to 0.5, and expect a better outcome. However, that might not be the case in RM training, as we have shown that as the tied ratio increases to 100%, the training result becomes worse (e.g., Table 2 and Figure 3). Another interesting bunch of active learning algorithms is the representation-based algorithms (e.g., CoreSet or MaxHerding), which require the queried samples to cover the feature space as much as possible. In that sense, our results imply that covering some samples close to the boundary (those “tied” ones) would benefit the learning (Table 2 and Figure 3) compared to the current practice of simply discarding these samples. We believe there are also other possibilities to improve the learning of RMs once all the fine-grained labels are included.

We are grateful for the questions and suggestions. We are glad to discuss with you in the following week if there are any further questions/comments.