Improving Context-Aware Preference Modeling for Language Models

Thank you for your detailed review and feedback. Please find our responses below.

The paper could benefit from testing the proposed method across a broader range of general benchmarks, such as MMLU and AGI-Eval, to assess how fine-tuning for context-aware preferences might affect the LLM's original capabilities or general applicability.

We agree that training generative models with context-aware rewards is a great direction, and part of our broader agenda. In particular, we believe this could improve sensitivity to the system prompts and diverse users, and to prompts with multiple instructions or constraints, which is something we have observed state-of-the-art models struggle with. The design decisions here are not obvious though (where does context for MMLU come from, for example, especially given its more objective nature) and add another layer of complexity that we believe would be better suited for future work.

It remains unclear how applicable the proposed method is to other LMs, especially those of different sizes or from different series.

This is a good point. We took the opportunity to use RPR to finetune one of the stronger 2B parameter reward models according to the Reward Bench leaderboard, which is based on Gemma (hf:Ray2333/Gemma-2B-rewardmodel-baseline). We used the same hyperparameters as used for the 7B Mistral RM in the paper. We obtained the following context-conditioned results, which we will include the paper (Mistral results included for reference, best Gemma model bolded):

Note: Multifaceted Bench is a new context-conditioned preference dataset — see general response.

We observe that finetuning the Gemma RM shows similar patterns to finetuning the Mistral RM, and that the finetuned model is in many cases competitive with the larger Mistral base model.

Thank you for your detailed review and feedback. Please find our responses below.

citations / Li et al. [1]

Thank you for bringing Li et al. to our attention. We agree it is closely related and will add it to the related work, and we will revisit our literature search for other recent work we may have missed.

As part of our rebuttal, we have run their Auto-J model, but found that both Ultra RM and the base Mistral RM outperform it on every dataset/benchmark we use, both unconditioned and conditioned. We note that Auto-J, including the datasets used in the Auto-J paper, seeks to directly address the unconditioned preference modeling problem, even if Auto-J generates criteria as part of its judging process.

Table 2 could be attributed to the model being trained on data from the same distribution as the test set.

We agree this is true for RPR and will make this more clear in our paper (see, e.g., the stars in the new data ablation table of the PDF attachment). However, we note that our finetuned CARM also improves performance on all other context-specific datasets, including the newly added Multifaceted Bench (see General Response), as is now more clearly shown in Figure 1 of the PDF attachment.

In Table 4, the model is prompted by context generated by GPT-4, leading to unfair comparisons with similar models

We disagree that these comparisons are unfair. In particular, all models—not just our CARM—have access to the same additional context, hence their improved performance relative to the “NC” column.

The lack of experiments illustrating that two-step preference modeling achieves better performance compared to traditional reward modeling on general preference datasets is a concern.

We agree it would be ideal to show that two-step preference modeling can achieve better performance on general preference datasets. However, this requires two parts: (1) better context-specific modeling, and (2) strong context inference or specification. While we show in this work how we can improve the context-specific modeling problem, the context inference problem remains difficult. We have tried to generate context with respect to unconditioned queries as part of our work as well, and found that getting current models to generate useful context is quite challenging (for evidence of this, see the Auto-J results noted above), which is why we believe one of the next steps for future work is to investigate how to best obtain strong context supervision (as we allude to at L186-187). We think supervision of some kind is critical in order to go beyond the “user's intent is inherently contained in the query” as you wrote.

Furthermore, we believe two-step preference modeling to be useful even if it could not improve on current unconditioned preference datasets such as Reward Bench. See next response.

Why is two-step preference modeling necessary? … why should the estimated intent be treated independently?

We think your last sentence here captures our motivation: “ambiguity exists in estimating intent, as different individuals may have inconsistent interpretations of intent for the same query.” Part of our argument is that intent need not be estimated solely from the prompt—indeed, it can be:

• specified for annotators (see L84-91), or via default rules similar to Constitutional AI, or via the system prompt
• inferred from past interactions in cases of persistent context (e.g. user profiles, L287; see revised Table in the attached pdf), or
• learned from data via some context supervision system.

To the extent that there are inconsistent interpretations of the same query, any such context will usefully disambiguate the query, which provides several advantages:

• (a) Given context, we reduce reliance on unstated, implicit assumptions made by annotators, thereby increasing overall agreement (see Ziegler at al., quoted at L88).
• (b) Using context we can improve steerability and pluralistic alignment (see Sorensen et al.).
• (c) Making the context explicit may also be useful to diagnose errors made by models.
• (d) Notably, making context explicit allows us to change the aggregation rule from a Borda count (see Siththaranjan et al.) to a more flexible Social Welfare Aggregation (as in Bakker et al., who find this to be effective as a consensus mechanism).

In the last case, (d), agreement with respect to unconditional preferences gathered from a group (and thus, performance on general preference datasets) seizes to become a good measure, as Equation (2) [L158-159] (or modifications of Equation 2 for non-EU SWFs) will not hold, so that explicit contextualized aggregation (of human preferences) becomes a better alignment target than overall preference (see Siththaranjan et al. for a concrete example of this).

Thank you for the authors' detailed response that solves some of my questions through additional experiments and explanations.

However, my core question still exists, which is ``whether two-stage modeling is better than traditional reward modeling''. I understand that using experiments to illustrate this can be difficult, but it should be an important part.

In real conversation and preference annotation scenarios, the intent of a query is ambiguous and has multiple possibilities. The two-stage approach may lead to better modeling results through decomposition, but it may also result in error accumulation.

Moreover, as Weakness1 pointed out, previous work has already explored developing evaluation criteria before conducting evaluations. This results in a limited contribution of this paper.

I think the author's research direction is promising, but there are still many points that can be improved. So, I keep my rating.

Thank you for your detailed review and feedback. Please find our responses below.

Under-supported claims / Assumption about Cardinality

The framing of this in the discussion is not as “claims” but rather a “conjecture” (L182) and a “hypothesis” (L188,L193). As noted in the text, we do find “some” (L181,L193) empirical support for each, in Tables 2 and 5. We leave a more detailed exploration of this to future work, as the focus of this work is on improving context-specific preference modeling.

Data curation process / “lacks specific details about the prompts, the criteria for selection, and the methodology for ensuring the validity of the samples”

As noted in the main text at L211 (right after introducing the datasets), the full details of the dataset curation process, including prompts, criteria for selection, and methodology for ensuring validity (e.g. L794), are detailed in Appendix B (L766-L815). If there is some shortcoming in the presentation there, we would appreciate a more specific critique.

Clarity / Enhancing the narrative flow

We have added a Figure 2 and revised Figure 1 (see pdf attachment) to clarify the framework and main results. We believe the narrative flow is strong, but would appreciate and happily consider any more specific critiques the reviewer has on this point.

Assumption related to Bernoulli distribution / Intent distribution

Modeling paired preference as sampling from Bernoulli distribution is standard; e.g., in the widely used Bradley-Terry model. Allowing for an intent distribution (as opposed to a single intent) seems like a natural modeling choice that broadens the scope of the model (it gets us Equation (3) in addition to the single intent Equation (4)). We do not believe these modeling choices require empirical justification, as we are not claiming in Sections 3.1-3.2 that real human annotators explicitly infer an intent or distribution of intents, or that real human annotators explicitly model their label as a Bernoulli distribution. Indeed, at L148-151 we adopt the (commonly used) Expected Utility model, which is used to arrive at Equations (3) and (4); but it is well known that humans systematically deviate from expected utility.

Our formalism is Sections 3.1-3.2 is best understood as any other idealized model: it offers analytical insights. In our case, our model is used to arrive at the bounds in Equation (3) and (4), which expands on and strengthens the motivation in Section 2, and suggests the discussion in Section 3.3. As with any model, it is up to the reader to determine whether the model in question is a reasonable, and useful enough, approximation to reality.

There is no empirical evidence or studies cited that demonstrate annotators' ability to accurately infer intents from prompts. Additionally, the variability in annotators' interpretations and the potential biases they introduce are not addressed.

This is a core motivation for our work, as set out at L82-85: “If we train models using non-basic preference annotations, the contextual biases and assumptions underlying those judgments may be implicitly embedded into the model [35, 48] … Rather than rely on annotators to integrate the correct distribution of contextual assumptions … ”, wherein it is suggested that we might not want to rely on annotators to “accurately infer intents” [i.e., contexts], as this is subject to “the variability in annotators' interpretations and the potential biases”. A descriptive study of the biases and assumptions made by real human annotators is beyond the scope of the present work.

Human validation

The human validation of the dataset labels was done by the authors (blind / response orders and criteria/scenarios randomized – we will add this detail to the paper). As the authors, we are the closest to the data, and are able to provide high quality labels. In our paper we reported 100 total labels (50 for RPR Criteria and 50 for RPR Scenarios; different prompts for each). Since submission, we have added an additional 100 labels (50 for each Criteria and Scenarios), achieving a total agreement of 97% for RPR Criteria and 95% for RPR Scenarios (100 labels each). This gives 95% confidence intervals as follows:

Thank you for your detailed review and feedback. Please find our responses below.

The current experiments do not demonstrate the general advantage of using a paired dataset with reversed preference labels.

Thank you. This is a good criticism. We realized this post submission and have prepared a data ablation that finetunes the base Mistral reward model on context-conditioned data from other distributions, including:

• Preference Collection (PC) (hf:prometheus-eval/Preference-Collection), in which the context is highly correlated with unconditioned preference, and
• a “one-sided” version of our RPR datasets, where kept only one of the two sides by having GPT-4 pick one of the two criteria or scenarios that it thought was more suitable for the given prompt.

The results are shown in Table 1 in the PDF attachment, and demonstrate the advantage provided by our RPR dataset. We will add this to our paper. Notably, the Multifaceted Bench (MF), released after the NeurIPS submission deadline, introduced an additional context-conditioned preference benchmark that is outside of our training distribution, where you can see that training on RPR strongly improves context-conditioned performance over the base model (and the PC model).

On widely used preference datasets, the model trained on the proposed dataset does not outperform the baselines.

If you are referring to the no context versions of HHH, Reward Bench, and Chatbot Arena, these are unconditional preference benchmarks, where it is not expected that our model, trained for context-conditioned preference queries, would do better than the base model. We report results here mainly to provide a reference figure, and also to show that fine tuning on RPR does not hurt the unconditioned preference prediction of the base model.

It only performs better on the test set, which is built in the same way as the training set.

This is not true. Our model not only performs better on the RPR test sets, but also: Multifaceted Bench (MF, see general response), HHH with context, Reward Bench with context, and Chatbot Arena with Context*. This is more clearly shown in Figure 1 in the PDF attachment, which we will add to our paper as Figure 2. Please note the errata for GPT-4 results in the general response.

Note that the preference labels on this test set are not verified by humans and could be unreliable.

Human validation of the dataset labels was done by the authors (blind / response orders and criteria/scenarios randomized). As the authors, we are the closest to the data, and are able to provide high quality labels. In our original submission we reported 100 total labels (50 for RPR Criteria and 50 for RPR Scenarios; different prompts for each) in Table 2 (pdf page 7). Since submission, we have added an additional 100 labels (50 for each Criteria and Scenarios), achieving a total agreement of 97% for RPR Criteria and 95% for RPR Scenarios (100 labels each). This gives 95% confidence intervals for human-RPR agreement as follows:

Some noise in human agreement is unavoidable. As noted in our paper (L73) there is significant inter-human disagreement on unconditioned queries, e.g., humans agree with each other only 65.7% of the time on AlpacaEval. Even the carefully curated MT-bench shows has agreement of only 81-82% on strict preference queries (Zheng et al., Table 5). Thus, our 90+% agreement between the human authors and the synthesized labels is quite high, likely as a result of the added context and multiple levels of filtering.