Influencing Humans to Conform to Preference Models for RLHF

Thank you for your review! We hope we have addressed your concerns regarding the practicality of our proposed approach and its potential extension to more complex domains in our response to all reviewers. Applying our approach to language-driven settings is an interesting direction for future exploration. To answer your other questions:

“Does influencing preferences toward a specific model potentially introduce bias in scenarios where the model may not be the best fit for human objectives?”

We assume there exists some preference model that an RLHF practitioner wishes to use during reward learning, and outline several interventions to influence the expression of human preferences towards this preference model. What that specific preference model should be, including if it is an adequate fit to learn the human objectives, remains at the discretion of the RLHF practitioner. Note that prior work in RLHF typically assumes a preference model; we propose providing the additional flexibility of influencing humans towards this preference model.

“In the Privileged Experiment, given that users see reward information directly, is there a risk they might over-rely on this data, reducing their engagement with the task?”

This is an important insight, however as noted in the paper (see Section 5.1), the Privileged experiment serves as a probe into whether or not it is possible to influence the expression of real human preferences. The Trained and Question experiments aim to provide a practical methodology for influencing human preferences towards a specific preference model, where this is no longer a concern.

Thank you for your review and for recognizing the novelty and significance of our paper. We also appreciate your careful attention to detail in pointing out several typos. Regarding the potential weaknesses you mentioned:

“I feel like the "% of partitions in which performance is near optimal" results are not very convincing to support the hypotheses presented.”

Our main objective, as expressed in Hypothesis 1 for each experiment, is to demonstrate that each intervention can influence humans toward a specific preference model, and we generally succeed in showing this. The figures you reference (Figures 6, 8, and 10) pertain to Hypothesis 2, which focuses on reward learning from a dataset of preferences influenced towards a specific preference model. With this distinction in mind, we will now further address your concerns.

“PRIVILEGED EXPERIMENT Figure 6: The difference between the P_regret and P_reward lines in both subplots is very minimal and does not hold for all number of partitions.”

We agree with your observation. However, to evaluate Hypothesis 2, it is important to focus on the gap between the blue/red lines and the gray line in Figure 6. The blue/red lines represent learning from preference datasets where humans were influenced toward the regret/partial return models, while the gray line reflects learning from the control condition’s dataset. A significant gap exists for all partition sizes except the smallest and largest, suggesting that presenting each segment’s statistic (blue/red lines) leads to learning more aligned reward functions with the preference model corresponding to this statistic than when learning from the control condition’s dataset (gray line). Comparison between the blue lines and the red lines is irrelevant to Hypothesis 2. Therefore, Figure 6 supports Hypothesis 2.

“TRAINED EXPERIMENT Figure 8: The P_regret trained preferences always lead to better performance -- this seems to indicate that the P_regret training paradigm leads to better understanding of the ground-truth reward function more than it supports Hypothesis 2.”

We agree with your observation that teaching humans to follow the regret preference model always leads to better performance when learning a reward function, but disagree with your proposed explanation that this is because the Trained intervention simply leads to a better understanding of the ground-truth reward function. We address this concern in the response posted to all reviewers. The results in Figure 8 are actually in line with prior work, which found that learning from preferences generated by the regret preference model induces more aligned reward functions (see Knox et al., 2022) and that preferences generated by the regret preference model—when learned from under the assumption that instead they were generated by the partial return preference model—results in a highly shaped reward function in a variety of conditions (see Knox et al., 2024). Therefore, influencing humans towards the regret preference model reasonably leads to better reward learning performance.

“QUESTION EXPERIMENT The P_regret question does not seem to me to focus subjects on a segment's deviation from optimality. Perhaps something like "Which path is farthest from optimal?" would have been better Figure 10: The difference between the P_regret and P_reward lines in both subplots is very minimal and does not hold for all number of partitions.”

We appreciate your careful attention to the wording used to influence humans toward the regret preference model. Since we elicited preferences from subjects without assuming technical backgrounds, we deliberately avoided terms like 'optimal,' which carry specific meanings in computer science but may not be as clear to the broader public. Regarding your observation about Figure 10, to evaluate Hypothesis 2, we focus on comparing the blue/red lines with the gray line as we described above for Figure 6. Specifically, we are interested in the effects of influencing humans toward a specific preference model (blue/red lines) versus not influencing them (gray line) on reward learning. The large differences between the Regret-Question and Partial-Return-Question condition’s lines (blue and red) compared to the control condition’s line (gray) support Hypothesis 2.

Thank you for your continued engagement with this work! Below are our responses

"Unless I am misunderstanding something, hypothesis 2 states that "presenting each segment's statistic leads to learning more aligned reward functions with the preference model corresponding to this statistic", the fact that the red and blue lines are so close to each other contradicts this because we would expect the red line to be much higher in the left plot and the blue line to be much higher in the right plot."

Your misunderstanding lies in the fact that hypothesis 2 can be restated as "presenting each segment's statistic leads to learning more aligned reward functions with the preference model corresponding to this statistic compared to not presenting such information". Therefore, to evaluate hypothesis 2, you should look at the difference between the red/blue lines and the gray line.

"“TRAINED EXPERIMENT Figure 8: The P_regret trained preferences always lead to better performance -- this seems to indicate that the P_regret training paradigm leads to better understanding of the ground-truth reward function more than it supports Hypothesis 2.” The problem is the same as above where we should expect the red line to be higher than the blue line in the left plot."

We are only interested in comparing a condition where we influence human subjects towards a specific preference model (red/blue lines) with the control condition (gray line). We should expect the red line to be higher than the gray line in the left plot but we do not see that. Hence, in Section 5.2, we claim that Hypothesis 2 is only partially supported; it is supported by the graph on the right not on the left.

We hope that we have clarified how to analyze Figures 6,8, and 10 with respect to Hypothesis 2, and that this sheds light on the Question expirement.

"Regarding the wording of the question, I maintain that I believe that it does not reflect deviation from optimality as the authors claim."

Perhaps you are correct; we do not necessarily dispute your claim and further investigating the wording of this question is an interesting direction for future work. Regardless, in this paper, we do not see statistically significant results from this 1 condition in the Question expirement. Therefore we are unable to (and do not) claim that this specific question influences human subjects towards the regret preference model. Again, this may be because the wording of the question is not sufficient as you claim, but such an investigation is out of scope for this paper.

We thank you for your review, and for your clear understanding of our work. We hope that our response to all reviewers has adequately addressed your concern that the experiments in our paper may not fully support the main claim. To summarize, additional results in Appendix H show that for all experiments, each condition’s dataset is more likely under the target preference model than the other condition’s datasets under the same preference model. This provides empirical evidence that the proposed interventions do in fact influence humans towards the target preference model, more so than towards the alternative preference model. To address your additional concerns around this point:

“...Figures 6 and 10...show that influencing participants to follow either one of the preference models leads to better reward inference with both, not only the targeted one. “

We agree with your observation here, and it highlights the general benefits of our proposed approach; influencing humans towards a specific preference model generally results in better reward inference with either preference model. This result is to be expected; partial return is a component of regret (see Eq. 7), and so influencing human preferences towards the partial return model may also influence their preferences towards the regret model and vice versa. We do not take this as evidence against any of our hypotheses.

“Figures 8 and 27, show that training people to follow a regret preference model is most effective for inferring their reward function, be that done with regret or partial return preference models. “

We also agree with your observation here, which also highlights a strength of our proposed approach. Training humans to follow the regret preference model benefits reward learning with either preference model. These results are in line with prior work, which found that learning from preferences generated by the regret preference model induces more aligned reward functions (see Knox et al., 2022) and that preferences generated by the regret preference model—when learned from under the assumption that instead they were generated by the partial return preference model—results in a highly shaped reward function in a variety of conditions (see Knox et al., 2024).

We have addressed your concern around the practicality of our proposed approach in the response posted for all reviewers and in Appendix A. Regarding your question, “Is training practical when the correct reward function is not known?”, we propose exploring whether we can train human subjects to follow a specific preference model in a simple domain, such as the grid-world used in our work, with the goal of influencing their preferences in a more complex, real-world domain where the ground-truth reward function remains unknown. We leave this direction to future work outlined in Appendix A; this paper serves as a strong motivation for such a direction.

To answer your remaining questions:

“For all three experiments, what was the mean Cross-Entropy Loss for the opposite preference model?”

We present these results in Appendix Figures 20-22.

“For the question experiment, would participants have understood the difference between the question “which path has better immediate outcomes” and “which path reflects better decision-making”? The second questions seems ambiguous to me, considering the decision-making could be good in both the short or long term.”

You make an excellent point. With regards to the second question, the wording was chosen to be somewhat ambiguous by design; we leave it up to the human subject to decide what constitutes “better decision-making”, including whether that means taking into account short-term or long-term decision-making. With regards to whether participants “would have understood the difference between the questions”, our empirical results do not measure exact understanding but do show that subjects were affected by the wording of the question, hinting at a strong level of understanding. We find that we can influence subjects towards the partial return model with statistical significance by only changing the wording of the question (see Section 5.3), indicating that even though the change in wording may appear subtle, it can have significant effects on how humans label preferences.

“Section D.5 in the appendix suggests that people generally are better aligned with regret-based preference models. This is also in line with the work of Knox et al. 2022. Could this natural alignment towards regret-based preference have made training people to follow partial return preference less effective?”

We find that, across all experiments, we can influence humans to follow the partial return preference model with statistical significance. Therefore, despite humans being generally more aligned with the regret preference model, all our proposed interventions prove effective at aligning humans with the partial return model.

Thank you for the thoughtful review! We have addressed your concern regarding the practicality of our proposed approach in the response posted for all reviewers and in Appendix A. With regards to your question “How could one actually implement the "trained" intervention in practice for an unknown reward function?”, we propose investigating whether we can now train human subjects to follow a specific preference model in a simple domain, such as the grid-world domain we used in this work, with the goal of influencing their preferences in a more complex, real-world domain where the ground-truth reward function is unknown. We leave this direction to future work—outlined in Appendix A—where this paper serves as a strong motivation for such a direction.

Additionally, we have also addressed your concern about disentangling the effects of simply teaching subjects more about the ground truth reward function rather than the chosen preference model; details in the Appendix of our paper support our hypothesis that this effect is likely relatively minimal, although future work will certainly focus on disentangling these factors more rigorously. Given your feedback, Appendix H now explicitly addresses your concern. The response posted for all reviewers and Appendix H therefore provides a partial answer to your question “How can we disentangle how much of the improvement in matching the human models is due to just learning the reward function better versus following the preference model?” We plan to further address this question in future work—now outlined in Appendix A—where we will elicit human preferences from a domain with a ground-truth reward function that is different from the one used for teaching subjects about a specific preference model.

In your paper summary, you state “the third [intervention] also seems to increase agreement but is not statistically significant given the sample size.” We point out that the third intervention—the Question experiment—does yield statistically significant results for the condition where we influence human preferences towards the partial return model (see Section 5.3).

Thank you for your attention to detail regarding the typos - we have corrected these in the revised manuscript. Regarding the addition of error bars to Figures 5, 7, and 9: We appreciate this suggestion but note that the choice of error bars requires careful consideration. The likelihoods of our given dataset are not normally distributed, hence the non-parametric statistical tests. Therefore, traditional error bars showing standard error or standard deviation would be misleading since these metrics assume normality. We note that the reported significance test results provide much of the value of error bars, though we plan to include them to maximize readability. We would like to spend more time thinking about what error bar metrics would be most appropriate to ensure they accurately reflect the statistical properties of our data. We are open to suggestions you may have as well!