Eliciting Human Preferences with Language Models

Baselines

Thanks for raising this point, as we could clarify better in the main text. The two primary baselines we consider are 1) No interaction (this is 0 on our graphs) and 2) Detailed prompts, where we have the participant write out their preferences. For the content preferences domain only, we also have access to a dataset of example news articles. We can use this to present two additional baselines: supervised learning and active learning. Unfortunately, in many domains, we do not have access to a large dataset of examples from the target domain; for example, there is no “natural distribution” over moral dilemmas or invalid email addresses. Thus, our experiments attempt to capture the performance of GATE in both settings.

In fact we see that as an advantage of our method: lots of domains do not have large, pre-existing corpuses of data which allow for supervised learning and/or active learning. Our method opens the possibility of learning preferences in those domains without pre-collecting a labeled dataset.

Why not train a reward model?

We agree that reward models are a useful way to model preferences. Unfortunately, reward models require a large amount of data; for example, [1] collected a dataset of over 64,000 examples. By contrast, in our setting we consider interactions with a human that last less than 5 minutes long, which would leave us with far too little data to train a reward model. Furthermore, another advantage of using language models is their ability to comprehend free-form text data, rather than needing to learn from examples. As our transcripts are mostly free-form text (particularly in the two generative questions cases), it is impossible to train a reward model with traditional supervised learning.

Mediating generative open-ended questions

Broadly speaking, there is a tradeoff between interaction time/effort and the degree to which an approach can elicit human preferences. As the reviewer has noted, open-ended questions tend to be on one end of the tradeoff, requiring slightly higher mental demand (although equivalent mental demand to 2 of the 3 baselines), but being the best performing out of the three. Yes-or-no questions, while worse than open-ended questions at eliciting preferences, still beats out all baseline approaches, while being much equivalent or less mentally demanding to all three baselines. We agree that it would be great if there were some approach that could push the Pareto frontier, which we leave to future work.

How is the framework in Section 2 connected to Section 3?

In short, the framework of Section 2 motivates a gap in the literature which we fill in Section 3.

Section 2 describes how existing learning paradigms can be described as task elicitation methods along two axes. First, methods can elicit information passively (as in supervised learning) or actively (as in active learning). Next, they can elicit information with examples (again as in traditional supervised learning) or by rich free-form inputs (such as prompting). However, no present method actively elicits free-form inputs. Section 3 describes our proposal of GATE, a framework for actively eliciting free-form task specifications, and we describe how we operationalize this framework for investigation in our paper. Section 4 then describes the concrete experiments we do given this operationalization.

In Figure 2, what does the * signify?

The asterisk signifies statistical significance at the 0.05 level. We have added this information to the caption in our newest revision.

What do 6/10 and 7/10 settings refer to?

Great catch – There were actually only 9 settings and GATE improves in 7 of them. Specifically, we examined the following 9 (domain, elicitation method) pairs from Figure 2 and counted in how many settings that GATE improves over user-written prompts, ignoring significance: (Content domain, generative open-ended questions): Yes (Content domain, generative yes-or-no questions): Yes (Content domain, generative active learning): No (Moral domain, generative open-ended questions): No (Moral domain, generative yes-or-no questions): Yes (Moral domain, generative active learning): Yes (Email domain, generative open-ended questions): Yes (Email domain, generative yes-or-no questions): Yes (Email domain, generative active learning): Yes

6/10 referred to a metric we had in a previous version of the paper, apologies for the confusion. We have taken this out in our newest revision and fixed the denominator to 7/9.

Does the term “time limit” refer to the 5-minute interaction period?

Yes, we have edited this in our newest revision to clarify.

[1] https://arxiv.org/pdf/2009.01325.pdf

Dear Reviewer j3g4,

Thank you for your review and feedback. As the discussion period is coming to a close, please let us know if our response has adequately addressed your concerns, or if you have any remaining questions and concerns. If not, we would appreciate if you could raise your score. We appreciate your hard work!

We thank the reviewer for their review!

Other preference elicitation formats

This is a great point! This is indeed relevant literature which we will discuss in related work. We believe that GATE is complementary to pairwise comparisons / choice-based elicitation, and we can definitely do them within the GATE framework (i.e. prompt the LM to generate pairwise queries, choices to select among), which would be a good topic for future work. We don't mean to claim that this paper has exhaustively characterized the space of possible GATE policies.

Some answer types are less flexible

We agree—generative active learning and generative yes/no questions both assume a binary set of responses. This might be limiting in real-world settings where the true answer might be “it depends,” or something more subtle. Interestingly, however, we do see some evidence that participants use the open-endedness of the chat interface to provide more details in these settings, rather than always replying with a single “yes” or “no.” And in general we believe that the advantage of our work is that it opens up the possibility for open-ended dialogue and answers that can more fully capture the space of nebulous human preferences, as we explore in the open-ended response settings we study.

Choice of models

We thank the reviewer for this note. While we consider the GATE framework in our paper to be the main contribution of the work, we agree that closed-source models are limited. We hope to revisit these experiments with open-source models in future work, especially as more able models are released.

Answer diversity vs. success of GATE

This is a great question! We did a preliminary investigation of whether we could correlating the % of answers that aren’t “yes” or “no” in a transcript (for the “yes-or-no questions” elicitation setting) vs. the delta p(correct) after the transcript. Unfortunately, we could not find a strong correlation in our preliminary investigation (p-values $0.4$, $0.5$, $0.9$ respectively for the email, moral, and content domains). Note that writing an explanation takes additional time, so transcripts with more explanations tend to also have less turns overall. This preliminary result suggests that directly answering the questions asked by GATE may be more useful than whatever explanation a user may decide to provide. However, more fine-grained analyses may be necessary to break down the nature of the additional explanation / fuzzy answer, and the direct contribution of that answer to the accuracy.

Combine proposed prompts to improve performance & reduce mental load

This is a good question. We believe that combining question types would be an interesting direction for future work, and could be better able to navigate tradeoffs between performance vs. mental load. We will add this discussion to future versions of our paper. We also found that the open-ended case does indeed sometimes ask yes-or-no questions.

We thank the reviewer for their review!

Overfitting and polarization

We agree it is crucial to prevent overfitting to people's preferences. Our work makes efforts to guard against two potential sources of overfitting. First, we guard against statistical overfitting of our classifiers by ensuring that models are always evaluated on how well they can predict preferences on held-out data from another distribution. The strong performance of our models in this setting indicates they are not overfitting to the training data. Second, we check that the model does not cause the humans to “overfit” by strengthening or modifying their own preferences (e.g. via echo chamber effects). Notably, Figure 3, right, demonstrates that people's preferences do not change after using GATE. That said, we agree that longer-term use of personalized recommender systems warrants caution and further study, and have included discussion of this in our ethical considerations section.

Morality

We completely agree that there is no single objective framework for moral reasoning. The point of these experiments in our paper was to see whether GATE could enable a language model to elicit information about an individual’s subjective, personal moral preferences. Interestingly, we do find that many of the questions asked by the language model align with the axes explored by moral foundations theory (e.g. exploring the tension between care and fairness), and have added discussion and citations to that effect in the newest revision.

We thank the reviewer for their review!

Long-term use of GATE

As the reviewer noted, our research received IRB approval for the experiments we carried out in our paper. These experiments involved short-term interactions with GATE; specifically, participants interacted with the model for five minutes. However, we did consider whether interacting with a language model would have an impact on user preferences (Figure 3, right). There, we found that participants did not have significantly different preferences before vs after interacting with the model; that is, using GATE did not materially alter the user’s preferences. We agree that the long-term use of language model assistants is worth caution and further study. In our ethical considerations section we will expand on our discussion of “thin slicing” and other harms to include discussion of potential attachments people may form with language models and how to avoid dependence and over-sharing. We will also caution against widespread long-term use of LM companions prior to these risks being better understood.

Discussion of work in other fields

We are happy to elaborate on these sentences. Many fields are concerned with understanding people’s cognitive processes, inner states, or personal preferences. For example, in cognitive psychology one might attempt to understand what strategies people use when attempting to solve math problems, and in user experience research one might attempt to understand how users understand an interface. However, these inner experiences are often hard to verbalize because people are not always skilled at introspecting and discussing what factors guide their preferences and actions. Thus, many fields study how to better elicit and measure these inner cognitive states, including by interviewing and asking questions of people, as we study in our work

Dear Reviewer djQq,

Thank you for your review and feedback. As the discussion period is coming to a close, please let us know if our response has adequately addressed your concerns, or if you have any remaining questions and concerns. If not, we would appreciate if you could raise your score. We appreciate your hard work!