Active Teacher Selection for Reinforcement Learning from Human Feedback

Thank you for the detailed feedback and suggestions. We found the result that querying noisier teachers can sometimes be more informative to be surprising ourselves, and are pleased that you found it insightful.

We particularly appreciate your suggestions of ways to improve the paper. In response, we’ve made the following revisions:

• We state assumptions more explicitly. When presenting the HUB problem, we state the assumptions that all actions take one timestep (which we agree is a simplification, but a standard one in sequential decision-making literature) and that all teachers reference the same utility function (which is also a simplification, but a necessary one as optimal preference aggregation is an unsolved problem and solving it is out-of-scope for this work). (Section 3, page 3)
• We add additional details to the simple HUB problem presented at the start of the paper to make it more concrete. (Section 1, page 1)
• We add a discussion of crowdsourcing including the proposed references and a few additional ones. (Appendix A, page 13)

In response to your questions about the method, we used a POMDP rather than contextual bandit framing because the contextual bandit doesn’t apply to our problem. In the HUB problem, the state (utility function, arm distributions over items, teacher parameters) never change, so there is no changing “context” to model. ATS maintains and updates a belief over the utility function and arm distributions, so this changing belief functions as a changing state. However, the belief changes in response to actions (for example, querying a teacher can update the belief over the utility function), violating the contextual bandit assumption that state changes are independent of actions. POMDPs are designed for this scenario, so we use a POMDP formulation and solve it with adapted POMCPOW. As an online Monte Carlo planner, POMCPOW generates simulations and determines the policy online (“on-the-fly”).

In response to your question about estimating utilities, we’d like to clarify that preferences depend on both the utility function and the teacher rationality parameter $\beta$ as shown in Equation 1. As a result, utilities cannot be calculated from preferences without also knowing $\beta$. Prior RLHF research typically makes the simplifying assumption that $\beta$ is known, allowing utilities to be calculated from preferences. However, we believe that this is not always a realistic assumption, so we also present a method to estimate $\beta$ (Section 4.2) and demonstrate the procedure in a real-world domain (Section 5.2).

Thank you also for suggesting an additional baseline. We believe that this baseline would be applicable, but would perform significantly worse than our existing Naive baseline (while being more complicated because it relies on ATS and a bandit algorithm). If we understand correctly, your proposed stage 1 would be comparable to a lower-reward version of the exploration period in the Naive algorithm. ATS selects which teacher to query based on (1) teacher query cost and (2) expected usefulness of teacher feedback for identifying higher-utility arms. (2) requires a belief over arm distributions, which ATS must pull arms in order to update. Therefore, if ATS only queries teachers in stage 1, it can only select teachers based on (1), which will result in it always querying the lowest-cost teacher (or choosing randomly, if there isn’t a unique lowest cost). This is comparable to the Naive algorithm always querying the same teacher during its exploration period. However, the Naive algorithm intersperses teacher queries (which impose a cost) with arm pulls (which earn a utility). Since the HUB problem is temporally discounted, it is better to intersperse positive and negative reward actions early on, rather than doing a block of negative reward ones first. Moreover, in the worst case the lowest-cost teacher will be essentially random, and teacher-only ATS won’t learn anything. Your proposed stage 2 (using a bandit algorithm) would also be more complicated than the post-exploration phase of the Naive algorithm (which is deterministic). If stage 1 learns a good utility function, the bandit algorithm may do a better job than the Naive one at finding the highest-reward arm. However, the Naive algorithm identifies the highest-reward arm quite frequently already, so it’s unlikely that this would offset the lost reward from front-loading teacher query costs.

Thank you for your review. We appreciate that you found the problem we present to be timely and interesting.

While your suggested baseline is interesting, it is extremely similar to one of the variants of the algorithm that we present (ATS). In the ATS-general variant, we add a binary action to decide whether to query a teacher, as you suggest. We agree that this is an important baseline for comparison, since standard RLHF systems similarly do not distinguish between teachers. We compare this to the ATS-specific variant, with a query action for each possible teacher, at the end of section 5.1 and Figure 6 (page 8). We find that ATS-specific performs significantly better than ATS-general.

ATS-general uses a POMDP formulation rather than a contextual bandit one as you suggest because the contextual bandit setting is not equipped to handle a continuous high-dimensional state space that changes in response to actions taken. In our problem, the state space is a belief (continuous) over multiple distributions (high-dimensional) that update in response to actions (for example, updating utility estimates based on teacher feedback). This violates the contextual bandit assumption that pulling arms doesn’t impact state, so we must use POMDPs instead.

We agree that it is informative to illustrate the amount of teacher queries. We do so in Figure 3c (showing how teacher queries vary across time and algorithm) and in Figure 10b (showing how teacher queries vary with teacher cost).

Thank you very much for your review. In response to your points:

• We’d like to clarify that we adapted an existing algorithm (POMCPOW) to our HUB problem by designing specialized rollout policies, rather than using an off-the-shelf planner. We’ve clarified this in the paper (final paragraph of Section 4.1, page 5). We further discuss the specific rollout policies that we designed and results from empirical evaluations in Appendix D.
• The HUB problem is not technically a bandit problem because the agent cannot observe utility from pulling arms. We realize that some of our wording was ambiguous, and made minor edits throughout to make this more clear. Because the items and teacher feedback are observed but the utilities are not, this is a partially-observable problem, and therefore a framework for sequential decision-making in partially observable domains is needed. We use POMDPs because they are well-suited to this.
• The utilities from querying arms and costs from querying teachers are both combined into the objective (discounted sum of reward), so the agent must trade off between them. We state this explicitly in Section 3 (page 3). We also investigate the impact of increasing teacher cost and how it trades off against performance and cumulative reward in Appendix E. In response to your specific question, querying a low-cost teacher has the opportunity cost of preventing the agent from pulling arms to earn utility, so continuously doing so won’t maximize cumulative reward.
• In Section 4.2 (previously 3.2), we show how to estimate $\hat{\beta}$ when $\mathcal{U}$ is not known. For completeness, we also show how to calculate $\beta$ directly when $\Delta_{ij}$ is known (which is a more lax requirement than knowing $\mathcal{U}$). However, this isn’t necessary. We’ve revised the wording in Section 4.2 (page 5) and Theorem 2 to make this more explicit.

Thank you also for pointing out confusing notation. In response we made these minor revisions:

• We changed the teacher costs from negative to positive values so that we can subtract them from the reward for the HUB-POMDP as you suggested. (Definition 4.1, page 5)
• We clarified that $\mathcal{U}^*$ and $\mathcal{D}^C*$ are the ground truth values (Section 3.1, page 4).
• We clarified that the legend applies to the full figure for Figure 3. (page 7)

Thank you for the detailed and helpful feedback. We’re pleased that you agreed with the significance of the teacher selection problem and found our mathematical foundation and methodology to be sound, and we appreciate the questions and suggestions on how to improve the paper’s clarity.

In response to your feedback, we made the following revisions:

• We added further explanation of “arms” and “items” to the introduction, and described a concrete example (in which arms are vending machines, items are fruit, and teachers are taste-testers). (Section 1, page 1)
• We added a “Background” section, including a discussion of the POMDP literature. We didn’t find the suggested papers on predictive state representations, undercomplete POMDPs, or off-policy evaluation sufficiently related to our work to include, but did add references to core papers on POMDPs and online simulation-based solvers. (Section 2, page 2)
• We intended for the teacher costs to be negative, so it’s correct to add them in Definition 4.1 (originally 3.1). However, we see that this was confusing, so we revised to assume the costs to be positive, and subtract them from the reward for the HUB-POMDP. (Definition 4.1, page 5)
• We’ve fixed broken links to the appendices throughout.

We also want to clarify the difference between $\mathbb{D}$ and $\mathcal{D}$. As defined in Definition 3.1 (originally 2.1), $\mathcal{D}$ refers to a specific joint distribution, whereas $\mathbb{D}$ refers to a space of joint distributions. We didn’t see a reference to $\mathbb{D}$ page 3 of the original submission, but we did see a reference to $\mathcal{D}^\mathcal{C}$. This notation is correct, as it refers to a specific joint arm distribution. Please let us know if we can clarify further.