On Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problems

We truly appreciate your comments and your thorough review. Let us answer the questions in your review as follows:

My biggest concern is that I found the logic of this paper a bit confusing…

My understanding is that, the authors are not trying to use theory to verify empirical success (which I was expecting), but rather, they use empirical success to prove the theory…

Yes, this is a good question. Overall, what this paper tries to do is propose a theory that can explain a phenomenon in practice. The logic of this paper is intended to be as follows:

Learning with human preference data is observed to yield much better performance in practice, compared to prior efforts of learning with human ratings. It still remains a question what actually causes this superior performance. Thus, we seek to explain this phenomenon by proposing a theory: human ratings are actually subject to more human bias in practice, similar to the model we propose in Section 3, while the generation of human preferences is more robust to such bias (the BTL model that people currently use is already faithful to the real world). It has been a common belief that it is more accurate for human annotators to compare than to rate directly [1,2], but granted, it might seem unfair at first to consider bias in the model for human ratings while keeping the BTL model for human preferences unaffected by bias. We justify this in Section 5, in which we consider bias in the BTL model too and find that using human preferences does not have a better sample efficiency than using human ratings in this setting, contrary to what happens in practice. This shows the superior performance of the human preference approach does not happen because its algorithm (pessimistic MLE) is more statistically efficient than the algorithm used for human ratings (standard LCB); this performance difference is more likely caused by a difference in model assumptions between the two approaches (i.e., one type of model/data is easier than the other).

Overall, the goal of this paper is to propose a reasonable theory for a real-world phenomenon. It would be possible to “use theory to verify empirical success” only if the mathematical formulation and underlying assumptions of the problem of interest are clear. However, it is not the case in this problem, since it is unclear what causes the human preference approach to be better in practice. (Clearly the basic model with subgaussian random reward from the theoretical literature falls short of explaining this.) For this reason, we think it is more appropriate to propose a (heuristic) theory and make sure it is in line with the real-world phenomenon and is also justifiable.

[1] Touvron, Hugo, et al. "Llama 2: Open foundation and fine-tuned chat models." arXiv preprint arXiv:2307.09288 (2023).

[2] Kevin R. Tarlow, Daniel F. Brossart, Alexandra M. McCammon, Alexander J. Giovanetti, M. Camille Belle, and Joshua Philip. Reliable visual analysis of single-case data: A comparison of rating, ranking, and pairwise methods. Cogent Psychology, 8(1):1911076, 2021. doi: 10.1080/23311908.2021.1911076.

Some other issue: the lower bound results (theorem 1 & 2) only considered the LCB algorithm. It will be more convincing to establish a universal and information-theoretic lower bound, i.e., a lower bound that holds for any algorithm.

This is a good suggestion. Theorem 2 can be made into a universal lower bound, but Theorem 1 is specific to algorithms that use pessimism. In fact, we tailor these lower bounds to specific algorithms because together with Section 5, it rules out any algorithmic factor that causes the human preference approach to perform better. It serves our message better to keep the algorithm in the picture. If we proved an info-theoretic lower bound, the message would become “the human rating data following the model in Equation (3) are fundamentally harder than human preference data following the BTL”. Although it still is something we want to show, it no longer has the additional benefit of ruling out the algorithmic factor.

Thanks for the detailed response! I really appreciate that the authors went through the logic of the paper again in detail. However, I still can't find the logic of the paper to make sense. If my understanding is correct, the theoretical conclusion of this paper is still not drawn from theoretical derivation but from empirical observation, as the author said in response

... this performance difference is more likely caused by a difference in model assumptions between the two approaches ...

To be more specific, the authors equivalently did the following in sequence: (1) proposed some theoretical assumptions (the biased ones), (2) did some derivation, (3) observed that the theoretical results don't match the empirical performance, (4) claimed that this mismatch is due to a wrong assumption that they started with, and finally (5) turned to the other assumption (the bias-free one) and claimed that this explains the benefit of learning from preference.

I still don't think this logic is convincing. In my opinion, the more correct and common way to show the provable benefit of something is the following: (1) propose some assumptions and some justification for the assumption, (2) do some derivation and get some results, and (3) observe that the theoretical results match the empirical performance. By comparing this logic with that of the paper, I found that this paper doesn't have an explicit part that shows some justification for the assumption. Instead, the way that they justify the assumptions is by empirical observation. In particular, they justify the "bias-free" assumption by the empirical success, which is exactly what they need to theoretically justify. So I feel there is a circular reasoning.

Moreover, the fact that the paper failed to show the superiority of learning from preference under equal assumptions (ie, the biased one) is unnecessarily due to that learning from preference is bias-free. Actually, this concluded assumption seems super strong in the sense that humans can't be bias-free anyway (but the authors removed the "bias" in the formulation in section 5.1, although I admit that the BTL has some randomness itself). I would suggest the authors to try some more refined assumptions or analyses instead of "bias" vs "bias-free", which also seems to trivialize the problem in the sense that "bias-free" should intuitively be better.

Again I greatly appreciate the detailed response by the authors. However, I still fail to see the rationale of this work. Please correct me if I missed any key parts. I still can't raise my assessment of this work for now since my biggest concern still exists.

We truly appreciate your comments and questions. Let us answer the questions in your review as follows:

It seems that the proposed algorithms are designed based on standard techniques, such as pessimism and MLE. The authors should elaborate more on their technical novelty. This is my main concern.

We appreciate you being clear about your main concern. In fact, the goal of this paper is not to propose new algorithms but to propose a theory that seeks to explain the superior performance of preference-based methods that people observe in practice. We theorize this is because human rating feedback is less robust to bias than preference feedback, so we propose a new model for it, which we believe is closer to what actually happens in practice than existing models. New theoretical results for these classic algorithms are derived under our new observation model. (As can be seen, this model considers a new type of bias and noise different from the existing results in the literature, and the standard LCB algorithm behaves quite differently under our new model. For example, it would normally converge under partial coverage defined in Assumption 1, but we have shown in Section 4.1 it would not converge under our model.)

It would be more clear to present conditions 1, 2 and 3 as assumptions. The authors should justify more on these assumptions. For example, why is condition 1 reasonable? Why the noise never changes the human preference?

You’re correct that Conditions 1-3 are assumptions on the human rating model. We can make it clearer in our revision. Your question about Condition 1 is an important one. It is more thoroughly discussed in the paragraph starting with “Apparently” in the middle of Page 4. In short, Condition 1 is necessary because it keeps the estimation problem statistically consistent or well-defined, i.e., there exists an estimator that converges to the optimal policy in probability. In fact, this is an important contribution of our human rating model in Equation (3) over the ones in the existing literature, which do not always admit a solvable problem. In addition, Condition 1 is in alignment with the assumption of the BTL model (human preference) that humans can rank two state-action pairs correctly in expectation, making the comparison between human rating and preference fairer, as explained in Remark 1 on Page 4.

As to why the noise never changes the human preference, we suppose you’re asking why the bias is not in the BTL model for human preference. This is indeed an important question in our theory. It might seem artificial at first to consider bias in the model for human ratings while keeping the BTL model for human preferences unaffected by bias. This is why we include Section 5, in which we consider bias in the BTL model too and find that using human preferences does not have a better sample efficiency than using human ratings in this setting, contrary to what happens in practice. Section 5 shows the superior performance of the human preference approach does not happen because its algorithm is more statistically efficient than the algorithm used with human ratings (standard LCB), but it is likely that this performance difference is caused by a difference in model assumptions between the two approaches (i.e., one type of model/data is easier than the other). This justifies why we theorize the bias affects human ratings but not so much human preferences. We will be happy to strengthen this discussion in our revision.

In Theorem 1, the setup $C^\star = 2$ seems too specific. Can the result be extended to the one that allows general $C^\star$ and depends on $C^\star$?

This is a good point. $C^\star = 2$ is actually not crucial in our lower bound construction; it can be any constant strictly greater than 1 and would only affect the final suboptimality by a constant factor. In fact, $C^\star = 2$ implies the data have very good coverage (the distribution shift is no more than a factor of 2 over all state-action pairs). The algorithm would only incur a worse constant lower bound under distribution shift with larger $C^\star$, which proves our point even better. Since the goal of this theorem is just to establish the suboptimality is a constant and does not decay as sample size increases, rather than to delineate a specific suboptimality rate, the current presentation is sufficient to demonstrate our point.

Please let us know if this reply is able to address your concerns, and don’t hesitate to let us know if you have any other questions or if you have any suggestions about revision. We look forward to your reply. Thank you!

We truly appreciate your comments and support. Let us answer the questions in your review as follows:

Given the assumptions on the bias transform $h$, I am not surprised that the preference-based feedback is rather invariant to such biases. So I am not sure if the final results are uncovering an informative phenomena.

Your question is completely valid. The theoretical results are not meant to be surprising. There are two reasons for this. First, it is already a common belief that it is more accurate for human annotators to give preference feedback than to give rating feedback [1,2], but there hasn’t been any work that formalizes this theoretically. Furthermore, it is also notable that the bias in our rating model is monotone and does not change the optimal policy of the problem, so it is actually not immediately obvious why such bias makes learning from human ratings harder. Second, while it is not surprising that the robustness of human preferences to bias is a factor of why preference-based methods perform better in practice, we make an important contribution by pinpointing this factor as the likely primary cause among other factors. Specifically, we rule out the algorithmic factor by showing the pessimistic MLE used in preference-based approach is no more statistically efficient than the standard LCB used in rating-based approach.

[1] Touvron, Hugo, et al. "Llama 2: Open foundation and fine-tuned chat models." arXiv preprint arXiv:2307.09288 (2023).

[2] Kevin R. Tarlow, Daniel F. Brossart, Alexandra M. McCammon, Alexander J. Giovanetti, M. Camille Belle, and Joshua Philip. Reliable visual analysis of single-case data: A comparison of rating, ranking, and pairwise methods. Cogent Psychology, 8(1):1911076, 2021. doi: 10.1080/23311908.2021.1911076.

A number of prior works on contextual bandits with preference feedback is not mentioned. While the overall approach is sufficiently different (they optimize a least squared loss), I think they should be mentioned for completeness.

Thank you for providing these references! We will include them in our revision.

How would you go beyond tabular setting? Would you say the pessimistic MLE algorithm can be easily extended to say, a kernelized or linear rewards over a compact domain?

There is no problem extending pessimistic MLE to a more general function approximation, as it has been studied in the linear function approximation setting by [1] and the general function approximation setting by [2]. However, how to extend our newly proposed rating model to the linear setting still needs investigation. Recall our model considers a general bias that is likely nonlinear in the real world. While a tabular algorithm is able to learn such biased reward, linear function approximation would suffer a huge approximation error; RKHS and more general function approximation like neural networks should be flexible enough to approximate such biased reward.

[1] Banghua Zhu, Jiantao Jiao, and Michael I Jordan. Principled reinforcement learning with hu- man feedback from pairwise or k-wise comparisons. arXiv preprint arXiv:2301.11270, 2023.

[2] Wenhao Zhan, Masatoshi Uehara, Nathan Kallus, Jason D Lee, and Wen Sun. Provable offline reinforcement learning with human feedback. arXiv preprint arXiv:2305.14816, 2023.

Please don’t hesitate to let us know if you have any further questions. Thank you!