Query-Policy Misalignment in Preference-Based Reinforcement Learning

We sincerely thank Reviewer EEB5 for the constructive comments. For each weakness and question, we provide the following responses.

W1. Query-policy misalignment not conclusively proven: While the paper proposes that query-policy misalignment is an interesting hypothesis to cause a problem in PbRL, there is no comprehensive evaluation or solid evidence to confirm this proposal.

• We really appreciate this valuable comment. We have added Figure 11 and Appendix D.1 to confirm this proposal. As all the recent off-policy PbRL methods are built upon PEBBLE, we compare the two methods: PEBBLE and PEBBLE + policy-aligned query selection in locomotion tasks. The policy-aligned query selection is proposed in Section 5.1 to address the query-policy misalignment.
  • We compute the log-likelihood of current policy $\pi$ using the queried segments at each query time. Figure 11(a) shows that the segments queried by PEBBLE exhibit a low log-likelihood of $\pi$, indicating that these segments fall outside the distribution of the current policy $\pi$. This demonstrates the existence of the query-policy misalignment.
  • Figure 11(a) shows that, if incorporating policy-aligned query selection into PEBBLE to confine the queries to the local policy-aligned buffer, there will be a substantial increase in the log-likelihood of $\pi$, which means the query-policy misalignment can be addressed.
  • Figure 11(b) shows that the performance of PEBBLE + policy-aligned query selection significantly surpasses that of PEBBLE.
• Overall, Figure 11 demonstrates that: (1) the query-policy misalignment issue does exist in typical PbRL methods and does cause feedback inefficiency; (2) using policy-aligend query selection to address the query-policy misalignment can result in a significant improvement in feedback efficiency.

W2. Lack of real human experiments: The testing and validation conducted in the paper relied on a scripted annotator which is not representative of real-world users, which could limit the generalizability and applicability of the approach.

Q1. The paper could be strengthened by testing the proposed method under real-world conditions, using actual human annotators instead of an oracle, to test the generalizability of their results.

• The use of scripted ground truth reward is to evaluate and compare these PbRL methods quantitatively, unbiasedly and quickly, which is a common practice in existing PbRL literature [1-6].
• We have included additional real human experiments in Appendix E to showcase the applicability and effectiveness of our method. Using feedback from real humans, our method also notably improves feedback efficiency compared to PEBBLE.

Q2. Providing more implementation details or pseudocode for the proposed method would add value to the paper and make it easier for others to understand, replicate, and build upon the proposed method.

• We thank the reviewer for this kind suggestion. We have provided the pseudocode in Appendix B and implementation details in Appendix C on Page 13-16. These details encompass the implementation framework, query selection scheme, data augmentation explanation, and hyperparameter settings.

Reference

[1] Deep reinforcement learning from human preferences, NeurIPS, 2017.

[2] PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training, ICML, 2021.

[3] SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning, ICLR, 2022.

[4] Reward Uncertainty for Exploration in Preference-based Reinforcement Learning, ICLR, 2022.

[5] Meta-Reward-Net: Implicitly Differentiable Reward Learning for Preference-based Reinforcement Learning, NeurIPS, 2022.

[6] Inverse Preference Learning: Preference-based RL without a Reward Function, NeurIPS, 2023.

Thanks a lot for this good suggestion!

As suggested, we have added more experimental details of human labeling process in Appendix E on Page 24-25 in our latest version paper. Specifically, we have provided the main Python code for segment video rendering, labeling instructions, and human preference collection on Page 24-25. The user interface is shown in added Figure 25. The human feedback frequency and total feedback remain consistent with the experimental setup outlined in Table 2. The supplementary material includes the videos of agent training processes in human experiments.

We sincerely thank the time and effort you have engaged in the review and discussion phase!

We sincerely thank Reviewer EPnJ for the useful comments. For each weakness and question, we provide the following responses.

W1. The focus is predominantly on off-policy PbRL methods, with limited exploration of on-policy PbRL methods, which naturally select on-policy segments to query preferences.

• As we discussed in "Conclusion and Discussion" on Page 9, although on-policy PbRL methods naturally select on-policy segments, they suffer from severe sample inefficiency issue as compared to off-policy PbRL methods. Due to the prolonged policy learning process, the query segments generated by slowly learned policies are often not very informative for rapid reward model learning, which in turn causes poor feedback efficiency in on-policy PbRL methods.
• We add the performance curve of PrePPO (on-policy PbRL method) [1] to Figure 5 and 6 on Page 8. We can see that PrePPO's performance is notably inferior to current SOTA off-policy PbRL methods.
• All the recent SOTA PbRL methods, including ours, are built upon off-policy framework. We discover that the query-policy misalignment can potentially harm feedback efficiency, and thus enforces query selection to align with the policy induced distribution $d^{\pi}$. Note that this is different from conventional on-policy RL, as our treatment is conducted on the query selection part (i.e., selecting proper query segments that are already in the replay buffer), rather than collecting on-policy samples for policy learning as in typical on-policy RL methods.

W2. The approach is quite simple and I’m not sure if it’s novel compared to methods that are more similar to on-policy methods.

• To the best of our knowledge, our paper is the first study that reveals the long-neglected query-policy misalignment causes query inefficiency in PbRL. Most existing studies focus on selecting the most "informative" segment for preference labeling. However, as demonstrated in our work, this can be less effective as expected. On the other hand, we provide a remarkably simple method to improve feedback efficiency in PbRL without any fancy query selection scheme.
• Second, as we have clarified in the previous response, our method is an off-policy method, simply using an on-policy PbRL method performs poorly due to sample inefficiency.

W3. The paper should compare to Liu et al. Meta-Reward-Net: Implicitly Differentiable Reward Learning for Preference-based Reinforcement Learning. NeurIPS '22, which is the current SoTA for PbRL.

• We appreciate the reviewer's helpful reminder! The performance of Meta-Reward-Net (MRN) and its comparison to QPA has been added in Figure 12, Appendix D.2. In most tasks, our method QPA consistently outperforms MRN. Essentially, MRN takes a good idea of adopting the performance of the Q-function as the learning target to formulate a bi-level optimization problem, which is orthogonal to our methods. Our technique policy-alignment query selection can be easily incorporated into MRN to further improve feedback efficiency (MRN + policy-alignment query selection).

W4. I thought the Figure 1, 2, 3 could be improved and explained better, both in the text, and in the figure, and in the caption.

• We thank the reviewer for the suggestion. We have added more explanations on Page 2-4. Furthermore, we have provided very detailed experiment settings and descriptions in Appendix C.

Q1. Could the proposed methods be adapted or combined with on-policy PbRL techniques to achieve even better results?

• Our proposed method focuses on the query selection and value learning from the replay buffer. In contrast, in on-policy PbRL, these processes occur within on-policy rollout trajectories, where the policy-aligned query selection and hybrid experience are no longer applicable. As we have explained in the previous response as well as the additional results of on-policy method PrePPO, the low sample efficiency and prolonged policy learning process in on-policy PbRL can hurt feedback efficiency. By contrast, our proposed method has achieved SOTA performance.

Are there specific scenarios or domains where the proposed method may not be as effective?

• When the reward function is highly non-smooth in the state-action space and drastically different in various regions, hard exploration might be needed to learn the reward model effectively. In such case, relying solely on policy-aligned query selection might lead to getting trapped in local solutions.

How does the system handle scenarios where human feedback might be inconsistent or contradictory?

• We have added new experiments with human feedback, which could potentially contain inconsistencies. The results are presented in Appendix E, we find our proposed QPA still achieves reasonable/good performance.

Reference

[1] Deep reinforcement learning from human preferences, NeurIPS, 2017.

We sincerely thank Reviewer LzS2 for the insightful comments. For each weakness and question, we provide the following responses.

W1. I wasn’t able to fully follow some of the logic regarding the justification of the sampling scheme design...

• We thank the reviewer for this helpful comment. We explain the connection between the analysis of Eq.(5) and the proposed technique as follows.
  • The proposed technique, policy-aligned query selection, allocates more queries according to the induced distribution $d^{\pi}$ of current policy $\pi$. This could potentially enhance the accuracy of reward learning within distribution $d^{\pi}$, enforcing the condition of $||\hat{r}\_{\psi} - r||\_{d^{\pi}} \leq \epsilon$.
  • The proposed technique, hybrid experience replay, samples more fresh transitions within the $d^{\pi}$ to update the Q function. This pays more attention to improving the quality of Q-function within $d^{\pi}$, enforcing the condition of $||Q\_{\hat{r}\_{\psi}}^{\pi} - \hat{Q}\_{\hat{r}\_{\psi}}^{\pi} ||\_{d^{\pi}} \leq \alpha$.
  • As shown in Eq.(5), these two conditions would lead to a concrete error bound on the approximated Q-value $\hat{Q}\_{\hat{r}\_{\psi}}^{\pi}$.
• We have taken the reviewer's suggestion and moved the analysis of Eq.(5) to the Appendix A. We thank the reviewer again for this helpful comment.

W2. At the same time, similar ideas have been explored in literature. The field of decision-aware model learning (or value equivariance model learning) used the idea that, in model-based RL...

• We really appreciate the reviewer for this valuable suggestion! Good point! We have added the discussion about the high-level connections between these model-based decision-making methods and our method in Section 2 on Page 3. This could help the readers to better grasp the core ideas and rationale behind our method.
• We have added the discussion and refrences of the following model-based decision-making (local decision-aware model learning) papers. Please let us know if we are missing any important relavant works.

Added relevant model-based decision-making references:

[1] Learning neural network policies with guided policy search under unknown dynamics, NeurIPS, 2014.

[2] Learning contact-rich manipulation skills with guided policy search, arXiv, 2018.

[3] One-shot learning of manipulation skills with online dynamics adaptation and neural network priors, IROS, 2016.

[4] Sample-based informationl-theoretic stochastic optimal control, ICRA, 2014.

[5] Autonomous helicopter control using reinforcement learning policy search methods, ICRA, 2001.

[6] Locally weighted learning for control, Springer, 1997.

[7] The value equivalence principle for model-based reinforcement learning, NeurIPS 2020.

[8] Value prediction network, NeurIPS, 2017.

[9] The predictron: End-to-end learning and planning, ICML, 2017.

[10] Value iteration networks, NeurIPS, 2016.

[11] Universal planning networks: Learning generalizable representations for visuomotor control, ICML, 2018.

[12] Live in the moment: Learning dynamics model adapted to evolving policy, ICML, 2023.

[13] Objective mismatch in model-based reinforcement learning, L4DC, 2020.

[14] Learning Policy-Aware Models for Model-Based Reinforcement Learning via Transition, L4DC, 2023.