Mixing Corrupted Preferences for Robust and Feedback-Efficient Preference-Based Reinforcement Learning

We sincerely appreciate the reviewer's comments. We are grateful because the reviewer's in-depth reviews improved our manuscript. We have updated our manuscript based on the comments and colored by blue. We respond each comment in detail one by one as below.

W1. Necessity of convexity assumption

In considering the assumption of convex states and actions, we concluded that it is not a crucial requirement for our method. The potential lack of real-world convexity in mixed trajectories does not negatively impact network learning. This is similar to the original mixup approach, which interpolates two image-label pairs that might not always be realistic. Despite this, mixup has shown considerable robustness against adversarial examples and incorrect labels, largely due to vicinal risk minimization.
The primary benefit of mixup is its capacity to generalize and regularize learning process, prioritizing the overall data characteristics over the strict realism of individual augmented samples based on convexity. Moreover, studies involving the augmentation of virtual trajectories have shown significant performance gains without explicitly checking the validity of generated rollouts [1, 2]. Consequently, we have decided to remove the assumptions related to convexity and validity from the description of our method.

References
[1] Yu, T., Lan, C., Zeng, W., Feng, M., Zhang, Z., & Chen, Z. (2021). Playvirtual: Augmenting cycle-consistent virtual trajectories for reinforcement learning. Advances in Neural Information Processing Systems, 34, 5276-5289.
[2] Shi, L. X., Lim, J. J., & Lee, Y. (2022). Skill-based model-based reinforcement learning. arXiv preprint arXiv:2207.07560.

W2. More thorough comparison with state-of-the-art PbRL methods

Thanks for the insightful comments. We have expanded our experimental comparison to include all environments on SURF, as detailed in Appendix D.4 (Figure 12-14). Furthermore, we have implemented two additional novel preference-based RL methods, RUNE [1] and MRN [2], which were only referenced in the Related Works section of our previous manuscript. We have also updated the tables detailing the hyperparameters for these methods in Appendix C (Table 3-5).

References
[1] Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for exploration in preference-based reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=OWZVD-l-ZrC.
[2] Runze Liu, Fengshuo Bai, Yali Du, and Yaodong Yang. Meta-reward-net: Implicitly differentiable reward learning for preference-based reinforcement learning. Advances in Neural Information Processing Systems, 35:22270–22284, 2022.

W3. Missing related works and comparison

We appreciate the suggestion to consider the paper on RL from diverse human preferences [1] as a related work addressing corrupted human preferences. We have included the paper in the Related Works section. However, we encountered a challenge in conducting a direct comparison due to the unavailability of their method's code. We have reached out to the authors for access to their code. We are hopeful for a positive response and plan to update our manuscript with additional experiments upon receiving it. Meanwhile, to ensure a comprehensive comparison, we have re-implemented other publicly accessible methods [SURF, RUNE and MRN], as mentioned in our response to W2.

References
[1] Xue, W., An, B., Yan, S., & Xu, Z. (2023). Reinforcement Learning from Diverse Human Preferences. arXiv preprint arXiv:2301.11774.

W4. Introducing mixup in the preliminaries

Thanks for in-depth review. We have introduced the Mixup technique in an additional preliminary section on Appendix A.

(Continue)

Q2. When might MCP not work effectively?

As mentioned in the conclusion section, the current implementation of MCP is designed for continuous action spaces. While MCP's reward learning is not inherently limited by the type of action space, experimental validation remains important. We recognize that its effectiveness in scenarios with discrete action spaces remains to be explored. This area will be focus of our future research efforts.

(Continue)

W5. No analysis with actual human-in-the-loop

We understand the value of including real human annotations in our study to further justify the effectiveness of MCP. However, we faced significant challenges, including the absence of a standardized protocol for human annotation and difficulties in maintaining control over various variables for fair comparison.
To ensure a rigorous and fair evaluation, we followed the practices common in numerous established preference-based RL studies [1-4]. These studies typically utilize the B-Pref benchmark [5] for evaluations. The B-Pref benchmark is renowned for its effectiveness in simulating human irrationality, making it a robust and controlled environment for methodological comparisons. This benchmark allows us to assess the performance of MCP in a consistent and reliable manner, compensating for the absence of direct human-in-the-loop analysis.

References
[1] Kimin Lee, Laura M Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning, pp. 6152–6163. PMLR, 2021b.
[2] Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. SURF: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=TfhfZLQ2EJO.
[3] Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for exploration in preference-based reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=OWZVD-l-ZrC.
[4] Runze Liu, Fengshuo Bai, Yali Du, and Yaodong Yang. Meta-reward-net: Implicitly differentiable reward learning for preference-based reinforcement learning. Advances in Neural Information Processing Systems, 35:22270–22284, 2022.
[5] Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking preference-based reinforcement learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021a. URL https://openreview.net/ forum?id=ps95-mkHF_.

Q1-1. Potential negative impact of corrupted preferences and how MCP mitigate the negative impact?

The negative impact of corrupted preference data is that it can lead to reward estimator that misinterprets user intentions, causing the agent to act inappropriately. Mixing preference data makes it more difficult to merely memorize the corrupted data, acting as a form of regularization [1]. The second benefit is that MCP improves data efficiency by creating unlimited new data, leading to a smoother reward distribution and enhanced generalization. Lastly, it also mitigates overconfidence problem [2], which can hinder the reward estimator learning. Without mixup, training with binary preference labels using binary cross entropy loss might cause significant reward divergence between two trajectories. This happens due to the tendency to assign overly low rewards to non-preferred trajectories (minimizing the likelihood of selecting non-preferred trajectories) and overly high rewards to preferred trajectories (maximizing the likelihood of selecting preferred trajectories). MCP adjusts for this imbalance in trajectory valuation. To evaluate MCP's confidence calibration, we additionally report the reliability diagram and expected calibration error [3] in Appendix D.6 and Figure 16.

References
[1] Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2018, February). mixup: Beyond Empirical Risk Minimization. In International Conference on Learning Representations.
[2] Thulasidasan, S., Chennupati, G., Bilmes, J. A., Bhattacharya, T., & Michalak, S. (2019). On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances in Neural Information Processing Systems, 32.
[3] Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017, July). On calibration of modern neural networks. In International conference on machine learning (pp. 1321-1330). PMLR.

Q1-2. Illustration and detailed explanation of newly generated trajectory

To clarify the concept of newly generated trajectories, we have added a visual example in Figure 7 of Appendix B. This illustration shows how two trajectories are mixed. Each trajectory segment consists of state-action sequences of length of $H$. A new trajectory is formed by linearly interpolating these sequences component-wise, based on the mixup ratio $\lambda$. Consequently, each component (state-action) of the mixed trajectory is a linear combination of corresponding components from the two original trajectories at the same indices.

Thank you for your feedback and for acknowledging our additional experiments. We understand the reviewer's concerns regarding (1) the absence of real human-in-the-loop experiments and (2) the lack of comparion with the closely relevant work [1].
In response to the first concern, which other reviewers have also highlighted as important, we have now included human-in-the-loop experiments in our revised manuscript. For more details, please refer to our General Response.
Regarding the comparison with [1], we agree that it is highly relevant study. However, a direct comparison has been challenging due to the unavailability of public reproduction and verification for [1].
While we haven't been able to address this concern directly, we believe our work provides a comprehensive analysis, including more baselines, varied environments, a higher number of runs, and additional in-depth analyses.

References
[1] Xue, W., An, B., Yan, S., & Xu, Z. (2023). Reinforcement Learning from Diverse Human Preferences. arXiv preprint arXiv:2301.11774.

Thanks for the reviewer's insightful comments on our method. We expect that addressing the concerns above must improve our manuscript. We have updated our manuscript based on the comments and colored by blue. We respond each comment in detail one by one as below.

W1. No analysis with actual human-in-the-loop

We understand the value of including real human annotations in our analysis to further justify the efficacy of MCP. However, we encountered challenges, such as the absence of a publicly verified human annotation protocol and difficulties in controlling other variables for a fair comparison.
To ensure a rigorous and fair evaluation, we followed the practices of numerous established preference-based RL studies [1-4]. These studies typically rely on the B-Pref benchmark [5] for evaluation. The B-Pref benchmark is widely recognized for its ability to simulate human irrationality effectively. We believe that this benchmark provides a suitable and controlled environments for comparing methodologies. It allows us to assess the performance of MCP in a consistent and reliable manner, even without direct human-in-the-loop analysis.

References
[1] Kimin Lee, Laura M Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning, pp. 6152–6163. PMLR, 2021b.
[2] Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. SURF: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=TfhfZLQ2EJO.
[3] Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for exploration in preference-based reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=OWZVD-l-ZrC.
[4] Runze Liu, Fengshuo Bai, Yali Du, and Yaodong Yang. Meta-reward-net: Implicitly differentiable reward learning for preference-based reinforcement learning. Advances in Neural Information Processing Systems, 35:22270–22284, 2022.
[5] Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking preference-based reinforcement learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021a. URL https://openreview.net/ forum?id=ps95-mkHF_.

W2. Necessity of convexity assumption + Q1. Proportion of invalid state-action pairs

In considering the assumption of convex states and actions, we concluded that it is not a necessary condition for our method. While the mixed trajectory might not always adhere to real-world convexity, this doesn't negatively impact the network learning. This is similar to the original mixup approach, which interpolates two pairs of image and label that might not represent feasible real-world scenarios. Despite this, mixup has shown considerable robustness against adversarial examples and incorrect labels, largely due to vicinal risk minimization.
The primary benefit of mixup is its capacity to generalize and regularize learning, focusing on the broader data rather than the precise validity of augmented samples based on convexity. Studies that have augmented virtual trajectories also reported significant performance gains without explicitly checking the validity of generated rollouts [1, 2]. Hence, we have decided to remove the assumptions regarding convexity and validity from the proposed method section to address these concerns.
Nonetheless, we agree that checking the proportion of valid values is also beneficial. We tried to manually set mixed states in the simulator and input them with mixed actions. Unfortunately, the DMControl environments do not offer this functionality. However, in our exploration with the Metaworld environments, specifically the Button Press scenario, we manually set one million mixed transitions from the replay buffer. We found that all mixed actions were feasible at their respective mixed states. This feasibility arises because all actions in Metaworld environments are unconditionally available, and the mixed values fall within the min-max range of spaces.

References
[1] Yu, T., Lan, C., Zeng, W., Feng, M., Zhang, Z., & Chen, Z. (2021). Playvirtual: Augmenting cycle-consistent virtual trajectories for reinforcement learning. Advances in Neural Information Processing Systems, 34, 5276-5289.
[2] Shi, L. X., Lim, J. J., & Lee, Y. (2022). Skill-based model-based reinforcement learning. arXiv preprint arXiv:2207.07560.

(Continue)

Q2. Ablation study with the original trajectories

Thanks for pointing this out. To address this question, we conducted an ablation study in three environments (Walker, Door Open and Sweep Into) using only mixed preference data. This was done by applying just the second term in Equation (8). Figure 15 depicts the learning curves when using only mixed preference data compared to using both original and mixed preference data. 'Concat (O)' refers the original MCP method using Equation (8), which concatenates original and mixed preferences for learning. On the other hand, 'Concat (X)' indicates MCP trained solely with mixed preference data. Our findings indicate that including original preference data in training leads to marginal performance improvements and faster convergence than training only with mixed preference data. However, it is noteworthy that MCP, when trained solely with mixed preferences, prominently outperforms PEBBLE in two of three environments (Walker and Door Open).

Q3. How to select the trajectories used for mixup and effect in performance

We do not have a specific process for selecting trajectories for mixup. As described in Algorithm 1 in Appendix B, within a given preference batch, each preference instance is mixed with another randomly chosen instance from the same batch. This implies that any sampling strategy, such as uncertainty-based sampling, influences only the selection of trajectories for annotation (same as PEBBLE), not the selection of trajectories for mixing within the batch.

Q4. Role and value selection of Beta distribution

We employed the Beta distribution for sampling the mixup ratio, following the approach in the original mixup paper [1]. For $\lambda \sim Beta(\alpha,\alpha)$, $\lambda$ tends toward extreme values (either 0 or 1) as $\alpha$ decreases. It leads to a mix that heavily favors one of the original trajectories. Conversely, a higher $\alpha$ results in a more balanced interpolation of the two trajectories. Because the optimal $\alpha$ might differ across domains, we conducted hyperparameter searches for DMControl and Metaworld. The results are described in Table 6 in Appendix D.3.

References
[1] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.

We sincerely appreciate the reviewer's meaningful comments on justification and statistical significance about our method. We acknowledge that other readers may also have same concerns and addressing them is crucial. We have updated our manuscript based on the comments and colored by blue. We respond each comment in detail one by one as below.

Q1. How mixup improves performance and robustness?

The negative impact of corrupted preference data is that it can lead to reward estimator that misinterprets user intentions, causing the agent to act inappropriately. Mixing preference data makes it more difficult to merely memorize the corrupted data, acting as a form of regularization [1]. The second benefit is that MCP improves data efficiency by creating unlimited new data, leading to a smoother reward distribution and enhanced generalization. Lastly, it also mitigates overconfidence problem [2], which can hinder the reward estimator learning. Without mixup, training with binary preference labels using binary cross entropy loss might cause significant reward divergence between two trajectories. This happens due to the tendency to assign overly low rewards to non-preferred trajectories (minimizing the likelihood of selecting non-preferred trajectories) and overly high rewards to preferred trajectories (maximizing the likelihood of selecting preferred trajectories). MCP adjusts for this imbalance in trajectory valuation. To evaluate MCP's confidence calibration, we additionally report the reliability diagram and expected calibration error [3] in Appendix D.6 and Figure 16.

References
[1] Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2018, February). mixup: Beyond Empirical Risk Minimization. In International Conference on Learning Representations.
[2] Thulasidasan, S., Chennupati, G., Bilmes, J. A., Bhattacharya, T., & Michalak, S. (2019). On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances in Neural Information Processing Systems, 32.
[3] Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017, July). On calibration of modern neural networks. In International conference on machine learning (pp. 1321-1330). PMLR.

Q2. Invalidity concerns about mixed state-action

We acknowledge that mixup-augmented data might lead to infeasible state-action pairs in complex real-world problems. However, our findings indicate that these interpolations, even if not entirely feasible in real world, do not hinder the reward estimator learning. This is similar to how the original mixup works by interpolating images and labels, which might not always result in realistic images. Still, the mixup has been effective in enhancing robustness against adversarial examples and incorrect labels, as well as improving data efficiency through vicinal risk minimization.
The key aspect of mixup lies in its generalization for the network's learning process, as mentioned in the previous question. This prevents the network from merely memorizing the whole data, rather than the strict validity of augmented samples. Furthermore, other studies that create virtual trajectories without verifying their real-world validity have also reported significant performance enhancements [1, 2]. In response to the reviewer's concerns, we have removed the convex and validity assumption in the proposed method section.

References
[1] Yu, T., Lan, C., Zeng, W., Feng, M., Zhang, Z., & Chen, Z. (2021). Playvirtual: Augmenting cycle-consistent virtual trajectories for reinforcement learning. Advances in Neural Information Processing Systems, 34, 5276-5289.
[2] Shi, L. X., Lim, J. J., & Lee, Y. (2022). Skill-based model-based reinforcement learning. arXiv preprint arXiv:2207.07560.

(Continue)

Q3. Statistical significance

Thanks for the suggestion. To address the concerns about statistical significance, we have now incorporated metrics from rliable library [1]. We have included these additional metrics in our analysis, including the interquartile mean and optimality gap, which can be found in Appendix D.4 and Figure 14. This approach provides a more comprehensive assessment of our method's performance compared to PEBBLE and SURF.
In our pursuit of statistically significant comparison, we conducted ten independent runs per each experiment, as is standard in studies like PEBBLE [2] and MRN [3]. This number of runs is double that used by SURF [4] and RUNE [5]. Note that in the field of deep RL, 3-10 runs are typical due to computational constraints. Also, the metrics used in rliable are designed to address such challenge, providing reliable estimates even with a limited number of runs.

References
[1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare.
Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304–29320, 2021.
[2] Kimin Lee, Laura M Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning, pp. 6152–6163. PMLR, 2021b.
[3] Runze Liu, Fengshuo Bai, Yali Du, and Yaodong Yang. Meta-reward-net: Implicitly differentiable reward learning for preference-based reinforcement learning. Advances in Neural Information Processing Systems, 35:22270–22284, 2022.
[4] Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. SURF: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=TfhfZLQ2EJO.
[5] Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for exploration in preference-based reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=OWZVD-l-ZrC.