Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learning

Weakness # 1:

• In the Human-in-the-Loop section of the Related Work, we briefly describe the five baselines used in our Experiments section, which include the preference learning algorithms: PEBBLE, SURF, RUNE, MRN, and the scalar feedback algorithm, Deep TAMER. To improve clarity, we will refer to these algorithms by name throughout this section. Additionally, we will emphasize that, like our method SDP, all the preference learning algorithms also learn a reward model, making this aspect more explicit in the revised section.
• We would also be happy to include additional relevant related work if you were able to make some suggestions? We think we have covered the most relevant works but are always interested in identifying things we’ve missed.

Weakness # 2:

• We will revise the submission to make it explicit that SDP never hurts performance by a statistically significant margin.

Question # 1:

• We augmented SDP with four preference learning algorithms: PEBBLE, RUNE, SURF, and MRN. Of these, SURF incorporates data augmentation within its algorithm. We compared the performance of the baseline algorithm both with and without the addition of SDP. These algorithms were chosen as baselines for two reasons: (1) they are popular preference learning methods in the literature, and (2) both these baselines and our work aim to improve the feedback efficiency of preference learning. In other words, the goal is to develop a preference learning algorithm that can learn an accurate reward model with fewer human queries.

Question # 2:

• SDP does assume access to the minimum environment reward (r_min). However, we note that this assumption is commonly made in previous works [1,2]. However, future work can consider ways to relax this assumption. For example, one possibility is to use the lower bound of the reward model’s output as r_min.

Question #3:

• In future work, we plan to provide a theoretical analysis of SDP. Specifically, in the context of learning from scalar feedback, where a reward model is learned through regression. We intend to demonstrate that the reward bias introduced by SDP does not outweigh the reduction in model variance. Additionally, we intend to evaluate the efficacy of SDP with human teachers in more complex environments, such as a real robot performing a pick-and-place task. This is crucial for further validating the applicability of SDP to real-world scenarios. These points will be added to Section 6 of our submission.

References:

• [1] https://arxiv.org/abs/2202.01741
• [2] https://arxiv.org/abs/2010.14500

Weakness # 1:

• The primary focus of this work is on improving the feedback efficiency of human-in-the-loop reinforcement learning (HitL RL algorithms). While it is true that the work builds on established techniques, such as the Bradley-Terry model, our approach addresses the specific challenge of how to make more efficient use of limited human feedback (scalar ratings or preferences over trajectories). This is an open problem in the HitL literature and has significant practical implications, as common HitL RL techniques often require a large number (up to tens of thousands) of human queries, making it costly and labor-intensive.

Question #1 / Weakness # 2:

• It is important to emphasize that the effectiveness of SDP relies on the transitions being pseudo-labeled to be sub-optimal. In Equation (4), we define sub-optimal transitions as those for which the difference between the true reward and the minimum reward is smaller than a small threshold, epsilon. If this assumption holds, the bias introduced by using an incorrect reward label will be minimal, and the reward model will benefit from the larger dataset, leading to a reduction in model variance. However, if this assumption does not hold, SDP can negatively impact performance. In Appendix D.7 (Figure 16), we presented an experiment where we replaced sub-optimal transitions with those from an expert policy. In this experiment, we observed a performance degradation, highlighting that when the reward bias is large—such as when using expert policy transitions—SDP fails to improve performance.

• Moreover, we evaluated the efficacy of SDP on two widely used testbeds for preference learning: the DMControl Suite and Metaworld, which are standard benchmarks in this domain [1-5]. Notably, 7 out of the 9 tested environments featured observation spaces exceeding 20 dimensions. In particular, we tested SDP on Quadruped-walk (main results) and Quadruped-run (ablation study), both of which have observation dimensions of 78. Future work could explore higher-dimensional settings, such as learning directly from pixel inputs.

Weakness # 3:

• We thank the reviewer for recognizing the effort we put into the human subject study. We appreciate your suggestion regarding the potential benefits of including direct interventions and corrections in the user study. We agree that learning from human interventions, such as trajectory corrections, can provide more detailed insights into human intentions. However, this research focused on learning from feedback (scalar signals or preferences). Therefore, to test the efficacy of our method with real human teachers, requires us to perform a human-subject study where participants are providing preference-based feedback.

• In general, preference-based feedback can be a more scalable and less labor-intensive method compared to direct interventions [6], and it has been shown to be effective in many real-world applications such as the rise of large language models. That said, we agree that future work should explore combining preference-based feedback with direct human interventions, which would allow for a richer understanding of human intent and could further improve the performance of preference learning algorithms.

References:

• [1] PEBBLE: https://arxiv.org/abs/2106.05091
• [2] SURF https://arxiv.org/abs/2203.10050
• [3] RUNE https://openreview.net/forum?id=OWZVD-l-ZrC
• [4] MRN https://openreview.net/forum?id=OZKBReUF-wX
• [5] Few shot: https://arxiv.org/abs/2212.03363
• [6] https://users.cs.utah.edu/~dsbrown/readings/assist_teleop.pdf

Weakness # 2:

• We have included a comparison of the effectiveness of SDP for users with and without a computer science background and found that there was no significant difference (Table 7, Appendix C).
• We have performed additional statistical testing comparing the effectiveness of SDP across the following demographic conditions: gender, age, and educational background. We found no significant differences across these dimensions. We have updated the manuscript with this result in Table 8, Appendix C. | Task | Feedback | Demographic | AUC | P Value | |--------------------|----------|----------------------------------|---------------------|---------| | Cartpole-swingup | 48 | Female | 8720.03 ± 3185.4 | 0.085 | | Cartpole-swingup | 48 | Male | 11770.22 ± 1669.96 |
  |Pendulum-swingup | 40 | Female | 8587.13 ± 2772.96 | 0.331 |Pendulum-swingup | 40 | Male | 10134.67 ± 4822.22 | |
  | Cartpole-swingup | 48 | Age 18-30 | 10109.54 ± 2370.61 | 0.313 | | Cartpole-swingup | 48 |Age 50-70 | 10923.05 ± 1292.83 | | | Pendulum-swingup | 48 | Age 18-30 | 8619.07 ± 3381.48 | 0.318 | | Pendulum-swingup | 48 |Age 50-70 | 10607.93 ± 5249.72 | | | Cartpole-swingup | 48 | Education: Bachelors or higher | 11044.55 ± 1783.25 | 0.794 | | Cartpole-swingup | 48 |Education: High school diploma | 6247.99 ± 5186.47 | | | Pendulum-swingup | 48 | Education: Bachelors or higher | 10188.88 ± 3858.95 | 0.742 | | Pendulum-swingup | 48 |Education: High school diploma | 7677.85 ± 3540.7| |

Question # 1:

• Yes, SDP can be more effective than traditional RL methods that learn from sparse rewards. Importantly, SDP learns a dense reward function from human feedback. RL algorithms historically have more difficulty learning in sparse reward settings for a variety of reasons: (1) credit assignment problem – when the agent receives a reward, it must figure out which states and actions it should credit for the reward, (2) slower training – the agent can only update its actor/critic when it receives a reward which infrequently occurs in the sparse setting. SDP, by contrast, can provide an intermediate reward, helping the agent learn more effectively.

Question # 2:

• In our experiments, we found that more complex environments (e.g., Quadruped-walk) performed well with the same amount of prior data as less complex environments (e.g., Cartpole-swingup). However, it is likely the case that in even higher dimensional environments, more sub-optimal data would be useful in order to achieve a better coverage of the sub-optimal state and action space.

Question # 3:

• We have evaluated SDP on both Quadruped-walk (main result – Figure 2) and Quadruped-run (ablation – Figure 4), each of which has an observation size of 78, whereas Humanoid from the DMControl suite has an observation size of 67. We observed that in Quadruped-walk, the addition of SDP was able to statistically outperform MRN and PEBBLE, and achieved comparable performance to SURF and RUNE. Similarly, in Figure 4, we found that SDP achieved greater learning efficiency than PEBBLE in Quadruped-run.

Question # 4:

• For any preference learning method, it is crucial that human teachers understand the objective of the task. To facilitate this, we provided detailed instructions outlining the task’s goal in our human-subject study. Additionally, we included video clips illustrating both task success and failure to ensure clarity. Familiarity with the preference collection interface also plays an important role in reducing errors. To address this, participants were given a practice round of providing preferences. These measures help ensure that the collected preferences are accurate and reliable. In Appendix B.1, we outlined in detail the protocol of our human-subject study.

Weakness # 2:

• One of the key strengths of SDP is its simplicity, which offers several advantages. Simpler algorithms are often easier to implement, debug, and maintain which can facilitate broader adoption.
• In addition, the use of sub-optimal data has been explored in the offline RL literature [1]. However, we note that in Appendix D.5, we included an experiment where we directly adapted this technique to the preference learning setting in the Walker-walk environment. Our results showed that the average final performance achieved using this adapted approach was nearly 4 times lower than the final performance of SDP. In addition, in Section 5.2, Figure 6 we performed an ablation of the components of SDP, the reward model pre-training phase, and the agent update phase. This study demonstrated that neither component, when used individually, leads to improved performance. Together, these experiments highlight that naively incorporating suboptimal data does not yield performance gains in the human-in-the-loop RL setting.

Weakness # 3:

• We conducted an experiment in the Cartpole-Swingup environment where we replaced the pseudo-labels of r_min​ with the labels generated by the preference-learned reward model. However, this ablation significantly underperformed compared to SDP. The primary issue with this ablation is that SDP relies on the reward model pre-training phase, which occurs before any human feedback is provided. In this ablation, however, the suboptimal data used to pre-train the reward model are pseudo-labeled with random rewards. These rewards are random because the preference-learned reward model has not yet been updated with human feedback. As a result, its outputs are random and fail to produce meaningful pseudo-labels, negating the usual benefits of the reward model pre-training phase. | Method | AUC | |---------------------|------------------------| | SDP | 21519.9 ± 2435.02 | | SDP without r_min | 7179.41 ± 505.97 |

Weakness # 4:

• While we agree that evaluating against methods utilizing suboptimal data could be interesting, we believe that comparing to an offline RL algorithm would not be appropriate due to the vastly different problem settings and assumptions. Our work focuses on human-in-the-loop algorithms, where reward models are learned in real time through scalar or preference feedback, requiring both human input and online interaction with the environment. In contrast, offline RL assumes access to a pre-collected dataset of transitions spanning from low- to high-quality policies and may not allow for additional online interaction with the environment. These fundamental differences make direct comparisons between the two approaches challenging and less meaningful.
• For an upper bound comparison, we argue that a better alternative is to consider the online RL algorithm SAC which learns directly from the ground truth reward function. This is a more appropriate benchmark for several reasons: (1) in our experiments in Section 5.2, the human-in-the-loop algorithms aim to approximate the ground truth reward function using preference or scalar feedback; (2) the human-in-the-loop algorithms we evaluate, including ours, use SAC as the base algorithm, ensuring consistency in comparison; and (3) as an online RL algorithm, SAC is expected to achieve higher performance than offline RL methods, which are inherently limited by the quality and diversity of the data they rely on.
• We have included the performance of SAC in all environments to provide a clear upper bound for evaluating our approach and the baselines. That said, if all reviewers agree that this would be an important benchmark, we are open to adding such results to our revised submission.

Weakness # 5:

• We agree that investigating more complex domains would be valuable for illustrating the efficacy of our method in more realistic and challenging environments, and we plan to explore this in future work. However, in this submission, we specifically chose Cartpole-Swingup and Pendulum-Swingup to ensure that we could gather a larger sample size of participants. This decision allowed us to conduct a more robust evaluation of SDP in a controlled setting, where we could assess its performance across a broader range of users.

References:

• [1] https://arxiv.org/abs/2202.01741