Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback

We thank the reviewer for the insightful and useful feedback, please see the following for our response.

[Q1: The main weakness is the limitation to control/locomotion tasks. ]

We would like to clarify that Uni-RLHF primarily focus on RLHF research for decision making (locomotion/manipulation/navigation…), building upon the settings established by a series of mainstream research works such as Preference-PPO[1], Pebble[2]. These works concentrate on decision-making tasks too, and we have developed a labeling platform and benchmark that can accommodate a broader range of decision making tasks.

Same as many widely used annotation tools, e.g. PixelAnnotationTool[3] for vision domain and Curve[4] for time series domain, Uni-RLHF also focuses on specific areas and hopes to build a robust eco-system for decision-making. Therefore, we are able to focus on solving specific problems in decision tasks annotation, such as a robust environment access methods, feedback format encoding, and so on. Also, we modify relevant description of introduction to avoid confusion and add the statement that we specialise in decision making domain. (See red text in the updated manuscript, page 1 and 7)

Based on the above core design concepts, we will also continue to maintain and update the platform to support more real-world problem environments/datasets and more modalities.

[Q2: The benchmark only compares offline RL, online RLHF is not ablated. ]

The reason we mainly use Offline RL is that we want to provide researchers with reusable labelled datasets to easily evaluate algorithms without additional labelling cost on a uniform benchmark. While decision making tasks in Online RL needs to learn from scratch, the experimental samples and annotations will be completely different from the algorithms, sampling methods, etc., which makes it difficult to reuse.

As section 3.1 mentioned, Uni-RLHF is able to support the online training mode. We add a more detailed implementation of the online asynchronous annotation process in the revision. (see Appendix E and Fig.8, page 21)

To better show the effectiveness of the online mode of Uni-RLHF, we add an extra online RLHF experiment in the revision (See Appendix E and Fig.9, page 23). After totally 200 queries of human annotation, walker can master the continuous multiple front flip fluently. Through our experiments we demonstrated that Uni-RLHF can access online training to allow agent to learn novel behaviours that are difficult to design reward.

[Q3: The different query samplers are not ablated. ]

Offline RLHF experimental results with crowdsourced annotation are based on the random sampler implementation because of limited resource. However, we provide several ways of sampler definition in our platform to facilitate flexible selection by users and researchers, and our focus is to provide an open and easily extensible platform to customise the appropriate query according to specific tasks. Therefore we hope to open up better tools and remain some open research problems to RLHF community like how to choose the sampler.

To better validate the role of different sampler, we collected new crowdsourcing labels and conducted the additional experiments to demonstrate the performance of different sampling methods. In each experiment, we collect 100 annotation labels (comparative feedback) using a different sampler separately and use IQL-MLP-CrowdSource method for simplicity. See Appendix D.3 (page 19) of our revision for detailed experimental setup and results.

We observe that suitable label sampler could improve the task performance, this is also consistent with a similar conclusion by OPRL[5]. More ablation experiments are in process and we will continue to update the results in rebuttal phases.

We thank the reviewer for the insightful and useful feedback, please see the following for our response.

[Q1: Uni-RLHF's large-scale crowdsourced feedback datasets may contain noise and bias, which can affect the performance of RL algorithms trained on these datasets. ]

We would like to clarify that noise and bias can't be avoided in pratical problems and annotations, that's what we're focusing on. We hope to estimate human irrationality and bias in realistic annotators to futher study the RL algorithms in more realistic scenario.

Several RLHF benchmarks[1][2] utilize predefined hand-engineered reward functions from the environment as scripted teachers, which can be considered a noise-free environment. Experimental results in Table.2 show that our crowdsourced human feedback can achieve competitive performance or outperform compared to noise-free scripted teachers.

In the data collection process, we try to minimise extreme noise and refine the dataset quality including continuous optimisation of task instructions, annotation example and sampling inspection.

We have added further analysis and visualisation of the trained reward model (See Appendix D.2, page 19, revised version paper) and experimentally demonstrated that the reward model trained by human feedback can be well fitted to trends of the hand-designed reward function (high relevance) and aligned with human intent (high prediction accuracy).

[Q2: The system's offline RLHF baselines may not be optimized for specific applications, which can limit their usefulness in practical settings. ]

We agree with the reviewer that the addition of more specific applications can increase the usefulness of the Uni-RLHF, which is also in line with our core motivation. Due to the high cost of labelling, previous research has mostly been in more ideal scenarios such as synthetic labels, however, we provide annotation tools for multi-user annotation and a large number of crowdsourced labels and this will facilitate the RLHF community to research in more realistic settings. Our main goal is to demonstrate the feasibility of a complete annotation pipeline and an open source reusable dataset. As for offline RLHF baseline, we only utilize the collected crowdsourced feedback datasets to verify the reliability of the Uni-RLHF system, which is not the core contribution of this work. We hope that more researchers will access their real-world problems in the platform to annotate and further optimise the framework based on this codebase.

[Q3: The system's reliance on human feedback may introduce additional costs and delays in the RL development process, compared to purely synthetic feedback. ]

It is worth noting that in the most of RLHF application scenarios, we do not have access to synthetic feedback. Purely synthetic labels exist only in ideal research simulation environments, as a wide range of researchers would struggle to afford the cost of crowdsourcing labels in all experiments. The process of generating synthetic labels is as follows: 1) utilize predefined hand-engineered reward functions as scripted teachers, 2) score of trajectories by scripted teachers to obtain synthetic labels. In application scenarios, RLHF is mostly suitable for environments where manual design rewards are difficult, when additional costs and delays are unavoidable.

Take SMARTS domain as an example, designing reward functions is extremely difficult, requiring extensive expertise and multiple rounds of training attempts for iteration, the final training reward (marked as oracle) consists of five reward items of widely varying magnitude: Distance, Near Goal, Mistake Event, Lane-Selection and Time Penalty. Adjusting the reward function also requires a significant amount of cost, whereas crowdsourcing feedback annotations enable simply to achieve competitive performance to oracle and more consistent with human intentions. (See Section 4.1.3, page 8 and Appendix D.1, page 18 for details)

---

[1] Lee K, et al. B-pref: Benchmarking preference-based reinforcement learning. arXiv preprint arXiv:2111.03026, 2021.

[2] Shin D, Dragan A, Brown D S. Benchmarks and Algorithms for Offline Preference-Based Reward Learning. Transactions on Machine Learning Research, 2022.

We thank the reviewer for the insightful and useful feedback, please see the following for our response.

[Q1: The claim for a "standardized feedback encoding format along with corresponding training methodologies" might be overly ambitious.]

Thank you for your valuable suggestion! After careful consideration, we focus on proposing a set of feedback encoding format which are easy to use. We then train them with possible training methodologies based on adapting and improving existing literature, like comparative and attribute feedback in the previous paper version. So we change the statement to standardized feedback encoding format along with possible training methodologies in the revision. (See Section 3.2, page 5)

Based on this core idea, we complement the revision with new crowdsourced labels of keypoint feedback and corresponding experiments (See Appendix D.5, page 20), so the dataset and benchmark contains comparative feedback, attribute feedback and keypoint feedback now. And we provide more possible training methodologies for the all 5/5 feedback types. (See Appendix G, page 5) We will continue to research and expand the platform and datasets for diverse human feedback.

[Q2: Comparative feedback interface also does not allow the user to indicate the strength of preference]

We agree with the reviewer that it might be very meaningful to further allow the user to indicate the strength of preference. We have added this option to the platform design through a slider like Anthropic's dataset[1]. We've also updated the description and cited the paper in the Appendix G.1, page 23:

It is worth noting that $y_\text{comp}$ can simply be extended to a finer-grained division of labels including the degree to which A is better than B. For example, $y_\text{comp} = (0.75, 0.25)$ represents that A is better than B with a smaller degree than (1, 0). Also, the expansion method can simply be trained along the Eq. (4) without any changes.

[Q3: the paper provides too few details on the way Atari experiments were performed]

Related to the previous point, the paper provides too few details on the way Atari experiments were performed. Appendix G and Section 4.1.1 imply that comparative feedback was collected, but Figure 5 (d) in the appendix -- that visual feedback was used instead.

Sorry for the confusion. We begin by clarifying that our atari experiments use comparative feedback for all, which is intended as an implementation of the CNN reward structure. In Figure 5 (d) in the appendix, we demonstrate that we can use the atari environment offline dataset for visual feedback annotation in the Uni-RLHF platform, but we do not provide the annotated dataset for visual feedback and the corresponding experiments due to limited resource in the current version, and we will continue to explore the performance gain of visual feedback in our future work.

The highlighting in Table 8 should be explained. My guess is that the best of the ST and CS labels is highlighted.

Yes, you are absolutely correct in your understanding! We use highlighting to mark the better one of the ST and CS labels. And, we increase the importance of atari experiments and provide clearer experimental details In the revised main text. (See Section 4.1.2 Atari Experiments, page 8)

[Q4: About some confusion in full results of Table 2]

We sincerely thank the reviewer for efforts to improve the quality of our papers, In revision, we have carefully examined the data and marks throughout the table, redesigned some of the annotation (blue highlight and *) and re-analysed experimental findings. (Table 2, page 8)

Star annotations in Table 2 are incomplete.

Sorry for the confusion. In the updated revision, we use * to mark the method that works best of all (usually Oracle). Since multiple algorithms are close in effect in many datasets, we take the best performing as the baseline, and if standard deviations in the rest of the algorithms intersect, they are similarly labelled as *.

Blue annotations in Table 2 are also significantly wrong.

We apologise for the unclear description. In our definition, blue is used to highlighted which of the CS-MLP and CS-TFM methods is better and we've carefully checked in the revised version.

Some CS-TFM results are missing from Table 2.

We filled in all the missing results in Table 2 and reanalysed them based on complete results.

About performance comparison using ST and CS.

Thanks for the correction. We modify the description in the revision to argue that CS can achieve competitive performance with ST in IQL backbone, even script teacher (ST) can be regarded as an expert with some degree. And other offline RL approaches (CQL and TD3BC) show that CS labels are better more consistently. We obtain a similar conclusion in atari domain (image observation and discrete action space).

First, I would like to thank the authors for an extensive revision that carefully takes into account the reviewers' suggestions.

• [regarding the inconsistencies in Table 2 and its analysis in the main text] The new version looks consistent and complete. I agree with the updated analysis in the paper as well.
• [regarding Atari experiments] I thank the authors for the clarifications in the reply and in the paper.
• [regarding the more complete discussion of RLHF training methods and the extra keypoint dataset and evaluation] I think this is a valuable addition to the system.

Regarding other reviewer's comments, I believe that the authors' response addresses the concerns well.

• Reviewer FdzJ points out some weaknesses of the system that are inherent in any RLHF approach where real human feedback is used. I agree with Reviewer FdzJ that the nature of work is incremental, but we both seem to think that the contribution is enough for publication.
• Reviewer BKHb points out missing experiments for a better understanding of the results, which the authors do provide, with the exception of labels for conversational models. I believe that a system with a focus on non-textual agents is still a sufficient contribution, even if it does not contain evaluation for online RLHF. The revised version goes even further and does include experiments on online RLHF as well.

Based on these considerations, I am happy to update my rating for the paper from 6 to 8.