Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries

1. Claim about performance of DQ (half):

(1) For query budget and fairness comparison discussion, please refer to point 1-(1)&(2) in our response to reviewer ufMA.

(2) For effectiveness, please refer to point 1-(3) in our response to reviewer ufMA. Besides, we show in Fig 2 in https://drive.google.com/drive/folders/1wR469npWztzkjyW0YF2H9C10LTI3wnTU that DistQ learning from only distinguishability preference feedback (DQ_d loss) performs far behind DistQ learning from only pairwise preference feedback (DQ_pairwise loss), which demonstrates that DQ (half) actually obtains much less information from a seemingly "roughly equal labeling budget" compared with the baselines. In such case, performance of DQ (half) shown in Fig 3 could be recognized.

At last, to clarify, the comment in line 345 saying DQ (half) "outperforms most baselines only except for MRN" is only made on Quadruped walk", which we think is consistent with Fig 3. As for other tasks, DQ (half) indeed has similar performance to some of the baselines, which is understandable given the above explanation.

2. Overestimation from easy experimental setup:

(1) We've conducted more experiments on harder control tasks. Please refer to point 1 in our response to reviewer pH8H and mQWc. Besides, as pointed out by reviewer mQWc, there also exist challenging control tasks to solve. We believe that for those tasks, subtle information in ambiguous queries does matter, and we demonstrate that our method can work.

(2) Note that our method balances query efficiency and user-friendliness by selecting both informative (ambiguous) and easy-to-answer queries as introduced in Sec 4.2, instead of only selecting easy samples to train on. In our experiments (see Sec 5.5), we also conduct an ablation study to compare the performance of our method and an alternative approach that samples queries only based on easiness (DQ (E) curve in plots). The results show that our method considering both ambiguity and user-friendliness is much better than DQ (E).

3. About testing DistQ on LLM Alignment:

In our current setting for DistQ, the reward model is an ensemble of three-layer neural networks with 256 hidden units, which is quite small compared with the reward model in LLM alignment. Therefore, it may need much more queries and computational resources for LLM alignment, which is hard for us to test given the limited time. We believe the high-level idea of DistQ can be applied for LLM alignment, and we leave this extension to future work.

1. About experiments on more challenging tasks:

Please refer to point 1 in our response to reviewer pH8H. Besides, it is worth mentioning that the suggested harder tasks are not evaluated in all our baselines either. Therefore, we need to determine workable hyper-parameters for all the methods as well as the backbone RL algorithm SAC, which can be time-consuming given the high difficulty of tasks. In our linked results, all methods follow their default hyper-parameter settings without tuning.

Also, there is possibility that these tasks are already challenging when a ground-truth reward function is available. Our method doesn't claim to improve the sample efficiency of the underlying deep RL algorithm. In contrast, we propose a method to circumvent the need of having to define a ground-truth reward function, by asking informative and easy-to-answer queries than other RLHF methods.

2. Additional experiments with real human feedback:

We have conducted the suggested experiments with real human involved and explained details in Sec 5.4 and Appendix D.3. To have a comprehensive understanding of the user study, we also provided an anonymous link of videos of selected queries and evaluation of trained agents in Appendix D.3 in our original version of paper.

1. About more experimental environments:

For the current version of our paper, we selected representative control tasks of different difficulty levels (in terms of query budget needed to accomplish the task) and of different types (locomotion and robotic manipulation) to demonstrate the performance of our method. Following the reviewers' suggestions, we now validate our method on more tasks, including the hard locomotion task Dog walk (hard), and robotic manipulation tasks Door open (easy) and Disassemble (hard). Results are shown in Fig 1 in this link (https://drive.google.com/drive/folders/1wR469npWztzkjyW0YF2H9C10LTI3wnTU). For the latter two tasks, our method can still perform better or competitively compared with other baselines. For the hard Dog walk task, however, all methods (including SAC with ground-truth reward function) fail to achieve it. A possible explanation is that, for such complex tasks, it takes a large number of queries and needs efforts to find workable hyperparameters for the backbone framework. Given the time limit, we haven't found proper settings for all methods and we've tried our best to test on as many tasks as possible. We will include more results in our final version if time permits.

2. About more analysis and experiments:

We deeply appreciate your approval of our method. Besides synthetic feedback, we also conduct user study with real human providing feedback as discussed in Sec. 5.4. We provide both statistical results and cognitive feedback from the participants here, which evidently support our argument of effectiveness and user-friendliness of DistQ. Larger scale user study may take a large mount of human labor. We will find efficient ways to enrich our user study in future.

3. About application to broader domains:

DistQ is built on top of video-based queries and online interleaving of reward learning and agent learning, so the whole method can be straightforwardly adapted to domains like robotic arm experiments and games. As for offline RLHF tasks, we may need to adjust the implementation of the query selection criteria. But the proposed query type and high-level ideas for query selection can still be used in such settings.

4. About discussion of more related papers:

Thank you for suggesting these recent related works. The first two mainly propose efficient and expandable platforms for RLHF experiments with real humans, which can be adopted for efficient large-scale user study. Considering your concerns about more extensive user study, we may utilize such platforms to perform user studies to further validate DistQ. The remaining two instead focus on refining RLHF approaches from different points of view. We will include a discussion about them in our final version.

1. About experimental settings:

(1) To address potential misunderstanding in the review, we first clarify some terminologies we used in our paper. One distinguishability query (DQ) consists of 2 pairwise comparison queries (PCQs). The human first chooses the more distinguishable PCQ and then chooses the preferred trajectory in this PCQ (line 202). The "query budget" in our paper refers to the number of PCQs (line 359) since all our baselines use PCQs. In the experiments section, for both DistQ and DistQ (half), the human chooses the preferred trajectory only from the chosen PCQ in a DQ, instead of "also the not-chosen pair" as stated in the review. That is, for each query in both settings, the human always makes 2 choices instead of 3. The difference between DistQ and DistQ (half) is that DistQ uses the same "query budget" as other baselines while DistQ (half) only uses half of the budget (line 364). See the first paragraph of Sec 5.2 for detailed instructions. We think both settings are consistent with our proposition.

(2) We then discuss the fairness in our experimental evaluation. As stated in the review, for one DQ, the human makes 2 choices. We understand that this may be considered unfair when comparing with baselines using PCQs where the human only makes 1 choice. That's why we design DistQ (half), which only uses 100 DQs (=200 human choices). However, it's also unfair to only count the number of human choices since such choices for DQ and PCQ apparently provide different amount of information. Therefore, we consider DistQ with full budget. See our explanations in the first paragraph of Sec 5.2.

(3) We next emphasize the effectiveness of DistQ. Given (2), it's hard to have a totally fair comparison among DistQ and baselines. The full and half budget settings actually provide a performance range of our method when compared with its baselines. Therefore, it's normal that DistQ (half) can't beat all its rivals in Fig 3. The fact that DistQ (half) can match or outperform baselines demonstrates in our opinion that it is possible to achieve good performance while asking informative and easier to answer queries. In addition, we believe that answering one DQ is easier than two PCQs (especially when not controlling for their hardness) based on our proposition. Besides, the goal of DistQ is to balance query informativeness and user-friendliness rather than only one of the two. Thus, we need to look at both performance (Fig.3) and query easiness (Fig.5, where our method outperforms others).

(4) As for the suggested setting DistQ (half-half), we think this is identical to DistQ in our paper from the view of query quantification.

2. About easiness measurement:

We consider the "predicted feedback" in terms of human side since we hope to quantify easiness by whether the human can provide correct feedback to queries. Similar query easiness measurement to our "wrongly predicted feedback" has also been used in previous work (see the 2nd paper in the references list). For the ground truth preference entropy of queries, it seems not very straightforward to see whether queries are easy to answer for humans compared with wrongly predicted number of feedback. But indeed, we acknowledge that this may also be a good measurement for easiness. We will use this entropy as a supplementary easiness measurement in our final version.

3. About essential references:

Thank you for suggesting these recent works. We will discuss how our paper is related to them in our final version.

4. About user study:

(1) For each method, 150 queries are answered. We conduct 10 rounds of evaluation after one training (line 402-406). During one training, the human answers 150 queries.

(2) The authors of PEBBLE conducted similar (but easier) user study on Quadruped agent, where 200 PCQs are needed. Besides, they only claimed that the agent could succeed but didn't report the success rate. PEBBLE fails to work in our experiments, because the setting is harder and with a more limited query budget.