Deep Reinforcement Learning from Weak Hierarchical Preference Feedback

Dear Reviewer roR3:

Thank you for the insightful and critical review of our work. We believe that your first two cited weaknesses stem from a misunderstanding, and that by enhancing the clarity of our paper, they can be addressed. We also include discussions on your second two points, which we believe only require minor adjustment to be addressed in our paper. Please let us know if you have any further questions.

Weakness 1: Lack of clarity

• The general setting we consider here is one where at each time step, we receive a state observation, take an action, and then receive the next state as well as a list of reward factors. This setting is the same as that commonly used for reward engineering. Our setting is described in detail in Section 3.1 of the submitted manuscript.
• Given this setting, HERON asks the human overseer to assign rankings over the order of the set of reward functions, $z_1, …, z_t.$ A concrete example would be (wait time > queue length > number of emergency stops) in traffic light control. This is described in the weak preference elicitation paragraph. These rankings are then used for learning.
• We make the assumption that a decision tree over the reward factors will be able to approximate the true reward. Using this assumption has two main benefits: (1) constructing the decision tree requires less human input than the typical linear combination that reward engineering employs and (2) decision trees are highly flexible, and can therefore provide a good approximation of the true reward in most practical scenarios. Our experimental results back such an assumption.

Weakness 2: reliable comparisons

• In HERON, the comparisons are coming from the algorithm itself, and there is no human involved in the preference elicitation stage (see Algorithm 1).
• Indeed, HERON is a new reward design algorithm, but we believe that it is a novel and significant contribution: no algorithms have been proposed to use the type of rankings that HERON uses, and HERON is a principled method that can outperform classical reward engineering approaches, only utilizing the rank of the reward factors and gaining robustness.

Weakness 3: Reliance on reward factors

• Thank you for bringing this point up. Indeed, HERON is designed for the classical setting where a set of reward factors is available. We believe that such a setting can still be useful, as demonstrated by our experiments in code generation, robotics, and traffic light control, all of which are practical environments where learning directly from human feedback would be extremely costly.
• Moreover, we believe that using reward factors could boost emerging applications which require reward learning directly from the input, such as LLM finetuning. For example, in LLM finetuning, we could use tools such as grammar checkers and unit tests as the reward factors to allow HERON to be used. Such tools could even be used in conjunction with human feedback.

Weakness 4: Experiments and comparison to RLHF

• A key advantage of including the trajectory as the input of the reward model is that doing so allows us to take the trajectory context into consideration, which reward engineering does not allow for. For example, in the code generation task, we would like the reward model to give a higher score to a program if it is almost correct compared to when it is completely incorrect. This can only be done by using the trajectory as the reward model input. Also, for code generation we fine tune a pre-trained model to obtain our reward model, allowing us to transfer domain knowledge obtained through pre-training into the reward model.
• However, we believe that the main novelty and contribution of our work comes from how we use the reward factors to elicit preference labels. Once we have these labels, any technique can be used to learn the reward function, including linear regression with the weak reward factors as inputs.
• To address your second point, we remark that our algorithm is not meant for the setting where a human annotator is used to compare each trajectory. Therefore we do not include RLHF in our main experiments. However, we do include a comparison of HERON to RLHF in Appendix F.

Question 1: Problem setting description

• We include a description of our problem in Section 3.1 of the submitted manuscript.
• We do not make any strong assumption on the ground truth reward, but we do expect that the hierarchical comparison rule created by HERON will be aligned with the comparison rule induced by the ground truth reward. We believe that this assumption is reasonable, as the human overseer has ranked the reward factors by importance.

Dear Reviewer yRzv:

Thank you for the detailed review of our work. We believe that your two main concerns on marginal improvements are addressable. As for the concerns about unconvincing comparisons, we believe there is a major misunderstanding. Our method is inspired by RLHF, but is not an RLHF method. We attempt to clarify in our response and in the updated paper. Please let us know if you have any further questions.

Weakness 1: Marginal Improvements

• We would like to point out that the improvements over comparable baselines are not marginal. In robotics, HERON results in a 7.8% improvement compared to reward engineering. For traffic light control, HERON beats all other baselines by 33%. For Coding, HERON can beat the CodeRL method by 3.9% as measured by Pass@K and by 5.2% as measured by filtered Pass@K, while greatly reducing the tuning cost. These improvements are displayed in Tables 1, 2, and 3 and Figure 3c of the submitted manuscript.
• Moreover, we would like to point out that our method is designed to learn from rankings, greatly reducing the tuning cost of reward design. Therefore the goal of our experiments is not necessarily to beat carefully tuned rewards, but to show that we can match them while only using rankings over reward factors.

Weakness 2: Unconvincing Comparison

• We believe there is a misunderstanding here. In our main experiments, we do not compare our method to RLHF techniques as RLHF considers a completely different setting than our paper considers (see section 3.1 for our problem description). Instead, RLHF serves as the motivation for our HERON algorithm, as mentioned in the introduction. HERON is designed as an alternative to reward engineering.
• We primarily compare our method with (1) policies trained with the ground truth reward (2) heuristic algorithms that receive the same amount of information of HERON (see the description of baselines on page 6) and (3) State-of-the-art reward engineering approaches for the Coding task. All of these algorithms either receive more or an equal amount of information in the reward design process than HERON, and are therefore meaningful baselines.

Question 1: Comparison with RLHF Baselines

• As we previously mentioned, our work is inspired by RLHF, but is not designed for the RLHF setting. Therefore it would not be meaningful to compare it to methods like PEBBLE and MetaReward-Net, which we consider to be orthogonal directions of work. However, we do include a comparison to RLHF based learning in Appendix F, where we further examine the difference between HERON and RLHF.

Question 2: Addressing Limitations

• HERON essentially proposes a new RL setting, where only weak feedback is given. The main limitation of HERON is that not every RL environment will contain an obvious ranking over the reward factors, as some factors may be equally important. We have added more discussion in Appendix L of the updated paper.

Question 3: Practical Significance

• Our main motivating problem is code generation, where reward engineering may suffer from approximation bias and require a large tuning effort. In contrast, ranking the reward factors will be easy. Concretely, CodeRL introduces 4 tunable reward parameters, while PPOCoder has 6. In this task, we show that HERON can achieve significantly better performance than state-of-the-art baselines, therefore demonstrating HERON’s practicality in the very useful task of code generation. Similarly, we achieve great results in traffic light control, which is a practical and real world application.

Dear Reviewer 9kSM,

Thank you for the insightful and clear review! We provide our responses to your questions and concerns below and look forward to further discussion.

Weakness 1: Lack of clarity regarding motivation of reward modeling.

• Note that we have explained our motivation of employing reward modeling in paragraph 5 of the introduction, where we state that we use a reward model to leverage human rankings over reward factors. Here we would like to summarize our motivation again:

• The main motivation of HERON is to replace reward engineering, which requires handcrafting the reward. HERON uses human rankings over reward factors to automatically generate preferences between trajectories. These comparisons are then used to train a reward model. The motivation of using a reward model is to convert the preferences between trajectories into a numerical reward that can then be used to train the policy.

• In the updated version we have added a clarifying sentence to paragraph 5, stating “The reward model provides us a principled manner to convert rankings over the reward factors into a numerical reward.”

Weakness 2: Lack of ability in personalization.

• This is an interesting point. Beyond changing the ranking, we can increase the flexibility of HERON by tuning the margin parameters in the HERON framework. For example, if we would like to increase the emphasis on a given reward factor, we can decrease the size of the margin for that reward factor. In this way, a greater percentage of the decisions would be taken according to the reward factor we wish to emphasize. In the updated version, we have added discussion about how to increase flexibility in section 4.3, writing “To further increase the flexibility of HERON, we could tune the margin parameter of each reward factor, therefore impacting the fraction of decisions made at each level of the decision tree.”
• In addition, in the case where we have two factors, we can randomly drop the comparison based on the second factor is we would like to increase the dependence on the first factor.
• Finally we remark that HERON has two additional benefits for enhancing the capability of reward modeling, which the linear combination of the reward factors does not have: (1) HERON allows us to take advantage of the contextual/state information contained in the trajectories by predicting the reward based on the trajectory and (2) for certain applications, e.g. code generation, it allows us to use pre-trained models as the initialization of the reward, and therefore vast amounts of domain knowledge (e.g., semantic and syntactic knowledge of code) can be transferred to the learnt reward.

Question 1: Learning curves in robotics environments

• We have included the curves in Appendix K of the updated paper. From these curves, we observe that HERON matches or beats the performance of reward engineering while being much more stable.

Question 2: Hyperparameters in Figures 3a and 3c.

• We have sufficiently tuned the baselines in both of these experiments, selecting the best policy (over 5 random seeds) from the learning rates {1e-6, 1e-5, 1e-4, 1e-3}. We believe that these two baselines have high variance due to the poor shaping of the reward, which can lead to instability in training.

Dear Reviewer rUJV,

Thank you for the thoughtful review and suggestions. We provide our response to your concerns and questions below.

Weakness 1: Lack of expressivity.

• Thank you for pointing this out. We remark that HERON and classical reward engineering are two different design principles, and which one can better capture the reward may be problem specific. However, decision trees can be highly non-linear, therefore allowing HERON to approximate a wide variety of reward functions.
• In practice, we typically find that the HERON framework does provide enough expressivity, as shown by Section 4.3 where switching the order of secondary and tertiary rewards correctly changes how the agent acts. We can further increase the flexibility of the HERON framework by tuning the margin parameters. For example, if we would like to increase the emphasis of the reward on a given reward factor, we can decrease the size of the margin for that reward factor. In this way, a greater percentage of the decisions would be taken according to the reward factor we wish to emphasize. We have added discussion about how to increase flexibility in section 4.3 of the updated paper.
• Finally we remark that HERON enhances the capability of reward modeling in two ways: (1) it allows us to take advantage of the contextual information contained in the trajectories (which reward engineering does not do) by predicting the reward based on the trajectory and (2) it allows us to use pre-trained models as the initialization of the reward, therefore vast amounts of domain knowledge (e.g., semantic and syntactic knowledge of code) can be transferred to the learnt reward.

Weakness 2: Lack of clarity on the setting.

• We believe there is a misunderstanding here. The general setting we consider is one where at each time step, we receive a state observation, take an action, and then receive the next state as well as a list of reward factors. This setting is the same as that commonly used for reward engineering. Our setting is described in section 3.1 of the submitted manuscript.
• With regards to the reward modeling setting, the only difference with RLHF comes from how we obtain the preference labels. The input to the reward model is still the trajectory segments, but now the preference label is obtained from HERON’s decision tree over the reward factors. This forms the key novelty of our algorithm. Otherwise all aspects remain unchanged from standard reward learning in RLHF. These details can be found on page 4 of the submitted manuscript .

Minor comments. Thank you for pointing out the error in Algorithm 1 and suggesting improvements for Figure 3a. We have fixed the algorithm in the updated paper and edited Figure 3a to make the curves more visible.