Deep Reinforcement Learning from Hierarchical Preference Design

Dear Reviewer Q7xT,

Thank you for the insightful review! We are glad you appreciate our work, and present our rebuttal to your review below.

Experimental Designs Or Analyses

The experiments could benefit from additional ablation studies to further isolate the impact of specific components of the HERON framework, such as the hierarchical decision tree and the preference-based reward model, on the overall performance.

• Note that we conduct an extensive ablation study in section 4.5. We find that the margin parameter is quite important.
• Furthermore, we also investigate the flexibility and robustness of HERON, finding HERON is both flexible and provides increased robustness.

Other Strengths And Weaknesses

A potential weakness is the limited exploration of scenarios with unclear hierarchical structures, which could restrict the framework's applicability in more complex real-world environments

• First, we remark that HERON is designed for settings where there is a clear hierarchy (see our discussion on suitable scenarios) and performance in other settings is a “bonus.” In settings with unclear hierarchies like robotic control (see section 4.4), we find HERON can still outperform reward engineering. Our intuition is that in these settings a roughly correct hierarchy provides sufficient information for good performance. If the signals have equal performance, we propose in line 402 to randomly flip the ranking of equal feedback signals. However, we leave this study for future work.

Questions

Could the authors provide more details on how HERON handles scenarios where the hierarchy of feedback signals is not clear or where signals have equal importance?

• See our above response to Other Strengths And Weaknesses.

How does the computational cost of HERON compare to other state-of-the-art methods, especially in terms of training time and resource requirements?

• We show the training time of HERON versus baselines in Figure 6a. HERON is around 25% slower than reward engineering, but greatly reduces tuning cost (Figure 6c). HERON is faster than other baselines such as the ensemble baselines as well.

Can the authors discuss the potential impact of HERON on multi-objective RL problems, where objectives may conflict?

• Thank you for the suggestion. As we mention in Line 420, HERON is not designed for multi-objective RL (MORL). In fact, HERON and MORL are solving different problems. MORL tries to train a policy on the pareto frontier among several reward factors. In contrast, HERON tries to find a way to combine feedback signals into a reward in a user-friendly way, such that the user can easily guide the agent’s behavior. This is useful in many real-world tasks—like code generation, aligning language models, or traffic light control—where some goals are naturally more important than others. To make this clearer, we’ll move the MORL discussion to the related work section and explain how it differs from our approach.

Thank you again for the insightful review, and please let us know if you have any other questions. We look forward to further discussion!

Dear Reviewer zX5S,

Thank you for your detailed review and insightful suggestions! We are glad you think our work is novel. We provide our response to your review below. We shorten some questions to save space.

Claims and Evidence

The authors hypothesize that HERON's reward function "may be more conducive to learning". However no empirical support to this claim is provided.

• We would like to clarify that this is an intuition rather than a definitive claim. Our hypothesis—that HERON's reward function may be more conducive to learning—was meant as a possible explanation for HERON's empirical performance advantage, not a conclusive statement. This hypothesis stems from the fact that many raw feedback signals are discrete or sparse, and reward models can serve as a way to smooth out this sparsity.
• We do find some supporting evidence in Figure 14, where HERON exhibits significantly smoother and more stable learning curves compared to reward engineering. This suggests that HERON's reward function facilitates more stable training dynamics, which aligns with our intuition.

Experimental Designs Or Analyses

However, I would like to single out HERON’s post-training reward scaling procedure in Section 4.2. This method appears to be an additional form of reward shaping, but it is not really motivated in the paper.

• Our post-training scaling is motivated by Figure 13, in which we found that reward modeling alone does not perfectly separate correct code from incorrect code at a global level. In order to further separate the two cases, we introduce a scaling parameter. Note that we only have a single parameter to tune, compared to the 4 used in CodeRL.
• The pass@50 results without reward scaling can be found below. HERON still performs decently (outperforming all baselines, which are carefully tuned in prior works), but we find that post-scaling can further improve performance. We will include the results in the appendix.

Table: Pass@50 Scores

Relation To Broader Scientific Literature

There is also a significant connection to multi-objective RL (MORL) literature...

• Thank you for pointing out reference [1], we will include it in our related work. As for MORL,we remark that MORL tries to find the pareto frontier among several reward factors. In contrast, HERON tries to find a way to combine feedback signals into a reward in a user-friendly way, such that the user can easily guide the agent’s behavior. We believe these are two orthogonal directions. We will move discussion of MORL to the related work section.

Essential References Not Discussed

A major missing reference and baseline is PEBBLE.

• PEBBLE is not directly comparable to HERON, as PEBBLE relies on human preference data for learning. HERON on the other hand is meant for settings where we try to design a reward function from some freely available feedback signals. Therefore PEBBLE cannot be used as a baseline in our experiments.
• Moreover, PEBBLE’s contributions (unsupervised pre-training and off-policy learning) are orthogonal to HERON’s contributions. Therefore we do not include PEBBLE as a reference.

Weaknesses

However, a significant limitation of HERON is its assumption of an underlying ranking of feedback signals.

• While HERON assumes an underlying ranking of feedback signals, this assumption is well-motivated by many real-world scenarios where feedback signals naturally have a hierarchy—for example, in traffic light control, code generation, and LLM alignments. In such contexts, ranking the relative importance of feedback signals is valid. We believe good results in these environments already represent a significant contribution.
• Moreover, as demonstrated in Section 4.4, HERON performs well even in the absence of a strict feedback hierarchy. This empirical resilience highlights its practical utility and broad applicability, even when the ranking structure is noisy or only partially specified.

Questions

Q1: See above response to Experimental Designs Or Analyses.

Q2: We find HERON can still perform well in these settings, outperforming reward engineering baselines (Table 6). HERON is relatively robust to shuffled rankings. In Figure 6, we show how HERON performs with inexact domain knowledge, i.e. only knowing which factors fall in the top 3 and which ones fall in the bottom 3. By one tuning iteration (i.e. the best of two shuffled rankings out of a possible 36), HERON can already outperform the best-case performance of reward engineering.

Q3: See above response to Claims and Evidence.

Thank you again for your thoughtful review, and please let us know if you have any further questions.

Dear Reviewer 7gbW,

Thank you for your detailed review and suggestions to improve our work. First and foremost we would like to address your review of our claims, in which you say we claim “HERON universally eases reward design”. Nowhere in the paper do we make this claim, and in fact we consistently state that our scope is limited to problems with hierarchical structure (see line 51 right, line 62 left, and line 436 left).

We present our rebuttal of the remainder of your review below. We shorten some questions to save space.

Experimental Design or Analyses

The choice of baselines is relatively outdated, lacking comparisons with [1] and [2]

• We believe there is a misunderstanding here. HERON is not comparable to preference learning methods. HERON assumes online access to an environment which gives the agent several feedback signals. We seek to design a reward function from these feedback signals, which is a classical setting in RL. On the other hand, [1] requires access to an offline preference dataset, while [2] requires access to online preference annotation. In contrast, HERON asks a human annotator to rank the importance of feedback signals (usually there are 3-7 such signals) one time. This is completely separate from the recommended papers. We include a detailed discussion on suitable scenarios in line 436.

It remains unclear if the experiments cover a sufficiently diverse range of scenarios...

• We conduct extensive experiments in 7 diverse environments, which we believe is sufficient to show the benefit of HERON (multi-agent traffic light control, code generation, LLM alignment, and four robotic control experiments). In robotic control experiments the feedback signals do not all have a clear hierarchy, yet we are able to beat or match the baselines in all environments. These results demonstrate that our method performs well in a diverse set of hierarchical or roughly hierarchical environments.

Relation To Broader Scientific Literature

A more comprehensive discussion of how HERON fits into and advances the current state-of-the-art would be beneficial.

• Thank you for the suggestion. We will add more works on SOTA approaches in our related works section, including [1] and [2]. Please note we already have a discussion on suitable settings for HERON (line 436).

Weaknesses

Weakness 1

• First, we want to point out that HERON is designed for settings where there is a clear hierarchy (see our discussion on suitable scenarios) and performance in other settings is a “bonus.” However, in environments without clear hierarchy, we find HERON can still perform decently, outperforming reward engineering baselines. See section 4.4 of our paper for more details.

Weakness 2

• Theoretical analysis is challenging, as it is difficult to precisely characterize the relationship between the feedback signals and the ground truth reward. For example, in traffic light control, defining an optimal combination of all relevant factors is inherently ambiguous.
• That said, we hypothesize that if theoretical guarantees were to be established, the convergence of the policy would likely depend on two key factors: (1) the statistical error in the learned reward, and (2) the distribution shift between the state visitation distributions of the sampling policy and the optimal policy.

Questions

Q1: See our response to weakness 1.

Q2: One approach is to learn a decision tree based on human preference data. This may work when the feedback signals have hierarchical structure and we only have a limited amount of human preference data. However, this would be a separate setting and we leave it for future work.

Q3: See response to weakness 2.

Q4: The main baseline we compare against is reward engineering, which has been the most popular approach for reward design over the past 20 years. We also compare against two baselines that have been used in MORL literature, the ensemble baselines. In relevant settings such as code generation, we compare to carefully designed and publicized reward functions like that of CodeRL. Finally, when available, we also compare training directly on the ground-truth reward. Again we note that those SOTA methods of [1] and [2] are not relevant in our setting, as they assume access to a large set of human preference data (either online or offline) which HERON is not designed to use.

Q5: HERON is relatively robust to suboptimal rankings. In Figure 6, we show how HERON performs with inexact domain knowledge, i.e. only knowing which factors fall in the top 3 and which ones fall in the bottom 3. By one tuning iteration, HERON can already outperform the best-case performance of reward engineering. This indicates HERON can perform well even with slightly noisy rankings.

Thank you for the detailed review, and please let us know if you have any further questions or need any clarification.

Dear Reviewer uDMx,

Thank you for your thoughtful and detailed review of our paper. We are glad you appreciate the novelty of our approach as well as our experimental analysis. We provide our rebuttal to your critiques below. We enumerate your questions and comments to save characters.

Experimental Design and Analysis:

1. Traffic Light Control:

1.1: How many samples were used?

• We present results over five random seeds. Indeed we find that the baselines have high variance, but HERON exhibits very low variance. To further validate the efficacy of our method, we have run a t-test for the last 1000 evaluation time steps in the traffic light control environment. Our method is significantly better than the reward engineering baseline, with p=0.003. This gain justifies the use of our algorithm.

1.2: Ground truth reward.

• The ground truth reward can be found on line 693, and it has been developed over several papers. It was finalized in [Zhang 2019]. Other baselines sometimes outperform it, as we extensively tune all baselines. This tuning can be seen as a type of reward shaping, which can improve performance.

1.3: Proportion of decisions made at different levels.

• If too many decisions are made at a single level, that effectively means that information from other levels of the decision tree are being completely ignored, which is not ideal. Decisions being made at all levels indicate information from all feedback signals is being incorporated into the reward, which is desirable and indicates the efficacy of our reward design. We compute the proportions in traffic light control by recording which level of the tree each decision was made at throughout training.

1.4: Figure 4.

• We only vary the second feedback signal to keep the reward realistic, as the first feedback (traffic throughput) is almost always viewed as most important in traffic light control. We show the policy can still achieve good performance with other hierarchies in Figure 10.

1.5: Figure 5.

• The original speed is 35, then we change it to 25, 30, 40, 45 (See Figure 5). We conduct 5 independent trials for each experiment and report mean and standard deviation over the runs.

2. For the code generation experiments, what is the number of trials used, and what is the variance in the results?

• We conduct training over 1 seed due to the large expense of these training runs, which is in line with prior works. However, when evaluating a policy on APPS, we evaluate each policy with 1 million generated programs (there are 5000 test problems, we generate 200 for each problem) and 100000 on MBPP. Treating each set of programs as an independent Bernoulli variable, we can conduct a t-test. When considering the largest value of K in pass@k in tables 1, 2, 3, 4 we find that the a t-test comparing HERON and the best baseline has p values < 0.05, indicating that HERON indeed outperforms the baselines in a statistically significant manner.

3. In the LLM alignment experiments, why was HERON-DPO compared with REINFORCE?

• We compare with REINFORCE as it is one of the most popular and high performing approaches for LLM alignment these days [1]. It is possible the baseline could do better with PPO, but this would require extensive tuning and PPO is shown to underperform REINFORCE for LLM alignment [1].

[1] Ahmadian, Arash, et al. "Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms."

4. Regarding the robotics experiment, could the authors clarify the details of Table 6?

• In table 6 we show the average ground truth reward obtained by each algorithm over the final 1000 iterations of training (training is 2 million steps total). We conduct experiments over 5 random seeds, and use the PPO algorithm to optimize the policies. The reward hierarchy is generally is-alive > moving forward > control cost > contact cost. We will make this more clear in the paper.

Questions

Q1: In code generation, we use a pre-trained language model as the initialization for the reward model. This is advantageous as it means reward training is faster and more accurate. Similarly for LLM alignment we could use a pre-trained LM as a reward, but opt for DPO in our experiments due to its simplicity.

Q2: DPO is most useful when the horizon length of the task is one, as in that case we can directly train the policy on preference comparisons, without having to train a reward model. This is the case in LLM alignment, which is where DPO is mainly used.

Thank you again for the detailed review. We are eager to know if your questions about our empirical evaluation have been satisfied, or if there are any more details we can give you.