PEAR: Primitive Enabled Adaptive Relabeling for Boosting Hierarchical Reinforcement Learning

We would like to express our gratitude to the reviewer for dedicating their valuable time and effort towards reviewing our manuscript. We deeply appreciate the insightful feedback provided, and we have thoroughly responded to reviewer’s inquiries in the responses provided below.

Weakness 1: Inconsistent definition: Section 3 states that the expert data contains only states. But in Section 4.2, the low-level regularization term uses actions from the expert data.

Response to Weakness 1: We agree with the reviewer and apologize for this oversight. We have corrected this in Section 3 of the rebuttal pdf rebuttal_pdf_link, and assume access to a small number of directed expert demonstrations $D=(e^i)_{i=1}^N$,

where $e^i=(s^e_0,a^e_0, s^e_1,a^e_1 \ldots, s^e_{T-1},a^e_{T-1})$.

Weakness 2: Addressing the non-stationarity issue in HRL is a main claim of the paper. However, the proposed method does not resolve this issue. The high-level policy still faces non-stationarity, as the transitions in its replay buffer, which are determined by the low-level policy, continue to change throughout the training process.

Response to Weakness 2: We appreciate the reviewer's deep understanding and insight. We agree that the transitions in its replay buffer still continue to change, and therefore may cause non-stationarity issue. In this work, however, we focus on mitigating non-stationarity by first adaptively relabeling a few expert demonstrations to generate efficient subgoal supervision, and then jointly optimizing HRL agents by employing reinforcement learning (RL) and imitation learning (IL).

We show in Figure 3, 4 and 5 that this indeed mitigates non-stationarity in HRL. However, our primitive enabled adaptive relabeling and subsequent joint optimization based approach can be added on top of approaches like $`HAC`$, which also relabel replay buffer transitions to handle non-stationarity. This is an interesting research direction, which we would like to explore in the future.

Question 1: Many existing works [1,2,3] also use IL loss from data to regularize high-level policy learning. What differentiates the proposed regularization method from these approaches?

Response to Question 1: The two main contributions of the proposed approach are $(i)$ adaptively relabeling expert demonstrations to generate efficient subgoal supervision, and $(ii)$ jointly optimizing using RL and IL using the generated subgoals. Although several prior approaches employ IL regularization, our approach has two major distinctions:

1. Our primitive enabled adaptively relabeling approach uses the lower level $Q$ function to generate efficient subgoals that are achievable by the lower level policy. These generated subgoals are used to regularize the higher level policy using IL regularization.
2. Our approach employs a joint objective (RL and IL). Since it employs RL, $`PEAR`$ is able to explore the environment for high reward predictions. Further, the IL term regularizes the learnt higher level policy to predict achievable subgoals, thereby mitigating non-stationarity.

Question 2: Why can we set the threshold of Q to 0 (Section 4.1) in all experiments? I believe this hyperparameter should vary, depending on the reward function specific to each task.

Response to Question 2: We would like to clarify that we experimentally found the value of $Q_{thresh}=0$ to work well in the tasks, which shows that the training stability is not hyper-volatile with respect to this hyper-parameter. Further, before comparing with $Q_{thresh}$, the $Q_{\pi^{L}}$ values are normalized using the following equation: $(Q_{\pi^{L}}-\text{min value})/(\text{max value} - \text{min value} )*100$, where $\text{min value}$ and $\text{max value}$ are the minimum and maximum values of $Q_{\pi^{L}}$ respectively. Thus, although we do vary $Q_{thresh}$ depending on the reward function specific to each task, we found that out of those values, $Q_{thresh}=0$ works well for all the tasks.

We hope that the responses address the reviewer's concerns. Please let us know, and we will be happy to address additional concerns if any.

Dear Reviewer

This is a gentle reminder that we have submitted the rebuttal to address your comments. We sincerely appreciate your feedback and are happy to address any additional questions you may have during this discussion period. We thank you again for taking the time to review our work.

Best regards,

Authors

Dear Reviewer,

We thank you once again for your time and efforts in reviewing our work and providing feedback on your rebuttal. This is a gentle reminder to revise our scores if you find it appropriate, and we are happy to address any additional remaining concerns. We are grateful for your service to the community.

Regards,

Authors

As per the reviewer's kind feedback and concerns, we have thoroughly provided our responses in the rebuttal and improved the final manuscript. We hope that we have sufficiently addressed the reviewer's concerns.

After our rebuttal, reviewer di4o maintained a score of 8, and reviewers Cpmf and u7xN both increased their scores to 6 for the paper. Since the deadline is approaching, we urgently request the reviewer to respond to our rebuttal and revise our scores if deemed appropriate. We sincerely hope that our work adds sufficient value to the research community, are we are happy to address any additional or remaining concerns. Once again, we are thankful for the time and efforts of the reviewer, which has allowed us to further strengthen the manuscript.

Dear Reviewer,

We thank you once again for your time and efforts in reviewing our work and providing feedback on your rebuttal. Since the deadline is approaching, we urgently request you to revise our scores if the rebuttal has satisfactorily addressed your concerns. After the rebuttal, reviewer di4o maintained a score of 8, and reviewers Cpmf and u7xN both increased their scores to 6 for the paper. We will be happy to address any additional remaining concerns.

Regards,

Authors

We are thankful to the reviewer for dedicating their valuable time and effort towards evaluating our manuscript. We deeply appreciate the insightful feedback provided, and we have thoroughly responded to reviewer’s inquiries in the responses provided below.

Weakness 1: As authors note, the method is currently reliant on expert demonstrations. However, many benchmarks exist which include other kinds of demonstrations, including human teleop. While the method may not perform well on these demonstrations just yet (as it is listed as an aim of future work), providing results on suboptimal demonstrations would help demonstrate concretely the strong and weak points of authors' method, and potentially provide insights on why it fails in these settings.

Response to Weakness 1: We thank the reviewer for this insight, and agree that experiments on suboptimal demonstrations might provide further intuitions on why it might fail in such settings and lead to potential future research problems. To this end, we perform additional experiments with sub-optimal demonstrations and provide them in Figure 16 of the rebuttal pdf rebuttal_pdf_link. Here are the findings:

1. We find that with the increasing number of sub-optimal demonstrations (referred as bad demos in the figure), the performance degrades (which is expected since the IL regularization term becomes sub-optimal).
2. We see from the results that the performance degradation with sub-optimal demonstrations becomes more prominent as the environments become harder, which implies that the imitation learning regularization term is crucial for mitigating non-stationarity in harder tasks.
3. Since our approach employs joint optimization objectives (RL and IL objectives), the IL objective might generate sub-optimal results when sub-optimal demonstrations are used. Therefore, we would like to explore how to adaptively set the regularization weight parameter in future work, one that is able to adjust (minimize) the IL objective weight in the presence of sub-optimal demonstrations.

Weakness 2: Furthermore, it seems inaccurate to state that PEAR uses only “a handful” of demonstrations, when Fig. 13 shows that generally 50-70+ demonstrations are needed to solve the provided tasks (with the exception of Franka Kitchen, which provides fewer demos).

Response to Weakness 2: We realise now that using the word “handful” may lead to confusion, and we have removed the keyword from the rebuttal pdf rebuttal_pdf_link.

Question 1: Have you tested on suboptimal demonstrations? Are there any interesting findings/results which may point to future avenues for research or failings of the method which should be investigated/improved upon?

Response to Question 1: Please refer to our response for Weakness 1.

Question 2: Were you able to find notable reasons for why the MSE-regularized learning objective would occasionally outperform the IRL-regularized version? Is there any relationship to task difficulty, data diversity, etc?

Response to Question 2: We did further experiments on additional seeds (shown in Figure 3 in the rebuttal pdf rebuttal_pdf_link) and found that MSE is only able to outperform IRL based regularization in rope manipulation environment. We believe that although IRL regularization consistently yields good results, it is difficult to train in rope environment due to its unique task structure. Moreover, the expert demonstrations in the rope manipulation environment might be sub-optimal, making it more challenging to learn the IRL regularization objective for this task. In future work, we aim to investigate efficient methods to further boost performance on this environment.

Question 3: In “Algorithm 2: PEAR,” line 8, shouldn’t the lower-level policy’s IL regularization be done with $D_g$? Since we are providing state $s^f$ from the goal dataset $D_g$ and subgoal supervision $s_g^e$ to the goal-conditioned low-level policy, then the policy predicts action $a$ , and we regularize this to be close to the dataset action $a^f$ (either with MSE or the IRL objective)?

Response to Question 3: We thank the reviewer for this comment and agree that this is indeed a typo. The reviewer is right and the lower-level policy’s IL regularization be done with $D^L_g$ (as also mentioned in Eqn 2). We have corrected this typo in the rebuttal pdf rebuttal_pdf_link.

We hope that the responses address the reviewer's concerns. Please let us know, and we will be happy to address additional concerns if any.

Question 1: Why is the Q_{Thresh} set to 0? If the low-level reward is -1 for not achieving the goal and 0 for achieving the goal, the Q value should be negative for any correct action that puts the agent on a path to achieve the goal.

Response to Question 1: We would like to apologize for the confusion and clarify that before comparing with $Q_{thresh}$, the $Q_{\pi^{L}}$ values are normalized using the following equation: $(Q_{\pi^{L}}-\text{min value})/(\text{max value} - \text{min value} )*100$, where $\text{min value}$ and $\text{max value}$ are the minimum and maximum values of $Q_{\pi^{L}}$ respectively. Since the $Q_{\pi^{L}}$ values are normalized before comparison, $Q_{thresh}=0$ is feasible. We would like to clarify that we experimentally found the value of $Q_{thresh}=0$ to work well in the tasks, which shows that the training stability is not hyper-volatile with respect to this hyper-parameter.

Question 2: Is the high level policy only trained with a dataset containing expert trajectories? Or is it also trained on its own interaction data? If it is trained on its own interaction data, is there any relabeling done on that?

Response to Question 2: The higher level policy is trained using joint optimization, where it employs reinforcement learning (RL) using its own interaction data (enabling efficient exploration), and additional imitation learning (IL) regularization. In this work, we do not perform relabeling on interaction data, but we would like to explore this direction in future work.

Question 3: I did not understand the “margin” component of the objective. Can you provide another explanation of this component?

Response to Question 3: In our primitive enabled adaptive relabeling procedure, we employ the lower level $Q$ function $Q_{\pi^{L}}$ to select efficient subgoals, which is conditioned on the expert states $s^e$. Since $Q_{\pi^{L}}$ is the lower level $Q$ function and is simultaneously trained, it might over-estimate and provide erroneous values for states $s^e_{unseen}$ that are unseen during previous training (a known issue with Q functions in RL). Therefore, we posit that the over-estimation issue can be avoided by adding a margin objective that penalizes the $Q$ values on out-of-distribution states. We state this as margin classification objective in the paper and empirically found this objective to stabilize training.

Question 4: Why does the paper characterize the number of demonstrations as a “handful” of demonstrations, when it uses 100 for most tasks?

Response to Question 4: We realize now that using the word “handful” may lead to confusion, and we have removed the keyword from the rebuttal pdf rebuttal_pdf_link.

Question 5: Have you experimented on any domains with image observations?

Response to Question 5: We did experiment with image observations in the maze navigation, pick and place, and rope environments, and found that $`PEAR`$ consistently shows impressive performance on the tasks and outperforms the baselines. However, due to resource and time constraints, we decided to not add those results to this paper draft. We will try to finish the experiments in time, and add them to the final paper draft submission.

We hope that the responses address the reviewer's concerns. Please let us know, and we will be happy to address additional concerns if any.

We are thankful to the reviewer for dedicating their valuable time and effort towards evaluating our manuscript. We deeply appreciate the insightful feedback provided, and we have thoroughly responded to reviewer’s inquiries in the responses provided below.

Weakness 1: My main concern with this approach is that the contribution seems to be marginal as I do not see a compelling reason why PEAR would consistently perform better than (a) HAC (Levy 2019) with replay buffer augmented with demonstration data

Response to Weakness 1: We thank the reviewer for raising this concern. Before stating our response, we would respectfully like to clarify the following:

1. $`PEAR`$ uses our novel primitive enabled adaptive relabeling approach to leverage expert demonstrations and generate efficient subgoal supervision for the higher level policy.
2. $`PEAR`$ uses a joint optimization objective that employs reinforcement learning (RL) and additional imitation learning (IL) regularization. RL allows the hierarchical policies to continuously explore the environment for high reward predictions, and IL regularizes the higher level objective to predict achievable subgoals for the lower primitive (and regularizes the lower level primitive to predict efficient primitive actions).

$`HAC`$ Limitation: Although $`HAC`$ tries to deal with non-stationarity using relabeling and subgoal testing as mentioned by the reviewer, it fails to mitigate non-stationarity in complex long horizon tasks (as seen in Figure 3).

$`PEAR`$ can explore using RL: Further, although $`PEAR`$ employs imitation learning regularization, it also uses RL in our joint objective formulation which allows it to explore the environment for high reward predictions, even in presence of sub-optimal demonstrations.

Comparison with HER-BC: We would also like to point out that although $`HER-BC`$ baseline works well in easier tasks, it is unable to outperform $`PEAR`$ and perform well in harder long horizon tasks. Further, we would like to point out that $`HAC`$ additionally suffers from non-stationarity issue in HRL, since it is hierarchical (unlike the non-hierarchical $`HER-BC`$).

Additional experiments: To support our claims, we perform additional experiments: $(i)$ HAC with demos ($`HAC-demos`$) and $(ii)$ Hierarchical behavior cloning ($`HBC`$) in Figure 15 in the rebuttal pdf rebuttal_pdf_link to demonstrate the efficacy of $`PEAR`$ over these baselines.

We would like to point out that it is not straight-forward to train the higher level in hierarchical approaches with behavior cloning, since we typically do not have access to higher level subgoal supervision ($`PEAR`$ uses primitive enabled adaptive relabeling to acquire efficient subgoal supervision). Therefore, for the following experiments, we employ subgoals extracted using fixed window based approach (as in Relay Policy Learning approach $`RPL`$):

1. $`HAC-demos`$ uses hierarchical actor critic with the RL objective and is jointly optimized using additional behavior cloning objective, where the lower level uses primitive expert demonstrations and the upper level uses subgoal demonstrations extracted using fixed window based approach (as in $`RPL`$).
2. $`HBC`$ (Hierarchical behavior cloning) uses the same demonstrations as $`HAC-demos`$ at both levels, but is trained using only behavior cloning objective (thus, does not employ RL).

As seen in Figure 15, although $`HAC-demos`$ shows good performance in the easier maze navigation environment, both $`HAC-demos`$ and $`HBC`$ fail to solve the tasks in harder environments, and $`PEAR-IRL`$ significantly outperforms both the baselines, demonstrating the efficacy of our primitive enabled adaptive relabeling and joint optimization based approach. Note that both $`HAC-demos`$ and $`HBC`$ suffer from non-stationarity issue in HRL, which accounts for their poor performance. This shows that our approach is able to efficiently mitigate non-stationarity in HRL. Finally, primitive enabled adaptive relabeling and subsequent joint optimization can be added on top of $`HAC`$, which is an interesting direction we would like to explore in future work.

Dear Reviewer

This is a gentle reminder that we have submitted the rebuttal to address your comments. We sincerely appreciate your feedback and are happy to address any additional questions you may have during this discussion period. We thank you again for taking the time to review our work.

Best regards,

Authors

Dear Reviewer,

We thank you once again for your time and efforts in reviewing our work and providing feedback on your rebuttal. This is a gentle reminder to revise our scores if you find it appropriate, and we are happy to address any additional remaining concerns. We are grateful for your service to the community.

Regards,

Authors

Thank you authors for the response to my questions. I increased my score by a point as I believe the approach is sufficiently new. However, I still do not understand why HAC is characterized as nonstationary. The reward for subgoal actions in HAC never depend on the value functions of states achieved by the current low level policy, they only depend on the value functions of states achieved by fully trained low level policies as a result of the hindsight transitions. The problem with HAC is not nonstationarity but rather exploration. Your HAC+demos experiment does not seem to do what I requested, which was a pure HAC with the replay buffers supplemented with the fixed window transitions from the demonstrations rather than incorporating the imitation learning objective objective into HAC. I also do not understand why a pure hierarchical behavior cloning approach, in which high level policy is trained to output states reached from demonstrations, would be "nonstationary". This objective would have no dependence on the states reached by the current low level policies.

Question 1: How do the authors justify the reliance on expert demonstrations in the first phase of PEAR compared to HRL methods that function without such requirements?

Response to Question 1: Please refer to our response to Weakness 1.

Question 2: Can the authors provide additional comparisons with recent HRL approaches to strengthen the positioning of PEAR within the current landscape?

Response to Question 2: We have performed additional experiments and provided in the rebuttal pdf rebuttal_pdf_link. Please refer to our response to Weakness 2.

[1] Wang, Vivienne Huiling, et al. "State-conditioned adversarial subgoal generation." Proceedings of the AAAI conference on artificial intelligence. Vol. 37. No. 8. 2023.

We hope that the responses address the reviewer's concerns. Please let us know, and we will be happy to address additional concerns if any.

We would like to express our gratitude to the reviewer for dedicating their valuable time and effort towards evaluating our manuscript, which has allowed us to strengthen the manuscript. We deeply appreciate the insightful feedback provided, and we have thoroughly responded to reviewer's inquiries in the responses provided below.

Weakness 1: Expert Demonstrations Requirement: The first phase of PEAR relies on expert demonstrations to generate subgoals, which raises concerns about the fairness of comparison with other HRL methods that do not require such demonstrations. This could affect the generalizability of the findings.

Response to Weakness 1: We agree with the reviewer that the proposed approach relies on expert demonstrations to generate subgoals, which can be challenging in environments where generating such demonstrations is computationally expensive (we also mention this in the discussion section of the paper).

However, we would like to respectfully point out that:

• collecting a few expert demonstrations is often feasible in such robotics control tasks, and indeed several prior works employ such expert demonstrations (covered in detail in the Related Work section).
• the main contribution of this work is to use primitive enabled adaptive relabeling to generate efficient subgoals, and subsequently leveraging them in our joint optimization based approach.

Although we compare our approach with several baselines that employ expert demonstrations (Relay Policy Learning $`RPL`$, Discriminator Actor Critic $`DAC`$, Behavior Cloning $`BC`$), the purpose of comparisons with baselines that do not use expert demonstrations are as follows:

• $`HAC`$: to show that $`PEAR`$ is able to better mitigate non-stationarity.
• $`RAPS`$: to show that $`PEAR`$ outperforms approaches that use behavior priors.
• $`HIER`$ and $`HIER-NEG`$: to show the importance of extracting subgoals using primitive enabled adaptive relabeling and subsequent joint optimization.
• $`FLAT`$: to show the importance of our hierarchical formulation (temporal abstraction and improved exploration).

Thus, we would like to respectfully point out that such comparisons offer additional insights into into the factors contributing to the superior performance of $`PEAR`$ compared to the baselines.

Additional Experiments: Further, in response to reviewer Cpmf, we perform additional experiments comparing our approach to two more baselines that employ expert demonstrations (Figure 15 in the rebuttal pdf rebuttal_pdf_link).

1. $`HAC-demos`$ uses hierarchical actor critic with the RL objective and is jointly optimized using additional behavior cloning objective, where the lower level uses primitive expert demonstrations and the upper level uses subgoal demonstrations extracted using fixed window based approach (as in $`RPL`$).
2. $`HBC`$ (Hierarchical behavior cloning) uses the same demonstrations as $`HAC-demos`$ at both levels, but is trained using only behavior cloning objective (thus, does not employ RL).

As seen in Figure 15, $`PEAR-IRL`$ significantly outperforms both the baselines, demonstrating the efficacy of our primitive enabled adaptive relabeling and joint optimization based approach.

Weakness 2: Lack of Recent Comparisons: The paper does not include comparisons with several hierarchical reinforcement learning methods published in the last three years. This omission limits the contextual relevance of the results and could misrepresent the state of the art, such as [1], [2].

Response to Weakness 2: Based on the reviewer's concern, we have performed and added additional experiments in the paper to compare our approach with $`SAGA`$ [1]. $`SAGA`$ employs state conditioned discriminator network training to address non-stationarity, by ensuring that the high-level policy generates subgoals that align with the current state of the low-level policy. We provide the comparisons in Figure 3 of the rebuttal pdf rebuttal_pdf_link.

We find that although $`SAGA`$ performs well in maze navigation environment, it fails to solve harder tasks, where $`PEAR`$ is able to significantly outperform the baseline. This demonstrates that $`SAGA`$ suffers from non-stationarity issue in harder long horizon tasks, and $`PEAR`$ is able to better mitigate non-stationarity issue and effectively solve long horizon tasks.

Weakness 3: Citation Issues: Many citations are sourced from CoRR ... and accurate referencing.

Response to Weakness 3: We thank the reviewer for pointing this out. We have corrected all the citations with their formal venue citations in the rebuttal pdf rebuttal_pdf_link.

Dear Reviewer

This is a gentle reminder that we have submitted the rebuttal to address your comments. We sincerely appreciate your feedback and are happy to address any additional questions you may have during this discussion period. We thank you again for taking the time to review our work.

Best regards,

Authors