PEAR: Primitive enabled Adaptive Relabeling for boosting Hierarchical Reinforcement Learning

Reviewer 3:

We thank the reviewer for their detailed, constructive feedback. We address the reviewer’s concerns as follows:

1. "The core idea of generating the subgoal by thresholding an environment-specific value has natural constraints."

Response:

The proposed approach does require setting $Q_{thresh}$ hyper-parameter but we empirically found that the performance is not unstable with varying hyperparameter value. Intuitively, this is a reasonable trade-off that renders the higher level subgoal generation a self-supervised process, thereby eliminating the need for expert guided annotations for segmenting subgoals. Additionally, please note that we empirically found that after we normalize $Q_{\pi^L}$ values of a trajectory, the $Q_{thresh}$ hyper-parameter value of $0$ works consistently well across all environments, which exhibits that setting $Q_{thresh}$ value does not cause significant overhead. We have added the hyper-parameter ablation analysis in Appendix A.4 Figure 10 for your consideration. Additionally, in real robotic experiments, we used the default $Q_{thresh}$ value of $0$ and PEAR was able to outperform the baselines in all experiments.

2. "The method of determining reachability via the low-level Q function concurs the approach in [Inter-level cooperation in hierarchical reinforcement learning.]"

Response:

The paper titled "Inter-level cooperation in hierarchical reinforcement learning" proposes an approach based on inter-level cooperation between multi-level agents by jointly optimizing the higher level and lower level functions using RL. Similar to our approach, the paper considers the lower level Q function to motivate the higher level agent to produce reachable subgoals for the lower level policy. However in their approach, the subgoals are encountered as a result of environment exploration of the higher level policy, which may be quite random. This leads to two major issues: (i) either the generated subgoals are very hard, which impedes effective lower level policy learning, or (ii) the subgoals are too easy, due to which the higher level has to do all the heavy lifting to solve the task. The issue is exacerbated in the beginning of training, due to which the policies have to extensively explore the environment before being able to come across good subgoal predictions. Due to these issues, the approach might be unable to perform well on multi-stage long horizon tasks. In our approach, we use a handful of expert demonstrations and employ our novel "primitive enabled adaptive relabeling" procedure for picking reachable subgoals for the lower level policy, and effectively utilize them for training higher level policy using additional imitation learning regularization. Notably, expert demonstrations can not be directly leveraged in raw trajectory form to train the higher level, and therefore require efficient data relabeling to segment them into meaningful subgoals. Moreover, since we periodically perform data relabeling, our approach always generates reachable subgoals for lower level policy, thereby mitigating non-stationarity issue prevalent in vanilla hierarchical learning approaches, and thus devising a practical HRL algorithm. In order to prevent over-estimation of the values on out-of-distribution states, we also employ an additional margin classification objective as explained in detail in Section 4.1. We also provide theoretical guarantees that justify the benefits of periodic re-population using adaptive relabeling, which are largely missing in recent contemporary work. We also demonstrate the efficacy of our approach in complex long horizon tasks like Franka kitchen environment, and also include environments with challenging dynamics and policy, especially the soft rope manipulation, which goes beyond some peer work that focuses on navigation-like tasks on 2D plane, where the goal space is limited. Finally, we demonstrate that PEAR is able to demonstrate good performance in challenging real world environments. We have improved the related work section based on the reviwer's concern and the discussion above, and added a brief discussion summarizing above points in the the paper.

I recognize and appreciate the authors' efforts to address the initial concerns raised. However, my reservations regarding the distinction between this paper and the prior work "State-conditioned adversarial subgoal generation" as well as the environment-specific hyper-parameter remain.

The response provided does not fully clarify how this work significantly diverges from the aforementioned prior work, particularly in terms of optimizing hierarchical policies. While the method of generating relabeled subgoals might differ, the core concept of using adversarial learning in the HRL context to generate achievable subgoals seems to align closely with the previous work. The novelty of this paper appears to hinge on the specific method of generating relabeled subgoals, but this alone may not constitute a substantial advancement.

Furthermore, my concern regarding the method of generating subgoals by thresholding an environment-specific value has not been adequately addressed. Without more comprehensive evaluations, it's challenging to assess the potential limitations or strengths of this approach. This aspect is crucial for understanding the practical applicability and robustness of the proposed method in diverse scenarios.

1. The response provided does not fully clarify how this work significantly diverges from the aforementioned prior work, particularly in terms of optimizing hierarchical policies

Response:

We understand the concern and provide clarifications to differentiate our method from the aforementioned work. Although our approach and the mentioned work both use adversarial training to generate achievable subgoals, our approach differs from the previous work in the following manner:

1. As explained before, the approach in the mentioned paper may yield degenerate solutions. However, we side-step this issue by using a handful of expert demonstrations for picking reachable subgoals for the lower level policy, and effectively utilizing them for training higher level policy using additional imitation learning regularization.

2. The relabeling approach in the mentioned paper relies on the approach in the paper HIRO, which assumes additional assumptions on the environment subgoal generation. In contrast, our approach assumes no such limiting assumptions.

3. We propose a hierarchical curriculum learning based approach, which always generates a curriculum of achievable subgoals for the lower primitive.

4. Apart from adversarial training, we derive a general imitation learning based regularization objective, where we can plug in various distance metrics to yield various imitation learning algorithms. We perform theoretical and empirical analysis for inverse reinforcement learning and behavior cloning regularization objectives.

5. Our approach provides a plug-and-play framework where we can plug in off-policy RL algorithm and IL algorithm of choice, which ultimately yields various joint optimization objectives for hierarchical reinforcement learning.

6. We also provide theoretical guarantees that justify the benefits of periodic re-population using adaptive relabeling, which is missing from the mentioned work.

7. We also demonstrate the efficacy of our approach in complex long horizon tasks, and also include environments with challenging dynamics and policy, which goes beyond some peer work where the goal space is constrained.

8. Additionally, we would like to point out that this work is actually concurrent to our work, as our work was in submission when this work was published.

We hope that the above mentioned points differentiate our work from the previous work, and elucidates the contributions of our approach.

2. Furthermore, my concern regarding the method of generating subgoals by thresholding an environment-specific value has not been adequately addressed

Response:

In our approach, $Q_{thresh}$ is used for automatically relabeling expert demonstrations to generate expert subgoal dataset for training higher level policies. Some prior works use fixed window based relabeling to generate expert subgoal demonstration dataset for the higher level policy [https://arxiv.org/abs/1910.11956]. In such approaches, the window size parameter $k$ is an environment specific value which has to be set. Moreover, this approach is basically a brute force approach where we consider multiple window sizes, and select the best window size from among them. However, intuitively, such approaches suffer from two reasons: (i) it requires environment specific window size hyper-parameter to be set, and (ii) fixed parsing approaches may generate sub-optimal expert subgoal dataset. Although our method uses environment specific $Q_{thresh}$ value, it side-steps the fixed parsing issue by using primitive enabled adaptive relabeling, which leads to better policies learnt from adaptively relabeled expert demonstrations. Moreover, our higher level subgoal generation process (primitive enabled adaptive relabeling) is a self-supervised process, which does not require an expert for segmenting subgoals.

Additionally, please note that the $Q_{thresh}$ hyper-parameter value of $0$ works consistently well across all environments, which demonstrates that setting $Q_{thresh}$ value does not cause significant overhead. Thus, we effectively just need to normalize $Q_{\pi^L}$ values of a trajectory, and then select $Q_{thresh}$ hyper-parameter value of $0$, which works consistently well across all environments.

We hope that the response addresses the reviewer’s concerns and clearly explains its contributions.

Reviewer 4:

We thank the reviewer for their detailed, constructive feedback. We address the reviewer’s concerns as follows:

1. "The learning curve figures are too small and very difficult to see. It would be much appreciated if the authors could fix this."

Response:

We apologize for the figure size and clarity issues. We have now fixed the issues in the paper. We hope this improves the overall clarity.

2. "In Algorithm 1 for Lines 5-13 it could improve clarity if the authors included comments on what each line is doing. I found this part a bit difficult to understand."

Response:

We thank the reviewer for pointing this out. We have now added the comments in Algorithm 1 wherever applicable. We hope that this improves the clarity of the algorithm psuedo-code.

3. "It seems it is necessary to manually set $Q_{thresh}$ for each environment."

Response:

The proposed approach does require setting $Q_{thresh}$ hyper-parameter but we empirically found that the performance is not unstable with varying hyperparameter value. Intuitively, this is a reasonable trade-off that renders the higher level subgoal generation a self-supervised process, thereby eliminating the need for expert guided annotations for segmenting subgoals. Additionally, please note that we empirically found that after we normalize $Q_{\pi^L}$ values of a trajectory, the $Q_{thresh}$ hyper-parameter value of $0$ works consistently well across all environments, which exhibits that setting $Q_{thresh}$ value does not cause significant overhead. We have added the hyper-parameter ablation analysis in Appendix A.4 Figure 10 for your consideration. Additionally, in real robotic experiments, we used the default $Q_{thresh}$ value of $0$ and PEAR was able to outperform the baselines in all experiments.

4. "I'm curious why PEARL-IRL outperforms PEAR_BC on Maze, Pick Place, Bin, and Hollow, but PEAR-BC outperforms PEARL-IRL on Kitchen. Do the authors have any ideas why this is?"

Response:

Our motivation for using inverse reinforcement learning (IRL) is that although behavior cloning (BC) works efficiently in many scenarios, it sometimes fails to perform well for complex tasks that require long term planning. However, the advantages of IRL are sometimes counterbalanced by the fact that they are really difficult to train. We believe that since it is a really difficult environment to train on, PEAR-BC slightly outperforms PEAR-IRL in Franka kitchen environment:

Success Rates:

Thus, although in general, PEAR-IRL works better than PEAR-BC in long horizon complex tasks, PEAR-BC might work better in some cases where it is hard to train an IRL objective. However, the performance does not differ significantly and we found that in real world tasks, PEAR-IRL consistently outperforms PEAR-BC in the experiments.

5. "Have the authors experimented with different numbers of expert demonstrations? I'm curious how the method would perform with more/less demonstrations."

Response:

Yes, we have performed ablations to analyze the effect of varying the number of expert demonstrations for each task (Please refer to Appendix A.4 Figure 9). Although it is subject to availability, we increase the number of expert data until there is no significant performance boost.

6. "How is $Q_{thresh}$ chosen for each environment?"

Response:

Firstly, we select $100$ randomly generated environments for training, testing and validation. Using this, for calculating $Q_{thresh}$, we compute success rate plots for various $Q_{thresh}$ values to select the best $Q_{thresh}$ parameter (the ablation analysis is shown in Appendix A.4 Figure 10).

We hope that the response addresses the reviewer’s concern. Please let us know, and we will be happy to address additional concerns if any.

Reviewer 2:

We thank the reviewer for their detailed, constructive feedback. We address the reviewer's concerns as follows:

1. "[Deal with the non-stationarity] I don't agree that this work ameliorates non-stationarity. Since PEAR can access expert samples when training the high-level policy by BC or IRL, the high-level dynamics remain consistent with the dynamics that generate the expert sample trajectories without any change. This is inconsistent with the claim of non-stationarity made by the authors, which raises concerns about the contribution in this regard."

Response:

We understand the reviewer's confusion and explain the claim regarding non-stationarity as follows: as explained in detail in Section 4, PEAR jointly optimizes HRL agents by employing reinforcement learning (RL) with imitation learning (IL) regularization. Since the higher level policy does not have direct access to expert subgoal dataset, PEAR using primitive enabled adaptive relabeling to generate efficient subgoal supervision for the higher level policy. While PEAR leverages imitation learning regularization using expert demonstrations to bootstrap learning, it also employs reinforcement learning to explore the environment and come up with efficient actions to execute in complex task environments. Due to this exploration and joint optimization, PEAR significantly outperforms the imitation learning baselines, as stated in the Experiment Section 5 and Table 1. Moreover, PEAR mitigates non-stationarity by periodically re-populating subgoal transition dataset $D_g$ after every $p$ timesteps according to the goal achieving capability of the current lower primitive. This generates a natural curriculum of achievable subgoals for the lower primitive, thereby ameliorating non-stationarity. We hope that this elucidates the motivation behind how and why PEAR deals with non-stationarity. Please let us know and we would be happy to clarify any other doubts and concerns regarding the motivation.

2. "[Performance limitation from expert] Since the high-level policy is optimized only through BC or IRL using expert samples and does not participate in the environment exploration process, the performance of expert samples significantly limits the upper-performance limit of PEAR. Therefore, the authors need to analyze the impact of expert performance on the upper-performance limit of PEAR and conduct experiments with expert samples of varying performance."

Response:

As clarified in the previous answer, PEAR employs reinforcement learning along with imitation learning regularization to explore the environment and devise efficient actions to execute in the complex task environments. Due to this, PEAR significantly outperforms various hierarchical and non-hierarchical baselines that may or may not employ imitation learning. Additionally, we perform ablations to analyze the effect of varying the number of expert demonstrations for each task (Please refer to Appendix A.4 Figure 9). Although it is subject to availability, we increase the number of expert data until there is no significant performance boost. We also conduct experiments where we vary the quality of demonstrations by adding in some g number dummy/garbage demonstrations with the N expert demonstrations, but since the algorithm is unable to learn from such garbage demonstrations, empirically the method performs as if it is being trained using (N-g) demonstrations, which is similar to the ablations analyzing the effect of varying the number of expert demonstrations for each task, and hence we omit the ablation results from the paper.

3. "[How to get $Q_{thresh}$] When performing adaptive relabeling, the authors require additional environment-specific Q values, which are not available in standard experimental settings and may limit the practicality of PEAR."

Response:

The proposed approach does require setting $Q_{thresh}$ hyper-parameter but we empirically found that the performance is not unstable with varying hyperparameter value. Intuitively, this is a reasonable trade-off that renders the higher level subgoal generation a self-supervised process, thereby eliminating the need for expert guided annotations for segmenting subgoals. Additionally, please note that we empirically found that after we normalize $Q_{\pi^L}$ values of a trajectory, the $Q_{thresh}$ hyper-parameter value of $0$ works consistently well across all environments, which exhibits that setting $Q_{thresh}$ value does not cause significant overhead. We have added the hyper-parameter ablation analysis in Appendix A.4 Figure 10 for your consideration. Additionally, in real robotic experiments, we used the default $Q_{thresh}$ value of $0$ and PEAR was able to outperform the baselines in all experiments.

The clarity of the figures in the experiments is still poor in your revision. It is hard for the readers to read the lines and legends.

I am a bit confused by the response. I guess the authors ignore the "not" term in my question.

I'd like to know the effectiveness of your method in two different cases. First, when using a RANDOM policy or a MEDIUM expert policy dataset from the D4RL datasets, how does your performance vary? Second, compared to the expert policy, how much improvement does your performance exhibit? Just adding some negative samples to the expert data is not expected, as this can easily be done with a classifier or low-weighted when RL training.

"You define $r_{ex}$ and claim it has been used for high-level learning. Could you point out the exact objective function that using it? As $r_{ex}$ is not revealed in Eq(1)-(8) in your paper."

Response:

As briefly mentioned in Section 3 para 1 in the paper, the overall off-policy reinforcement learning objective is to maximize expected future discounted reward distribution: $J = (1-\gamma)^{-1}\mathbb{E}_{s \sim d^{\pi}, a \sim \pi(a|s,g), g \sim G}\left[r(s_t,a_t,g)\right]$.

For the high level policy, $r_{ex}$ will denote $r(s_t,a_t,g)$, which is the the extrinsic reward that the higher level policy receives from the environment. This represents the RL objective $J$ which is used in the joint optimization objectives in Eqns 5-8. The reviewer is right to point out that we do not explicitly use the term $r_{ex}$ later in the paper which may cause confusion, but we used this term to clearly differentiate between the higher level extrinsic reward $r_{ex}$ (the reward that the higher level policy achieves from the environment), and the lower level intrinsic reward $r_{in}$ (the reward that the lower policy gets from the higher level policy) in the hierarchical setup. We hope that this clarifies the doubt. Please feel free to let us know and we would be happy to further clarify.

"What I want to know is not whether adding some negative samples to the expert data has a huge impact on performance"

Response:

We conducted separate experiments where we incrementally added bad/negative demonstrations to the expert demonstration dataset and compared its success rate performance. As expected, the performance degrades when the quality of number of bad demonstrations increases. However, since the approach also uses reinforcement learning along with the imitation learning objective, the performance drop is not significant and the policy is still able to show robustness to bad demonstration trajectories. We have added an ablation study in Appendix Figure 11 for maze navigation, pick and place, rope manipulation and franka kitchen environments (The experiments for bin and hollow environments are still running and we will upload the results soon).

We thank the reviewer for their detailed, constructive feedback. We address the reviewer's concerns as follows:

1. "For example, in section three it is stated that no actions are part of the demonstrations: Notably, we only assume access to demonstration states and not actions. However, in section 4.2, expert data $(s^f, a^f, s^f_{next})$ is used for imitation learning on the lower level. $a^f$ is explicitly used as an expert action here."

Response:

When we train the higher level policies using IRL or BC, the higher level joint optimization procedures (Eqn 1 and Eqn 3) do not require expert actions $a^f$, which is what we initially wanted to convey through the statement. However, expert actions are indeed required when lower level policies are trained, using IRL or BC joint optimization. We understand that our previous statement may lead to confusion for the reader, and thank the reviewer for pointing this out. We have removed the statement and clarified in the paper draft wherever applicable (Section 3 and otherwise).

2. "In Algorithm 1 in line 12, there are furthermore triplets $(s_j,s_w,s_k)$ added to the expert subgoal dataset $D^e_g$ where runs from the initial to the current goal index. Adding $s_k$ seems to be unnecessary as in section 4.2, only the first two entries in the triplet are actually used for imitation learning. It therefore seems to me that Algorithm 1 could be significantly simplified."

Response:

Considering the triplets $(s_j,s_w,s_k)$, the reviewer is absolutely right that $s_k$ is not used in the joint optimization objectives. Nevertheless, we had added the triplets to the dataset $D^e_g$ so that the dataset transitions are consistent with the replay buffer transition storage format used by Soft Actor critic for off policy reinforcement learning. We could remove the $s_k$ entry from the transitions, which simplifies the notation and improves space complexity, but honestly we are not sure how this would simplify Algorithm 1 significantly. Please feel free to let us know if we missed something and we would be happy to add the improvements.

3. "In the second to last paragraph of section 4.1 there are furthermore triplets $(s^e_0, s^e_i,a_i)$ sampled from $D_g$ to train the lower level. This is again in contradiction to what was actually added to the dataset $D_g$."

Response:

In the second to last paragraph of section 4.1, in $Q_{\pi^{L}}(s^e_0,s^e_i,a_i)$, $s^e_0$ and $s^e_i$ are sampled from dataset $D_g$, whereas $a_i = \pi^{L}(s^e_{0},s^e_i)$, which is the action generated by the lower primitive policy. We have also shown this explicitly in Algorithm 1 when computing $Q_{\pi^{L}}(s^e_{init},s^e_i,a_i)$ where $a_i = \pi^{L}(s^e_{i-1},s^e_i)$.

4. "In section 4.2 BC parameters $\zeta_{L}$ and $\zeta_{H}$ for the lower and higher parameters are introduced which are distinct from the parameters $\theta_{L}$ and $\theta_{H}$ of the low- and high-level policies. It is not clear what is meant by that. For example, the BC objective in equation (1) optimizes $\zeta_{H}$ but nothing in it actually depends on $\zeta_{H}$."

Response:

Initially, we had used $\zeta_{L}$ and $\zeta_{H}$ parameters to explicitly depict the policy parameters in the BC objective, but we understand now that this causes confusion, as correctly pointed out by the reviewer. Hence, we have removed the $\zeta$ variables from the draft and now $\theta$ are used to consistently depict the policy parameters.

5. "So $\Pi_{D}^{H}$ is a probability distribution which generates a dataset $D_H$ but $\pi_{D}^{H}$ then is a policy from $D_H$. So $D_H$ does consist of policies then? Apparently not because the second sentence says it is representing the expert dataset. So how can $\pi_{D}^{H}$ be from that dataset then? Is it some kind of empirical policy only defined on the data? This is completely unclear at this point."

Response:

$\Pi_{D}^{H}$ represents some unknown distribution over policies from which we can sample a policy $\pi_{D}^{H}$. The expert dataset $D_H$ does not consist of policies; rather, it consists of state action transitions. Let us assume that these transitions are generated from an unknown empirical policy $\pi_{D_H}^{H}$. Thus, if we know $\pi_{D_H}^{H}$, we can use it to predict actions for the state transitions present in the dataset $D_H$. We hope that this clarifies the confusion.

Dear reviewer,

We hope our response clarified your initial concerns/questions. We would be happy to provide further clarifications where necessary.

Response to rebuttal

I thank the authors for their detailed answers to my questions and concerns.

1. Ok, great that this point could be clarified.

2. As reviewer hXvK also remarked, Algorithm 1 is quite hard to understand with its four nested four loops. Introducing $w=i-1$ as an additional variable does not seem to help with that nor does adding $s_w$ to $D_g^e$. You are right, removing these parts would not be a significant simplification but it would be a start for making the algorithm more readable (together with the comments you added).

3. I mean the second term where it explicitly says $(s_0^e, s_i^e, a_i) \sim D_g$ where $a_i$ is used as a low-level action. This seems to be in contradiction to Algorithm 1 which adds triplets of states to $D_g$.

4. Thank you for clarifying the notation. This makes it easier to understand the algorithm.

5. Thank you for the explanation. The formulation in the paper is not in line with what you write here, however. The paper reads “Let $\Pi^H_D$ and $\Pi^L_D$ be higher and lower level probability distributions which generate datasets $D_H$ and $D_L$ , and $\pi^H_D$ and $\pi^L_D$ are policies from datasets $D_H$ and $D_L$ .”, i.e., it explicitly states that $\Pi^H_D$ generates a dataset $D_H$ and $\pi^H_D$ is a policy from $D_H$. This paragraph does not properly introduce these objects but rather mixes them up.

6. Notation (like $\kappa$) should be introduced properly, in my opinion, even if it is also used in other papers. In the OPAL paper $\kappa$ is assigned a clear role (the initial state distribution), in Theorem 1 in the submission, it just appears without an explanation. It is not clear to me, why $\kappa$ as an arbitrary distribution makes sense in this context (and why the importance sampling ratio should be bounded).

7. What is usually meant by “the non-stationarity issue of HRL” is the changing behavior of the lower level during training. As the proposed algorithm uses the off-policy method SAC on the higher level, it suffers from this non-stationarity issue. Repopulating the subgoal dataset $D_g$ does not mitigate this kind of non-stationarity as it is caused by the lower level, not the higher level. That is not to say that the contribution of how $D_g$ is repopulated is not valuable, but the claim that PEAR mitigates the non-stationarity issue of HRL seems misleading to me.

8. I agree that sparse rewards are highly relevant. I just wanted to point out that the baselines without IL components do not stand a chance with very sparse rewards. It is good, however, that an IL baseline was considered. Thank you for the explanation regarding the hyper parameter tuning.

9. Thank you for the explanation and for adding missing seeds. I agree with Reviewer uEcd that the figures are still not ideal. The color of the shaded regions does not match the color of the mean, for example.

10. to 14. Great, thank you for your effort. I will give some examples where the article is missing. First paragraph of the introduction “a large amount of environment interactions”, “the higher level policy predicts subgoals”. In the same paragraph this should be plural like this “HRL suffers”. There are many more such small issues in the paper. I know these are small things but they are distracting while reading the paper. I would encourage the authors to try to fix these.

Overall, the proposed algorithm seems interesting to me and the experimental results look good. However, I still have concerns about the presentation of the paper that could not be resolved by the response of the authors. I would appreciate further clarficiations.