Stochastic Subgoal Representation for Hierarchical Reinforcement Learning

Thank you for your further clarification on Q1. The choice of using a fraction for the variable $\mathbf{F}$ and a softplus for $\mathbf{Z}$ in our loss function is based on specific insights related to the nature of these spaces:

1. Subgoal Space $\mathbf{Z}$ Regularization: Our primary goal is to regularize the absolute distance (using softplus function) in the subgoal space $\mathbf{Z}$. This is crucial since $\mathbf{Z}$ refers to the subgoal space, where maintaining and controlling the absolute distances is key to effective subgoal representation.
2. Relative Distance in Intermediate Latent Space $\mathbf{F}$: For $\mathbf{F}$, we use the ratio (fraction) as a relative distance measure to avoid two potential issues: (a) the need for an extra hyperparameter to balance absolute distances in both $\mathbf{Z}$ and $\mathbf{F}$ spaces, which would arise if the softplus function were used for both; and (b) the undue emphasis on the absolute distance in $\mathbf{F}$, a factor that is less crucial in our context. Our primary concern in $\mathbf{F}$ is the ratio between high-level and low-level transitions, effectively captured by a 'relaxed' relative measure rather than the absolute distances. This 'relaxed' approach provides more adaptable control over the transitions in $\mathbf{F}$.

These design choices are driven by the distinct roles and requirements of the subgoal space and intermediate latent space, ensuring that each space is optimized in a way that best serves its purpose in the overall model. We hope the above explanation clarifies our choice regarding the two variables.

Q1: Learning objective

A1: Thank you for your insightful query regarding our learning objective's formulation. Our logarithmic term is based on the softplus function, as introduced in Dugas et al. [2001]. This choice offers two advantages over the more commonly used hinge loss function for contrastive learning:

1. No Hyperparameter Requirement: Unlike the hinge loss, the softplus function does not require setting a margin as a hyperparameter.
2. Smooth Behavior: The hinge loss exhibits discontinuity around margin planes, which can be problematic in GP regression. In contrast, the softplus function decays exponentially, avoiding the abrupt cut-off characteristic of the hinge function. This design ensures that even triplets meeting the margin criteria contribute to the loss, enabling continuous pushing/pulling samples as close or as far as possible. The goal is to regularize the absolute difference in $\mathbf{Z}$ space as the subgoal space. To further enhance feature discrimination and the interaction between $\mathbf{F}$ and $\mathbf{Z}$, we utilize the ratio as a relative distance measure in $\mathbf{F}$ as the auxiliary loss. This aims to encourage closer intermediate latent representations between low-level state transitions with smaller ratios, and to promote separation in the intermediate latent representations between high-level state transitions with larger ratios. It is the relative ratio that matters, as this term does not account for the absolute difference.

Reference:

Dugas C, Bengio Y, Bélisle F, Nadeau C, Garcia R. "Incorporating second-order functional knowledge for better option pricing." Advances in Neural Information Processing Systems. 2000;13.

Q2: relationship between f and z

A2: Thank you for your question regarding the role of $f$ and $z$ in the context of this paper. You are correct in your understanding that $f$ represents an intermediate encoding of the state space $s$, and $z$ is the latent subgoal embedding that we are ultimately interested in. The process flow involves mapping the original state space to an intermediate space $f$ through encoding, which is then used as an input to the GP to obtain $z$, i.e., the subgoal representation. The GP enables denoising the representation, taking into account the uncertainty and correlations in the input space. Eq. 1 provided defines a two-step process in a GP based model:

1. $z_i$, the subgoal representation of $s_i$, is governed by a Gaussian Process. This GP is characterized by a mean function, which is assumed to be zero, and a covariance function defined by the kernel $\kappa(s_i, s_j)$. The kernel function dictates the covariance or degree of similarity between any two points in the state space, thereby determining the relationship between the values of $z_i$ at different states $s_i$ and $s_j$.

2. $f_i$ is then defined as $z_i$ with an added Gaussian noise $\epsilon$, where $\epsilon \sim \mathcal{N}(0, \sigma)$. This introduces a stochastic element to the latent variable, creating the observed representation $f_i$ that encompasses the “stochasticity” not captured by the GP.

The dependence of $f$ on $z$ indicates that $f$ is not just a direct observation from the state space but is a smoothed representation informed by the GP's interpretation of the state space. The addition of noise $\epsilon$ to form $f$ acknowledges that real-world observations may deviate from the GP's idealized $z$, reflecting the inherent uncertainty of real-world environments, capturing the unpredictability and randomness that are typical in practical scenarios. This addition not only improves the model's generalization capabilities to new or less frequent states but also facilitates exploration in the learning process, which is vital for effective policy development in HRL.

Q3: HSSR computational complexity, compared to HESS, without randomness

A3: In comparing the average inference time across 10,000 episodes using 5 different random seeds, HSSR exhibits an approximately 3% longer inference time than HESS.

If the clarifications and revisions we've made address the concerns you highlighted, we would greatly appreciate your consideration for re-evaluating your initial rating. We look forward to any additional feedback you may have.

Q1: Can the authors provide more evidence (e.g. show some meaningful bounds) on why GP could help with subgoal representation learning?

A1: Thank you for your question regarding the use of Gaussian Processes (GPs) in subgoal representation learning. We agree that providing more theoretical evidence, such as meaningful bounds, would strengthen our argument for using GPs in this context. We plan to include additional theoretical analysis in our future work to more rigorously demonstrate the advantages of GPs in modeling the latent subgoal space. Compared to existing approaches in learning subgoal representations for HRL, such as predefined subgoal spaces or deterministic representations learned via neural networks, our method introduces a significant advancement. We formulate a posterior distribution over the latent subgoal space, employing GPs to account for stochastic uncertainties in the learned representation. This approach allows for a more flexible and robust representation, adapting to the complex and uncertain environments typically encountered in HRL. We hope this response provides a clearer understanding of our approach and the future direction of our research in this area. Thank you again for your valuable feedback.

Q2: Is it possible to test the proposed method in environments other than MuJoCo simulations? Some subgoal-sensitive tasks, such as NetHack or MineDojo could be interesting for experimenting.

A2: Thank you for recommending that we extend our method's testing to environments beyond MuJoCo, such as NetHack and MineDojo. The advantage of using learnable subgoal representation is its adaptability to various settings without necessitating manual specification of subgoal environments. This flexibility allows our method to be effectively utilized in diverse environments, including NetHack and MineDojo.

We chose MuJoCo for its established benchmarks in HRL research, following the lead of studies like HIRO, HRAC, and LESSON. While integrating new environments would greatly enhance our research, the limited timeframe of the rebuttal period makes this challenging. Comprehensive testing in additional environments requires significant time and resources for proper integration and analysis. We value your recommendation and plan to explore diverse environments in our future work to further validate and strengthen our method.

Following your constructive comments, we have provided detailed responses and made targeted revisions to our manuscript. If our clarifications have resolved the issues you raised, we would be thankful for a re-evaluation of your initial rating. Your further feedback is always welcome.

Q3: Figure 6(Left) suggest that LESSON with \sigma=0.1 doesn't degrade as much compared to the fully deterministic case -- \sigma=0.0, whereas 0.15 can significantly hurt LESSON; so I doubt that sigma=0.1 is a noise that is high enough that would make existing baselines. Figure 6(Left) should also include HESS which is the strongest baseline other than LESSON. HESS is missing in Figure 6(Left)

A3: Indeed, LESSON with $\sigma=0.1$ doesn't degrade as much compared to the $\sigma=0.0$ case in this dense reward setting. However, as illustrated in Fig. 6 (Left) of the revision, such degradation is more pronounced in the sparse external reward setting, which is the primary setting also investigated in Fig. 5. Due to the lack of dense learning signals and the increasing presence of environmental stochasticity, HESS and LESSON start to show deteriorating degradations compared with HSSR.

Our initial comparison didn’t include HESS as it is an extension of LESSON, incorporating an active exploration strategy on top of the deterministic subgoal representation method proposed by LESSON. Our objective was to isolate the impact of the stochastic vs. deterministic nature of subgoal representation methods in the face of environmental uncertainties, without the additional complexity introduced by active exploration strategies like those in HESS. In the revised version, we have now incorporated HESS into Figure 6 (Left) to ensure a more comprehensive comparison.

Q4: learning objective formulation

A4: Our logarithmic term is based on the softplus function, as introduced in Dugas et al. [2001]. This choice offers two advantages over the more commonly used hinge loss function for contrastive learning:

1. No Hyperparameter Requirement: Unlike the hinge loss, the softplus function does not require setting a margin as a hyperparameter.
2. Smooth Behavior: The hinge loss exhibits discontinuity around margin planes, which can be problematic in GP regression. In contrast, the softplus function decays exponentially, avoiding the abrupt cut-off characteristic of the hinge function. This design ensures that even triplets meeting the margin criteria contribute to the loss, enabling continuous pushing/pulling samples as close or as far as possible. The goal is to regularize the absolute distance in $\mathbf{Z}$ space as the subgoal space.

For $\mathbf{F}$, we use the ratio (fraction) as a relative distance measure to avoid two potential issues: (a) the need for an extra hyperparameter to balance absolute distances in both $\mathbf{Z}$ and $\mathbf{F}$ spaces, which would arise if the softplus function were used for both; and (b) the undue emphasis on the absolute distance in $\mathbf{F}$, a factor that is less crucial in our context. Our primary concern in $\mathbf{F}$ is the ratio between high-level and low-level transitions, effectively captured by a 'relaxed' relative measure rather than the absolute distances. This 'relaxed' approach provides more adaptable control over the transitions in $\mathbf{F}$.

These design choices are driven by the distinct roles and requirements of the subgoal space and intermediate latent space, ensuring that each space is optimized in a way that best serves its purpose in the overall model. We hope the above explanation clarifies our choice regarding the two variables.

A straightforward formulation like $\Delta_\mathbf{f}^1 - \Delta_\mathbf{f}^k + \Delta_\mathbf{z}^1 - \Delta_\mathbf{z}^k$ with weight coefficients would linearly combine absolute distances in two spaces. This approach might not capture the intricate distances in either space effectively and could lead to excessive hyperparameter tuning and unstable training.

Reference:

Dugas C, Bengio Y, Bélisle F, Nadeau C, Garcia R. "Incorporating second-order functional knowledge for better option pricing." Advances in Neural Information Processing Systems. 2000;13.

Q1: setup of stochastic environments being a bit artificial; why not perturbing the entire state space?

A1: Thank you for your valuable feedback regarding our experimental setup, particularly our approach to simulating environmental stochasticity. Our study follows the precedent set by recent works such as HIGL (Kim et al., 2021) and HRAC (Zhang et al., 2020). Specifically, HIGL added Gaussian noise at $\sigma=0.05$ to the $(x, y)$ position to model stochastic environments. Similarly, HRAC experimented with adding Gaussian noise levels of $\sigma=0.01, 0.05,$ and $0.1$ to the $(x, y)$ position, describing $\sigma=0.1$ as the "most noisy scenario."

In our experiments, we consistently applied Gaussian noise at $\sigma=0.1$ across all tests, adhering to these established benchmarks. We further extended our investigation to include a $\sigma=0.15$ setting, as shown in Fig. 6 (Left). This decision aimed to thoroughly test our method's performance in highly stochastic settings, which are arguably more demanding than those used in previous studies.

Your suggestion to perturb the entire state space in future work is insightful, yet it requires careful consideration. While we recognize that adding Gaussian noise to the $(x, y)$ positions as a proxy for environmental stochasticity is a simplification, these coordinates are the primary dimensions that are directly influenced by such stochasticity. Other dimensions, like torso contact forces, joint angles, and velocity, pose challenges in quantifying their respective impacts from the same level of stochasticity. Emphasizing the need for extensive experimentation with hyperparameters and design choices, such an investigation goes beyond the scope of this paper. It would require a dedicated research effort to adequately address these complexities. We hope this clarification provides a better understanding of our experiment settings and their context within the broader scope of HRL research. We remain open to further discussion and are grateful for the opportunity to enhance our work based on your insights.

Q2: the experiment settings can only deal with local, independent Gaussian noises rather than the inherent uncertainty that is a result of lack of exploration (i.e. unexplored state space).

A2: Our approach, which learns a posterior distribution over the subgoal representation, is specifically designed to address the limitations you've highlighted. Traditional deterministic subgoal representation methods often struggle in novel state regions due to insufficient historical information to form accurate subgoal representations. This can lead to an underfitting of the learning objective and an inability to accurately capture the underlying dynamics in these new areas of the state space. In contrast, our method explores the intrinsic structure in the state space through learnable kernels, and forms a posterior distribution over the subgoal representation. This approach is particularly effective in underpinning representations in novel state regions, offering a significant degree of resilience against environmental stochasticity, including those inherent uncertainties induced by unexplored spaces.

To empirically validate our approach, we conducted comprehensive experiments in deterministic environments, as detailed in Fig. 8 (Appendix Section) of our paper. The results clearly demonstrate that our stochastic subgoal representation method offers performance improvements in handling the inherent uncertainty induced by unexplored state spaces, compared to the deterministic subgoal representation method LESSON. We hope this explanation clarifies how our method effectively deals with both types of uncertainties – the local, independent Gaussian noises and the inherent uncertainty from unexplored state regions.

Q1: Environment settings

A1: Thank you again for your valuable feedback. We have updated our manuscript to include references to both HIGL and HRAC in the context of stochastic environment settings. Given that our work is one of the first to concentrate on stochastic subgoal representation learning in HRL, we find the lack of established benchmarks in this specific area. Hence, we opted to enhance the noise configuration adopted in existing HRL works to rigorously test our method's resilience and the performance deterioration in baseline methods under increased stochasticity.

Moreover, we emphasize that environmental stochasticity is only one of several challenges addressed in our evaluations. Other aspects, such as sparse external rewards and the use of image inputs, also contribute to the complexity of learning subgoal representations. Our method demonstrates its merits and advantages in these varied and challenging settings, making it arguably one of the most comprehensive among existing HRL works.

We agree that real-world robotic tasks often encounter noise in multiple dimensions, presenting a broader challenge. Our reference to “more extensive study” pertains to future research exploring the quantification of noise level for each dimension of the state space, which falls outside the current scope of our work.

Q2: Figure plotting

We are grateful for your suggestion to enhance the quality of our figures. In the camera ready version, we will ensure consistent smoothing, use distinctive colors, and provide clear breakdowns in our figures for improved visual clarity and consistency.

Q3: Ablation study on loss function

A3: Regarding the ablation study on the loss function, Section 5.3 introduces HSSR-BL-B as a baseline model utilizing a contrastive learning objective, similar to that in LESSON. In Fig. 5, we present a comparison between HSSR-BL-B and HSSR, showcasing the advantages of our proposed learning objective that integrates the softplus function.

If we have adequately addressed your concerns, we would greatly appreciate a re-evaluation of your initial rating. We appreciate any further feedback you may have.

Q1: This paper is hard to understand without introducing their algorithm. The review felt confused about the poorly notations such as \phi(s), etc. The disturbing technical communication is below the standard of ICLR.

A1: While all other three reviewers appreciated the technical clarity of the paper (presentation: good, good, good), we acknowledge the importance of clarity in presenting and the reviewer’s suggestions. We have improved the paper and answer below each question in detail. Hope this satisfies the reviewer. We are more than willing to improve the paper writing and presentation further if needed.

Q2: What is the proposed complete algorithm?

A2: Due to the limited space in the main manuscript the detailed algorithm is provided in Appendix Section A.1. In the main paper, in Section 3.1, we introduce our two-level HRL framework, with Soft Actor Critic (SAC) being utilized at each HRL level. We have included the SAC RL training objective in the updated manuscript (Sec. 3.1) to improve clarity. We hope this clarifies your concern.

Q3: Poor notations

A3: In light of your comments, we wish to emphasize that our notation is broadly in line with established conventions in HRL and GP literatures. To clarify:

• The notation $\phi(s)$ aligns with established conventions in HRL literature, as used in HESS [Li et al., 2022] (3rd line, last paragraph on page 2) and LESSON [Li et al., 2021] (4th line, 4th paragraph on page 2).
• $\mathbf{Z} = \left( \mathbf{z}_1, \mathbf{z}_2, \cdots, \mathbf{z}_N \right)$ is similarly defined in [1] (3rd line, first paragraph on page 5).
• $\Delta_\mathbf{f}$ and $\Delta_\mathbf{z}$ are shorthand notations representing distances, used for brevity. These forms are consistent with conventions found in algebra literature, such as those detailed in Section 3.2 of [2].

We hope this clarification helps, and we are open to further suggestions to improve our manuscript.

References:

• [1] Lee, J., Bahri, Y., Novak, R., Schoenholz, S.S., Pennington, J. and Sohl-Dickstein, J., 2017. Deep neural networks as Gaussian processes. ICLR 2018.
• [2] Berg, T., 2020. Intermediate Algebra. Kwantlen Polytechnic University.

Q4: Is the number of subgoals fixed? What is this number?

A4: Thank you for inquiring about the number of subgoals. In our experiments, the high-level policy is designed to generate a subgoal every $k$ timesteps, with $k$ set to 50 for all MuJoCo tasks, as detailed in A.4.1 in the Appendix section.

Q5: What is the relationship between g and z?

A5: Your question brings up an important aspect regarding the relationship between $g$ and $z$. In our framework, the high-level policy samples a subgoal $g_t$ from the latent subgoal space abstracted by the representation function $\phi(s)$. The low-level controller then strives to achieve these subgoals, with $z_t = \phi(s_t)$ serving as the mapping from the state space to the subgoal space. Essentially, $g$ represents the action taken by the high-level policy within the subgoal space, while $z$ is the corresponding subgoal space representation of the state $s$, as illustrated in Fig. 1.

Q6: How to interpret Fig.4 without marking which goal is associated with which segment of the trajectories?

A6: We are grateful for your constructive feedback regarding Fig. 4. Acknowledging the confusion caused by our initial presentation, we have revised the figure to enhance clarity. The subgoals are now color-coded to match their corresponding states, facilitating a more intuitive understanding of their associations with specific trajectory segments.