Hierarchical Empowerment: Towards Tractable Empowerment-Based Skill Learning

Thank you for your feedback.

Missing comparisons to Unsupervised Goal-Conditioned Algorithms (MEGA, Skew-fit, EDL, etc.)

We disagree that a comparison with the unsupervised GCRL algorithms (MEGA, Skew-fit, EDL) is necessary. Given the importance of Empowerment as a way to find the space of reachable states and thus quantify the size of an agent’s skillset, there needs to be algorithms that optimize the full Empowerment objective. But MEGA, Skew-fit, and EDL do not optimize the full Empowerment objective. Rather they optimize two different objectives. In one process, they optimize $H(Z|s_0)$ or $H(S_N|s_0)$ (e.g., training an autoencoder on visited states to obtain a goal space $p(z|s_0)$). At the same time, but in a second separate process, they train and goal-conditioned policy using goals sampled from the goal space to minimize $H(Z|s_0,S_N)$ or $H(S_N|s_0,Z)$. The clear failure case for this approach is when there is randomness and the goal states from the goal space are not consistently achievable. In this situation, this strategy may not calculate empowerment accurately (i.e., find the largest space of reachable states.) because the skill space is not optimized to learn skills that are consistently achievable. Thus in our baselines, we just considered algorithms that optimize the full Empowerment objective.

“Albeit biased, previous empowerment-based approaches can work without a model.”

It’s difficult to compare our empowerment-based work with earlier results because both our (a) skill space learned for each skill start state and (b) the distribution of skill start states seem to be vastly larger than prior work (i.e., our $p(z|s_0)$ and $p(s_0)$ are significantly more entropic), particularly in our larger settings. Storing and sampling biased $(s_0,z,s_N)$ tuples from a replay buffer seems unlikely to scale to very large domains. For instance, given that the replay buffer contains stale (i.e, biased) data, you can only use the recent past. But then in large domains, much of the state space may not be continually visited so if you cut off your replay buffer, then your empowerment optimization may be missing those reachable states.

Section 3.1 Clarifications

$g$ represents the reparameterization trick. For our uniform goal space distributions, each dimension of $z_i = -\text{halfwidth (output by goal space policy)} + 2*\text{halfwidth}*\text{eps}$, in which eps is a sample from a unit uniform random variable. $h$ is just a projection to the goal space (e.g., the (x,y) coordinates from the state).

The line "instead of sampling the skill z from the fixed variational distribution" was just referring to how the log of the variational posterior $\log q(z|s_0,s_N)$ is calculated, not how the skill $z$ is sampled. Instead of computing $\log N(z; \text{mean}=s_N,\text{std}=\sigma_0)$ we use $\log N(z + h(s_0);\text{mean}=s_N,\text{std}=\sigma_0)$. This causes the skill $z$ to reflect desired change in state $s_0$ (e.g., desired change in the x and y coordinates from the skill start state $s_0$) and not actual actual states. This is helpful because it initially centers the learned goal space on the start state $s_0$.

Where is the model used and where is it required?

We use the model to help generate transitions for all the goal-conditioned and goal-space policies. The goal-conditioned policies need unbiased (prior state,action,next state) tuples and the goal-space policies need unbiased (skill start state, goal space action, epsilon, skill end state) tuples. The model is not required for the base level goal-conditioned policy because a replay buffer of (prior state, action, next state) transitions would be unbiased. A model is also technically not required for the goal space policies because the Q function from the goal-conditioned policies at the skill start states could be used as the critic for the goal space policy but we chose to use the model to generate transitions to train a separate goal space critic.

State coverage is not apples-to-apples with DIAYN and DADS

For our largest setting, our episode lengths were 2000 actions, which is likely significantly larger than in DIAYN and DADS papers. However, our implementation of DIAYN and DADS could not learn skills to achieve goals in a 20x20 box so it seems very unlikely the extra time would make a noticeable difference in their state coverage. In regards to the simulator difference, the authors of the Ant domain in Brax note that it was derived from the MuJoCo Ant domain. We also have experience implementing Ant in MuJoCo and the domains seem similar so it seems unlikely our results would be too different in MuJoCo.

Thank you for your feedback.

Clarify the sentence “The entropy term $H_{\phi}(Z|s_0)$ encourages the goal space policy $\mu_{\phi}(s_0)$ to output larger goal spaces.”

The output of the goal space policy includes the half-lengths of each dimension of the uniform goal space distribution. The entropy term $H_{\phi}(Z|s_0)$ in the Goal-Conditioned Empowerment objective incentivizes the goal space policy to output larger halfwidths, which is what we meant by “larger goal spaces.” The dimensionality of the goal space is fixed and we can clarify that in the paper.

Why is a model of the transition dynamics needed?

The primary need for the model is to generate unbiased (state, subgoal action, next state) tuples for training the above-base level goal-conditioned policies with RL. Given that the goal-conditioned policies at each level are continually changing, storing and sampling these transitions from a replay buffer would provide biased samples.

In addition, we have tried to argue that existing empowerment-based skill-learning methods (e.g., DIAYN) that train the posterior distribution $q_{\phi}(z|s_0,s_N)$ with maximum likelihood will also need a model when the distribution of skill-start states $p(s_0)$ is highly entropic. Maximum likelihood training of $q_{\phi}(z|s_0,s_N)$ requires on-policy tuples (s_0,z,s_N), which require executing the current skill-conditioned policy $\pi_{\theta}(a|s,z)$ from the skill start state $s_0$ for $N$ actions. Without a model, this would require executing the skill-conditioned policy for every skill $z$ at every start state $s_0$ to get unbiased (s_0,z,s_N) tuples. This is not practical when p(s_0) and p(z|s_0) are highly entropic (i.e., when you need to learn a large set of skills from each state in a large set of states). The alternative of storing the (s_0,z,s_N) tuples (e.g., p-HER [1]) in a replay buffer will provide biased samples because the skill-conditioned policy is continually changing. Having a model would enable the user to execute the skill-conditioned policy from the necessary (s_0, z) combinations in parallel.

Justify the statement "hand-crafted goal space may only achieve a loose lower bound on empowerment".

We will break this down into two possible cases. In the simpler case of the human trainer setting too small of a goal space (i.e., there are additional goals outside the hand-crafted goal space that can be achieved), $H(S_N|s_0)$ (the upper bound of empowerment assuming a discrete goal space) will be smaller than when the goal space includes all of the achievable goal states. This would produce a loose lower bound on empowerment if the hand-crafted goal space is significantly smaller than the achievable goal space.

In the case of the human trainer setting the goal space too large (i.e., there are goals in the goal space that cannot be achieved in $N$ steps), then the goals that cannot be achieved will yield skill end states $s_N$ that are redundant as they can be achieved by other goals (e.g., the goal $s_N$). This will lower $H(S_N|s_0)$ unless all the redundant skill end states $s_N$ are distributed uniformly over the reachable state space $S_N$. If the redundant states are not uniformly distributed across $S_N$, this again can produce a loose lower bound on empowerment.

In our case of the continuous goal space distributions, it’s more complicated because there is no upper bound on the conditional entropy $H(S_N|s_0,Z)$, which can reach infinity. But clearly in practice there will be situations when the above two cases apply and the empowerment that is achieved is significantly lower than what is possible with a learned goal space.

Missing citation to previous work on HRL.

Can you provide citation [4] again as the citation cannot be seen in your rebuttal? Thanks.

Results do not offer experiments that separate the effects of empowerment and the goal-conditioned hierarchical policies.

We may be misinterpreting the question, but we are not sure how to separately assess them because we are not aware of other ways to build a goal-conditioned hierarchy in an unsupervised manner. Our work shows that empowerment can be a way to generate each level of the goal-conditioned hierarchy in an unsupervised manner.

[1] Choi et al. Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learning. ICML 2021

Thank you for your feedback.

We agree with your characterization of the strengths and weaknesses of the algorithm. In regards to selecting the number of levels $k$, our main criteria was that we wanted each level to learn a short sequence of actions given the difficulties of credit assignment in RL. Given that the $k$-th root of the number of actions in an episode, (i.e., $(\text{actions per episode})^{\frac{1}{k}}$), reflects the lengths of the policies each level needs to learn, we were looking for the $k$-th root to be some small number (ideally under 10). Although increasing the number of levels can make learning long horizon skills more tractable, the main drawback is that it increases the wall clock time of training increases. Training the top level goal-conditioned and goal-space policies in the hierarchy requires simulating $N^k$ primitive actions, which increases exponentially with $k$. $N$ here refers to the maximum number of actions executed by each level.

Thank you for your feedback.

We agree with your characterization of the weaknesses of the algorithm. The large standard deviations for the 1 skill level agent in the “Half Cheetah – Big” domain in Figure 3 and the 2 skill level agent in Figure 4 was due to there being a trial in which the agent failed to learn anything meaningful in phase 2. The poor performance in phase 2 was a result of learning an inadequate skillset in phase 1 of training. The large standard deviation for the 2 skill level agent in “Ant Field – Big” early in training is something we occasionally noticed in our experiments and is likely due to the difficulty of the phase 2 task. In the large domains, the phase 2 task is a very sparse reward goal-achieving task. Sometimes due to the lack of reward signal, the phase 2 policy will saturate the tanh activation function and then become stuck proposing subgoals at the extremes (in this case, the corners) of the learned goal space box. The policy can become stuck because the gradients are near zero when the tanh activation function saturates. Ultimately though, in all our experiments, the phase 2 policy of the agent with more levels policy always recovered. A simple way to alleviate this issue would be to increase the goal-achieving threshold for the phase 2 task in order to make the reward signal less sparse.