Learning Large Skillsets in Stochastic Settings with Empowerment

Since there has been no rebuttal from the author, and another reviewer gave a rating of 8, I am considering lowering my rating to convey that, despite the significant amount of work put into the paper, the ultimate outcome appears to be a considerable waste of resources for no significant contribution. This should not be encouraged.

Thanks for the reply, I want to make some comments on your reply:

About contribution and "implicit issues ":

1. The bound still only contributes to the approximation level. All the bounds in Appendix A come from that (8) is a better approximation than (7).

2. Directly comparing (8) with (7) is unfair and potentially misleading. As I mentioned in my original review, existing methods that use (7) alternatively update $(\pi,\phi)$ and $\psi$ and it is totally capable of learning $\pi_1$ in Figure 1. There are two steps for each update, first update $(\pi,\phi)$ with (7) then update $\psi$ with (6). Directly comparing (8) with (7) is like comparing $I(\pi_{k+1},\phi_{k+1},\psi_{k+1})$ with $I(\pi_{k+1},\phi_{k+1},\psi_k)$, insufficient to show the bound's long-term significance as $k\rightarrow \infty$. In optimization literature, block coordinate descent[3](alternative update) is possible to converge to the same solution as full gradient descent(simultaneous update).

3. Last but not least, you seem to not fully understand the most important point, the “implicit issue” of $I(S;Z)$. I was clearly referring to [1][2]. You should read them before you make further comments. Even if you spend more effort to make a perfect approximation of $I(S; Z)$, fundamental limitations persist. This undermines the significance of your contribution, despite your substantial effort.

About diversity:

This concern is not addressed. Your response just states a well-known property of mutual information, but you are unaware of the more recent theoretical results in Unsupervised RL and what "diversity" means for good downstream task adaptation. I can still use my initial review about "rigor" to reply to your rebuttal.

About skill number:

(Cover & Thomas, 2006) is a textbook and I am not sure what you mean by referring to it.

There is a reason why I asked are you assuming $q_\psi(z|S)=1$ after training. Because only when $q_\psi(z|S)=1$, $\hat{I}(S|Z)=E[\log \frac{q_\psi(z|S)}{\phi(Z)} ]=E[-\log \phi(Z)]$ is log of skill number. However, I did not see how this assumption could be justified.

About computation:

Then what is the training time for existing methods in the same setting? If you are training in parallel, then what is the memory usage?

About baselines:

The formulation of LSD and CSD does not directly resemble $I(S; Z)$ because they are aware of the limitations of $I(S; Z)$ and proposed new objectives to overcome them. CIC also saw the limitation and they optimized the $I(\tau; Z)$, where $\tau$ is the trajectory. Despite the approximation bound, it could result in much more diverse behavior because the space of trajectories is inherently much larger than the state space.

The CIC paper is not accepted because there might be potential inconsistencies between their reported results and their code. However, LSD and CSD can still serve as strong baselines, particularly for higher-dimensional control environments like Ant.

I strongly recommend that you stay updated with the latest research in this area. I apologize if my language seemed overly direct, and I truly appreciate the effort you've put into this work. However, I believe it might be beneficial to reconsider the current research direction in light of these recent developments.

[1] Lipschitz-constrained unsupervised skill discovery.

[2] Controllability-aware unsupervised skill discovery

[3] Wright S J. Coordinate descent algorithms

Page 1 of 2

Thank you for your feedback and apologies for the delay.

• Contribution is limited as only changing search procedure for (skill-conditioned policy $\pi$, variational parameters $\psi$)

We do not believe that characterizing our contributions as limited is accurate. Our first contribution of proving that existing approaches (e.g., DIAYN) only maximize loose lower bounds on the mutual information between skills and states should be important to the community as it provides a new and principled reason for why other papers (e.g., (i) Campos et al., 2020 and (ii) Levy et al., 2023) have shown that existing approaches typically only learn small skillsets. (Note that the simple proof for why our approach can learn a tighter bound on the empowerment of a state is provided in section A of the appendix in the updated draft. The proof was previously left out because we felt the wording we provided in the original manuscript was sufficient). In addition to this contribution, we then provide an alternative objective that provides a tighter lower bound to the mutual information between skills and states. Yes, this objective just involves training a variational posterior conditioned on the skill-conditioned policies and distribution over skills that are being considered, but it provides a more principled way to maximize mutual information. Further, we also introduce a new actor-critic architecture to optimize this tighter variational lower bound as it is not clear how to maximize this new objective with existing techniques. Empirically, we also provide the first results in a stochastic setting for an algorithm that maximizes the mutual information between skills and states. We believe these contributions should be sufficient for a conference paper.

• $I(Z;S)$ is “not a good measure of skillset diversity”, “unclear relationship to number of skills”,

These comments are not true. The mutual information between skills and states given (i) a particular distribution of skills $\phi$ and (ii) a skill-conditioned policy $\pi$ computes the number of skills from the ($\phi,\pi$) skillset that can be executed and decoded from the skill-terminating state with arbitrarily low error (Cover & Thomas, 2006). Note, that this number of skills is a rate (i.e., the number of skills per use of the channel between skills and states). Thus, maximizing mutual information will produce skills that target different skill-terminating states (i.e., will optimize skillset diversity) because the skills need to be decoded with arbitrarily low error. Mutual information can also thus represents the “number of skills” in a skillset. When we define “number of skills” as mutual information, we can emphasize that this is a rate (number of skills per channel use). Also, we are not the first to use variational mutual information between skills and states to mean “number of skills” (e.g., Gregor et al. 2016).

• Results show $I(Z;S)$ has an “implicit issue”

I am not sure what “implicit” issue you are referring to and none of our results produce a skill-conditioned policy that was similar to the initial skill-conditioned policy. In the one example chosen in the review, Ant, the agent went from having a minimal skillset to having a massive skillset with ~19.7 million skills (i.e., a mutual information of 16.8 nats). Although these skills did not produce much translation they produced specific joint movements.

• Computationally expensive approach requiring “LLM-equivalent” compute for simple tasks

A few comments here. First, the reviewer takes issue with the approach of searching over different distributions of skills and skill-conditioned policies. But mutual information depends on these two distributions, so searching over these two distributions to maximize mutual information is principled.

Second, yes the approach requires extensive compute, but we are learning large skillsets with this compute. Sure you can argue that many of these skills could be unnecessary. But there also may be many situations in which a lot of these skills are necessary. For instance, there may be a setting where an ant agent needs to be able to open a variety of door handles. In this case, the ant agent may need to achieve states in which one of its legs is at a certain angle moving with a certain angular velocity. In this type of situation, the agent would need to have skills that are not limited to (x,y) translation. In addition, for downstream applications such as empowerment-based reinforcement learning where an agent takes actions to maximize an empowerment reward, the empowerment of states needs to be known so agents need to know all the skills they have in a state. Unsupervised skill-learning Algorithms like LSD and CSD that do not actually maximize mutual information would not be sufficient for this application.

Thank you for your response.

See below regarding your concerns about the implementation.

1. The gradient of the mutual information critic $Q(s_0,\pi)$, in which $\pi=f_{\lambda}(s_0)$ (ignoring $\phi$), with respect to $\lambda$ only requires the gradients of the mutual information critic with respect to each dimension of $\pi$. That is, $\nabla_{\lambda} Q=\nabla_{\pi} Q \cdot \nabla_{\lambda} \pi = \sum_{i=1}^{|\pi|} \nabla_{\pi_i} Q \times \nabla_{\lambda} \pi_i$ (i.e., $\nabla_{\pi} Q \cdot \nabla_{\lambda} \pi$ is the dot product between (i) the gradients of the critic with respect to each parameter in $\pi$ and (ii) the gradients of each parameter in $\pi$ with respect to $\lambda$). This means that the parameter-specific critics that only measure the changes in mutual information from changes to a single parameter are sufficient to know the gradient with respect to $\lambda$ and the critics do not need to approximate changes to multiple parameters.

2. The need to either have or learn a model of the transition dynamics was why we attached our follow up paper that shows how mutual information can be maximized using (a) our actor-critic architecture and (b) only off-policy transition data $(s_t,a_t,s_{t+1})$. This algorithm maximized a lower bound to a tight upper bound to the mutual information between skills and states, and this objective can be optimized with off-policy data. The results from the off-policy algorithm matched the performance of Skillset Empowerment in all tasks despite not having or needing to learn a simulator of the environment.

Despite the limitation of needing to learn a simulator, we still believe Skillset Empowerment is a sufficient contribution as it provides a different way to maximize the mutual information between skills and states. Yes there are other ways to learn skillsets in an unsupervised methods (e.g., the works by Park et al. that the reviewer has mentioned), but these works do not maximize the mutual information between skills and states and there may be a need for agents to maximize this quantity and learn a large number of the available skills in a state (e.g., empowerment-based RL). In addition, our approach is the first unsupervised skill learning algorithm to learn skills in stochastic settings, which does not seem minor.

Further, Skillset Empowerment seems to provide a more principled approach to address the problem of nonstationarity that can hinder unsupervised skill learning approaches. In unsupervised skill learning, the reward changes with changes to the skill-conditioned policy. This does not affect Skillset Empowerment because the action is the skill-conditioned policy and so the action is a sufficient statistic for the reward. But in other approaches this is not the case as the action is still the primitive action so each change to the policy will change the reward associated with a (state, skill, primitive action) tuple.

3. The variational mutual information performance metric we used should not be a problem. The algorithms that we compared against, VIC and GCRL, should be learning policies in which the true posterior $p(z|s_n)$ is approximately diagonal gaussian so modeling the variational posterior as diagonal gaussian should not be problematic. Sure it's possible there could be some noise in whether the gaussian variational posterior can match the true posterior, but this would affect all approaches and should disappear with more trials. I also do not remember observing this in any trial.

It is only accurate around your current greedy policy, nonparametric sensitivity analysis should be much more efficient for this.

For example, a much more simplistic way of implementation is to sample a large number of $(\pi', \phi')$ around current $(\pi, \phi)$ and learn $\psi$ for all of them, evaluate their objective values and find the best one.

My evaluation was done before the discussion on implementation details. The further discussion only serves as potential suggestions

Your implementation relies on the assumption that of certain smoothness of $Q(\pi)$. Without this assumption, you need to retrain all of your critic neural networks for every new greedy $\pi$, which is extremely inefficient. However, this assumption again limits the expressiveness of $Q(\pi)$.

I suggest to look at techniques from zero-order(Derivative-free) optimization[1]. There should be efficient ways to sample and test $(\pi,\phi)$ skillsets.

Also, many newer methods such as LSD,CSD and VISR(APS), are actually highly related to and much more intricate than what you mentioned:

" However, in practice this problem can be easily overcome by adding a small amount of stochasticity to the policy (e.g., in our implementation we used a diagonal gaussian policy with a fixed small standard deviation). After this change, in practice you will not get infinite posteriors because there will be some overlap in the states targeted by different skills."

We can consider them as methods focusing on the "quality" of learned skills. My suggestion is that you find an approach with acceptable computational demand, learn skills with both better quantity and quality, and demonstrate its superiority against more recent methods. Achieving this would be a strong contribution to the URL community.

[1] Larson et al. (2019): "Derivative-free optimization methods"

No all parameters of the skill-conditioned policy are being updated at the same time. By "one dimension at a time", I was just referring to all the $Q(s_0,\pi_i)$ just approximating mutual information from changes to one parameter in $\pi$.

The inner loop that trains $Q(s_0,\pi_i)$ is approximating a tighter lower bound to the mutual information $I(Z;S_n|\pi_i)$ so it will not have the same local max issue. Note that the inner loop consists of a series of updates to first the variational posteriors $q_{\psi_i}(z|s_n,\pi_i)$ and then a series of updates to the $Q(s_0,\pi_i)$ functions. Yes, this is more computationally expensive but should not have the same local max issues which our theory and empirical evidence demonstrate.

Thank you for your feedback and apologies for the delay.

• Missing proof that Skillset Empowerment measures tighter bound on empowerment

We have added the simple proof in section A of the appendix that Skillset Empowerment learns a tighter bound on the empowerment of a state relative to existing approaches (e.g., DIAYN) that train the variational posterior by minimizing the KL divergence between the posterior of the current skillset and the variational posterior. We had left this out originally because we felt the wording in the original manuscript was sufficient, but have now added it to the paper. A tighter bound on empowerment results in part because for any ($\phi,\pi$) skillset learned by existing methods, our approach learns a higher variational mutual information for the same $(\phi,\pi)$ skillset because our approach can learn a more accurate variational posterior $q_{\psi}$. Our approach may then find a different $(\phi_{SE},\pi_{SE})$ skillset with larger variational mutual information.

• Approach assumes a model of the transition dynamics, which means unsupervised RL is not needed.

A couple of comments here. First, measuring the empowerment of a state has other uses in addition to solving downstream MDPs. For instance, empowerment-based RL (Klyubin et al., 2005) in which agents maximize an empowerment reward, requires that agents are able to measure the empowerment of states.

Second, it is true that this approach requires a model but we have already shown how this can be overcome (see paper provided in supplementary materials). This paper maximizes a principled replacement for the mutual information between skills and states that only requires an off-policy dataset of transitions. This updated objective matched the performance of Skillset Empowerment in all domains.

• Compute requirements are high

This is true but it is also learning very large skillsets with the high compute. Our approach also provides a principled way to maximize the mutual information between skills and states because it maximizes a tighter bound on the mutual information. Previous empowerment-based approaches have required less compute but are also less principled as they only maximize a loose lower bound on mutual information between skills and states.

• Missing qualitative examination of skills

Is there something in particular you had in mind? In the $H(S_n|Z)$ visualization we show the skill-terminating states targeted by each skill. Given the state coverage shown in the $H(S_n)$ visual, we are learning skills that target nearly all reachable skill-terminating states.

• Missing downstream task performance

Given that we are trying to maximize mutual information between skills and states, we believe this mutual information is the most important metric and is a metric that many other mutual information maximizing algorithms have not provided (e.g., APS, CIC), which makes it unclear how well these other approaches work. Also, general purpose agents need to be able to achieve a vast number of tasks so showing good performance on one or two tasks is not illustrative of a widely capable agent.

Thank you for the feedback and apologies for the delay.

• Argument for loose lower bound is not sufficiently rigorous

Section A in the appendix provides the proof that Skillset Empowerment learns a tighter bound on empowerment than existing approaches that train the posterior to match the true posterior of the current skillset. The tighter bound on empowerment results in part because the maximum variational mutual information learned by existing approaches must be less than or equal to the variational mutual information learned by our approach for the same pair of (distribution over skills $\phi$, skill-conditioned policy $\pi$) because our approach learns a more accurate posterior distribution. In addition, our approach can find a $(\phi,\pi)$ skillset with larger mutual information.

• More precisely defining variational objectives using by existing approaches

Sure we can be more precise here but regardless of whether the posterior is mapping skill-terminating states to skills (e.g., VIC (explicit option version)) or states at each time step to skills (e.g., DIAYN), they both will be loose lower bounds as they are not also conditioned on the skillset distributions $(\phi,\pi)$.

• Confusing notation for distributions

Yes we can improve the notation as $\phi$ both represents the distribution over skills $\phi(z|s_0)$ and a scalar quantity representing the side length of the box that represents the distribution over skills. In the camera ready, we will switch to describing the distribution over skills as $p(z|s_0,\phi)$, which depends on the parameter $\phi$.

• Having $|\pi|$ critics is not sufficiently expressive

I am not sure I follow. Each parameter-specific critic approximates how the mutual information of a skillset changes with a small change to one of the parameters in the skill-conditioned policy. This neural network should be sufficiently expressive to approximate this quantity.

• Baseline visualizations

We did add the state coverage for GCRL in stochastic four rooms (Figure 17) in which the GCRL agent just learns to move to the corners. We can add the VIC agent as well. Though because the VIC agents did not learn anything meaningful, there should not be too much to see.

• Why uniform distribution over skills?

Our decision to use a uniform distribution over skills in the shape of a $d$-dimensional cube was done for a few practical reasons. First, it is easy to compute the entropy of a $d$-dimensional cube as it is just the volume of the cube. Second, this distribution is defined by a single scalar which simplifies the search problem of finding the mutual information-maximizing $\phi$. Third, if our approach was to be used for solving downstream tasks, it is simple to use this cube as the action space for the downstream policy that outputs skills. Fourth, as you are maybe hinting at, the uniform distribution does have the largest entropy.

• Limited to simple domains

The domains are simple because maximizing the mutual information between skills and states conditioned on a particular distribution over skills $\phi$ and skill-conditioned policy $I(Z;S|\phi,\pi)$ requires a lot of compute. Approximating this mutual information for a single $(\phi,\pi)$ skillset, let alone many $(\phi,\pi)$, requires sampling a large number of skills from the distribution over skills and executing them in the environment with $\pi$. So it should not be surprising that we applied our approach to simple domains. Note that we still did apply our approach to the Brax Ant environment where our agents were able to learn $\sim 20$ million skills despite being compute constrained.

Simple domains are also helpful because we can more easily approximate the mutual information between skills and states learned by our approach and the baselines. This is something that is missing from prior works in which they provide an algorithm that they claim maximizes the mutual information between skills and states but then do not provide results showing the actual learned mutual information between skills and states (e.g., the algorithms APS and CIC). They only show results on downstream tasks, but then it is unclear whether the mutual information maximization component is working as intended.

• Missing baselines to CSD, APS, CIC

The purpose of our paper was to target the premise from which all these algorithms were built. Park et al. state that the purpose of LSD and CSD was in part because they believed $I(Z;S)$ could not be maximized in deterministic settings so an alternative objective was needed. Liu et al. said APS was designed because they claimed that maximizing $I(Z;S)$ does not encourage exploration. Laskin et al. designed CIC in part because they believed the issue with maximizing a variational lower bound was that the variational bound should be implemented with contrastive learning. Our paper shows that these reasons are not correct. We show that the problem is that the typical variational lower bound objective (e.g., DIAYN) is a loose lower bound on the true mutual information $I(Z;S)$ because the variational distribution $q_{\psi}(z|s_n)$ is not conditioned on the skillset distributions ($\phi,\pi$) . Our empirical work, in which we are able to successfully measure empowerment in a variety of deterministic and stochastic settings, shows that unlike the claims of the above authors (i) empowerment can be measured in deterministic settings, (ii) exploration is built in to the objective (provided a simulator is available, though we address this limitation in the paper in the supplementary materials), and (iii) a variational mutual information can be maximized without contrastive learning.

• Visualize Skills

The entropy visualizations and in particular the $H(S_n|Z)$ visual show the skill-terminating states targeted by specific skills $z$. The $H(S_n)$ visual shows the distribution of skill-terminating states targeted by 1,000 randomly sampled skills. The behavior of the agent in the intermediate time steps should not be relevant.

Thanks for your replay. I am still concerned about the experiments, and I keep my score.