Answer: Reviewer 8kZQ

We thank the reviewer 8kZQ for their very positive comments on the paper presentation and for their insightful comments on C-learning. We hope to address their questions in our response.

We did find that the application of C-learning is limited based on the dimensionality of the problem. The application of C-learning to the high-dimensional state spaces of the MuJoCo environments was a challenging aspect of this work, as detailed in Appendix D. We therefore did not try RGB environments, as this would require further fine-tuning of C-learning on a convolutional architecture. However, recent contributions [1][2] building upon C-Learning illustrate that learning this measure in such environments is feasible, making this application an interesting direction for future work. We will include these references in the discussion on the limitations of C-learning in Appendix D.

[1] Eysenbach, Benjamin, et al. "Contrastive learning as goal-conditioned reinforcement learning." Advances in Neural Information Processing Systems 35 (2022): 35603-35620. [2] Zheng, Chongyi, Ruslan Salakhutdinov, and Benjamin Eysenbach. "Contrastive Difference Predictive Coding." The Twelfth International Conference on Learning Representations.

Answer: Reviewer axfE

We thank Reviewer axfE for their comments. In our general response, we address their specific concern regarding a more comprehensive comparison of LEADS with other baselines. Below, we will address the remaining questions and concerns about our study.

Beyond the maximum coverage objective:

We appreciate the reviewer raising this insightful question, which opens up important avenues for discussion. As supported by multiple studies [1][2][3][4], exploration represents a critical step in deriving effective policies, especially in hard exploration problems. We conjecture that initializing, for example, a goal-reaching algorithm using the replay buffer obtained via LEADS would yield better success rates, particularly in hard exploration problems. For instance, in Figure 2, in the Hard-maze environment, the transitions sampled by skill 4 are essential for training any goal-reaching policy over goals in that part of the maze. We also conjecture that another expected advantage could be the design of reusable/composable skills in a hierarchical setting, but we consider it out of the scope of this study and reserve it for future work. We propose mentioning this direction in the conclusion section.

Different number of skills:

As with other skill-based algorithms, LEADS's coverage changes with the number of skills. However, the skills learned by LEADS evolve throughout the entire training procedure to visit unexplored areas. Hence, one can expect LEADS to require fewer skills to achieve the same final coverage performance than algorithms that learn static skills, like DIAYN. Choosing the number of skills involves a tradeoff between the acceptable computational runtime of the algorithm (which increases with the number of skills) and the desired efficiency in state space coverage. This choice is also highly problem-dependent. Dynamic adaptation of the number of skills is a challenging and open topic, covered by works such as [5]. Although this remains an open question, for LEADS we worked with a hand-chosen, predetermined number of skills and did not focus on this aspect, as is common in most works in the literature. We propose including this discussion in Appendix C on Hyperparameters.

We hope these answers address the concerns of the reviewer and lead to further exchanges.

[1] Deepak, Pulkit et al. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pages 2778–2787. PMLR, 2017.

[2] Yuri, Harrison et al. Exploration by random network distillation. arXiv preprint arXiv:1810.12894, 2018.

[3] Adrià, Pablo et al. Never give up: Learning directed exploration strategies. arXiv preprint arXiv:2002.06038, 2020.

[4] Zhaohan, Shantanu, Miruna et al. BYOL-explore: Exploration by bootstrapped prediction. Advances in neural information processing systems, 35:31855–31870, 2022.

[5] Kamienny, Tarbouriech et al.(2022, April). Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reaching. In ICLR 2022.

Answer: Reviewer 98FT

We thank Reviewer 98FT for their comments. In the following, we address their specific questions and concerns about our study.

Hand environment:

Indeed, the environment used in our tests is HandReach. We will make this clear in the experimental section.

Theoretical study of LEADS

We agree with the reviewer's comment about the benefits of a deeper theoretical understanding of the LEADS algorithm and specifically the interplay between the mutual information objective and an explicit exploration objective. Through this study, we aim to demonstrate that mutual information maximization can lead to exploration when coupled with an appropriate objective, such as the maximization of the uncertainty measure we propose, along with a sufficient successor state measure. A theoretical understanding of this interplay motivated the writing of Appendix A, which we hope serves as a starting point for this discussion. More specifically, the analysis of the exploration term in Equation 7, detailed in Appendix A.1, draws a link between the uncertainty measure we use and a specific Kullback-Leibler divergence, providing a practical understanding of its maximization.

In future work, we hope to build upon this theoretical insight to derive more formally motivated objectives based on this study.

Thank you for your response.

The connection to the KL divergence does help motivate the method a bit better. I have increased my score to a 7

Answer: Reviewer QVxs

We thank Reviewer QVxs for their comments. In our general response, we address several of the concerns raised by the reviewer. Below, we specifically address their additional questions and concerns about our study.

Statement in line 164:

The sentence "natural interplays between skills" in the text refers to the interaction or coordination required among the different skills during the maximization of Mutual Information. Specifically, the reversed form of Mutual Information (MI) is:

$$\mathcal{I}(S,Z) = \mathbb{E}_{\substack{z \sim p(z) \\ s \sim p(s|z)}} \left[\log \left(\frac{p(z|s)}{p(z)}\right)\right].$$

To maximize this quantity, the distribution $p(z|s)$ in the log term must be as concentrated as possible on the skill $z$ that was used to sample this state. Therefore, the probability of any other skill on that state must be minimal. The term "natural interplays" thus informally describes the necessary coordination between skills to ensure that each skill is uniquely associated with specific parts of the environment, leading to maximal MI.

In contrast, maximizing the lower bound given in Equation 4:

$$\mathbb{E}_{\substack{z \sim p(z) \\ s_2 \sim p(s|z) \\ s_1 \sim p(s|z)}} \left[\log(m(s_1, s_2, z))\right]$$

corresponds to increasing the probability mass on state $s_2$ (sampled in $p(s|z)$) starting from $s_1$ (also sampled in $p(s|z)$) for the skill $z$, independently of the probability mass of other skills on these two states. The maximization of this quantity does not account for the interactions between skills.

In Equation 5, we derive a new lower bound that reintroduces these interplays:

$$\underset{\substack{z \sim p(z) \\ s_2 \sim p(s|z) \\ s_1 \sim p(s|z)}}{\mathbb{E}} \left[\log \left(\frac{m(s_1, s_2, z)}{1 + \sum_{z' \in \mathcal{Z}} m(s_1, s_2, z')}\right)\right]$$

In this formulation, the interplays between skills are captured in the denominator of the log term. To maximize this log expression, the denominator must be minimal. This occurs when the probability of transitioning to state $s_2$ from state $s_1$ (which were sampled using skill $z$) is minimal for all other skills $z'$.

To conclude, the term "natural interplays" refers to whether the relationships and interactions between skills are preserved in the maximization process of these expectations. We propose to include the above reasoning as an explanation of the phrase in a new short appendix section.

Eigenoptions in related work:

In the study of Eigenoptions, the authors derive a way to decompose a given task into different options for which they propose automatic learning. We agree that the study could be relevant in the related work subsection "Successor features for MI maximization". We will include the following reference [1] in the final version of the paper.

Statistical Analysis of the Results:

We thank the reviewer for their suggestions and will modify the results in the final section. Specifically, we will use Standard Error and a Paired t-test in our final results. Below, we include p values using the paired t-test, which we will add to Appendix B. We note that the significance indication in Table 1 does not change when using the paired t-test; LEADS is significantly superior to all other methods on the Umaze, Hard, and Fetch Slide environments, and no other method significantly outperforms all other methods.

Hyperparameter tuning:

We propose the inclusion of the following statement in the article: "Hyperparameters were determined through manual testing on the Hard-Maze environment. Resulting hyperparameters are displayed in Table 3. Default values from the literature were used as hyperparameters for other methods." We include hyperparameters of all methods in the included code (https://anonymous.4open.science/r/LDS-BE6B).

Figure 1 clarification:

In the Figure 1, only the states denoted by the skills are reachable. We propose making the unreachable states gray and indicating this in the figure explanation.

Clarification on the Introduction of the First Bound and its Benefits:

We believe the reviewer refers to the use of Jensen's inequality to derive the first lower bound in Equation 4. The mutual information can be defined as: $\mathbb{E}_{(s_2, z) \sim p(s, z)} \left[ \log \left( \mathbb{E}s_1 \left[ m(s_1, s_2, z) \right] \right) \right]$, with $s_1 \sim p(s_1|z)$.

Suppose we wish to maximize this quantity using SGD. For each sample $(s_2,z)$, we need to compute the expectation $\mathbb{E}_{s_1 \sim p(s|z)} \left( m(s_1,s_2,z) \right)$ (because the log is not linear). In practice, this is computationally expensive because it requires sampling a mini-batch of $s_1$ states for every single $s_2$ state. Instead, we use Jensen's inequality, leveraging the concavity of the log function to derive the lower bound:

$$\mathbb{E}_{\substack{z \sim p(z) \\ s_2 \sim p(s|z) \\ s_1 \sim p(s|z)}}\left[\log(m(s_1,s_2,z))\right]$$

Maximizing this new quantity can be done with SGD by sampling three independent mini-batches at each iteration. We propose including the above explanation in a new appendix section that expands on section 3.1

[1] Machado, M. C., Rosenbaum, C., Guo, X., Liu, M., Tesauro, G., & Campbell, M. (2018, February). Eigenoption Discovery through the Deep Successor Representation. In International Conference on Learning Representations.