ComSD: Balancing Behavioral Quality and Diversity in Unsupervised Skill Discovery

We would like to thank you for your careful reading and expert review. We will answer your questions one by one.

For weakness:

Q1. original version of CIC

We agree that the estimator employed by the original CIC is similar to our contrastive estimator. But they still differ in the use of negative pairs, and our contrastive estimator is proven to be more accurate when the skill vector $z$ is sampled by a uniform distribution. In addition, CIC finally gives up this estimator in their final version and conducts corresponding experiments, which means they also found it harmful for exploration but didn’t solve it. The diversity hurt brought by giving up the conditioned entropy estimator is also ignored by CIC. Moreover, we also notice that CIC authors made experimental mistakes on contrastive estimator in first version (see their ICLR 2022 rebuttal). They actually didn't employ this estimator in their experiments and then totally gave up this estimator in their methodology. In this situation, we think (i) employing a contrastive estimator can be regarded as a contribution, and (ii) successfully incorporating a more accurate estimator through a novel weighting algorithm for exploration-exploitation balance is meaningful. It is a simple but effective method, like CIC and APS. (CIC actually enhances APT reward with a state-skill contrastive encoder and APS actually augments APT reward with a successor feature.) In addition, our evaluation experiments are also much more comprehensive and reasonable than CIC, which could help future works on skill evaluations.

Q2. Additional experiments with ComSD w/o SMW

Actually, we have conducted similar experiments (compared with ComSD w/o SMW) in our ablation study on numerical results and skill analysis. As mentioned in Section 3.3, the intervention of exploitation reward simultaneously improves behavioral diversity and reduces the activity of learned skills, which is also observed in experiments by CIC. This means a tuned constant $\alpha$ cannot achieve a balance between diversity and quality, which explains why CIC gives up contrastive estimator and motivates our SMW.

Q3. Mismatch of numerical results

These results come from BeCL. Concretely, for the results of DIAYN, APS, SMM and CIC, we run their official implementation and choose the similar results reported from URLB, CIC, and BeCL papers. Actually, we also notice that there are mismatches between these papers.

For minor issues:

1. Thanks for your correction, and we will revise it! We want this citation to be a parenthesis following the method name, which seems to cause confusion.

2. Thanks for your correction, and we will revise it!

For questions:

Q. Why not use a small and fixed $\beta$

As mentioned in Section 3.3, we use the exploitation reward to improve behavioral diversity which is ignored in CIC. However, the naive intervention of exploitation reward simultaneously improves behavioral diversity and reduces the activity of learned skills. SMW is designed to maintain this diversity improvement while alleviating the exploration hurt. If we use a fixed low coefficient ($\beta = w_{low} = 0$), the intrinsic reward $r_{ComSD} = r_{exploration}+\beta(z)\cdot \alpha r_{exploitation}$ degenerates into $r_{ComSD} = r_{exploration} = r_{CIC}$, i.e., ComSD actually degenerates into CIC. The behavioral diversity will be ignored again and the skill combination performance will drop severely. See section 4.2 for skill combination performance and see Appendix D for visualized skills of CIC and ComSD. If only skill finetuning evaluation is considered (i.e., the behavioral diversity is ignored like that in CIC), the SMW is of no use because it's designed for diversity-and-exploration balance. SMW enables ComSD to achieve competitive performance on both adaptation tasks simultaneously.

References

CIC Unsupervised reinforcement learning with contrastive intrinsic control. NeurIPS 2022

BeCL Behavior contrastive learning for unsupervised skill discovery. ICML 2023

URLB Urlb: Unsupervised reinforcement learning benchmark. NeurIPS 2021

CSD Controllability-Aware Unsupervised Skill Discovery. ICML 2023

APS Aps: Active pretraining with successor features. ICML 2021

DIAYN Diversity is all you need: Learning skills without a reward function. ICLR 2018

SMM Efficient exploration via state marginal matching. arxiv 2019

We would like to thank you for your careful reading and detailed reviews. We will answer your questions one by one.

Q1: SMW motivation.

As mentioned in Section 3.3, the intervention of exploitation reward simultaneously improves behavioral diversity and reduces the activity of learned skills. The larger the exploitation coefficient $\alpha$, the greater the skill diversity but the lower the behavioral activity (lazier exploration). With the proposed dynamic weight $\beta$ (SMW), the skill space is actually divided into different regions with different learning objectives. In the high-exploitation skill space ($flag(z)$ is large), the agents pay more attention to behavioral diversity, while in low-exploitation skill space ($flag(z)$ is low), the agents try their best to explore without considering diversity. In the middle interval ($flag(z)$ between $f_{low}$ and $f_{high}$), the agents can learn skills at different activity levels (corresponding to different $\beta$ value). With SMW, the agent takes both diversity and exploration into account, learning diverse skills at different activity levels.

We will explain our motivation more detailedly in our revision, thanks!

Q2: Difference from CIC.

I’m sorry there’s a typo in Section 4.1 causing a great misunderstanding. Thank you for pointing this out. The correct statement is "For CIC, ComSD follows it in state entropy estimation but first proposes to employ contrastive results for explicit conditioned entropy maximization." Concretely, CIC only uses a single exploration reward (for state entropy estimation) but doesn’t design an exploitation reward (for conditioned entropy estimation), i.e., CIC doesn’t explicitly maximize conditioned entropy by RL. The intrinsic reward of CIC is: $r_{CIC} = r_{exploration}$. By contrast, ComSD employs contrastive exploitation reward to estimate conditioned entropy and uses a dynamic weighting algorithm to alleviate its impact on dynamic behaviors. Its intrinsic reward is: $r_{ComSD} = r_{exploration}+\beta(z)\cdot \alpha r_{exploitation}$.

Q3: non-technical expression

Thanks for your suggestions! We will modify our expression.

References

CIC Unsupervised reinforcement learning with contrastive intrinsic control. NeurIPS 2022

For minor questions and suggestions:

1. Thanks for your careful reading, but in our settings, when $flag(z)$ equals $f_{high}$, the beta is indeed $w_{high}$.

2. Thanks for your correction, and we will revise it!

3. To reduce instability, we pre-train several skill agents and conduct several adaptation experiments. For example, we pre-train two walker agents with different seeds. For each agent, we train three meta-controllers with different seeds.

References

CIC Unsupervised reinforcement learning with contrastive intrinsic control. NeurIPS 2022

BeCL Behavior contrastive learning for unsupervised skill discovery. ICML 2023

URLB Urlb: Unsupervised reinforcement learning benchmark. NeurIPS 2021

CSD Controllability-Aware Unsupervised Skill Discovery. ICML 2023

APS Aps: Active pretraining with successor features. ICML 2021

DADS Dynamics-aware unsupervised discovery of skills. ICLR 2020

We would like to thank you for your careful reading and detailed suggestions. We will answer your questions one by one. We will also revise our paper according to your suggestions (e.g., more intuitive motivation and a detailed explanation of SMW).

For weakness:

Q1: SMW motivation.

As mentioned in Section 3.3, the intervention of exploitation reward simultaneously improves behavioral diversity and reduces the activity of learned skills. The larger the exploitation coefficient $\alpha$, the greater the skill diversity but the lower the behavioral activity (lazier exploration). With the proposed dynamic weight $\beta$ (SMW), the skill space is actually divided into different regions with different learning objectives. In the high-exploitation skill space ($flag(z)$ is large), the agents pay more attention to behavioral diversity, while in low-exploitation skill space ($flag(z)$ is low), the agents try their best to explore without considering diversity. In the middle interval ($flag(z)$ between $f_{low}$ and $f_{high}$), the agents can learn skills at different activity levels (corresponding to different $\beta$ value). With SMW, the agent takes both diversity and exploration into account, learning diverse skills at different activity levels.

Q2: Difference from CIC.

I’m sorry there’s a typo causing a great misunderstanding. Thank you for pointing this out. The correct statement is "For CIC, ComSD follows it in state entropy estimation but first proposes to employ contrastive results for explicit conditioned entropy maximization." Concretely, CIC only uses a single exploration reward (for state entropy estimation) but doesn’t design an exploitation reward (for conditioned entropy estimation), i.e., CIC doesn’t explicitly maximize conditioned entropy by RL. The intrinsic reward of CIC is: $r_{CIC} = r_{exploration}$. By contrast, ComSD employs contrastive exploitation reward to estimate conditioned entropy and uses a dynamic weighting algorithm to alleviate its impact on dynamic behaviors. Its intrinsic reward is: $r_{ComSD} = r_{exploration}+\beta(z)\cdot \alpha r_{exploitation}$

Q3: CSD and benchmark

• We also notice that CSD says 'MI-based methods often end up discovering 'static' skills with limited state coverage'. But we don’t completely agree with this view. In practice, MI-based methods can produce extremely continously dynamic behaviors in challenging locomotion (CIC) and enable efficient adaptation across different fields (APS,DADS).
• DMC (URLB) is challenging for unsupervised skill discovery. CIC found that DMC is much more difficult than gym (used in CSD) due to the lack of extrinsic signals on agent balance. According to CIC, most methods that succeed in openAI gym fail in DMC. Across methods that have been evaluated on DMC, only CIC and our ComSD can generate continuously dynamic robot behaviors.
• We have also considered manipulation tasks. However, we find that the objectives of robot skill discovery and manipulation skill discovery are totally different. In challenging locomotion (DMC), we want the agent to learn different behaviors (postures) at different activity levels. Both dynamic behaviors and static postures are useful, which motivates our method design. The learned skills are hard to judge only by moving distance. However, in manipulation, the objective is to move the arm or small objects as far as possible. Evaluation of skills is also limited to movement distance and directions (Only moving far away is considered a good skill, while direction is the only indicator of distinguishing skills), which means many interesting behaviors are ignored. For example, continuously moving behaviors (like arms moving up and down) and moving at short range (or moving a different distance) are regarded as useless skills in CSD's experimental settings. The ability of discovering continuously dynamic behavior and static postures can't be well evaluated in current manipulation experimental settings.

Q4: Comparison with LSD and CSD

We agree with you and will consider more comparisons. However, we did not limit the comparison to MI-based methods on purpose. Actually, we compare our method with current SOTA methods (CIC, BeCL) on DMC (URLB) and employ all skill discovery baselines used in their papers. The comparison with five popular baselines, including SOTA, is enough to support our conclusion. In addition, changing the benchmark for CSD leads to unfair comparison (e.g., we don’t have proper hyper-parameter settings). This may explain why CSD was also not compared with CIC and APS. Considering the reasons above, we didn’t make a comparison with LSD and CSD, which follows CIC and BeCL.

For minor questions and suggestions:

continued.

Thanks for your detailed review and suggestions! We appreciate your affirmation and will answer your questions one by one.

For main weakness:

Q: Additional related work DISk

Thanks for the additional reference. Surely we missed an important related work, and we will cite it in our background! However, we can’t agree that we make an incomplete comparison because we follow recent advanced works (including SOTA) in baseline selection. Concrete reasons are shown as the following:

• First, we have compared our method with CIC and BeCL, the current SOTA methods on DMC (URLB). We also employ all unsupervised skill discovery baselines used in their papers. Comparisons with five popular baselines, including SOTA, are enough to support the conclusion.
• Second, DISk provides a new perspective on skill discovery but is not suitable for direct comparison here. The main reason is that they set different policy networks for different skills, while all methods in our paper only use one skill-conditioned agent. In addition, DISk is evaluated on openAI gym but not on DMC. It is difficult to select the appropriate number of skills for DISk on a new benchmark and ensure a fair comparison without proper hyper-parameters. These may explain why current SOTAs (CIC and BeCL) on DMC also don’t employ DISk as a baseline and why DISk is not compared with APS and SMM.

For other issues:

Q1: The notation r_exploration and r_exploitation look quite similar to each other.

Thanks for your advice. We will revise the notation!

Q2: SMW motivation.

As mentioned in Section 3.3, the intervention of exploitation reward simultaneously improves behavioral diversity and reduces the activity of learned skills. The larger the exploitation coefficient $\alpha$, the greater the skill diversity but the lower the behavioral activity (lazier exploration). With the proposed dynamic weight $\beta$ (SMW), the skill space is actually divided into different regions with different learning objectives. In the high-exploitation skill space ($flag(z)$ is large), the agents pay more attention to behavioral diversity, while in low-exploitation skill space ($flag(z)$ is low), the agents try their best to explore without considering diversity. In the middle interval ($flag(z)$ between $f_{low}$ and $f_{high}$), the agents can learn skills at different activity levels (corresponding to different $\beta$ value). With SMW, the agent takes both diversity and exploration into account, learning diverse skills at different activity levels.

Q3: pre-training steps.

We follow previous advanced work (CIC, BeCL) and benchmark (URLB) on training step settings.

References

DISk One After Another: Learning Incremental Skills for a Changing World. ICLR 2022

CIC Unsupervised reinforcement learning with contrastive intrinsic control. NeurIPS 2022

BeCL Behavior contrastive learning for unsupervised skill discovery. ICML 2023

URLB Urlb: Unsupervised reinforcement learning benchmark. NeurIPS 2021

APS Aps: Active pretraining with successor features. ICML 2021

SMM Efficient exploration via state marginal matching. arxiv 2019