Efficient Skill Discovery via Regret-Aware Optimization

Q1: Collapsing Concern

Thank you for raising this insightful question. Our method is specifically designed to prevent skill collapse. As shown in Eq. (15), the skills are maintained within a population $P_z$. We address the diversity concern (i.e., avoiding convergence to a single point) from three perspectives:

1. KL divergence constraint (Eq. 16):
   We enforce a divergence between the newly generated skill distribution $\pi_{\theta_2}$ and existing skills in $P_z$ using KL divergence. This ensures new skills remain distinct.
2. Minimum variance and overlap check:
   The sampler $\pi_{\theta_2}$ (Gaussian) is constrained to maintain a minimum std (1e-1). We also check for distributional overlap before adding new skills, preserving diversity.
3. Empirical validation (Figure 7):
   Figure 7 shows a consistent decrease in average regret, mainly driven by SAC. As SAC improves, $\pi_{\theta_2}$ keeps adapting, avoiding collapse to a single mode.

Please let us know if further clarification is needed!

Q2: State-dependent Skill $z_{\text{updated}}$

Thank you for your careful observation.

• When $z$ is fixed across time, Eq. (4) tends to force straight-line trajectories. In skill-asymmetric environments, skills often share key states, making this problematic (Figure 5).
• Allowing $z_{\text{updated}}$ to vary with $\phi$ and time $t$ enables distinct trajectory curves, even with shared segments, better capturing behavioral nuances.

Advantages:
Improves skill diversity (Figures 5 & 6), better covers complex regions (e.g., maze corners), and outperforms METRA in skill expressiveness.

Disadvantages:
Increases learning complexity, as seen in Figure 3(a), where RSD is slightly less efficient in simpler environments.

Discussion on Related Work

We appreciate the reviewer highlighting TLDR[Bae et al., 2024], which is cited in Lines 303, 329, and Appendix A.1 (L608). Our method focuses on skill sampling scheme, whereas TLDR relies on RND and KNN-based exploration strategies. We will clarify this in the revised related work section.

Weakness Response

1. Clarity of Policies $\pi_{\theta_1}$ and $\pi_{\theta_2}$

We revised the start of Section 3 using a bullet-point format to clearly distinguish. We believe this change helps readers immediately grasp the roles of each policy.

2. Organization of Section 3

Our intention was to follow a “global-to-local” structure for clarity.

• Base reward is defined earlier in Preliminaries (Eq. 4).
• We were concerned that Keeping Section 3.1 ahead preserves motivation for why $\pi_{\theta_1}$ minimizes and $\pi_{\theta_2}$ maximizes regret.

3. Why $\pi_{\theta_1}$ Minimizes Regret (Eq. 8)

Great point — we clarify:

• Regret is estimated via Eq. 7.
• If $\pi_{\theta_1}$ has converged for skill $z$, regret (value improvement) is near zero.
• Thus, minimizing regret implies convergence under that skill.

Note: $\pi_{\theta_1}$ is still trained with standard SAC loss (Eq. 13, Alg. 1 line 10).

4. Intuition Behind Eq. (11)

For example, if $\||\phi(s_T)\|| \leq 1$ and we set $\phi(s_0) = 0$, then by telescoping: $\|| \sum_t^{T-1} \phi(s_{t+1}) - \phi(s_t) \|| \leq 1$. To simplify, we enforce Eq. (11), and one solution is: $\|| \phi(s_{t+1}) - \phi(s_t) \|| \leq 1/T$.

5. Additional Results

We added experiments in the Kitchen environment (pixel observations, robotic control), following METRA’s setup and reporting success out of 7 tasks.

As shown in the tables below, our method shows both higher sample efficiency and stronger final performance. Interestingly, at dim = 32, our method exhibits faster efficiency gains (e.g., Δ@300k = +0.51) compared to dim = 24 (Δ@300k = +0.36), suggesting that our efficient approach benefits more from a larger latent space. This trend is even clearer when visualized in the learning curves.

Performance @ Skill Dim = 24

Performance @ Skill Dim = 32

6. Grammatical Errors

We have addressed the mentioned issues. For the Lipschitz constraint, METRA uses L2 in the paper, though their code supports L1. Line 193 has also been revised for rigor.

Thank you for your time and for recognizing our work! Due to space constraints, some explanations have been shortened — please feel free to reach out if any clarification is needed.

We thank the reviewer for their feedback.

Q1: Unique Coordinates Metric

1. Origin of the Metric: The metric is inspired by the Policy State Coverage measure introduced in the METRA paper (Section 5.3), where it was used to evaluate the spatial coverage of skill policies—referred to as x-y Coverage in the Maze environment. In our work, we rename it to Unique Coordinates to improve specificity, as noted in Section 4.1 (line 312).
2. Definition: The metric essentially counts the number of unique 2D coordinates visited by the agent during skill executions. We extract the x and y coordinates from states (hence the name Coordinates), then count the number of distinct positions (hence using Unique).
3. Computation: For each algorithm:
   • Extract trajectory x-y states for each skill;
   • Discretize the x-y values; (e.g., rounded to integers);
   • Count the number of unique coordinate pairs across all rollouts.
4. Significance: This metric reflects how diversely the skills explore the state space. As supported in prior work, reaching distinct regions of the environment implies higher skill diversity(Figure 4).

We will include this clarification in the revised version.

---

Q2: Additional Experiments

1. Kitchen: It uses pixel-based observations and involves manipulation tasks, making it different from Ant-based settings.
2. Motivation: Following (issue), pixel-based tasks are indeed computationally demanding. We prioritized the Kitchen environment as it is both challenging and practically meaningful under time constraints.
3. Experimental Design: Following METRA’s setup, we report the number of completed tasks (7 in total) using mean ± std over three seeds {0,2,4}.

Performance @ Skill Dim = 24

Performance @ Skill Dim = 32

Our method shows both higher sample efficiency and stronger final performance. At dim = 32, our method exhibits faster efficiency gains (e.g., Δ@300k = +0.51) compared to dim = 24 (Δ@300k = +0.36), suggesting that our efficient approach benefits more from a larger latent space. This trend is even clearer when visualized in the learning curves.

---

Q3: Hyperparameter Sensitivity and Stability

1. Ablation: We include ablations (Appendix E) for critical hyperparameters, such as the regret window size (window) and regularization weights (alpha_1, alpha_2).
2. Tuning: We argue that hyperparameter tuning in our method is guided rather than purely search-based, as they primarily influence skill diversity. For instance, the KL divergence term in Eq. (16) provides a direct and interpretable signal for adjusting alpha_1.
3. Empirical Settings:
   • Ant/Maze: Used a fixed setting (alpha_1=5, alpha_2=1, window=15) across experiments.
   • Kitchen: Used lighter settings (alpha_1=1, alpha_2=0, window=8) due to inherent diversity.

While we also considered adaptive schemes (e.g., Lagrange multipliers), we avoided them to maintain a lightweight design.

---

Q4: Related Work

1. Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills
   This paper is cited in our Appendix B. It shares similar experimental setups with ours. However, we did not include it as a primary baseline because it focuses on simpler 2D navigation tasks and does not scale easily to high-dimensional settings.
2. Behavior Contrastive Learning for Unsupervised Skill Discovery
   Thank you for the recommendation. Despite conceptual similarity, it lacks demonstration in high-dimensional environments. We also tried contrastive losses but observed unstable learning, potentially due to conflicting gradients in METRA. We acknowledge this as a promising direction.
3. Choreographer: Learning and Adapting Skills in Imagination
   An insightful suggestion. The key differences are:
   • It's a model-based RL approach, which increases sample efficiency but at a higher computational cost;
   • It's exploration-agnostic, relying on separate exploration strategies (as noted in their Appendix A), whereas our method integrates exploration directly into the skill learning process.

Although these works tackle skill discovery, we address different challenges. We will add these references to our revised Related Work section.

We hope these resolve your concern!

We sincerely thank the reviewer for the thoughtful and constructive feedback. Below we address each point in detail.

Q1. Statistical Reporting

We appreciate the reviewer’s emphasis on statistical rigor.
All experiments were conducted using five independent random seeds: {0, 2, 4, 8, 16}.

• In Figure 3, the bold lines indicate the mean, and the shaded areas represent the standard deviation across these runs.
• In Table 1, we originally reported the mean performance only. In the revised version, we will add uncertainty measures to each entry. For example, in the Maze2d-large environment:

RSD (ours):     FD↓ = 0.535 ± 0.152, AR↑ = 0.811 ± 0.046
METRA-d:        FD↓ = 0.658 ± 0.154, AR↑ = 0.782 ± 0.043

These results demonstrate that RSD achieves competitive performance with comparable variance, supporting the robustness of our method.

Q2. Applicability Beyond METRA

Thank you for the insightful question regarding generality. We certainlly plan to explore this framework with other models (for example in VLA) in future work.

Our claim that RSD is a general framework refers to the min-max optimization formulation in Equation 8, which is algorithm-agnostic and, in principle, applicable to a wide range of unsupervised skill discovery (USD) methods.

In this work, we chose to implement RSD on top of METRA rather than LSD. This is because METRA, an improved version of LSD, provides key theoretical properties (Equation 3) that we leverage. In Section 3.2 (Equations 2 and 13), we introduce targeted modifications to enhance performance in skill-asymmetric environments.

Q3. Epochs in Figure 6

We appreciate the reviewer’s comment regarding Figure 6.

The relationship between epochs and timesteps is reflected in Algorithm 1 (lines 2 and 5):

timesteps = epoch × num_trajectories × max_length

As described in Appendix C.1, we use:

• num_trajectories = 16
• max_length = 300

So the approximate mapping is:

4000 epochs   → 2e7 timesteps  
8000 epochs   → 4e7 timesteps  
14000 epochs  → 7e7 timesteps  
18000 epochs  → 9e7 timesteps

In the revised version, we will redraw Figure 6 with explicit timestep annotations and improve color contrast, as suggested, to improve readability.

Q4. Timestep Scales in Figures 3 and 7

Thank you for noting the difference in timestep scales.

This design choice is intentional, as the two figures serve different purposes:

• Figure 3 highlights early-phase learning efficiency (shown up to 5e6 timesteps), with focus on the 0–2e6 range.
• Figure 7 illustrates long-term convergence and stability, plotted up to 1e7 timesteps.

To avoid confusion, we will explicitly clarify this in the caption of Figure 7 in the final version.

Additional Comments

1. Ablation Study Referencing

We agree that the ablation study contains important insights. In the revised version, we will:

• Explicitly reference Appendix E in the main text
• Summarize key findings to highlight contributions of individual components

2. Missing AntMaze2D in Appendix D

Thank you for catching this. Appendix D contains visualized results for experiments Table 1. We will:

• Clarify this reference in the main text
• Add explanatory notes in Appendix D to improve readability

3. Zero-shot Evaluation Concern

We appreciate your thoughtful concern. The goal of our zero-shot experiment is to evaluate practical utility.
Indeed, RSD achieves superior zero-shot performance partly due to higher coverage — which we see as a desired property.
The improved zero-shot success rate is a direct consequence of more effective skill discovery, which we believe substantiates our central claim.

Typos and Presentation Issues

We thank the reviewer for the attention to detail. All identified typos and presentation issues have been carefully corrected in the revised version to improve clarity and presentation quality.

Once again, we sincerely appreciate the reviewer’s time and valuable, constructive feedback. We believe it has significantly enhanced the clarity, rigor, and overall quality of our paper.

Q1: Motivation of "Policy Strength"

1. Definition:

   The term policy strength, derived from "the strength of the policy", refers to the capability of a policy to accomplish its designated objective—corresponding to a specific skill in our context.

2. Why we use "policy strength":

   In this work, we adopt the term policy strength to concisely describe the capability of the policy under skill-conditioned training. More specifically, we are interested in how well the policy can be optimized given a particular skill, and whether it has converged, or how far it is from convergence. This perspective helps us assess whether a skill has been sufficiently explored or still requires further training attention.

3. Motivation and necessity:

   Initially, we considered using phrases like "the performance of the policy", but we were concerned that performance might be misinterpreted as referring to the cumulative reward. This can be misleading in unsupervised skill discovery, where intrinsic rewards are not always directly comparable between different skills. In contrast, policy strength focuses on the optimization dynamics and convergence behavior of the policy, thus avoiding ambiguity tied to reward metrics.

We acknowledge the term may be unconventional and will revise the introduction to clearly define it.

---

Q2: More Experimental Evaluation

Thank you for encouraging broader validation.

1. Choice of Maze Environment

   We extend METRA’s evaluation by including Maze, a more challenging setup for skill discovery [1]. As seen in Figure 5, METRA underperforms here due to the skill-asymmetry issue, which our method specifically targets.Additionally, we assess zero-shot performance on downstream tasks, following standard protocol.

2. Evaluation on the Kitchen Environment

   In response to your concern, we additionally include experimental results on the Kitchen:

   • Kitchen Environment: Kitchen involves pixel-based observations and more complex robot-manipulation tasks, which are evaluated in METRA.
   • Why Kitchen: As noted in this issue, pixel-based tasks are indeed computationally demanding. We chose Kitchen for its practical challenge and feasibility under limited time.
   • Experimental Setup: Following METRA’s setup, we measure the number of completed tasks (7 total) using mean ± std over three seeds {0, 2, 4}. We evaluate across different training steps and skill dimensions (dim=24, 32).

     Performance @ Skill Dim = 24

     Model	Step = 100k	Step=200k	Step = 300k	Step = 400k	Δ@400k
     METRA	2.72±0.19	3.20±0.27	3.93±0.67	3.94±0.36
     Ours	2.41±0.02	3.61±0.02	4.29±0.11	5.08±0.45	+1.14

     Performance @ Skill Dim = 32

     Model	Step = 100k	Step = 200k	Step = 300k	Step = 400k	Δ@400k
     METRA	3.21±0.71	3.66±0.69	3.83±0.90	3.99±0.94
     Ours	3.51±0.23	3.91±0.43	4.34±0.39	4.55±0.35	+0.56

   • Results Summary: Our method outperforms METRA in both sample efficiency and final performance. At skill dim = 32, our method exhibits greater gains with training progress (e.g., Δ@300k = +0.51 vs. +0.36 at dim=24). This trend is even clearer when visualized in the curves.

---

Q3: Baseline Selection

1. Baseline Selection

   Our method is built upon METRA’s framework. Our primary contribution lies in enhancing the sampling efficiency by incorporating ideas from Prioritized Level Replay (PLR), enabling more effective skill discovery within the METRA setup.

2. Why we do not compare against intrinsic reward-based methods

   As discussed in Appendix A.1, we chose not to include other intrinsic reward methods for the following reasons:

   • RND-based methods improve exploration but are not tailored for skill discovery.
   • KNN-based methods are limited in high-dimensional representation space.

   Crucially, our focus is different: we optimize skill distribution during training using a regret-driven min-max formulation, unlike the uniform sampling in METRA and related work.

We hope this explains our baseline decisions.

---

Q4: Define $\phi$ on Line 131

We will add the following definition to improve clarity:

A representation function $\phi: \mathcal{S} \rightarrow \mathbb{R}^d$, which maps states into a skill-aligned latent space that shares the same dimensionality as the skill variable $z \in Z$.

---

[1] Campos, Víctor, et al. "Explore, discover and learn: Unsupervised discovery of state-covering skills." International conference on machine learning. PMLR, 2020.