First and foremost, we would like to thank you for your thorough review and insightful comments on our work. Your feedback is invaluable and we appreciate the opportunity to clarify and improve upon the points you have raised, especially about the choice of baselines and about the missing references in the related work.

Following your feedback we have updated our claims to reflect better our contribution and improved our discussion of the algorithms we chose as baselines and why other approaches were not included. We agree with the reviewer that the pointed references are relevant for this contribution and included them in the paper.

One of the main claims of the paper is that SCOPE "extends the traditional concept of goal to trajectory-based skill." This is simply wrong and clearly misrepresents a large body of literature on goal-conditioned RL and HRL in general. The claim that the proposed method extends the traditional concept of goal to trajectory level skill seems to me like a fundamental misunderstanding of what goals can represent in HRL. The idea that goals are associated with a particular state (meaning x,y,z positions) is a very narrow point of view of the numerous ways they are instantiated in the literature. Goals essentially depend on the goal space, which can encode any kind information we could imagine, including temporally extended features such as speed or feet contact rate.

Thank you for your insightful comments regarding our claim, we appreciate the opportunity to clarify our stance.

• We acknowledge that trajectory-based goals exist in the literature, and we certainly did not intend to imply that trajectory-based goals are non-existent in the field.
• Rather, we wanted to draw an analogy between the traditional concept of goal, where goals are associated with specific states [1, 2, 3] and our method.
• In GCRL, with this traditional concept of goal, the objective is to achieve a desired goal $g$, i.e. to reach a state that is in the subset $\\{\left. s \in \mathcal{S} \right\vert \phi(s) = g\\}$. In contrast, our objective is to execute a desired skill $z$, i.e. to sample a trajectory that is in the subset $\\{\left. (s\_t)\_{t \geq 0} \in \mathcal{S}^\mathbb{N} \right\vert \lim\_{T \rightarrow \infty} \frac{1}{T} \sum\_{t=0}^{T-1} \phi\_t = z\\}$ (see Section 3 for a longer discussion). Therefore, SCOPA tries to reach a goal in average over a trajectory rather than at state level.
• However, we agree that the sentence is misleading, so we have reformulated the claims about trajectory-based goals in the paper and we have also reformulated a paragraph in Section 3. We hope that this addresses your concerns.

[1] Goal-Conditioned Reinforcement Learning: Problems and Solutions. Liu et al. 2022
[2] Universal Value Function Approximators. Schaul et al. 2015
[3] Hindsight Experience Replay. Andrychowicz et al. 2017

The authors compare to a few HRL algorithms that learn skills, these algorithms have their rewards defined through DIAYN. This is an outdated baseline that has been beaten numerous times in the literature. The algorithm is compared to a variety of algorithms, but the HRL baselines essentially build their reward on DIAYN, which is not a good algorithm anymore. A recent state-of-the-art algorithm building on a quantity closely connected to the successor features is the idea of Laplacian based options [1]. A comparison to such a strong algorithm would be necessary to really attest to the algorithm's performance.

We understand your concerns, but in the present work, we do not directly compare our approach to DIAYN, but to state-of-the-art algorithms that build on top of DIAYN and that are way closer to our work. Thank you for the reference, however, the code for [1] is not available, so we could not compare to this baseline.

Why refer to DOMiNO as Reverse SMERL? DOMiNO presents strong results using methods that do not rely on DIAYN, this would be a much better comparison.

We agree with the reviewer that DOMiNO and Reverse SMERL are two different algorithms and we did not intend to say otherwise. To make this clear in the paper, we have added a footnote the first time we mention Reverse SMERL: "originally introduced as a baseline of DOMiNO".

We originally considered including DOMiNO in the baselines. However, DOMiNO is not designed to learn pre-defined skills like DCG-ME or like SMERL/Reverse SMERL can do by using DIAYN+prior. Consequently, for DOMiNO the notion of "distance to skill" does not exist and thus, the policies cannot be evaluated using traditional QD metrics.

"DOMiNO learns policies that are not skill-conditioned, hence the skills are not controllable." This is not right. Conditioning a policy on a N possible goal is the same as to learning N different discrete skills. It is simply a question of architecture.

We agree with the reviewer that this sentence needs to be updated to improve clarity. The new sentence is: Specifically, DOMiNO trains $N$ policies that are different from one another, but these policies are not trained to execute pre-defined skills. As originally stated, DOMiNO is not designed to control skills, neither this can be done without changing the algorithm fundamentals.

Our evaluation metrics expect the policies to take as input the skill $\boldsymbol{z}$, thus all our baselines return policies which (1) can take skills $\boldsymbol{z}$ as input, and (2) are trained to execute those skills. DOMiNO does not verify that property (see answer above).

Indeed, DOMiNO's policies are trained to maximise the distance in the state-action occupancy of other policies. Thus, DOMiNO returns $N$ policies exhibiting diverse behaviours; but those policies are not trained to follow any skill. Hence, DOMiNO does not solve the same problem as SCOPA and we believe that a comparison would be unfair to DOMiNO.

Solving Problem P1 amounts to maximizing the return while executing a desired skill" This is in essence the main goal of this paper [3].

Thank you for the reference, we have added a citation in the related works section.

More details on Figure 5 would be needed.

We added more details to the caption of Figure 5.

Despite building skills on the SF, no mention of [4] is made throughout the paper, which is closely related.

That is indeed very related even though [4] learns its features, while ours are pre-defined. Thank you very much, we have added a reference to [4] in the related work section of the new version.

"SF captures the expected future states visited under a given policy" This is not right, it is the expected future feature occupancy, not the state.

Thank you very much for spotting this mistake. We fixed it in the new version.

We once again thank you for your constructive critique. Your feedback has been instrumental in refining our paper, and we hope that our responses adequately address your concerns. We are committed to contributing valuable and clear research to the field, and your input has been essential in guiding our efforts towards this goal. Should you have any remaining doubts or concerns, please do not hesitate to share them with us. We will be more than happy to incorporate your feedback, as we did with the initial review, to further improve the quality of our work.

The qualitative results do not convey very effectively the diversity of the learned skills The qualitative experiments do not convince me that the algorithm learns diverse skills. Figure 2 is very poorly presented, how is the distance exactly calculated? What is the x axis? Why use a task dependent d_eval? It is very unclear how this exactly relates to diversity. Previous work presents diversity in a better way and such visualization would help the paper a lot.

The qualitative results are inspired by many published papers in different communities, such as Quality-Diversity [1, 2] and GCRL [3]. In our opinion, the best way to visualize the diversity of learned skills is to watch the videos on our website mentioned in the abstract.

We agree that distance and performance profiles may be difficult to interpret at first sight. Those are also inspired by previous works in the Quality-Diversity [2, 4] community and in the GCRL community [3]. They can be summarized as follows: for a score $x$ on the x-axis, the score profile shows the percentage of the skill space where the score is higher than $x$. Hence, the higher the distance profile, the closer the policy gets to the target skills. To make it easier to read, we added an illustration in Appendix F that shows how those distance and performance profiles are plotted, and how to interpret them. This new illustration is now referenced in the caption of Figure 2.

We also added the missing labels in Figure 2. Thank you very much for spotting this.

The idea of $d\_{eval}$ is simple. We intend to calculate the performance of our skill-conditioned policy. However, we want to measure the performance of our policy only on the skills that are successfully executed by the policy. For example, consider a robot whose objective is to minimize energy consumption, and where the skills are target velocities of the robot. If the policy is conditioned on the skill [5., 0.] m/s, but the policy does not move, then it achieves great performance (because it minimizes energy consumption). However, the skill [5., 0.] is not successfully executed. In that case, we want to ignore the performance.

In the end, $d\_{eval}$ is just a tool we use for plotting the performance results. It is used to filter out all the skills that the policy does not execute successfully before plotting the performance heatmaps and profiles. For each sampled skill $\\boldsymbol{z}$, we perform rollouts with $\pi(.|., \\boldsymbol{z})$ and measure the distance between $\boldsymbol{z}$ and $\frac{1}{T}\sum\_{t=1}^T$ $\boldsymbol{\phi}\_t$. If the distance is higher than $d_{eval}$, then the skill is not considered as "successfully executed", and we don't take its performance into account.

Across all tasks, $d_{eval}$ is chosen equal to $5\delta$; we tested other values from $\delta$ to $8\delta$, and the conclusions drawn by the plots remained the same. However for values of $d_{eval}$ below $3\delta$, the results were merely readable when presented in the paper: the performance profiles of most baselines get very close to 0, and their performance heatmaps are nearly empty, as $d_{eval}$ is too small. You can find the values of $d_{eval}$ given in Appendix C Table C.1.

[1] Robots that can adapt like animals, Cully et al. (2015)
[2] Benchmarking quality-diversity algorithms on neuroevolution for reinforcement learning, Flageat et al. (2022)
[3] Rapid locomotion via reinforcement learning, Margolis et al. (2022)
[4] Don't Bet on Luck Alone: Enhancing Behavioral Reproducibility of Quality-Diversity Solutions in Uncertain Domains, Grillotti et al. (2023)

How are the skills chosen exactly? Do they use the hand-defined features from Section 5.1 (feet contact, angle, ...)? How does the method avoid learning multiple times the same skill?

• The feature functions are user-defined for each task, see Section 5.1 and Appendix C for tasks details.
• The skill executed over a trajectory is defined as the average of the features. Therefore, the skill space corresponds to all the possible average of sequence of features.
• The target skills are uniformly sampled in the skill space as explained in Algorithm 1 and in Section 4.2.
• We could imagine better skill sampling strategies but this is beyond the scope of this work.

6. In (2), which is used to train the network, delta' is needed to compute the targets for the cross-entropy loss. I checked the SCOPA hyperparameters in the Appendix and I couldn't find the values for delta'. Is the need for delta' lost somewhere?

• We have reformulated the proposition as well as Section 4.1 to improve the clarity of the paper. Specifically, we have removed $\epsilon$ and $\delta^\prime$, and thus only the hyperparameter $\delta$ remains.
• The value of $\delta$ is task specific and you can find it in Appendix C.1.
• The heuristic we use to choose the value of $\delta$ is the side length of the skill space divided by 100.
• Keep in mind that $\delta$ represents the maximum acceptable error between the desired skill and the observed skill, and needs to be chosen by the user. For example, with the jump features, the skill space is $[0., 0.25]^2$ m, and we take $\delta = 0.0025 \text{m} = 2.5 \text{mm}$. With the angle features, the skill space is $]-\pi, \pi]$, and we take $\delta = 0.06$.

7. Typically in goal-conditioned RL, you have a goal, and the reward function that you optimize is associated to the goal. This paper states that the P1 generalizes goal-conditioned RL, but to me it is just a different objective. I.e., you still have a main task reward augmented by the skill/goal objective. Thoughts on this? Am I misunderstanding?

The statement that P1 generalizes GCRL was based on two arguments:

• First, if we remove the (optional) objective maximization in P1 (i.e. red term in Section 4.1), then we end up only with the constraint. The constraint is responsible for executing skills, and an analogy can be drawn between "executing a skill" in our work and "reaching a goal" in GCRL. In that sense, we stated that P1 generalizes GCRL, because we can include an (optional) objective maximization in addition to "executing a skill".
• Second, we can show that the feature space (or goal space) $\Phi$ is a subset of the skill space $\mathcal{Z}$. In that sense, we stated that P1 generalizes GCRL to trajectory-level goals. For example, the feet contact tasks illustrate that the feature space is finite equal to $\\{0, 1\\}^2$ whereas the skill space is infinite equal to $[0., 1.]^2$.

However, the arguments rely on an analogy, and we agree with you that P1 is just a different objective and that it is not a generalization in any rigorous sense. Therefore, we have reformulated the corresponding sentence and paragraph at the end of Section 3. The sentence now reads as follows: "Problem P1 amounts to maximizing the return while executing a desired skill. If we ignore the objective and only consider the constraint, we can show that our problem is related to GCRL."

We once again thank you for your constructive critique. Your feedback has been instrumental in refining our paper, and we hope that our responses adequately address your concerns. We are committed to contributing valuable and clear research to the field, and your input has been essential in guiding our efforts towards this goal. Should you have any remaining doubts or concerns, please do not hesitate to share them with us. We will be more than happy to incorporate your feedback, as we did with the initial review, to further improve the quality of our work.

4. In the experiment section, we state: "we combine with four different feature functions that we call feet contact, jump, velocity and angle." Where do these feature functions come from?

• In Section 5.1, we describe the tasks used in the experiments, and then in the four subsequent paragraphs we give additional information about the four feature functions and indicate where they come from, see more details below.
• Velocity feature function comes from GCRL literature [1]. The skills for velocity can be directly opposite to solving the task (going forward).
• Angle feature function comes from GCRL literature [1] and has been used in robotics to rotate a body in space to a target orientation. We wanted to push the boundaries of what is possible with the humanoid like moonwalking, running sideways, changing angle during rollout and so on. We believe that the dexterity coming from angle skills could definitely be useful in downstream tasks like soccer play, see bit.ly/scopa.
• Feet contact feature function comes from QD literature [3] and has been extensively studied to solve downstream robotics tasks.
• Jump feature function is a new feature function we introduce in this work for three reasons. First, the jump features present a significant challenge because unlike velocity, the jump features information is not directly part of the state of the agent and because the $\max$ operator makes the features function non-linear. Second, to achieve a jump skill $z = \frac{1}{T} \sum_{i=0}^{T-1} \phi_i = 10 \text{cm}$, the agent is forced to oscillate around that value because of gravity, showing the benefit to define skills as the average features instead of trying to reach the features 10cm at every time steps. Third, jumping is a useful skill for downstream tasks like jumping over hurdles.
• The feature functions are not learned in this work, but it has been shown that it is possible [4] and would be the subject of a follow-up work. Whether it is a limitation or not depends on the need of the user of the algorithm. If the goal of the user is to learn skills like jumping, then it is very useful to be able to provide the feature function directly. However, if the goal of the user is simply to learn skills to adapt to unforeseen situations like changes in friction, then it could be desirable to learn the feature function.

We have improved the clarity and readability of Section 5.1, thank you for pointing out missing information about the feature functions used in this work.

[1] Goal-Conditioned Reinforcement Learning: Problems and Solutions. Liu et al. 2022
[2] Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping. Mezghani et al. 2023
[3] Policy Gradient Assisted MAP-Elites. Nilsson et al. 2021
[4] Autonomous skill discovery with Quality-Diversity and Unsupervised Descriptors. Cully et al. 2019

5. Where does the delta in P2 come from? When you reformulate P2 into (1), where does the delta go?

• We relax the constraint from Problem P1 using the $L^2$ norm and $\delta$, a threshold that quantifies the maximum acceptable error between the desired skill and the observed skill. The goal is to transform the "equality constraint" from P1 into an "inequality constraint" that is less restrictive and easier to satisfy. Relaxing a constraint that way is a common trick in constrained optimization. We have made adjustements in the paper to improve the clarity on this point.
• When we reformulate P2 into Equation (1), $\delta$ goes into the Lagrange multiplier. At the end of Section 4.1, we give the objective of the Lagrange multiplier and you can see that $\delta$ appears in the expression. The general idea is that if the constraint is not satisfied (blue term greater than $\delta$), then the Lagrange multiplier will increase to put more weight on the constraint. On the contrary, if the constraint is satisfied (blue term less than $\delta$), then the Lagrange multiplier will decrease to put more weight on the objective.

First and foremost, we would like to thank you for your thorough review and insightful comments on our work. Your feedback is invaluable and we appreciate the opportunity to improve upon the points you have raised, especially about the clarity of the paper related to the definition of skills and to the method.

1. Will the code be released? While the hyperparameters are available, I do have reproducibility concerns. There are many moving parts (e.g., learning the model, the constrained optimization, etc.) that to the naked eye seems sensitive to implementation details and hyperparameters.

• Yes, the full code will be released in a containerised environment that will ensure that these results can easily be reproduced.
• The code has already been submitted as supplementary material in our original submission.
• We acknowledge that there are 5 moving parts, namely the actor, value function, successor features, Lagrangian network and world model. However, we have made no change to the components that were already present in DreamerV3 (actor, value function and world model) and the hyperparameters of the successor features are exactly the same as the value function.
• We have no concern about convergence issues or instabilities during training as it is actually the algorithm showing the least variance across random seeds, see Figure 4 in Section 5.4 and Figure A.7 in Appendix A.1.

2. Is this definition of skill commonly used? As I understand it is not. What was the reasoning behind this definition? If there is a citation for this definition of skill, that would be appreciated and helpful to the paper.

• We define the skill executed by the policy as the expected features under the policy's stationary distribution. The expected features have been used before in [1, 2, 3] to characterize the behavior of the policy. However, to the best of our knowledge, we are the first to condition the policy on the expected features. Thank you for pointing that a citation was missing, and that the explanation was unclear. We have added the citation and reformulated the definition of skills. Please let us know if any point is still unclear.
• To explain the reasoning behind this definition, it is useful to draw the analogy with GCRL. In GCRL, the objective is to achieve a desired goal $g$, i.e. to reach a state that is in the subset $\\{\left. s \in \mathcal{S} \right\vert \phi(s) = g\\}$. In contrast, our objective is to execute a desired skill $z$, i.e. to sample a trajectory that is in the subset $\\{\left. (s\_t)\_{t \geq 0} \in \mathcal{S}^\mathbb{N} \right\vert \lim\_{T \rightarrow \infty} \frac{1}{T} \sum\_{t=0}^{T-1} \phi\_t = z\\}$ (see Section 3 for a longer discussion). Thus, the skills can be seen as trajectory-level goals. Additionnaly, it has been shown that Mutual Information Maximization methods used in URL optimize the same objective (up to a constant and to a mapping between goal and skill) than GCRL [4].
• Therefore, we argue that our definition of skill is very similar to the definition of goal used in GCRL.
• Finally, one of our contributions is to show that our definition of skill is expressive enough to produce diverse behaviours like jumping, running in every direction, running while spinning and walking with different feet contact rates. We added a few sentences to highlight those limitations in the final paragraph.

[1] Improving Generalization for Temporal Difference Learning: The Successor Representation. Dayan. 1993
[2] Successor Features for Transfer in Reinforcement Learning. Barreto et al. 2017
[3] Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality. Zahavy et al. 2022
[4] Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learning. Choi et al. 2021

3. I am confused about this description of distance to skill: "To evaluate the ability of a policy to achieve a given skill z, we estimate the expected distance to skill, denoted d(z), with n = 10 rollouts while following skill z." What is the mathematical definition of distance to skill? Is this just Euclidean distance? If I'm missing the mathematical definition somewhere, I apologize.

The mathematical definition of distance to skill is Euclidean distance between desired skill and observed skill. Thank you for highlighting the need for clarity regarding the distance to skill evaluation metric, we have reformulated the corresponding paragraph to include the previously missing mathematical definition.

First and foremost, we would like to thank you for your thorough review and insightful comments on our work. Your feedback is invaluable and we appreciate the opportunity to improve upon the points you have raised, especially the limitations of the skill definition and the overall clarity of the paper.

The skill space is a linear combination (span) of the feature space, i.e., a skill vector can be represented as only a linear combination of state observations. While this might be limiting in terms of how expressive or arbitrarily complex skill representation can be learned, it's not a major concern because the assumption is quite mild and also a crucial part to make the use of SF possible; in fact, rich skill spaces would be actually more difficult to learn than having such inductive biases.

If you are interested, we have updated the notations and the comparison between our definition of skill and goals in GCRL in Section 3.

Figure 2: the x-axis of the plots lacks labels, difficult to figure out what they would exactly mean.

Thank you for pointing out inaccuracies in Figure 2, we have added the previously missing x-axis label.

Proposition: it'd be good to mention what \Psi(st, z) denotes here -- the successor feature.

Thank you for your suggestion regarding the clarity of our proposition. We have now clarified the notation Psi in the proposition.

Sample efficiency -- how many samples is needed to learn successful skills and solve the task? Would the method still work if the entire algorithm is trained without the use of model, in a model-free fashion?

• We now specify the number of environment interactions for all baselines in Appendix E. In all experiments, each algorithm is run for a maximum budget of 10 million environment interactions, and we ensured that is enough for all of them to converge.
• Empirically, we have found out that around 100,000 to 200,000 environment interactions are needed to learn successful skills and to consider the task solved with SCOPA.
• Currently, the world model is an essential part of our algorithm as SCOPA learns to predict the features in order to estimate the successfor features and be able to backpropagate through the world model with Equation 2.

We once again thank you for your constructive critique. Your feedback has been instrumental in refining our paper, and we hope that our responses adequately address your concerns. We are committed to contributing valuable and clear research to the field, and your input has been essential in guiding our efforts towards this goal. Should you have any remaining doubts or concerns, please do not hesitate to share them with us. We will be more than happy to incorporate your feedback, as we did with the initial review, to further improve the quality of our work.

First and foremost, we would like to thank you for your thorough review and insightful comments on our work. Your feedback is invaluable and we appreciate the opportunity to clarify and improve upon the points you have raised.

MAJOR

1- Theory:

It is not clear if the MDPs are assumed to be continuous, discrete, or finite for the theoretical results. This should be made explicit.

In the original proposition you reviewed, the MDP was considered continuous. However, to improve the clarity of section 4.1, we have replaced the proposition and moved the original one to Appendix B.2.

The new proposition is easier to understand and assumes the MDP to be discrete; and this is now explicit.

The motivation for the given definition of a skill is not given. It is not clear why it is an average instead of a sum or any other function over the sequence of features. It is also not clear why the average gives the same probability to each feature in the trajectory. Shouldn't it be an expectation over the sum of features?

We agree that the motivation for the definition of skill in our original submission was lacking. We have updated Section 3 with better notations, two examples of tasks with two different types of skill and an analogy between the definition of goal found in GCRL literature and our definition of skill to help the readers understand better.

We define the skill executed by the policy as the expected features under the policy's stationary distribution. The expected features have been used before in [1, 2, 3] to characterize the behavior of the policy. However, to the best of our knowledge, we are the first to condition the policy on the expected features.

In particular, the reason why we do not want to take the expectation over the sum of features, is because

• In the non-discounted case, the infinite sum of features may diverge.
• In the discounted case, the infinite sum is not as easily-interpretable as the average. For example, in the first example given in Section 3 where the features characterize the velocity of a robot, it makes sense to try to follow a velocity in average, but not in sum.

Thank you for pointing that the explanation was unclear. Please, let us know if you have any remaining doubts or concerns.

[1] Improving Generalization for Temporal Difference Learning: The Successor Representation. Dayan. 1993
[2] Successor Features for Transfer in Reinforcement Learning. Barreto et al. 2017
[3] Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality. Zahavy et al. 2022

The explanations for Proposition 1 and Algorithm 1 are scattered. This made hypothesis 1 of Proposition 1 especially hard to understand.

We agree that the way the proposition and the algorithm were introduced was a bit unclear. We have updated Section 4.1 to make the proposition more straightforward and now reference Algorithm 1 in section 4.2. We hope that the paper have gained in clarity and readability, please let us know if you have any remaining concerns.

2- Baselines and empirical significance:

It is not clear if the 10 replications/runs used for each plot are different runs of each algorithm (e.g Algorithm 1 for SCOPA), or if they are using the same models trained from a single run. This should be clarified, as it affects the significance of the empirical results.

Each experiment is replicated 10 times with independent random seeds, and each replication learns a different model from scratch. We have now added the precision: "Each experiment is replicated 10 times with random seeds and each replication is independent and trains all components from scratch." at the beginning of Section 5.3 Evaluation Metrics.

To evaluate the distance to skill metrics and performance metrics we perform 10 rollouts for each algorithms and each skills, for each of the 10 replications.

While the authors have attempted to make their evaluation metrics fair by modifying the main baselines (SMERL, REVERSE SMERL, and DCG-ME) to learn the same skills that SCOPA is trying to learn, I think this is extremely wrong. While giving the baselines the same additional priors as SCOPA is good, the authors shouldn't have modified the actual optimisation objectives or skills being learned them.

We are sorry for the misunderstanding but we did not modify any of the baselines. All baselines are strictly the same as in their original paper.

We have updated the table D.3 in Appendix with the objective for all baselines. Additionally, you can find more information about the baselines in Section D in Appendix. Especially, a detailed explanation about SMERL's optimization objective is now provided in Section D.1 in Appendix.

If you could specify the section of our paper that caused this confusion, we would be eager to offer a detailed clarification.

MINOR

The proposed approach relies on the state features being predefined/known.

The feature functions are not learned and this is not the focus of the present work, but augmenting SCOPA with unsupervised features is a possible future direction of research, as mentioned in the conclusion. Whether it is a limitation or not depends on the need of the user of the algorithm. If the goal of the user is to learn skills like jumping, then it is very useful to be able to provide the feature function directly. However, if the goal of the user is simply to learn skills to adapt to unforeseen situations like changes in friction, then it could be desirable to learn the feature function.

The abstract and introduction could do a better job of explaining the contributions and what SCOPA is since I couldn't tell the difference between it and UVFAs or USFAs until Sec 3.

The abstract and introduction have been reformulated to better emphasize the contributions of SCOPA. Thank you for helping clarifying the paper.

Some of the figures in the experiments section are too far from where they are referenced for the first time. E.g Fig 2 and 3.

We have fixed this issue in our updated version. Thank you for making the paper better.

There are a couple of typos or grammar errors throughout the paper.

Thank you very much for spotting those typos, they have been corrected.

We once again thank you for your constructive critique. Your feedback has been instrumental in refining our paper, and we hope that our responses adequately address your concerns. We are committed to contributing valuable and clear research to the field, and your input has been essential in guiding our efforts towards this goal. Should you have any remaining doubts or concerns, please do not hesitate to share them with us. We will be more than happy to incorporate your feedback, as we did with the initial review, to further improve the quality of our work.

3- Evaluation Metrics:

The distance to skill and performance metrics used are specific to the type of skills learned by SCOPA. Despite this, the authors spend a lot of time comparing their approach to the baselines using these metrics, which is extremely unfair to the baselines. Naturally, this made the baselines look extremely terrible, inconsistent with the great results shown in those prior works.

The metrics used in this paper are not new, but instead well established in the Quality-Diversity literature.

• DCG-ME relies on a descriptor function that must be defined by the user, and that takes a trajectory as input and gives the skill executed over that trajectory as output. Naturally, in the present work, we chose the descriptor function to be the average of the features over the trajectory. Thus, the distance to skill and performance metrics are completely adapted to this baseline. Morever, DCG-ME has been evaluated with similar metrics in its own paper [4].
• For SMERL and Reverse SMERL, we agree that the metrics are less adapted because they are not per se QD algorithms, but rather algorithms built for robustness using diversity mechanism (namely DIAYN). However, last year a spotlight ICLR paper used rigorously the same metrics to compare SMERL to QD algorithms [5]. However, regardless of QD metrics, we evaluate on downstream tasks too for a full and fair comparison.

To evaluate the skill reachability, we take inspiration from [1], whose graphs are similar to the distance profiles and distance heatmaps used in our paper. To evaluate performance across the skill space, the metrics from Quality-Diversity are adapted [2] and have been used in previous work [3].

We understand your concern and believe that you raise a good point. However, that is why we have also evaluated the different approaches on downstream adaptation tasks. Moreover, we would like to highlight that the adaptation tasks are literally borrowed from SMERL paper itself (see Figure 1 for Ant - Obstacle, see Figure 2 for Humanoid - Hurdles and Humanoid - Motor Failure in SMERL paper).

[1] Rapid locomotion via reinforcement learning, Margolis et al. (2022)
[2] Benchmarking quality-diversity algorithms on neuroevolution for reinforcement learning, Flageat et al. (2022)
[3] Don't Bet on Luck Alone: Enhancing Behavioral Reproducibility of Quality-Diversity Solutions in Uncertain Domains, Grillotti et al. (2023)
[4] MAP-Elites with Descriptor-Conditioned Gradients and Archive Distillation into a Single Policy, Faldor et al. (2023)
[5] Neuroevolution is a Competitive Alternative to Reinforcement Learning for Skill Discovery, Chalumeau et al. (2023)

Just like prior works, I think the authors should have focused more on the "Using skills for adaptation" experiments for the comparison with the baselines. These experiments demonstrate how useful the diverse skills learned by each method are for downstream tasks. The authors should include such experiments for a variety of downstream tasks (similar to prior works).

In our original submission, we had 3 downstream tasks including 1 hierarchical task and 2 few-shot adaptation tasks. Since then, we added 2 extra few-shot adaptation tasks:

• Ant - Gravity task scaling gravity from 0.5 (low gravity) to 3. (high gravity).
• Walker - Friction task varying the friction with the ground from 0. (no friction) to 5. (high friction).

Hence, our number of downstream tasks is similar to prior work:

• SCOPA is evaluated on 1 hierarchical task, and 4 few-shot adaptation tasks (5 different types of perturbation).
• SMERL has 0 hierarchical tasks, and 9 few-shot adaptation tasks in total (3 different environments times 3 different types of perturbations).
• [1] compares SMERL and QD algorithms on 1 hierarchical task and 7 few-shot adaptation tasks (3 kinds of perturbations).

[1] Neuroevolution is a Competitive Alternative to Reinforcement Learning for Skill Discovery, Chalumeau et al. (2023)

The authors use MBRL (DreamerV3) to speed up the learning of their actors and critics, but do not do the same for the main baselines (SMERL, REVERSE SMERL, and DCG-ME). This makes the experiments extremely unfair towards the baselines. If the authors did not want to go through the additional effort of augmenting the baselines with DreamerV3, then I think they also shouldn't have augmented SCOPA with DreamerV3.

The world model we use is an integral and essential part of our algorithm, and showing that we can extend DreamerV3 to also predict features and learn a successor features (i.e. a skill critic analogous to a classic performance critic) is one of the main contributions of this paper, which to the best of our knowledge was never done in the litterature.

Furthermore, we trained each baseline until convergence for a maximum budget of 10 million environment interations (which is well enough time for all baselines to converge) and we do not compare the convergence speed of SCOPA against the baselines with learning curves in Section 5.4 Results. Our evaluation methods only focus on the final performance metrics achieved by each algorithm after the completion of the training process.

Finally, we did not want to modify the original implementations of the baselines we compare to, because augmenting the baselines with DreamerV3 runs into implementation issues that are subject to debate on how to actually make it work, and would result in different algorithms entirely. For example, DCG-MAP-Elites cannot possibly be extended because it would require performing full rollouts in imagination (see $1, 2$ for examples of model-based QD algorithm), whereas DreamerV3 performs only 15 steps in imagination and uses bootstrapping to estimate the critics.

[1] Dynamics-Aware Quality-Diversity for Efficient Learning of Skill Repertoires, Lim et al. (2021)
[2] Efficient Exploration using Model-Based Quality-Diversity with Gradients, Lim et al. (2023)

The authors used actor/critic networks with hidden layers of size 512 for SCOPA (and its variants), but only 256 for the main baselines (SMERL, REVERSE SMERL, and DCG-ME). The authors did not explain why they made this decision. Additionally, the number of iterations used for the main baselines (or possibly function evaluations for DCG-ME) was not stated. These make it hard to evaluate the significance of the empirical results, and I think it also puts into question the claims and conclusions based on them.

SMERL, Reverse SMERL and DCG-ME are all baselines that have been fine-tuned on similar locomotion tasks like Walker, HalfCheetah, Ant, Humanoid. Therefore, we initially made the decision to keep the hyperparameters coming from the original papers for all baselines.

However, we understand your concern and we believe that you raise a very good point. We have run all baselines with hidden layers of [512, 512] and observed a significant decrease in performance for Reverse SMERL and DCG-ME. For SMERL, the performance are not significantly better or worse. For this reason, we decided to keep the original architectures for our baselines, and the justification has been added to the hyperparameters section in Appendix.

We think that this additional evaluation increases the significance of the empirical results, and hope that this addresses your concerns.

Furthermore, we now specify the number of environment interactions for all baselines in Appendix E. Each algorithm is run for 10 million environment steps, and we ensured that is enough for all of them to converge.