Periodic Skill Discovery

Dear Reviewer qG2Z,

We sincerely thank you for your detailed and insightful feedback. We have prepared our responses below:

---

Q1. The paper introduces a practical inductive bias. Will it be eventually be surpassed by simply increasing the data, scale and compute of a more general method?

A1. We appreciate your comment. Even as generalist methods grow in data, scale, and compute, an explicit inductive bias for periodicity will continue to fill a role that pure scale alone cannot guarantee—much like convolution remains valuable in vision despite ever-larger transformers.

At the same time, this inductive bias is not at odds with future progress in large-scale models; it complements it. Since one of the PSD’s contribution is to discover a maximally diverse repertoire of useful periodic behavior in an unsupervised manner, it can be plugged in as a lightweight sub-module that a large, general model calls whenever rhythmic style or behavior-style rewards should be injected. In this way, the large model continues to handle broad task understanding and high-level decisions, while PSD provides a principled, sample-efficient mechanism for adding rhythmic nuance, thereby enhancing versatility rather than competing with scale.

Q2. Applicability of PSD to offline datasets.

A2. Thank you for your insightful comment regarding the applicability of PSD to offline datasets.

The main prerequisite is to know or estimate the period variable $L$ for each trajectory in the offline dataset, which we believe is a tractable problem.

For instance, we could estimate $L$ for a given trajectory using a Fourier transform to find its dominant frequency, similar to our analysis in Figure 3. Alternatively, one could add an auxiliary reconstruction loss to the existing PSD objective to find the period $L$ that best represents a trajectory's periodic features.

Once the period $L$ for each offline trajectory is estimated, an offline skill learning process is available. The recipe would be as follows:

1. For each trajectory in the offline dataset, first estimate its period, $L$.
2. Use these estimated (trajectory, L) pairs to train the PSD encoder and its circular latent space.
3. Finally, with the learned representations, train an RL policy with $r_{psd}$ using a standard offline RL algorithm.

Additionally, an offline-to-online approach could allow the agent to discover skills with periods that go beyond the range covered by the original dataset, while remaining consistent with the behaviors it contains. Thank you again for the valuable suggestion.

Q3. As far as I can tell the current setup cannot represent different gaits of the same period.

A3. We appreciate your excellent comment. The primary contribution of this work lies in autonomously discovering a diverse range of periods, rather than directly targeting multimodality within a single period.

However, your comment raises a very interesting direction for future work. Inspired by your idea, encoding the representation onto a more general geometry, such as the surface of a sphere (instead of a circle), would allow different gaits to be represented even for a single period. For example, different gaits of the same period could be encoded as distinct circles of the same circumference on the sphere's surface. We agree this is a very exciting future direction to explore.

Q4. Question about the state space of PSD.

A4-1. Your presumptions are correct. The default state representation for PSD does not include the global XY base position and consists only of proprioceptive states like joint angles and heading angles, as described in Appendix B. This corresponds to the default observation space in Gym environments and is a common practice in prior work [1, 2, 3] on robot learning for generating diverse behaviors.

A4-2. When we trained the Ant agent with the full state (including the XY coordinates) using only the PSD objective, we observed that the agent vigorously wiggled in place, making body movements in sync with the period. This behavior is the result of the optimization satisfying our objective, as it makes both the joint angles and the global XY coordinates periodic.

A4-3. As stated in our previous response, PSD can discover periodic behaviors even when the state includes the XY coordinates. However, a conflict arises when the agent must also satisfy a specific downstream task. If an external task reward encourages non-periodic changes in the XY position (e.g., by rewarding forward velocity), this would directly conflict with the PSD objective, which requires the latent representation to be identical after $2L$ steps, and would therefore hinder the learning process.

[1] Peng, Xue Bin, et al. "Amp: Adversarial motion priors for stylized physics-based character control." ACM Transactions on Graphics (ToG) 40.4 (2021): 1-20.

[2] Peng, Xue Bin, et al. "Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters." ACM Transactions On Graphics (ToG) 41.4 (2022): 1-17.

[3] Li, Chenhao, et al. "FLD: Fourier Latent Dynamics for Structured Motion Representation and Learning." The Twelfth International Conference on Learning Representations.

Q5. Need a clear outline of the problem statement in the introduction for online unsupervised skill discovery.

A5. Thank you for your suggestion. We will add an explanation of the basic setup of unsupervised skill discovery, along with the problem it aims to address and its significance, in the second paragraph of the introduction.

Q6. Use of the one-step periodic constraint as the intrinsic reward instead of the diameter constraint.

A6. That is an excellent question regarding our reward design! We chose the one-step periodic constraint instead of the diameter constraint for two primary reasons: learning efficiency and compatibility with off-policy RL algorithms.

A6-1. On a practical level, the one-step reward provides a dense, per-timestep learning signal, which is significantly more effective for policy training than the sparse reward from the diameter constraint that is only evaluated over an $L$-step horizon.

A6-2. More fundamentally, the use of a single-step reward is crucial for general compatibility with standard off-policy algorithms (e.g., SAC, DDPG). In contrast, a reward based on the diameter constraint would be inherently a multi-step, sparse reward, as it depends on states that are $L$ timesteps apart. This choice was made to ensure that PSD is easy to use and broadly compatible with standard off-policy RL algorithms that learn from single-step transition tuples. However, as you suggest, a more advanced implementation could certainly be developed to handle the multi-step diameter constraint, perhaps by leveraging n-step returns, which is an interesting direction for future work.

Thus, we chose this approach to ensure a dense learning signal and broad applicability with standard off-policy RL algorithms.

Q7. Line 182 "Since we want to discover maximally diverse period" is either bad grammar or confusing at least.

Thank you for your valuable suggestion. We found that the original wording was inconsistent with the tone of the paper.

We have revised the sentence as follows: "To enable the agent to discover a maximally diverse range of periods without any prior knowledge of the agent-specific period ranges, we introduce an adaptive sampling method that …”

Q8. Is their a practical maximum of discovered period? It would seem like you could come up with policies of essentially unboundedly high period and with low periods of a single timestep (e.g. staying still).

A8-1. Thank you for the question. Your understanding is correct. the period’s upper & lower bounds dynamically expand based on whether the policies at the boundaries satisfy the periodic conditions.

A8-2. Regarding the practical maximum, while there is no hard-coded upper bound, the range is limited by what is dynamically feasible for the agent, especially when maintaining performance on a specific task. For example, there is likely a maximum period at which an agent can still maintain a forward velocity of 2m/s.

A8-3. For the lower bound, since $L$ must be a positive integer in our objective, the shortest possible period corresponds to $L=1$, resulting in a period of 2 timesteps.

Q9. Connection to quality diversity methods such as MAP-Elites.

Thank you for this excellent suggestion. You are correct that our approach has a link to quality diversity (QD) methods such as MAP-Elites. However, we would like to highlight a key difference. While this QD methods typically operate on a pre-defined, static grid, PSD dynamically adjusts the range of the descriptor ($L$) based on the agent’s performance (sum of reward) and the embodiment's dynamics.

Following your suggestion, we will further investigate this connection and add a discussion of both the similarities and this key distinction to our Extended Related Work section.

---

Thank you again for your valuable and insightful review. Also, please let us know if our responses have addressed your questions. If anything requires further clarification, please do not hesitate to let us know.

Thank you for the detailed response to my review.

Q1 - satisfactory answer

Q2 - This is a fair answer - it would be nice to see this as an experiment but I don't think necessary for acceptance

Q3 - Again this would be nice to see but not necessary for acceptance

Q4 - I think this is still my biggest gripe with the method. Rather than being simply plug-and-play, this does severely limit the applicability of this method to apply to observation spaces where we know a priori which features we wish to be periodic. In the symbolic space this is not too large an issue as you can simply mask out the non-periodic features, however this is not possible in e.g. pixel-based environments. Your results with pixel-based observations would seem to refute this, however my intuition is that the only reason these worked is because the rendering of the floor is periodic (the checkerboard pattern). You could imagine that in navigating a non-periodic environment e.g. a landscape with variable terrain, then PSD with pixel observations would be forced to return to its original position every cycle to satisfy the periodic constraint (as with the wiggling ant). It would be interesting to try something like co-learning a projection in the latent space to a lower dimensional space, and then only forcing the periodicity to apply in this lower dimensional space. The hope would then be that the features we didn't wish to be periodic (e.g. XY coords) could be 'projected out' and ignored.

Having said all this, I think the novelty of the method in this paper is still sufficient, but this could be an interesting route forward to make this method more generally applicable.

Q5 - thank you for adding this

Q6 - This makes sense and is a fair justification, it would be nice to have it justified in the paper

Q7 - Thanks I think that reads better

Q8 - Thank you for clarifying, I can see now that with the forward velocity constraint this makes more sense

Q9 - Thank you for adding this comparison.

Overall I am satisfied with the authors response. I think that more could be done with this method to increase its usability, especially wrt cases where only part of the latent space is periodic, however I think the paper is a significant enough contribution already. I will increase my score to a 5 and advocate for the acceptance of this paper.

Dear Reviewer N2n9,

We are grateful for your careful reading and valuable suggestions.

We have summarized the weaknesses and questions you raised and provide our responses below:

---

Q1. The discrete step size adaptation of $L$ feels somewhat ad hoc and lacks theoretical justification.

A1. In the standard MDP framework, time progresses in discrete steps. Consequently, $L$ in the PSD objective is intended to encode timestep information. Therefore, it should be a positive integer. To discover maximally diverse periods — while respecting the fact that the minimum unit of time is one step — we set the default step size of the adaptive sampling method to its minimal feasible value, $N=1$.

Q2. A more systematic quantitative analysis across multiple seeds would strengthen the evaluation. Could you repeat the experiments for Figures 3, 5, and 6 across multiple seeds to assess the robustness of PSD?

A2. Following the reviewer’s suggestion, we have conducted additional experiments to address this.

A2-1. (Figure 3) To quantitatively compare the diversity of learned skills, we measured the frequency width — defined as the difference between the maximum and minimum frequencies of the skill trajectories (i.e., the gap between the highest and lowest frequency of bar in Figure 3) — and averaged the results over 3 seeds. A larger value for this metric indicates a broader range of periodic behaviors.

As the results show, PSD consistently discovers a much wider range of periodic behaviors compared to the baselines across all embodiments. The frequency width, which measures the diversity of the discovered periods, is significantly larger for PSD in every case.

We excluded DADS and DIAYN from this analysis as their policies collapse to a nearly static "yoga pose"—moving slightly at the beginning of an episode and then remaining stationary—which results in a frequency close to zero.

A2-2 & A2-3 (Figures 5 & 6). As you suggested, we have run the experiments for Figures 5 and 6 across multiple seeds to assess their robustness. While the NeurIPS rebuttal policy prevents us from including the new plots, we can summarize our findings.

The circular latent space structure shown in Figure 5 is learned robustly across all seeds. Interestingly, we often observe that different random seeds result in qualitatively different gaits for the same period, suggesting our method can find multiple diverse solutions. Our analysis for Figure 6 also confirms that the latent representation accurately reflects temporal distance, suggesting the learned distance converges to its optimal value.

Due to the time/character limits of the rebuttal period, we will provide the detailed quantitative results for a wider range of $L$ values during the upcoming author-reviewer discussion period. Also, the full multi-seed visualizations for these figures will be included in the final version of our paper.

Q3. The focus on locomotion tasks may limit the method’s generality and significance.

A3. PSD objective is theoretically applicable to any domain that involves periodic components, as it does not require domain-specific knowledge for optimization. For instance, in manipulation settings, it can be applied to tasks such as rotating a cube with fingers, turning a valve wheel, or mixing a salad bowl. Moreover, given a set of low-level policies $\pi_{low}(a|s,z)$, where $z$ represents a low-level skill such as ‘pick’, ‘move’, or ‘place’, a high-level policy $\pi_{high}(z|s,L)$ can be trained with the PSD objective to select skills $z$ as actions for composing complex, looped behaviors. We believe this could be an interesting future extension of PSD.

Q4. Lacks clear justification for hyperparameters of adaptive sampling method.

A4. To justify our hyperparameter choices for the adaptive sampling method, we conducted an ablation study (averaged over 3 seeds) on the Ant environment. The results show a clear trade-off between the final range of discovered periods and the stability of the expansion process.

For instance, the less conservative threshold set ($\alpha=0.5, \beta=0.8$) with a larger step size ($N=3$) initially discovers the widest range. However, this aggressive search proves unstable, as shown by the high number of "range rollbacks" (28.7). This instability eventually causes the discovered range to collapse later in training (from 52 at 200k episodes to 42 at 300k). A high number of rollbacks generally indicates a potential for instability in long-term training.

In contrast, the setting ($\alpha=0.4, \beta=0.9, N=1)$ expands the range to a comparable margin (47.3) with minimal rollbacks (2.0), demonstrating a much more stable and efficient learning process. We checked that this parameter set performs well across all embodiments, and therefore selected it for providing the best balance between discovering a diverse skill repertoire and maintaining training stability.

Q5. Could you provide an intuitive explanation for why a Fourier transform of PCA-projected, normalized skill trajectories is informative? What does this metric reveal about the learned behaviors?

A5. For Figure 3, we normalized the skill trajectories for each dimension using statistics computed from random action rollouts. This was done to account for the different scales of the state dimensions and to visualize the periodicity of each dimension fairly. In addition, for the 1D FFT, we applied PCA to reduce the trajectories to a single dimension and visualized them in the frequency domain. This analysis revealed that, compared to the baselines, PSD exhibits behaviors spanning a wider range of frequencies, as indicated by the broader distribution of bars in the frequency domain chart.

Q6. How is the latent space dimensionality chosen and how does it affect the resulting behaviors?

A6-1. As shown in Table 2, we found that using {3, 6}-dimensional latent space works well in practice across all embodiments. However, we observed that a 2-dimensional latent space does not perform well for complex agents such as Ant or Humanoid. We hypothesize that, since the PSD objective does not explicitly constrain the circles for each $L$ to lie on the same plane, having additional degrees of freedom allows different circles to occupy different planes. This, in turn, helps stabilize the embedding during training. Conversely, when the latent space is restricted to only 2 dimensions (i.e., a plane), this flexibility is lost, which leads to instability.

A6-2. Thank you for your suggestion to add a note that the 2D visualizations are the result of PCA projections! We will include this in the final version. If you have any additional ideas to further improve the reader’s understanding, please feel free to share them, and we will be happy to include them.

Q7. In pixel-based settings, robot limbs are differently colored. Was this necessary for the method to work? An ablation or explanation would clarify its impact.

A7. For symmetric agents like the Ant, limb coloration is a common practice to disambiguate limbs from pixel observations [1]. However, it was not strictly necessary for our specific Ant task (forward locomotion) or for the asymmetric Cheetah. Therefore, while helpful for feature learning, it is not an essential design choice for our method to work.

[1] Park, Seohong, et al. "Ogbench: Benchmarking offline goal-conditioned rl." arXiv preprint arXiv:2410.20092 (2024).

Q8. Discussion around where PSD may fail or underperform.

A8. PSD may underperform in settings with large, persistent external disturbances (e.g., constant interference from another agent) that make periodic behavior infeasible. We will add this discussion to our limitations section.

Q9. What aspect of vanilla PSD is supervised? Could METRA-PSD also be enhanced with extrinsic rewards for improved downstream performance?

A9. PSD itself is unsupervised as its objective is task-agnostic. The METRA-PSD setup is described as "fully unsupervised" because it combines two unsupervised methods without external supervision. As you suggest, extrinsic rewards can optionally be added to this framework to bias exploration, but this is a separate design choice.

---

Thank you again for your valuable review. We hope our responses have addressed your questions.

Dear Reviewer B3mW,

We sincerely appreciate your constructive and insightful comments, which have been invaluable in helping us further strengthen our work.

We prepared our response below:

---

Q1. Lack of ablation studies on the use of the adaptive sampling method to isolate its impact.

A1-1. We appreciate your comment. The adaptive sampling method is crucial for autonomously discovering diverse periods without supervision or prior knowledge. It operates by proposing progressively more challenging periods based on the agent's performance (average cumulative sum of $r_{psd}$). Without this method, the agent fails to learn a wide range of periods, even if those wide range of periods are sampled from the beginning of training. This is analogous to curriculum learning, where an agent masters a difficult task by tackling increasingly challenging goals, rather than being presented with the most difficult ones from the beginning.

A1-2. To isolate its effect, we conducted an experiment in which a wide range of periods was sampled from the start, but without the adaptive sampling method. After training, we rolled out policies conditioned on each period $L$ and compared their sum of $r_{psd}$ with those from our full method using adaptive sampling.

• Ant

  	L = 5	L = 16	L = 28	L = 40
  w/ adaptive	198.1 ± 1.2	188.1 ± 2.1	184.7 ± 2.9	179.5 ± 4.5
  w/o adaptive	152.4 ± 3.7	187.8 ± 4.2	185.5 ± 2.2	183.2 ± 1.9

• Halfcheetah

  	L = 5	L = 8	L = 11	L = 15
  w/ adaptive	187.8 ± 2.5	179.6 ± 3.4	177.3 ± 6.2	175.1 ± 2.2
  w/o adaptive	185.0 ± 2.7	177.2 ± 3.3	175.9 ± 2.1	167.6 ± 6.3

• Humanoid

  	L = 10	L = 26	L = 43	L = 60
  w/ adaptive	373.0 ± 8.4	351.9 ± 6.7	376.6 ± 9.5	385.2 ± 5.9
  w/o adaptive	367.1 ± 2.5	115.5 ± 6.1	91.2 ± 14.7	100.9 ± 6.9

• Hopper

  	L = 5	L = 20	L = 35	L = 50
  w/ adaptive	351.4 ± 4.9	360.1 ± 11.1	379.3 ± 9.6	353.3 ± 9.1
  w/o adaptive	309.8 ± 25.6	367.1 ± 8.4	372.2 ± 2.9	337.6 ± 6.7

• Walker2D

  	L = 5	L = 23	L = 41	L = 60
  w/ adaptive	379.6 ± 6.4	362.1 ± 12.2	366.6 ± 12.1	359.5 ± 10.4
  w/o adaptive	342.7 ± 4.8	371.0 ± 4.6	345.4 ± 4.7	301.8 ± 8.2

Based on the results, our adaptive sampling method is crucial for learning a wide range of high-quality skills, as it consistently outperforms the non-adaptive baseline. This is particularly evident at the extremes of the period range, which highlights its effectiveness in discovering a truly diverse skill repertoire.

Q2. Generalization of PSD to non-uniform or compositional sequences.

A2. Thank you for this insightful question. While the primary focus of this work, similar to other methods in unsupervised skill discovery, is on learning a rich repertoire of low-level skills, we believe that combining PSD with a hierarchical RL (HRL) framework could enable more complex, compositional tasks.

For example, consider a set of low-level policies $\pi_{low}(a|s,z)$, where $z$ represents a low-level skill such as ‘pick’, ‘move’, or ‘place’ and is capable of both periodic and non-periodic behaviors. Based on this, we could then train a high-level policy $\pi_{high}(z|s,L)$ with the PSD objective to encourage it to select skills $z$ as actions for creating complex, looped behaviors. This could be an interesting future direction.

Q3. Need for stronger statistical support for downstream task performance results.

A3. Thank you for your helpful suggestions. Following your valuable comment, we expanded our experiments from 5 to 10 seeds to provide stronger statistical support.

For the hurdle and friction tasks, we observed that PSD outperforms the baselines across all embodiments. We hypothesize that the (relatively) high standard deviation of PSD is attributable to the fact that it leverages a more diverse range of periodic behaviors as low-level skills compared to other baselines.

Q4. Number of distinct skills learned by PSD and its effect on downstream task performance.

A4-1. Thank you for the question. During the training of PSD, we uniformly sample 4 values of $L$ from the range $[L_{min}, L_{max}]$, including both bounds. We found this strategy sufficient, as our model can also achieve the desired periods for interpolated $L$ values after convergence. For more details, please refer to Appendices B.2 and C.2.

A4-2. We appreciate your comment. Based on your thoughtful question, we conducted experiments on all downstream tasks, comparing PSD trained with 4 discrete skills and 7 discrete skills within the same bound $[L_{min}, L_{max}]$.

As the results show, increasing the number of discrete skills sampled during training from 4 to 7 does not lead to a significant improvement in downstream task performance. Across all tasks, the performance of both settings is comparable within their standard deviations.

Q5. How were the trajectories presented in Figure 5 selected?

A5. If your question is about which state components were used to construct the trajectories, we selected the velocities and positions of joints of Ant and Walker2D that best highlight the periodicity. For clear visualization, we normalized these values to lie between -1 and 1. As described in the caption, we also switched the target period $L$ at fixed time intervals within a single episode.

Q6. Questions on training setup and skill selection for downstream tasks.

A6-1. We trained the low-level skill policy for all methods with the same number of environment steps for each embodiment: 40M for Ant and HalfCheetah, and 80M for Humanoid, Hopper, and Walker2D. We also closely followed the hyperparameters reported in their original papers.

For training the high-level policy for the downstream tasks, we ran 100k episodes across all experiments.

A6-2. For all baseline methods, we designed the high-level policy to have access to the full set of discovered skills, matching the skill distribution $p(z)$ used during low-level policy training. For PSD, we uniformly sampled fixed number of $L$ values for $\pi(a|s,L)$ from the range $[L_{min}, L_{max}]$.

Q7. Questions on the exploration reward.

A7. Thank you for raising this point. Since PSD is designed to learn how to behave within a task rather than directly address what to do, the specific task setting needs to be predefined when evaluating its performance. For this reason, we added a exploration reward to all methods for fair comparison. Importantly, this exploration reward did not negatively affect the quality of the skills required for downstream tasks. This type of setup where an auxiliary unsupervised reward is combined with the main learning objective is also widely used in prior work on intrinsic motivation [1, 2, 3].

[1] Lee, Youngwoon, et al. "Learning to coordinate manipulation skills via skill behavior diversification." International conference on learning representations. 2019.

[2] Li, Jiahui, et al. "Two Heads are Better Than One: A Simple Exploration Framework for Efficient Multi-Agent Reinforcement Learning." NeurIPS. 2023.

[3] Bae, Junik, et al. "TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representations." Conference on Robot Learning. PMLR, 2025.

---

Thank you again for your valuable and insightful review. We hope our responses have sufficiently addressed your questions, and we would be happy to provide any further clarification if needed.

Dear reviewer B3mW,

Thanks for your reviewing efforts so far. Please respond to the authors' response.

Thanks, Your AC

Dear Reviewer DTax,

Thank you for your valuable and insightful comments. We have carefully considered your feedback and prepared our response below:

---

Q1. Visualizations on website could be more informative.

A1. Thank you for your thoughtful suggestions! We will incorporate your feedback into the final version of our project page and paper. If you have any additional ideas to further improve the reader’s understanding, please feel free to share them, and we will be happy to include them.

Q2. Questions on the state space of the encoder.

A2-1. Thank you for your insightful question. The encoder's state representation does not include the global XY base position and consists only of proprioceptive states like joint angles, velocities, and heading angles, as described in Appendix B. This corresponds to the default observation space in standard Gym environments and is a common practice in prior work [1, 2, 3] on robot learning for generating diverse behaviors.

A2-2. Furthermore, from the perspective of our downstream task setup, including the global XY position would be particularly problematic. The exploration reward we use encourages consistent forward movement, which makes the XY position an inherently non-periodic states. This would directly conflict with the PSD objective, which requires the latent representation to be identical after $2L$ steps, and would therefore hinder the learning process.

A2-3. We verified this with an experiment on the Ant environment. When we removed the exploratory reward, provided the full state (including the XY coordinates), and trained the agent using only the PSD objective, we observed that the agent vigorously wiggled in place, making body movements in sync with the period. This behavior is the result of the optimization satisfying our objective, as it makes both the joint angles and the global XY coordinates periodic. We will include this result in the final version of our project page.

Therefore, PSD can successfully discover periodic skills even with the full state, which demonstrates the strength of the PSD objective.

[1] Peng, Xue Bin, et al. "Amp: Adversarial motion priors for stylized physics-based character control." ACM Transactions on Graphics (ToG) 40.4 (2021): 1-20.

[2] Peng, Xue Bin, et al. "Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters." ACM Transactions On Graphics (ToG) 41.4 (2022): 1-17.

[3] Li, Chenhao, et al. "FLD: Fourier Latent Dynamics for Structured Motion Representation and Learning." The Twelfth International Conference on Learning Representations.

Q3. Questions on the state coverage of PSD.

A3-1. We appreciate your constructive comments. We would like to clarify that PSD is not designed to encourage spatially far-reaching, state-covering trajectories. Instead, its primary goal is to achieve high coverage in the frequency domain by autonomously discovering a diverse repertoire of skills with different periods, as demonstrated in Figure 3.

A3-2. As mentioned in our response to Q2, an external reward is not necessary for learning periodic behaviors. Our method focuses on learning the style of motion (how to behave). The optional task reward simply provides a clear objective (what to do), which allows the various learned periodic styles to be demonstrated and evaluated more effectively.

Q4. From Figure 8 it seems that the XY coverage of PSD+METRA is worse than only METRA (based on their own results) - is this the case?

A4. Yes, your observation is correct. To quantify this, we measured the state space coverage by discretizing the XY plane (or X-axis for Walker2D) into a 1x1 grid and counting the number of unique cells visited over 32 random skill rollouts, averaged over 5 seeds.

The results confirm that the XY coverage of METRA+PSD is lower than that of METRA alone:

The reason for this trade-off is detailed in our response to your next question (A5).

Q5. Similarly, a more in-depth analysis of the trade-offs of PSD+METRA would strengthen the claim that they can easily be combined. Is the loss worse for both methods when combining them?

A5. Thank you for your question. Conceptually, METRA and PSD align well, as both methods address temporal aspects of skills: METRA modulates the temporal direction ($z$), while PSD modulates the temporal length ($L$). However, as you suggest, combining them involves a trade-off. While our mutual conditioning approach ($\phi_L(s, z), \phi_m(s, L)$) solves the primary issue of skill collapse, it introduces a secondary trade-off related to representational capacity.

By conditioning each encoder on an additional skill variable, we increase the complexity of its input space. This can slightly diminish the individual performance of each component compared to when it is trained in isolation. Therefore, while the combined policy $\pi(a | s, z, L)$ is significantly more versatile—offering independent control over both skill direction and period—this increased modality comes at the cost of a slight reduction in the peak performance of the individual METRA and PSD objectives compared to their standalone counterparts.

---

Thank you again for your valuable and insightful review.

Also, please let us know if our responses have addressed your questions. If anything requires further clarification, please do not hesitate to let us know.

Dear reviewer DTax,

Thanks for your reviewing efforts so far. Please respond to the authors' response.

Thanks, Your AC