Interplay Between Task Learning and Skill Discovery for Agile Locomotion

I have not received any comments from the authors to answer my queries. As such I will not be changing my initial review scores.

We thank the reviewer for their insightful review and for bringing up interesting discussion points. Please find our response below.

Q1.1: Related Works

Thank you for suggesting [1][2][3] as additional related works. [1] introduces a method for learning diverse behaviors to generalize across varying environments while accomplishing tasks. [2] employs a dual optimization approach with Lagrange multipliers to effectively balance the trade-off between quality and diversity. [3] builds on this by extending the approach to quadrupeds, enabling navigation around obstacles. These works are highly relevant to our work, and we will include them in the paper.

Q1.2: Additional Baselines

Thank you for suggesting additional baselines. We attempted to incorporate DoMiNiC [3] as an additional baseline but faced challenges with reward engineering, preventing successful integration into our environment. Instead, we have included DIAYN and Robot Parkour Learning in our evaluation. For DIAYN, we implemented a discrete version using PPO with a total of five discrete skills. This addition expands our evaluation to include exploration-based methods (RND), mutual information-based skill discovery methods (DIAYN), and distance maximization-based skill discovery methods (METRA). We also add Robot Parkour Learning as a specialized reinforcement learning baseline with a complex reward structure. We trained the parkour policy with soft dynamics for 70% of the training process, then fine-tuned it with hard dynamics for the remaining 30%, following the procedure outlined in the original paper. However, despite adhering to the reward terms and procedure described in the paper, we were unable to train a usable parkour policy for the Crawl task.

For skill discovery methods, we sampled a new skill at the beginning of each episode and kept it fixed throughout that episode. For Ours (fixed z) we used a single fixed z value for all the episodes. The table below summarizes our findings, reporting the average number of obstacles passed (maximum of 3) over 100 episodes:

From the table, we observe that our method performs on par with Robot Parkour Learning. This demonstrates that combining skill discovery with task rewards using our framework achieves approximately the same performance as a reinforcement learning policy with manually tuned reward functions but without the need for labor-intensive reward engineering. We also want to emphasize that the success rate of our method depends on the skills sampled at the start of each episode. These findings align with the observations reported in Table 1 of our main paper. We show that conditioning the policy on a fixed skill for all the eval episodes could enable our method to consistently perform well across all tasks. Notably we see a ~90% increase in the success rate for Crawl, and a near perfect success rate for Leap and Climb tasks outperforming even Robot Parkour Learning.

Moreover, we find that DIAYN performs well on the Climb task but struggles with the Leap task and fails on the Crawl task. We attribute this to the nature of DIAYN's objective function, which maximizes mutual information (MI) using KL divergence. This approach often leads to less distinctive behaviors, as KL divergence is fully maximized when two distributions have no overlap. Beyond this point, there is no additional incentive to further distinguish between distributions. As a result, DIAYN suffers from poor exploration compared to distance-maximization-based skill discovery methods, which are better suited for generating diverse and effective skills.

[1] S. Kumar, A. Kumar, S. Levine, and C. Finn, ‘One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL’, in Advances in Neural Information Processing Systems, Curran Associates, Inc., 2020, pp. 8198–8210. doi: 10.48550/arXiv.2010.14484.

[2] T. Zahavy et al., ‘Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality’, presented at the The Eleventh International Conference on Learning Representations, Sep. 2022.

[3] J. Cheng, M. Vlastelica, P. Kolev, C. Li, and G. Martius, ‘Learning Diverse Skills for Local Navigation under Multi-constraint Optimality’, in 2024 IEEE International Conference on Robotics and Automation (ICRA), Yokohama, Japan: IEEE, May 2024, pp. 5083–5089. doi: 10.1109/ICRA57147.2024.10611629.

Q2: Hardware Validation

We appreciate the reviewer’s suggestion to include hardware results and their concerns about unnatural motion. However, due to time constraints and the labor-intensive nature of hardware experiments, we are unable to provide hardware results for our method at this stage. While our approach generally produces natural-looking motions, we acknowledge that some tasks may not directly translate to real-world applications. The primary objective of this work is to demonstrate an effective method for combining skill discovery techniques with task rewards, eliminating the need for additional approaches like reward engineering to achieve agile motions. Investigating the applicability of our method to real-world robotic tasks remains an exciting avenue for future research.

Q3: Smoothness of skills and distribution of successful skills in the skill space.

Thank you for this insightful question, which prompted us to perform additional analysis. To evaluate the smoothness of the skills, we conducted a simple experiment on the Crawl task. Specifically, we manually identified two skill latent values: $z_{fail}$, corresponding to behaviors that fail to complete the task, and $z_{success}$, corresponding to behaviors that successfully complete it. For each of these values, we sampled 10 skills from the ranges $z_{fail}$ ±0.05 and $z_{success}$ ±0.05 , with intervals of 0.01, as our skill latent space is one-dimensional. The table below shows the average number of obstacles passed for these 10 sampled skills, providing insight into the smoothness of the skill transitions.

We observe that all the skills sampled near $z_{fail}$ result in failure, while those in the vicinity of $z_{success}$ consistently succeed, as shown in the results above. This indicates that the skills are smooth with respect to their corresponding behaviors.

Q4: Can METRA work with goal conditioned policices?

Yes, we believe METRA can work effectively with goal-conditioned policies. We understand your concern, but METRA and goal-conditioned policies operate on different sub-dimensions of the state space. For instance, in the crawl task, we use the linear velocity of the robot's base as the goal, while simultaneously using the robot's base height as input to METRA. These two sub-dimensions do not overlap. In other words, the robot can follow a specific linear velocity while exploring diverse heights of its base. By trying out different base heights, the robot discovers a solution to the task—crawling. Furthermore, even when sub-dimensions overlap, METRA and goal-conditioned policies can still complement each other. For example, in the leap task, consistently following the goal speed of 0.7 m/s would prevent the robot from successfully jumping over the gap. To make the jump, the robot must deviate from the goal speed momentarily to achieve the necessary dynamics, allowing it to continue following the desired speed for the rest of the episode. Without this deviation, the robot fails to clear the gap and crashes. In this scenario, the task reward guides the policy to adhere to the goal velocity as much as possible, while METRA enables the robot to explore different velocities when required. Lastly, we confirm that we used a skill discovery framework as our exploration method. We have clarified this in lines 315–319.

Q6: Shouldn’t lambda start decreasing as the model trains further?

First, intuitively, the value of λ should decrease when less exploration is needed. This assumes that the diversity reward should work as an exploration module. If the diversity reward does not serve as an exploration module, there would be no incentive to reduce the value of λ. This is indeed the case in our approach. As training progresses and most skills converge into a narrow solution space, exploring different skills tends to produce similar behaviors and task rewards. We refer to this phenomenon as “positive skill collapse,” which we discuss in detail in Section 4.3.

In summary, since different skills no longer produce diverse behaviors at the later stage of training, there is no incentive to decrease the λ value.

M1: Section 3.1

This is indeed a typo, the action space is pertubations in the joint angles form the default position. We will fix this is in the paper.

M2: Figure 6b) and Fig 8.

Thank you for pointing out the mislabelling in the plots. We have fixed them in the paper.

M3: Does the METRA encoder use the same observations as the policy?

No, the METRA encoder only gets certain states as inputs. These vary for different tasks. For Climb and Crawl tasks we use robot height and in Leap we use velocity in forward direction as the input to METRA encoder. We also mention this in Section 4.1, line 304.

Q1: Detailed formulation of rewards

We have added the mathematical formulation of the rewards in Appendix A.2.

Q2: Standard deviation being too high

The high standard deviation in our results can be attributed to the following factors:

i) Discrete Metric for Evaluation: We use the number of obstacles passed as a discrete performance metric, which inherently introduces variability. Since this metric represents distinct thresholds rather than a continuous range, small differences in performance can result in significant deviations in the reported averages.

ii) Variance in Convergence Across Seeds: Our method exhibits variability in the number of training epochs required to converge across different random seeds. This inconsistency in convergence times contributes to fluctuations in the performance metrics, leading to a higher standard deviation.

Q3: Lambda Curve Fig. 8

We would like to clarify that when we mentioned the lambda values staying within the range of 0.2–0.4, we were specifically referring to the mean of the lambda values. The primary point we intended to emphasize was the relatively constant behavior of lambda during the Leap task while it is constantly increasing for the Crawl and Climb tasks. This variation highlights how lambda adapts differently depending on the task dynamics and requirements.

To avoid any ambiguity, we will revise the text in the paper to make this explanation clearer and more precise.

Q4: Comparison against fixed lambda value

We would first like to clarify that we do not consider a single optimal λ value to exist for the entire training process. Instead, the purpose of adapting λ during training is not to find one "correct" value, but to dynamically adjust it based on the current training phase.

For example, at the beginning of training, λ should remain small so that the agent focuses solely on the task reward. This is beneficial because the agent can develop a straightforward forward-moving policy without requiring additional exploration. However, when the agent encounters an obstacle, λ should increase to encourage exploration and allow the agent to discover a solution. If λ were large from the beginning of the episode, the agent might not focus effectively on maximizing the task reward. Similarly, as the task is being solved, the optimal value of λ may change depending on the requirements of that phase.

In summary, the optimal λ value varies across different phases of training, and our method provides a way to automatically adjust λ to maximize task rewards at each stage. Therefore, we do not aim to find a single λ value that performs best across all stages of training. This is why we did not include the final λ value from the training run in our results.

That said, if you are curious, we conducted additional experiments using the final λ value from the "crawl" task (0.58) and reran the experiments with a fixed λ value of 0.58. We have provided the plot in the supplementary material. Although our method was able to solve the task, using a fixed λ value of 0.58 introduced challenges in solving the task effectively. If needed, we are willing to run further experiments with corresponding fixed λ values for other tasks.

Thank you for your thoughtful review. Below, we provide detailed answers to your comments.

W2: Additional Baselines

Thank you for suggesting additional baselines. In response, we have incorporated both DIAYN and Robot Parkour Learning into our evaluation. For DIAYN, we implemented a discrete version using PPO with a total of five discrete skills. This addition expands our evaluation to include exploration-based methods (RND), mutual information-based skill discovery methods (DIAYN), and distance maximization-based skill discovery methods (METRA). We also added Robot Parkour Learning as a specialized reinforcement learning baseline with a complex reward structure. We trained the parkour policy with soft dynamics for 70% of the training process, then fine-tuned it with hard dynamics for the remaining 30%, following the procedure outlined in the original paper. However, we were not able to train a usable parkour policy for Crawl even after following the reward terms and process as mentioned in the paper.

For skill discovery methods, we sampled a new skill at the beginning of each episode and kept it fixed throughout that episode. For Ours (fixed z) we used a single fixed z value for all the episodes. The table below summarizes our findings, reporting the average number of obstacles passed (maximum of 3) over 100 episodes:

From the table, we observe that our method performs on par with Robot Parkour Learning. This demonstrates that combining skill discovery with task rewards using our framework achieves approximately the same performance as a reinforcement learning policy with manually tuned reward functions, but without the need for labor-intensive reward engineering. We also want to emphasize that the success rate of our method depends on the skills sampled at the start of each episode. These findings align with the observations reported in Table 1 of our main paper. We show that conditioning the policy on a fixed skill for all the eval episodes could enable our method to consistently perform well across all tasks. Notably we see a ~90% increase in the success rate for Crawl, and a near perfect success rate for Leap and Climb tasks outperforming even Robot Parkour Learning.

Through our evaluation, we find that DIAYN performs well on the Climb task but struggles with the Leap task and completely fails on the Crawl task. We attribute this to the nature of DIAYN's objective function, which maximizes mutual information (MI) using KL divergence. This approach often leads to less distinctive behaviors, as KL divergence is fully maximized when two distributions have no overlap. Beyond this point, there is no additional incentive to further distinguish between distributions. As a result, DIAYN suffers from poor exploration compared to distance-maximization-based skill discovery methods, which are better suited for generating diverse and effective skills.

W3: Hardware Validation and Unnatural Motion

We appreciate the reviewer’s suggestion to include hardware results and their concerns about unnatural motion. However, due to time constraints and the labor-intensive nature of hardware experiments, we are unable to provide hardware results for our method at this stage. While our approach generally produces natural-looking motions, we acknowledge that some tasks may not directly translate to real-world applications. The primary objective of this work is to demonstrate an effective method for combining skill discovery techniques with task rewards, eliminating the need for additional approaches like reward engineering to achieve agile motions. Investigating the applicability of our method to real-world robotic tasks remains an exciting avenue for future research.

W4 & 5: Task-related plots and Failure cases in Videos

Thank you for your suggestion. We kindly request the reviewer to clarify the specific type of plots they would like us to include. Additionally, we will incorporate more video experiments in line with the reviewer's recommendation.

W3.2: Additional Baselines

Thank you for suggesting additional baselines. In response, we have incorporated both DIAYN and Robot Parkour Learning into our evaluation. For DIAYN, we implemented a discrete version using PPO with a total of five discrete skills. This addition expands our evaluation to include exploration-based methods (RND), mutual information-based skill discovery methods (DIAYN), and distance maximization-based skill discovery methods (METRA). We also added Robot Parkour Learning as a specialized reinforcement learning baseline with a complex reward structure. We trained the parkour policy with soft dynamics for 70% of the training process, then fine-tuned it with hard dynamics for the remaining 30%, following the procedure outlined in the original paper. However, we were not able to train a usable parkour policy for Crawl even after following the reward terms and process as mentioned in the paper.

For skill discovery methods, we sampled a new skill at the beginning of each episode and kept it fixed throughout that episode. For Ours (fixed z) we used a single fixed z value for all the episodes. The table below summarizes our findings, reporting the average number of obstacles passed (maximum of 3) over 100 episodes:

From the table, we observe that our method performs on par with Robot Parkour Learning. This demonstrates that combining skill discovery with task rewards using our framework achieves approximately the same performance as a reinforcement learning policy with manually tuned reward functions, but without the need for labor-intensive reward engineering. We also want to emphasize that the success rate of our method depends on the skills sampled at the start of each episode. These findings align with the observations reported in Table 1 of our main paper. We show that conditioning the policy on a fixed skill for all the eval episodes could enable our method to consistently perform well across all tasks. Notably we see a ~90% increase in the success rate for Crawl, and a near perfect success rate for Leap and Climb tasks outperforming even Robot Parkour Learning.

Through our evaluation, we find that DIAYN performs well on the Climb task but struggles with the Leap task and completely fails on the Crawl task. We attribute this to the nature of DIAYN's objective function, which maximizes mutual information (MI) using KL divergence. This approach often leads to less distinctive behaviors, as KL divergence is fully maximized when two distributions have no overlap. Beyond this point, there is no additional incentive to further distinguish between distributions. As a result, DIAYN suffers from poor exploration compared to distance-maximization-based skill discovery methods, which are better suited for generating diverse and effective skills.

Due to time and computational constraints, we are unable to conduct additional experiments with more seeds and baseline comparisons for the Wall-Jump task at this time. However, we will ensure that these additions are included in the final version of the paper.

W4: Related Works

We appreciate the reviewer’s suggestions and agree that [1][2] are highly relevant works leveraging motion priors to achieve agile locomotion. Similarly, [3][4] are valuable contributions that aim to combine unsupervised reinforcement learning and imitation learning to address the challenge of complex reward engineering. We will update the related works section of our paper to include these references.

[1] Han, Lei, et al. "Lifelike agility and play in quadrupedal robots using reinforcement learning and generative pre-trained models." Nature Machine Intelligence 6.7 (2024): 787-798.

[2] Yang, Ruihan, et al. "Generalized animal imitator: Agile locomotion with versatile motion prior." arXiv preprint arXiv:2310.01408 (2023).

[3] Peng, Xue Bin, et al. "Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters." ACM Transactions On Graphics (TOG) 41.4 (2022): 1-17.

[4] Escontrela, Alejandro, et al. "Adversarial motion priors make good substitutes for complex reward functions." 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022.

We thank the reviewer for the constructive feedback and suggestions to improve our work. Please find our answers to the questions below.

W1, W2: Hardware Validation, Unnatural Motion and physics paramters

We appreciate the reviewer’s suggestion to include hardware results and their concerns about unnatural motion. However, due to time constraints and the labor-intensive nature of hardware experiments, we are unable to provide hardware results for our method at this stage. While our approach generally produces natural-looking motions, we acknowledge that some tasks may not directly translate to real-world applications. The primary objective of this work is to demonstrate an effective method for combining skill discovery techniques with task rewards, eliminating the need for additional approaches like reward engineering to achieve agile motions. Investigating the applicability of our method to real-world robotic tasks remains an exciting avenue for future research.

We will add the details about the physics parameters in the appendix of the paper

W3.1: Evaluation environments

As the reviewer mentioned, we evaluated our methods on Leap, Crawl, and Climb tasks as well as on a challenging wall-jump task. We followed Robot Parkour Learning's task environment for evaluation. We will be happy to evaluate on more difficult tasks on the reviewer's suggestion.

Thank you for the efforts in providing additional comparisons against baselines. However, I am keeping my original evaluation of the work, primarily due to the absence of real robot evaluation. Given the recent developments in locomotion research, it has become standard for most works, including agile locomotion methods, to demonstrate their approach on real hardware. Considering the limited novelty of the proposed method, real robot evaluation is essential to justify publication.

Additionally, I have the following minor concerns: Unnatural Motion: The generated motions appear unnatural in certain scenarios, which raises questions about the applicability of the method to real-world tasks. Lack of Multi-Seed Results: For the most challenging task, the wall-jump, results have not been evaluated across multiple seeds. This makes it difficult to assess the robustness and consistency of the proposed method.