HuRi : Humanoid Robots Adaptive Risk-ware Distributional Reinforcement Learning for Robust Control

We sincerely appreciate your valuable comments and the time you have taken to review our paper. In the following sections, we address the concerns you raised and summarize the revisions made to the paper. To facilitate your review of these modifications, we have attached the revised manuscript.

The paper presentation needs a lot of work. There are numerous small things and typos that add up and make the paper difficult to understand. First, the citations are not well formatted and this makes reading the manuscript difficult. Secondly, the notation is not consistent: sometimes x is used for the state, and some other times s is used for the state. Another example is that the paper is about humanoid robots and then in the theorem the authors state "We describe the locomotion problem of quadruped robots". Overall, the presentation and writing of the paper requires quite some work.

A: In the revised version, we corrected all citation formatting to improve clarity. We also unified the notation throughout the manuscript to avoid ambiguity. Furthermore, we extensively revised the paper to eliminate typos and optimized its structure to enhance readability. Thank you very much for your careful review.

As far as I can tell (given the difficulty of understanding the manuscript described above), the authors claim that the novelty lies in a) the usage of an adaptive mechanism for adapting the β parameter of the distortion function, and b) the training of the RND network for β adaptation. Although I can see value in the adaptation mechanism, the motivation of using the RND network is rather weak. Similarly, the evaluation and discussion on why it works better is again rather weak. I would have expected a more thorough discussion and analysis since RND seems to be the critical factor in the performance gains.

A: In the revised version, we clarified the motivation for using RND in lines 300-303. Previous studies have categorized uncertainty into intrinsic uncertainty and parameter uncertainty. To more accurately measure environmental uncertainty, we used the IQR module to measure intrinsic uncertainty and the RND module to measure parameter uncertainty. To further validate the roles of RND and IQR, we conducted ablation studies detailed in Appendix A.5. The results show that combining both types of uncertainty yields better performance than measuring a single type of uncertainty.

The experimental section requires more work. For example, in Fig. 3 (and the describing text) we do not know if the authors ran just one training and smoothed out rewards or they ran multiple seeds and take some statistics. Similarly, for the experiments shown in Fig. 4, the authors do not mention how many seeds they used. They do mention the number of seeds for the experiments shown in Tab. 1. Overall, this goes back to presentation; the manuscript requires substantial work to make it understandable.

A: In the revised version, we significantly improved the descriptions of the experimental settings. We randomly selected five seeds for each method and repeated the training process five times. The aggregated results are presented in the updated Figure 5, further validating the stability and feasibility of our method. Additionally, we explicitly stated the number of seeds and repetitions used for each experiment. Finally, we included hyperparameters and details of the training and testing processes in Appendix A.3 to facilitate reader understanding.

In the current form, we cannot really evaluate the real-world experiments. Figures 6,7 and 8 do not showcase anything that is written in the text. We cannot validate what's happening here. The video showcases a lot of nice behaviors, but we need a more rigorous evaluation here. It is very difficult to evaluate the performance of the policy at the moment.

A: Currently, in the revised version, we added an additional real-world experiment, demonstrating stable walking on four different ground friction conditions. Additionally, we are conducting comparisons between HuRi and the baselines on the physical robot. However, the testing process is time-consuming. We are working on obtaining quantitative experimental results for HuRi and the baselines, and we will update these results in the paper within the next two days. We greatly appreciate your patience and understanding.

Connect to the previous response:

I do not get how the proposed approach is specifically suited for humanoids (which the author attempt to motivate, but fail imho).

A: Since humanoid robots rely on bipedal balance control, they are more prone to losing balance under external disturbances compared to other robots. Our approach mitigates this issue by perceiving risks and adjusting the risk preference of the policy accordingly. We demonstrated in both simulation and real-world experiments that our method outperforms others in terms of stability and robustness. While we argue that our method is particularly suitable for humanoid robots, it is not limited to them. Additional experiments in Appendix A.6 show that our method is also effective on quadruped robots.

Dear Reviewer,

Please provide feedback to the authors before the end of the discussion period, and in case of additional concerns, give them a chance to respond.

Timeline: As a reminder, the review timeline is as follows:

November 26: Last day for reviewers to ask questions to authors.

November 27: Last day for authors to respond to reviewers.

Thank you for recognizing our efforts in revising the paper. We sincerely appreciate your valuable comments and the time you have taken to review our paper. We address the concerns you raised:

My comment now becomes "the performance gain is not significant enough" to justify a whole novel approach. Overall, it's hard to see the algorithmic contribution of the manuscript.

A: In terms of the algorithm, we propose for the first time a comprehensive approach to measure uncertainty in the environment by considering both intrinsic uncertainty and parameter uncertainty. By combining these two types of uncertainty, we can more accurately assess the environmental uncertainty. To demonstrate the superiority of our method, we tested it under out-of-distribution disturbances. In the presence of external force interference, our method improved the success rate by nearly 20% compared to the baseline. Under load and mass disturbances, the velocity tracking error of our method consistently remained the lowest. This advantage is even more pronounced in real-world experiments. In scenarios involving external forces on the robot, the success rate of our method (65%) was significantly higher than that of the baseline (35%). When additional loads were applied to the center of mass and feet, we tested the velocity error rates at speeds of 0.3 m/s, 0.6 m/s, and 0.9 m/s. Our method consistently maintained the lowest velocity error rate. With the inclusion of quantitative real-world experiments, the advantages of our method became even more evident.

Then why argue that your method is particularly suitable for humanoid robots?

A: We would like to provide further clarification on this point. Compared to quadruped robots, wheeled robots, and humanoid robots exhibit smaller stability margins, making them more sensitive to disturbances. Our method, through distributional reinforcement learning, allows for risk modeling of the surrounding environment (such as terrain noise, sensor noise, etc.) during the robot's walking process. This guides the robot to explore the action space more safely and efficiently, leading to improved generalization and robustness in real-world deployments. While previous research on safety reinforcement learning has achieved success on quadruped robots, quadrupeds have larger stability margins, and recent studies have demonstrated that they are capable of overcoming, rather than avoiding, such risks. In contrast, for the current and future application scenarios of humanoid robots, robustness and safe walking are more critical than agility. Therefore, we place greater emphasis on risk-aware policies, which makes our approach particularly well-suited for humanoid robots.

I would really like to see the quantitative results on the physical robot.

A: We have updated the revised version. In the revised version, we included quantitative real-world experiments to further validate the effectiveness of our approach. First, we employed a pendulum system to generate a fixed lateral impact force and tested the success rate of the robot at a velocity of 0.6 m/s. A failure was defined as the robot falling over. Experimental results demonstrated that our method achieved the highest success rate, outperforming other methods. Additionally, we evaluated the velocity error rate under the condition of adding a 5 kg load to the robot's center of mass. Through these experiments, we further assessed the impact of load on the robot's stability. Another setting of experiments was designed to apply tensile force on the robot's legs by adding loads to its feet, testing the velocity tracking error rate under this condition. The results showed that, in both scenarios, our method consistently maintained the lowest velocity error rate, demonstrating that our method is more effective at resisting out-of-distribution disturbances. These experiments verified the effectiveness of our method under complex conditions. We firmly believe that these results clearly highlight the superiority of our approach, fully addressing the reviewers' concerns.

Thank you for your detailed review and valuable suggestions. We hope that our revisions have addressed all your concerns and questions. Please let us know if there are any remaining issues.

Dear Reviewer 1gwC,

 We sincerely appreciate your insightful comments and valuable suggestions on improving our work. As the final deadline approaches, we would kindly like to ask whether our responses have sufficiently addressed your questions and concerns. We are happy to engage in further discussion or provide any clarifications needed, and we welcome any additional feedback to strengthen our work before the rebuttal period ends.  Thank you for your time and thoughtful reviews. 

Kind regards,

 Authors

We sincerely appreciate your valuable feedback and the time you have taken to review our paper. In the following, we address the issues you raised and summarize the revisions made to the paper.

Effectiveness of Wang function needs to be verified. Considering add some experiments to verify the effectiveness of Wang function as it is one of the core contribution of this paper, for example, adjusting the risk sensitivity of CVaR with intrinsic uncertainty and parameter uncertainty and compare with your method.

A: In the revised version, we employed the CVaR distortion function in combination with the RND module and the IQR module to adaptively adjust the risk sensitivity. This experiment is documented in Appendix A.8. The results indicate that, despite trying multiple hyperparameter configurations, using the CVaR distortion function consistently led to training failures. We have conducted a detailed analysis of the reasons for these failures in Appendix A.8, further demonstrating that the Wang function is more compatible with the RND and IQR modules.

The real robot experiment is weak, authors should considering comparing your method and baselines on real robot.

A: Currently, in the revised version, we added an additional real-world experiment, demonstrating stable walking on four different ground friction conditions. Additionally, we are conducting comparisons between HuRi and the baselines on the physical robot. However, the testing process is time-consuming. We are working on obtaining quantitative experimental results for HuRi and the baselines, and we will update these results in the paper within the next two days. We greatly appreciate your patience and understanding.

The authors claim that "our method exhibits strong robustness in agents and bridges the gap between simulation environments and the real world", however, there is no content in the paper shows how the method benefits sim2real.

A: We sincerely apologize for the inaccurate wording in the original manuscript. In the revised version, we have updated the statement to read: "we demonstrate that our method exhibits strong robustness in agents and successfully validates sim-to-real transfer." Additionally, we have detailed our sim-to-real settings in lines 480-483.

Considering other recent work focusing on blind humanoid locomotion, such as [1], have much better performance on locomotion, I wonder if risk-aware reinforcement learning is harmful to locomotion task.

A: Current humanoid robots navigating extremely complex terrains, such as stairs, often require additional historical information inputs. Partial observations can only capture local information and fail to comprehensively reflect the complexity of the environment. Many existing methods, such as [1], address partial observability by incorporating historical information to expand the observation space. Our method focuses on enhancing safe exploration during the training process to improve the model's generalization ability, enabling it to resist out-of-distribution disturbances. Therefore, it does not incorporate historical information as input. We further validated that our method does not negatively impact locomotion tasks. First, we trained our method and baselines on multi-terrain environments and tested these models. The experimental settings and results are presented in Appendix A.7. The results show that our method achieved the highest reward and terrain level during training and the highest survival rate under external force disturbances during testing. Second, we conducted additional multi-terrain tests on a quadruped robot. The training setup and testing results are also included in Appendix A.6. During testing, we assigned the quadruped robot a challenging and risky task—walking down from a high platform. Our method achieved a significantly higher survival rate compared to the baseline. Both experiments conducted on multi-terrain settings demonstrate that our method does not harm locomotion tasks.

[1] Advancing humanoid locomotion: Mastering challenging terrains with denoising world model learning, Xinyang Gu, Yen-Jen Wang, Xiang Zhu, Chengming Shi, Yanjiang Guo, Yichen Liu, Jianyu Chen, RSS 2024

Dear Reviewer,

Please provide feedback to the authors before the end of the discussion period, and in case of additional concerns, give them a chance to respond.

Timeline: As a reminder, the review timeline is as follows:

November 26: Last day for reviewers to ask questions to authors.

November 27: Last day for authors to respond to reviewers.

Thank you for your patience. We have updated the revised version and incorporated your suggestions by adding comparisons with other baselines in the real-world experiments.

The real robot experiment is weak, authors should considering comparing your method and baselines on real robot.

A: We have updated the revised version. In the revised version, we included quantitative real-world experiments to further validate the effectiveness of our approach. First, we employed a pendulum system to generate a fixed lateral impact force and tested the success rate of the robot at a velocity of 0.6 m/s. A failure was defined as the robot falling over. Experimental results demonstrated that our method achieved the highest success rate, outperforming other methods. Additionally, we evaluated the velocity error rate under the condition of adding a 5 kg load to the robot's center of mass. Through these experiments, we further assessed the impact of load on the robot's stability. Another setting of experiments was designed to apply force on the robot's legs by adding loads to its feet, testing the velocity tracking error rate under this condition. The results showed that, in both scenarios, our method consistently maintained the lowest velocity error rate, demonstrating that our method is more effective at resisting out-of-distribution disturbances. These experiments verified the effectiveness of our method under complex conditions. We firmly believe that these results clearly highlight the superiority of our approach, fully addressing the reviewers' concerns.

Thank you for your thorough review and helpful suggestions. We hope we have addressed all your concerns and questions. Please let us know if there are any concerns.

Dear Reviewer 18Yh,

First and foremost, we sincerely thank you for your valuable feedback on our paper. Based on your suggestions, we have made several revisions, which are summarized as follows:

● We have elaborated on and demonstrated the role of the Wang_function.

● We have provided clarifications for certain terms and experimental details, and have modified sections that could lead to misunderstandings.

● We have added additional experimental comparisons with baselines in real-world experiments.

● We have empirically demonstrated that our method does not compromise the robot's motion control capabilities.

Once again, thank you for your time and effort in reviewing our work. We look forward to your feedback.

Best regards,

Authors of 14098

Thank you for taking the time to review our paper and for providing such helpful comments! We are glad that you like our ideas. You have raised several valuable suggestions, which we address below. Please let us know if you have any remaining questions or concerns.

Statistical robustness concerns: Figure 3 appears to show results from a single training seed, as evidenced by sharp performance dips. Multiple seeds are needed to validate learning stability, and success rate curves across seeds would better demonstrate robustness claims.

A: In the revised version, we selected five random seeds for each method and repeated the training process five times. The experimental results are shown in the updated Figure 3, further validating the stability and feasibility of our method.

Limited exploration of reward structure impacts: While the method shows improvements, there's no analysis of how dependent these gains are on their specific reward formulation. Recent work [1] has shown phase/clock-based rewards may not be optimal for robust locomotion. A comparison with alternate reward structures would strengthen their claims.

A: To demonstrate that our method does not depend on specific reward formulations, we trained and tested our approach on a quadruped robot using other reward formulations. Details of the training and testing are provided in Appendix A.6. The training results show that our method achieved higher rewards and reached higher terrain levels faster than the baselines, further proving its ability for safe exploration during training. During testing, we assigned the quadruped robot a challenging and risky task—walking down from a high platform. Our method achieved a significantly higher survival rate compared to the baseline. These experiments demonstrate that our method performs well under different reward formulations.

Uncertainty measurement complexity: The combination of RND with distributional RL seems potentially involved. I did not quite understand the justification of why RND is needed on top of distributional uncertainty estimates. Did the authors try to explore whether simpler approaches could achieve similar results, for example just using RND?

A: In the revised version, we clarified in lines 282–289 of the manuscript that environmental uncertainty is divided into intrinsic uncertainty and parameter uncertainty. Intrinsic uncertainty can be directly measured using the IQR module for return distributions. To estimate environmental uncertainty more comprehensively, we used the RND module. The motivation for the RND module is explained in lines 300-303. We also conducted ablation studies, the details of which are presented in Appendix A.5. In these experiments, we tested a method that uses only the RND module. The results show that our method outperforms the approach with only the RND module, demonstrating that combining both types of uncertainties is more effective. Detailed analyses are provided in Appendix A.5.

For improved readability, best scores in Table 1 should be highlighted in bold or black.

A: In the revised version, the best scores in Table 1 are highlighted in bold. Thank you very much for your thorough and thoughtful review.

Section 4.2's experimental description needs clearer organization: explicitly state that the evaluation focuses on two distinct robustness tasks (handling continuous forces and sudden impacts) with corresponding training setups. Current wording creates ambiguity about whether evaluations were conducted on out-of-distribution scenarios.

A: In the revised version, we clarified the experimental settings to ensure that readers understand our evaluations were conducted on out-of-distribution scenarios. Additionally, we included a table in Appendix A.3 listing the domain randomization parameters used during training and testing to emphasize that our evaluation was conducted under out-of-distribution disturbances.

[1]Marum, Bart Jaap van et al. “Revisiting Reward Design and Evaluation for Robust Humanoid Standing and Walking.” ArXiv abs/2404.19173 (2024): n. pag.

Dear Reviewer,

Please provide feedback to the authors before the end of the discussion period, and in case of additional concerns, give them a chance to respond.

Timeline: As a reminder, the review timeline is as follows:

November 26: Last day for reviewers to ask questions to authors.

November 27: Last day for authors to respond to reviewers.

Connect to the previous response:

Real world experiments lack statistical or more analytical significance. Please include some information on how the experiments show that the risk-awareness is important and that your instantiation of the risk-aware framework out performs other real-world methods. To do this, you might report success rates and safety violations that the varying methods exhibit.

A: Currently, in the revised version, we added an additional real-world experiment, demonstrating stable walking on four different ground friction conditions. Additionally, we are conducting comparisons between HuRi and the baselines on the physical robot. However, the testing process is time-consuming. We are working on obtaining quantitative experimental results for HuRi and the baselines, and we will update these results in the paper within the next two days. We greatly appreciate your patience and understanding.

We are grateful for your valuable comments. In the subsequent sections, we address the concerns you raised. For your convenience in reviewing these modifications, we have attached the revised manuscript.

There are a series of minor typo errors as well as incorrect latex formatting. While most of these issues don't hinder understanding, they make it harder to read the paper.

A: In the revised version, we have corrected these spelling and formatting issues and improved the organization and language of the paper. Thank you very much for your careful review.

Presentation of method is unclear: throughout section 3, the authors introduce various components of their method. Because each component is described mainly in text, the structure of dependancies and framework is difficult to understand. Figure 2 is helpful, but not fully informative. I would suggest including an Algorithm Block that goes through each component step-by-step. Further, it would be helpful to include labels for all the equations/expressions that are splitting up the paragraphs throughout the method section.

A: In the revised version, we improved the organization of the method section by assigning sub-section titles to each module, making the structure clearer. We also updated Figure 2 to better illustrate the relationships and dependencies among components. Additionally, we included a detailed algorithm block in Appendix A.4 to describe the full step-by-step process.

Usage of SR(λ): This term is introduced around line 229 but never really described in great detail. In general, this paragraph from 226 to 240 is confusing and would benefit from being described through an algorithm block.

A: To provide a clearer explanation of the SR(λ) method, we added a detailed description of its algorithmic process in Appendix A.4 of the revised version.

The quantile loss in line 241 does not include the distribution over which the expectations are being taken.

A: In the revised version, we clarified that $\mathcal{T} Z_\theta(s)$ is the target distribution in line 236. The energy loss calculated in the equation represents the difference between the Critic's output distribution and the target distribution.

MSE loss in line 247 has incorrect notation that make its meaning ambiguous (mismatched paranthesis).

A: In the revised version, we corrected this notation error. Thank you for pointing this out and for your careful review.

$g_\beta^\text{Wang}(\tau)$ in line 254 is introduced without any additional supporting information. In the following paragraph, the authors describe the nuances of β but not where the function g is ever used. Please clarify its position in the method.

A: In the revised version, we clarified in line 264 that represents $h_\beta^\text{Wang}(\tau)$ the distorted quantile. To avoid confusion with the target network symbol 𝑔 introduced later, we replaced the notation $g_\beta$ with $h_\beta$.

Poor reward comparison experiment: The results in Figure 3 do not include any error bars or indicate any notion of statistical significance. It is necessary for the authors to run these experiments over multiple seeds and report aggregated performance. Also, is reward the right metric of performance? I would imagine that violation of safety constraints is a more important feature to report on.

A: In the revised version, we selected five random seeds for each method and repeated the training process five times. The aggregated results are presented in the updated Figure 3. These results further demonstrate the stability and feasibility of our method. Additionally, in lines 72-79, we explain that our method promotes safe exploration during training compared to other approaches that use fixed scalar risk parameters. To support this claim, we presented results showing higher rewards achieved by our method, which reflect the effectiveness of safe exploration during training.

Robot specific simulated experiments: While the results reported in the remainder of the simulated experiment section are important, they seem more like results that would be found in a pure robotics paper.

A: In the revised version, beyond the simulation experiment section, we added several extended experiments in Appendix. These include: Ablation studies demonstrating the effectiveness of jointly estimating intrinsic and parameter uncertainties; Experiments on a quadruped robot to verify that our method does not depend on specific reward structures; Multi-terrain testing that confirms our method improves the robot's motion control capabilities; Comparisons with the CVaR distortion function to show that the Wang function is more compatible with our approach. These additional experiments further highlight the generalizability and theoretical soundness of our method.

Dear Reviewer,

Please provide feedback to the authors before the end of the discussion period, and in case of additional concerns, give them a chance to respond.

Timeline: As a reminder, the review timeline is as follows:

November 26: Last day for reviewers to ask questions to authors.

November 27: Last day for authors to respond to reviewers.

The clarity of the updated manuscript has improved since the review process began. However, the manuscript is still lacking the rigorous, detailed, and most importantly, clear, descriptions of the method and reasoning. For instance, the expectations in equations 3,4, and 5 still do not have the distributions over which they are taken.

Dear Reviewer 7TAc,

First and foremost, we sincerely thank you for your valuable feedback on our paper. Based on your suggestions, we have made several revisions, which are summarized as follows:

● We have provided clarifications for certain terms and experimental details, and have modified sections that could lead to misunderstandings.

● We have added additional experimental comparisons with baselines in real-world experiments.

● We have included additional ablation studies in the appendix to further substantiate our method.

Once again, thank you for your time and effort in reviewing our work. We look forward to your feedback.

Best regards,

Authors of 14098

Hello Authors,

The clarifications you have made do in fact help the paper and I appreciate the effort. The increased clarity helps the paper and I am willing to increase my score to a 5.

However, I am still not satisfied with the presentation of the technical details and preliminaries, especially in sections 3.2 and 3.3. There is still a lack of rigor in the presentation of the mathematical elements of the method.

I appreciate all your efforts in improving the paper thus far.

Best, Reviewer 7TAc