Response to Reviewer 9L23

Comment

Thank you for your review and comments. To respond/comment on your listed weaknesses:

W1: We have fixed the typos in our paper. The task is to tracking the user command including linear velocity in x-axis, linear velocity in y-axis and yaw angular velocity, The range differs from terrain type, for example, the command of linear velocity in x-axis on slopes and rough slopes is [-3.0, 3.0] m/s and [-1.0, 1.0] m/s on stairs, the range of linear velocity in y-axis is [-1.0, 1.0] m/s on all terrains, the range of yaw angular velocity on slopes and rough slopes is [-3.0, 3.0] rad/s and [-1.0, 1.0] rad/s on stairs. We also update a more clear decription of technical details in $\color{red}{section 3}$ in current version.

W2: We have done more evaluation of our method.

• We conduct quantitative comparison with various baselines in both simulation and real world in $\color{red}{section 4.3}$ and $\color{red}{section 4.4}$ in current version.

• We already clearify the task description of both simulation and real world benchmarks. In simulation benchmarks, we test the trackng error across various terrains and command ranges. In real world benchmarks, we set up three types of terrains to evaluate the performance of real world deployment, including success rate, number of stairs, maximum weights of payload, and other metrics. You can checkout the details in $\color{red}{section 4.3}$ and $\color{red}{section 4.4}$ in current version.

• We compared our method with the baselines including the regression method, RMA, MoB, Built-in MPC and ours w/o auxiliary loss, the experimental results show that our method outperform other baselines in both simulation and real-world. The experimental details are in $\color{red}{section 4.3}$ and $\color{red}{section 4.4}$.

• We conduct performance analysis of our method and other baselines across different terrains, command ranges and command types. For our tasks, the only target is to track the velocity commands. You can checkout the details in $\color{red}{section 4.3}$ and $\color{red}{section 4.4}$ in current version.

• We conduct extensive ablation studies on contrastive learning and regression. To be more specificly, we perform ablation study with the following methods:

  • Baseline: Training the policy with no auxiliary task, but with a history encoder of the same architecture training as our method, the history encoder is trained with policy update.
  • Ours w/o Velocity Input: Training the policy the same as our method, but set the position of velocity input to 0 for the policy network.
  • Ours w/o Velocity Loss: Training the policy the same as our method, but without the loss of velocity estimation and set the position of velocity input to 0 for the policy network.
  • Ours w/o Latent Input: Training the policy the same as our method, but set the position of latent input to 0 for the policy network.
  • Ours w/o Latent Loss: Training the policy the same as our method, but without the loss of dynamics-aware estimation and set the position of latent input to 0 for the policy network.
  • Regression: Training policy with velocity estimation and dynamics-aware estimation, but the auxiliary is to simply regress them in latent space.
  • Oracle: Training policy with a history of full observations.

  The experiments include simulation and real-world, showing that our method reaches good performance across simulation and real-world. You can see $\color{red}{section 4.5}$ for detail information.

Questions

Q1: We answered this question in W1. A more clear decription of technical details can be found in $\color{red}{section 3}$.

Q2: We answered this question in the point 1,2,3 of W2. We compared our method with the following baselines including the regression method, RMA, MoB, Built-in MPC and ours w/o auxiliary
task, the result shows that our method ou-perform in both simulation and real-world, the experiments are in $\color{red}{section 4.3}$ and $\color{red}{section 4.4}$.

Q3: We answered this question in the point 5 of W2. We perform ablation studies with various methods. The experiments include simulation and real-world, showing that our method reaches good performance across simulation and real-world. You can checkout $\color{red}{section 4.5}$ for detail information.

Q4: We answered this question in the point 4 of W2. Our new experiments in $\color{red}{section 4.3}$ and $\color{red}{section 4.4}$ include the performance of our method. In simulator, we test the tracking error over different terrains in different command ranges; in real world we setup a benchmark including stairs, unseen terrains and anti-disturbance, our method also out-performs in almost all the task.

We also conduct real-world experiments to compare our method with RMA[1], MoB[2] and Built-in MPC, the detailed benchmark setting are in $\color{red}{appendix.2}$, which is a little long for comments here. The results show that our method have good performance in real world application:

Reference

[1] Kumar, et al. RMA: Rapid Motor Adaptation for Legged Robots

[2] Margolis, et al. Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior

In order to compare our method with other blind locomotion method, we conduct a tracking error($\color{red}{the \ Euclidean \ distance \ between \ real \ velocity \ and \ target \ velocity}$) benchmark in simulation. The tracking error on slopes and rough slopes are tested in the range $[-1.0,\, 1.0] \,\text{m/s} \times [-1.0,\, 1.0] \,\text{m/s} \times [-1.0,\, 1.0] \,\text{rad/s}$ and $[-2.0,\, 2.0] \,\text{m/s} \times [-1.0,\, 1.0] \,\text{m/s} \times [-2.0,\, 2.0] \,\text{rads/s}$. All other tests are conducted within the range $[-1.0,\, 1.0] \,\text{m/s} \times [-1.0,\, 1.0] \,\text{m/s} \times [-1.0,\, 1.0] \,\text{rad/s}$. These ranges correspond to linear velocity in the forward direction, lateral directions, and angular velocity about the vertical axis, respectively.

When testing the linear velocity tracking error, the angular velocity target is set to 0. For the angular velocity tracking error tests, the linear velocity target is 0. The 'combined velocity' tests are conducted without setting either velocity component to 0. We uniformly sample test commands within these ranges. The environmental conditions are set to $\text{level}^{\text{terrain}} = 1, 2, 3, 4$, with an equal distribution of 25\mathrm{percent} for each. $\text{level}^{\text{terrain}} = 1, 2, 3, 4$ means that the slopes and rough slopes have a inclination of 4.44, 8.89, 13.33, 17.78 degree respectively, the rough slopes also have uniform noise of amplitude 1.78, 2.56, 3.33, 4.11 cm, the stairs' heights are 7, 9, 11, 13 cm respectively, the discrete obstacles' heights are 6.11, 7.22, 8.33, 9.44 cm respectively.

The compared methods include ours, regression version of ours, baseline, RMA[1] and MoB[2], the results are shown in the following table:

The unit of tracking error for linear velocity and angular velocity are $m/s$ and $rad/s$ respectively. In most of the complex terrains, our method outperforms other methods sigificantly.

Reference

[1] Kumar, et al. RMA: Rapid Motor Adaptation for Legged Robots

[2] Margolis, et al. Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior

Response to Reviewer Yqja

Comment:

Thank you for your review and comments. To respond/comment on your listed weaknesses:

W1: Thanks for your advice, we carefully checked the typos and incorrect reference in our paper, we have already corrected them in a the current version.

W2: We have done some ablation study on the use of velicity estimation, while only using implicit internal model can already achieve satisfying performance, the experimental results show that using velicity estimation can improve the performance. However, the implicit internal model plays the main role of performance. You can refer to our new ablation study in $\color{red}{section 4.5}$ both in simulation and real-world.

W3: We will answer the questions in the following contents.

Questions

Q1: Yes, our "dynamics model" is not the typical "dynamics model" or "world model" in Reinforcement Learning. We do not rollout on "dynamics model" to generate synthetic interaction data. Instead, the "dynamics model" is trained to help the policy understanding that under current state, if we do not take any action, how will the robot move. This in fact reflects the properties of the physical world, letting the policy determine how to react to achieve our command. More specifically, we treat the external environment as a kind of system disturbance(the impact of friction, resitution, terrain, etc), then use the "dynamics model" or "internal model" in our work to simulate the successor state of the system in order to estimate the potential disturbance. We think that "dynamics-aware model" should be a more accurate description, we already corrected this term in current version.

Q2: We have done the ablation study between contrastive learning and regression, it shows that contrastive learning performs better, the superior performance comes from the utlization of batch level information and discriminative representation. Furthermore, we also conduct ablation studies on different variants of our method. You checkout the details in $\color{red}{section 4.5}$.

Q3: Thank you for your advice on experiments, we conduct the ablation on the number of prototypes in $\color{red}{section 4.6}$, we also investigate the influence of velocity estimation(both input and auxiliary loss) and latent (both input and auxiliary loss) in $\color{red}{section 4.5}$, which shows that the latent(dynamics-aware) representaion plays a main role in the performance.

Response to Reviewer FQfg

Comment:

Thank you for your review and comments. To respond/comment on your listed weaknesses:

W1: The main difference between our work and "world models" is that we do not need the rollout process on "world models" to generate synthetic interaction data. In our work, the "dynamics model" is trained to help the policy understanding that under current state, if we do not take any action, how will the robot move. This in fact reflects the properties of the physical world, letting the policy determine how to react to achieve our command. More specifically, we treat the external environment properties(the impact of friction, resitution, terrain, etc) as a kind of disturbance, then use the "dynamics model" or "internal model" in our work to simulate the successor state of the system in order to estimate the potential system disturbance. For clarifcation, we refer to use "dynamics-aware model" in current version.

Questions

Q1: The relation between our work and "world models" is described in the comment to weakness. Besides, world models take historical state and action as input, while our method takes only historical state as input. Therefore, our method can not be considered as a world model. In fact, it comes from the idea of Internal Model Theory[1], which simulates the successor state of the robotic system in order to estimate the system disturbance brought by the environment. We add our inspiration and the relation between our method and world model in $\color{red}{section 3.3}$.

Reference

[1] Rivera, et al. Internal model control: PID controller design

Response to Reviewer SBNA

Comment:

Thank you for your review and comments. To respond/comment on your listed weaknesses:

W1: Previous works[1, 2, 3] use a mimic learning framework to predict unavailable terrain information in latent space, which entails additional performance reductions.Our method directly treats different terrains as perturbations, and try to learn the best response under different observations with the help of critic module, which is able to sense terrain information. A vivid analogy is that previous work trains a dog that can see the terrain and then blindfolded it to achieve prior performance; the dog we trained has never seen the terrain, and learns to react directly to a variety of different situations.

W2: Perceptive locomotion require to utilize a wealth of sensor information, such as depth cameras, ToF cameras or LIDAR, to capture height information about the surrounding terrain, which plays a crucial role. However, most of the sensors mentioned above are very expensive and fragile, and the vast majority of quadruped robots are not equipped with these sensors and can only use feedback information from a basic IMU and motors. For example, Mini-cheetah only equips IMU and joint encoder, and Unitree Go1's default depth camera is hard to tackle. Besides, our policy do not always hit the vertical edge, in most of the environments, it spontaneously use its front leg to sense the front terrain, the videos in demo shows this strategy.

W3: We are very thankful for your advice on experiments, we have already conducted extensive experiments in current version, including simulation benchmarks $\color{red}{section 4.3}$, real-world benchmarks $\color{red}{section 4.4}$, ablation study $\color{red}{section 4.5}$ and influence of prototypes $\color{red}{section 4.6}$.

W4: We are sorry for the wrong setting of oracle, we only include the current oracle observation in the policy input rather than a history of oracle observations, which may decrease the performance and learning speed of oracle policy, the new curve is also included in the paper $\color{red}{section 4.5}$.

W5: We are working on refine and adding comments to our code, we will try our best to release the code as soon as possible. In the current revision, we add our training hyper parameters in the appendix, which can be used directly for training.

Questions

Q1: The hybrid internal model is co-trained with the policy. In each iteration, we first collect trajectories, then train the hybrid intrernal model with the proposed method, after that, we froze the hybrid internal model and update the actor-critic part. We refined the describtion in our paper $\color{red}{section 3.5}$

Reference

[1] Kumar, et al. RMA: Rapid Motor Adaptation for Legged Robots

[2] Lee, et al. Learning Quadrupedal Locomotion over Challenging Terrain

[3] Miki, et al. Learning robust perceptive locomotion for quadrupedal robots in the wild

The training curve is in $\color{red}{section4.5}$, which can not be included in the comments due to the limitation of openreview. However, the real-world experiments(same as the comparision method) can show the improvement of our dynamics-aware modeling and contrastive learning method:

We also conduct real-world experiments to compare our method with RMA[1], MoB[2] and Built-in MPC, the detailed benchmark setting are in $\color{red}{appendix.2}$, which is a little long for comments here. The results show that our method have good performance in real world application:

Reference

[1] Kumar, et al. RMA: Rapid Motor Adaptation for Legged Robots

[2] Margolis, et al. Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior

In order to compare our method with other blind locomotion method, we conduct a tracking error($\color{red}{the \ Euclidean \ distance \ between \ real \ velocity \ and \ target \ velocity}$) benchmark in simulation. The tracking error on slopes and rough slopes are tested in the range $[-1.0,\, 1.0] \,\text{m/s} \times [-1.0,\, 1.0] \,\text{m/s} \times [-1.0,\, 1.0] \,\text{rad/s}$ and $[-2.0,\, 2.0] \,\text{m/s} \times [-1.0,\, 1.0] \,\text{m/s} \times [-2.0,\, 2.0] \,\text{rads/s}$. All other tests are conducted within the range $[-1.0,\, 1.0] \,\text{m/s} \times [-1.0,\, 1.0] \,\text{m/s} \times [-1.0,\, 1.0] \,\text{rad/s}$. These ranges correspond to linear velocity in the forward direction, lateral directions, and angular velocity about the vertical axis, respectively.

When testing the linear velocity tracking error, the angular velocity target is set to 0. For the angular velocity tracking error tests, the linear velocity target is 0. The 'combined velocity' tests are conducted without setting either velocity component to 0. We uniformly sample test commands within these ranges. The environmental conditions are set to $\text{level}^{\text{terrain}} = 1, 2, 3, 4$, with an equal distribution of 25% for each. $\text{level}^{\text{terrain}} = 1, 2, 3, 4$ means that the slopes and rough slopes have a inclination of 4.44, 8.89, 13.33, 17.78 degree respectively, the rough slopes also have uniform noise of amplitude 1.78, 2.56, 3.33, 4.11 cm, the stairs' heights are 7, 9, 11, 13 cm respectively, the discrete obstacles' heights are 6.11, 7.22, 8.33, 9.44 cm respectively.

The compared methods include ours, regression version of ours, baseline, RMA[1] and MoB[2], the results are shown in the following table:

The unit of tracking error for linear velocity and angular velocity are $m/s$ and $rad/s$ respectively. In most of the complex terrains, our method outperforms other methods sigificantly.

Reference

[1] Kumar, et al. RMA: Rapid Motor Adaptation for Legged Robots

[2] Margolis, et al. Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior