Hierarchical World Models as Visual Whole-Body Humanoid Controllers

We thank the reviewer for their valuable feedback. We address your comments in the following.

Q: Although the motion produced by Puppeteer is more natural, why does the humanoid robot always lean forward while moving?

A: It is difficult to give a definitive answer to this question, but we conjecture that it may be one of two reasons, or more likely a combination of the two: (1) leaning forward with the torso creates momentum while being a relatively stable gait so it is possible that the high-level policy learns such behavior for this reason, and (2) there are many instances in the human MoCap dataset where the human leans forward while performing highly dynamic motions, which could bias the low-level tracking policy towards such a gait. For context, we include videos of various reference motions from the MoCap dataset along with our learned tracking policy on our project webpage; you will notice that the human reference motion in clip 9 (bottom row, second to right) leans forward while making a quick turn. We agree that it would be very interesting to study how different mixtures of MoCap data (e.g. excluding all “running” clips from the pretraining data) influence the behavior of learned policies, and we will be happy to include some exploratory results on this in the camera-ready version if the reviewer believes it would add value to the manuscript.

Q: Can the proposed method be validated in real-world experiments?

A: We agree that deploying our method on a real humanoid robot is a natural next step (which we intend to pursue), but believe that it would be outside of the scope of this manuscript. Deploying a learned policy on a relatively new hardware platform such as the Unitree H1 / G1 will require substantial hardware engineering and sim-to-real design effort, while our work focuses more on the algorithmic foundations. That said, we do believe deploying our method to real hardware would be possible without significant algorithmic changes.

Q: Can this framework be generalized to manipulation tasks in HumanoidOlympics?

A: Yes, this should indeed be possible, and we are interested in pursuing this in the future. However, we would like emphasize two things: (1) HumanoidOlympics is a strictly state-based benchmark whereas our method and benchmark is designed specifically for visuo-motor control (image inputs), and (2) both HumanoidOlympics and HumanoidBench (the two alternatives to our benchmark) were developed concurrently with our work.

We believe that our response addresses your main concerns. However, if that is not the case please let us know if you have any additional comments. We would greatly appreciate it if the reviewer could provide us with precise and actionable feedback such that we can fully address your concerns. Thanks again for your time and valuable feedback!

Thanks for your reply! However, as the authors do not conduct any experiment I proposed in the original review, I change my scores as follows: Rating: 5 and Confidence: 4. I believe HumanoidBench (https://arxiv.org/abs/2403.10506) is not a concurrent work, and this paper should include manipulation tasks, as they would be much more suitable and easier to evaluate motion naturalness than the locomotion tasks designed in the paper.

Furthermore, I do not think the results of locomotion naturalness are compelling enough to support the major claims in the paper especially the authors do not really compare the motion clips produced by this method with the real motion capture data in the user study but only compare with regular RL method.

We thank the reviewer for their valuable feedback. We address your comments in the following.

Q: the main advantage of this method over TD-MPC2 is the resulting naturalness of the motions. Can the authors elaborate on why they think the proposed method improves this aspect?

A: We attribute the naturalness of motions to pretraining on a large collection of human MoCap clips. By first training a single, reusable tracking (low-level) agent to track reference motions from this dataset, and then subsequently training a task-specific high-level agent to perform downstream tasks by providing commands to the low-level agent, our method implicitly leverages the behavioral priors obtained during pretraining. This is, to some extent, validated by our ablations in Figure 8 that vary the number of MoCap clips used during pretraining – we observe a strong correlation between number of MoCap clips and tracking performance. Lastly, per request of reviewer HShf we have added 4 new evaluations of our low-level tracking agent and its relevant ablations, which are included in our reply here https://openreview.net/forum?id=7wuJMvK639&noteId=y6ra8KpCFC as well as in Appendix B of our updated manuscript.

Q: can the authors elaborate on how the low-level policies exactly track the high-level commands? Since the low-level receives a sequence of commands does it keep using until the tracking error is below a threshold, or is it only used for a single step independent of the outcome of applying the one-step action?

A: The low-level agent is provided a command (or sequence of commands) by the high-level policy at every step, and both levels will replan a sequence of high-level commands + low-level actions regardless of whether the previous command was achieved or not. This design decision is consistent with the pretraining phase in which the tracking agent is trained to track a reference motion. We have added the sentence “The high-level policy outputs commands at a fixed frequency regardless of whether the previous command was achieved” to Section 3.2 (L215-216) which can be found in the updated manuscript. We hope that this makes it more clear, and thank the reviewer for pointing it out.

Q: on the methods side of things, the paper extends td-mpc2 to a hierarchical architecture. Can the authors compare the method to the hierarchical version of Dreamer [1]?

A: We agree that this would be an interesting comparison in principle. However, our empirical results in Figure 5 indicate that DreamerV3 does not achieve any meaningful performance even on the state-based humanoid control tasks (same as SAC), while TD-MPC2 converges in <1M environment steps. These results are in line with concurrent work [2] that also finds TD-MPC2 to outperform DreamerV3 by a large margin in state-based humanoid control. We thus decided to proceed with TD-MPC2 as our base algorithm in the remainder of our experiments which are all vision-based. We will be happy to include results for a hierarchical DreamerV3 in the camera-ready version if the reviewer believes that this will be informative, but we do not expect results to change much compared to the non-hierarchical version.

[2] Sferrazza, C., Huang, D., Lin, X., Lee, Y., Abbeel, P., “HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulation“ (2024)

We believe that our response addresses your main concerns. However, if that is not the case please let us know if you have any additional comments. We would greatly appreciate it if the reviewer could provide us with precise and actionable feedback such that we can fully address your concerns. Thanks again for your time and valuable feedback!

Thank you for your rebuttal. Unfortunately, the current form of the paper and the answers from the rebuttal fail to convey the contribution of this work. The paper mainly advances the naturalness of the humanoid motion. The reason for the emergence of this more natural motion (according to the authors) is the pretraining of the low-level tracking agent on a human MoCap dataset, which represents expert interactions with a very narrow distribution. Without extensive comparison of the proposed approach to other hierarchical frameworks that should equally leverage the MoCap dataset, the value of this work is unclear. Model-based methods are typically advantageous due to their sample efficiency. However, it is unclear whether this advantage remains in the presence of a dataset that can facilitate pretraining a low-level tracking policy. Such low-level trackers reduce the complexity of the problem and the advantage of using a TD-MPC style high-level planner is unclear. I advise the authors to include a comparison with other hierarchical frameworks based on Dreamer, as well as model-free methods. Otherwise, the value of this work is unclear. The work looks at an interesting problem and could potentially make a nice contribution. However, in its current form, it is not ready for publication and I still propose rejecting it unless the mentioned baselines are included in the comparison.

We are pleased to hear that our ablation with TD-MPC2 at the low level and SAC at the high level addresses your concern to some extent.

We are currently running the same ablation with DreamerV3 at the high level, but these experiments (3 tasks, 10 seeds each) will take a while since DreamerV3 is significantly slower than SAC/TD-MPC2. We estimate 3M environment steps (as is reported for our other methods) to take approx. 6 days. We will keep you posted with results as training progresses.

In the mean time, we would like to provide some additional context and push back on this claim a bit:

This baseline is a step in the right direction but is not sufficient on its own as SAC typically requires a substantially higher amount of samples than MBRL methods.

We do not believe there is substantial evidence for this in existing literature in the context of SAC vs. DreamerV3. The DreamerV3 paper does not compare against SAC at all, and does not consider humanoid control tasks, but several other related works have made this comparison. In particular, the TD-MPC2 paper benchmarks SAC and Dreamer-V3 on DMControl (including humanoids albeit with a lower DoF than we consider in this work) and find that DreamerV3 and SAC mostly have comparable data-efficiency but that DreamerV3 tends to converge to a higher asymptotic performance: approx. 750 vs. 600 mean reward on DMControl excluding Humanoid tasks, and approx. 850 vs. 500 reward on the Humanoid Walk task of DMControl. For reference, TD-MPC2 achieves approx. 900 reward on Humanoid Walk in 2M environment steps, whereas DreamerV3 achieves its 850 reward at around 12M environment steps. These numbers are echoed by Humanoid-Bench, which likewise benchmark SAC, DreamerV3, TD-MPC2 on state-based humanoid locomotion tasks. On this benchmark, TD-MPC2 performs significantly better than either method both in terms of asymptotic performance and data-efficiency, and SAC + DreamerV3 generally have similar convergence rate but DreamerV3 has higher asymptotic performance across the board compared to SAC. Our current results on our proposed benchmark are in line with previous results. We observe that both our hierarchical approach and single-level TD-MPC2 solves humanoid control tasks in <3M environment steps, whereas SAC and DreamerV3 fail to learn within 3M steps.

Thank you for working hard on implementing some of the proposed baselines. I strongly believe that they are needed to clearly understand your contribution. The results so far are looking very promising and I'm looking forward to the final results. Can you please elaborate on the experimental setup for this comparison. Mainly, I'm interested in how much effort was put into hyper-parameter tuning of the baselines in comparison to your method (especially when it comes to things like horizon length, latent space dimensionality/ general network architecture, batch size and learning rates).

As promised, we would like to update the reviewer with preliminary results for the baseline that consists of a high-level DreamerV3 w/ a low-level TD-MPC2 agent (pretrained as in our method). We run this baseline on three tasks: stand, walk, run with 10 random seeds per task. Results are shown on this link as well as in Figure 5 of our revised manuscript. We have results for up to 1.7M environment steps at the moment (56 hours of training) and expect it to take another 3 days for training to complete. However, we believe that the conclusion is clear: a TD-MPC2 backbone (which uses planning) is crucial to performance in whole-body humanoid control, at every level of the (hierarchical) architecture. We summarize our experimental results wrt this particular contribution as follows:

• Single-level SAC and single-level DreamerV3 achieves no meaningful performance on our tasks
• Single-level TD-MPC2 achieves good data-efficiency and asymptotic performance but produces highly unnatural motions
• High-level SAC + low-level TD-MPC2 achieves no meaningful performance
• High-level DreamerV3 + low-level TD-MPC2 achieves non-trivial performance on a limited number of tasks and random seeds but is unstable and prone to divergence
• Our method, Puppeteer, achieves strong data-efficiency, asymptotic performance, and produces natural and human-like motions
• Disabling planning at any level of our hierarchical approach degrades performance significantly (Figure 8)

We hope that these additional baselines (high-level SAC / DreamerV3 + low-level TD-MPC2) address the reviewer's concerns.

Edit: We have updated the manuscript + link shared here with current "DreamerV3 w/ low-level TD-MPC2" baseline results one last time before paper revision closes. We believe that performance for this baseline is unlikely to improve with additional training but will run it to completion.

We thank the reviewer for their valuable feedback. We address your comments in the following.

Q: How relevant is the metric of "naturalness" in real-world humanoid control, and is it sufficient to evaluate humanoid trajectory tracking effectively?

A: This is a great question. We do believe that policies that behave “naturally” and “human-like” are inherently valuable. For example, the physical safety of a robot and its surroundings is often a priority in robotics applications, which is not unlike when humans perform everyday tasks. It is desirable for humans and robots alike to complete tasks in ways that are efficient yet reliable and safe, and humans do tend to prefer when other agents (e.g.) behave in a predictable manner. This preference is, to some extent, validated in our user study. Biasing an RL policy towards human MoCap data is an effective way to embed behavioral preferences in a humanoid setting. Preferences can in principle be specified via reward engineering, but designing a reward that accurately conveys specific preferences can be very non-trivial compared to a more data-driven approach like ours. We focus on humanoids in this work, but our approach could in principle be applied to other embodiments for which a dataset of reference motions are available. At a high level, you can think of our problem setting as conceptually similar to the alignment problem in e.g. LLMs.

Q: It would be beneficial to include a comparative study on the survival rate or survival time when using the final-trained model

A: Agreed! Table 1 of our manuscript already compares gait + average episode length (survival time) over the course of training. Following your suggestion, we now report episode length at 1M environment steps + at convergence. We generally observe that TD-MPC2 achieves similar episode return and survival times as Puppeteer at convergence, but that Puppeteer has significantly higher survival times both throughout training and at the 1M snapshot. We conjecture that the behavioral prior of the low-level tracking agent is especially helpful in the early stages of training.

We bold numbers that are at least one std.dev. apart. We have updated Table 1 with these new metrics in our revised manuscript.

Q: It would be helpful to include baseline experiments focused on "zero-shot generalization."

A: Great suggestion! We have added baseline results to our zero-shot generalization experiments in Figure 9 of our updated manuscript. The figure is also viewable here https://i.imgur.com/gP0MLW0.png for your convenience. We observe that Puppeteer generally is more robust to changes in gap length than the TD-MPC2 baseline, which we attribute to its more stable gait. Videos of our method performing this task with varying gap lengths are shown on our project webpage.

Q: Please provide details on memory usage, training time, and control (or inference) time across different methods.

A: Thank you for the suggestion! We already provide training times and hardware requirements of Puppeteer in Section 4.1 of our manuscript, but agree that a more comprehensive comparison to baselines would be informative. We have added a comparison of wall-time, inference time, and GPU memory for Puppeteer, TD-MPC2, SAC, and DreamerV3. Overall, wall-time and system requirements of Puppeteer are mostly comparable to that of TD-MPC2 across both state-based and visual RL. SAC runs approx. 3.6x faster than Puppeteer, but does not achieve any meaningful performance on our benchmark; DreamerV3 runs approx. 2.3x slower than Puppeteer and also does not achieve any meaningful performance.

These results are now included in Appendix C of our updated manuscript, along with a detailed description of how the numbers are obtained.

Edit: Added DreamerV3 numbers to the table above.

(1/2)

We thank the reviewer for their valuable feedback. We address your comments in the following.

Q: Lack of low-level tracking performance evaluation. There is no evaluation or metrics for the tracking accuracy of success rate.

A: While we predominantly focus on the utility of the low-level tracking agent for downstream tasks, we agree that a more thorough evaluation of the tracking performance itself would be informative. We include 4 new evaluations: (1) a collection of tracking visualizations for readers to qualitatively evaluate our approach, (2) success rate as defined by [1] Luo, Z., Cao, J., Kitani, K., & Xu, W. (2023), (3) average tracking error, and (4) CoMic [2] score across all 836 clips. Results are shown below, and have also been added to our revised manuscript along with more precise definitions of the metrics used.

We also appreciate the additional references; they are now cited in the related work section of our updated manuscript. Thanks again for your suggestions!

[2] Hasenclever, L., Pardo, F., Hadsell, R., Heess. N., Merel, J., “CoMic: Complementary Task Learning & Mimicry for Reusable Skills”, 37th International Conference on Machine Learning (2020)

Q: The lack of interface design discuss. [...] To me, the idea of training low-level tracking policy for skills reuse is a long-standing idea, but the interface of this hierarchy matters a lot. I'd love to see more comparison on this.

A: We agree that the interface between hierarchies is important, and that including a comparison to e.g. an interface based on latent actions (as opposed to our explicit 3D end-effector joint commands) would be informative. We experimented with such an approach in the early stages of our project but found our current approach to be more reliable for whole-body humanoid control. Including such a comparison will take longer than the span of this rebuttal period, but we are committed to adding it for the camera-ready version. Would that address your concern?

Q: The source of naturalness is unclear. The low-level tracking policy might be conditioned on human motion pirors, but if the tracking policy is good enough, it should be able to produce what TD-MPC2 achieves. Also, if the advantage of this paper is sample-efficiency and naturalness, a key baseline here would TD-MPC2+ AMP, which is missing.

A: We agree that further exploring how the data mixture and design choices affect generated motions would be really interesting and informative. If the reviewer believes that this would add significant value to the manuscript, we will be more than happy to add a TD-MPC2+AMP baseline to the camera-ready version, along with an additional ablation of how different mixtures of MoCap data (e.g. excluding all “running” clips from the pretraining data) influence the behavior of learned policies. We would like to point out, however, that the algorithmic complexity of AMP (inverse RL with a learned GAN discriminator + unsupervised skill discovery) is significantly more complicated than our approach (RL without any bells and whistles + no hyperparameter-tuning), and that the original AMP work only achieved downstream task transfer in simple state-based (i.e., no visual inputs) reaching and velocity control tasks while using several orders of magnitude more environment steps to do so. For example, AMP required approx. 800M steps to learn the reaching task, whereas our method learns visuo-motor control tasks such as running in an environment with randomized obstacles in as little as 3M steps.

We believe that our response addresses your main concerns. However, if that is not the case please let us know if you have any additional comments. We would greatly appreciate it if the reviewer could provide us with precise and actionable feedback such that we can fully address your concerns. Thanks again for your time and valuable feedback!