Thank you for your review and comments. Below are our responses to the listed weaknesses and questions:

Weaknesses:

The paper should be revised for some typos, e.g., in the paragraph Terrain Passability Experiment, there is a missing reference to a figure.

We appreciate the reviewer for pointing out the typographical issues in the manuscript. Specifically, we have corrected the missing reference to the figure in the "Terrain Passability Experiment" section.

Questions:

Q1: From Fig 4, it does not look like learning a world model helps much. The yellow and green curves converge to similar values and in almost the same number of steps.

Due to possible color differentiation issues, the distinction between our two curves may not have been clear; we have updated $\color{red}{Fig. 4 }$ for better clarity. From the updated figure, the blue curve (our method) outperforms the red curve (without the world model) in several aspects. In the "Mean reward" plot, the blue curve shows greater stability and slightly higher rewards, particularly after 4000 steps, indicating that the world model reduces training instability and improves final performance. In the "Terrain level" plot, the blue curve surpasses the red curve after 2000 steps, adapting to complex terrains faster and achieving higher final levels. These results highlight the advantages of incorporating the world model in terms of training efficiency, stability, and adaptability.

Q2: What is the difference between a world model and a dynamics estimator? It would be nice to clearly point out which part of the system is the world model and which part is the dynamics estimator in Fig. 2.

Thank you for your feedback. The description of the world model, including its two key components—the dynamics estimation module and the physical interaction regression module—was mentioned in $\color{red}{line\ 239 }$ of the main manuscript. Below is a brief description of their roles:
$\bullet$ Dynamics Estimation Module:
This module predicts the system's next state based on the observed state, helping the agent understand how its actions influence future states. It is the core component of the world model, responsible for capturing the fundamental dynamics of the environment.
$\bullet$ Physical Interaction Regression Module:
This module leverages the complete observation information provided by the critic network to guide the actor network in optimizing its latent variables. This process expands the observation space of the actor network, compensates for the limitations of partial observations, and helps the agent better understand environmental dynamics and interaction relationships.
The details can be found in $\color{red}{Section\ 3.2.2. 3.2.3 }$. In addition, Thank you for suggesting an update to Fig. 2. After careful consideration, we decided not to modify the figure to preserve its readability and simplicity. Adding detailed information about the dynamics estimation and physical interaction regression modules directly in the figure could make it overly complex and harder to interpret.

Q3: The ablation on the dynamics estimation module is missing. There is also no ablation on the latent variable regression. These ablations are mentioned in the text but are not in the figures.

We apologize for the lack of explicit references in the manuscript, which may have made it difficult to locate our ablation experiments. These experiments, including the ablation of the dynamics estimation module, latent variable dim, and time window length, are detailed in $\color{red}{Appendix\ A.3}$. We have addressed this oversight in the revised manuscript by ensuring clear references in the main text.

Dear Reviewer,

Please provide feedback to the authors before the end of the discussion period, and in case of additional concerns, give them a chance to respond.

Timeline: As a reminder, the review timeline is as follows:

November 26: Last day for reviewers to ask questions to authors.

November 27: Last day for authors to respond to reviewers.

Thank you for your review and comments. To respond/comment on your listed weaknesses:

Weaknesses:

W1: The presentation could be improved with more attention to figure and text quality. For instance, the text in many figures is too small and difficult to read without close inspection, and there is a missing figure reference on page 8. Additionally, the formatting and choice of symbols in mathematical equations could be refined for better readability. Some claims in the paper are also questionable, such as: “However, for humanoid robots, these methods can only handle relatively simple environments and have not yet fully addressed dynamic control issues in complex terrains.” Prior research has shown that RL can handle challenging terrains for humanoid robots [1,2,3].

$\bullet$ We appreciate the reviewer’s feedback regarding figure and text quality. We have reviewed and updated all figures to ensure the text is legible. Specifically, the font size of figure labels and captions has been increased, and we have ensured proper alignment and clarity. Additionally, the missing figure reference on page 8 has been corrected.
$\bullet$ Currently, learning-based humanoid robot locomotion methods still face significant challenges in terms of task success rates and the complexity of tasks they can handle. Even in industrial applications, achieving long-term stable locomotion for humanoid robots on complex terrains remains a major challenge. Regarding the works mentioned by the reviewer as being highly effective, we provide the following analysis:
[3] relies on external sensors and depth cameras to accomplish complex tasks, which introduces dependency on additional hardware.
[2] focus of this study is on designing minimalistic reinforcement learning controllers to validate the sufficiency of hardware designs based on learning-based control. However, the terrains involved in their experiments are not particularly challenging.
[1] proposes Denoising World Model Learning, while our approach employs a fundamentally different world model. In the revised manuscript, we have added comparative experiments in simulation to evaluate the terrain level progression curves of both methods. $\color{red}{Fig. 13}$ demonstrates that our method achieves a faster progression in terrain complexity.

W2: The evaluation would be more robust with additional quantitative results from real-world experiments. The sim-to-real gap can be significant, and while the authors show the policy performs in both simulation and real settings qualitatively, simulation results alone may not fully demonstrate real-world effectiveness without quantitative results, as performance could differ substantially.

Thank you for your feedback and for emphasizing the importance of quantitative results from real-world experiments for evaluation. We fully understand that real-world assessments can provide more robust validation of the proposed method.

However, due to the safety concerns and high risks involved in testing humanoid robots in unstructured environments, conducting large-scale real-world experiments is not feasible within the scope of this study. Instead, we have focused on demonstrating the zero-shot sim-to-real transfer capability of our method through qualitative real-world experiments and robust testing in simulation.

Although we have not provided quantitative real-world results, we have ensured that our simulation environment closely mirrors real-world scenarios. This includes domain randomization, curriculum learning, and leveraging dynamic prediction to reduce the sim-to-real gap. These techniques collectively improve the generalization and robustness of the proposed method across diverse and unpredictable environments.

W3: Finally, the gait in the supplementary video is not particularly impressive, as humanoid walking videos have become common. The stride length is short, and the speed seems slow relative to the robot’s scale. In the stair-climbing experiment at the end, the walking also appears less robust.

Thank you for your feedback on the gait and stair-climbing experiments in our supplementary video. While the robot’s stride length and speed may not appear remarkable, our primary focus is to demonstrate the proposed framework's adaptability and robustness in challenging real-world scenarios, rather than optimizing gait parameters such as speed or stride length.
The stair-climbing experiment specifically highlights the robot's ability to navigate uneven and unpredictable terrain, showcasing the strength of our method in handling complex dynamics and partial observability—key factors for reliable locomotion in unstructured environments.
We appreciate your observations and will explore further optimization of gait parameters in future work.

Thank you for your review and comments. To respond/comment on your listed weaknesses:

Weakness

W1: We applied external forces to the robot's legs in real-world environments to simulate disturbances. $\color{red}{Fig. 9}$ The experiments demonstrated that our method enables the robot to swiftly adapt to changes, adjust its gait, and maintain balance under external disturbances. Detailed results can be found in the supplementary video.
W2: The baseline method in our work is the classical PPO-based RL algorithm, which is widely adopted in both recent humanoid locomotion studies [1][2][3] and classical legged robot RL research. It is commonly used as a benchmark for performance comparison in these fields, demonstrating its relevance and validity as a representative baseline. We have included comparisons with other advanced architectures. $\color{red}{Fig. 4}$
W3:
$\bullet$ “Sufficient Information” is presented in contrast to "partial observations" mentioned earlier in the text. Sufficient information additionally includes environmental data (e.g., friction coefficients) and interaction information between the robot and the environment. $\color{red}{Line\ 234}$
$\bullet$ In $\color{red}{Line\ 249}$, this refers to the mechanism where the actor network leverages the hidden variables from the critic network during training. Since the critic has access to privileged information (asymmetric setup), its hidden variables encode richer contextual data. These variables serve as supervision signals for the actor network, helping it learn better representations of the environment-robot interactions. More details can be found in $\color{red}{Section\ 3.2.3}$
$\bullet$ “Contact Mask”: This indicates the contact states of the robot's feet with the ground; “Clock Inputs”: Clock input refers to a periodic signal input used in the reinforcement learning framework to provide temporal information. $\color{red}{Line\ 260}$
$\bullet$ critic_trans: Encodes the complete observation space, including privileged information, to generate contextualized features for the critic; actor_trans: Processes the partial observation data, utilizing temporal dependencies to produce robust policy outputs. tips:we refer to these two functions as $f_{\psi}$. $\color{red}{Section\ 3.2.2\ 3.2.3}$
W4: has been updated. $\color{red}{Conclusion}$
W5:
$\bullet$ Change to "Environmental information and robotic motion information originate from inherently different domains." $\color{red}{Line\ 72}$
$\bullet$ "Affordance-Based" is Similar to usable. $\color{red}{Line\ 90}$
$\bullet$ Space added. $\color{red}{Section\ 2}$
$\bullet$ Change to "θ_xyz". $\color{red}{Line\ 262}$
$\bullet$ "Better capture" refers to the ability of our method to better capture the interaction dynamics between the robot and its environment compared to other baselines. This comparison highlights the effectiveness of our approach in modeling these interactions. Tips: We have revised this section and removed this term. $\color{red}{Section\ 3.2.5\ Reconstruction\ Loss }$

Question

Q1:
Training Cost: The training process was conducted on a machine equipped with an NVIDIA RTX V100 GPU. (40G) Training typically converges within 2 days for our standard configurations.
Deployment Cost: For real-world deployment, the inference runs efficiently on an NVIDIA Jetson NX, a lightweight edge device. The Transformer-XL model achieves real-time performance (100 Hz control loop) without requiring additional computational resources.
Q2: In our work, the Oracle method is designed as a baseline that has access to privileged information, including proprioceptive data and privileged data. This baseline employs an MLP architecture to process the combined input of proprioceptive and privileged information. Our method, in contrast, which introduces several advantages: Temporal Modeling Capability、Implicit Environmental Interaction Modeling、Dynamic Prediction and Observation Expansion
Q3: We referred to the experiments in [4]. We selected the outputs of 6 neurons from the hidden layer, which has a total of 128 dimensions, for visualization. The visualization shows changes in the hidden layer responses as the robot moves from plane to an incline and back to plane. These changes intuitively demonstrate how the robot adapts to different terrains, reflecting the network’s ability to recognize and respond to terrain changes, as well as the real-time adjustment of its control strategy.
Q4: Refer to W4.

Reference

[1] Gu et al., Advancing Humanoid Locomotion: Mastering Challenging Terrains with Denoising World Model Learning, RSS 2024
[2] Cui et al., Adapting Humanoid Locomotion over Challenging Terrain via Two-Phase Training.Corl 2024
[3] Marum et al., Learning Perceptive Bipedal Locomotion over Irregular Terrain. ICRA 2024
[4] Ilija Radosavovic et al., Real-world humanoid locomotion with reinforcement learning.Sciencerobotic 2024

Dear Reviewer,

Please provide feedback to the authors before the end of the discussion period, and in case of additional concerns, give them a chance to respond.

Timeline: As a reminder, the review timeline is as follows:

November 26: Last day for reviewers to ask questions to authors.

November 27: Last day for authors to respond to reviewers.

Weaknesses

W2：Why is the baseline chosen a good representation of what is available as state of the art for humanoid locomotion?

In our previous response, we provided an explanation for why the baseline was chosen. To further address the reviewer's suggestion, we have added an additional experiment comparing our approach with the method proposed in [1]. This comparison was made based on the feedback from another reviewer, who pointed out that [1] demonstrated superior performance and suggested it would be valuable for us to include a comparison.

Specifically, [1] proposes Denoising World Model Learning, while our approach employs a fundamentally different world model. In the revised manuscript, we have added comparative experiments in simulation to evaluate the terrain level progression curves of both methods. $\color{red}{Fig. 13}$ demonstrates that our method achieves a faster progression in terrain complexity.

Tips："terrain level" refers to the complexity of the terrain, and it is determined by assigning a level to different terrain attributes, such as the height of steps. For example, in a "stair up" environment, stair heights ranging from 0.5 cm to 1.2 cm are divided into different levels, with 0.5 cm being level 0 and 1.2 cm being level 6.

W4: The conclusion section is hard to follow, especially where many different vague terms are used in a single sentence, for example, "This framework incorporates the world model concept, forming self-awareness, environmental understanding, dynamic prediction, and observation expansion through historical sequences". It is not clear how self-awareness or environmental understanding has been addressed with the approach.

Thank you for your valuable feedback on our conclusion section. We appreciate your point about the clarity of certain terms. We realize that some of the terminology used, such as "self-awareness" and "environmental understanding," was not the most appropriate. To address this, we have revised the conclusion to provide a clearer explanation of the framework and its contributions.

The updated conclusion now reads:

"In this work, we propose Interactive World Models for Humanoid Robots (HuWo), a novel framework designed to tackle the challenges of humanoid robot locomotion in complex environments. Our framework leverages the world model concept within an asymmetric actor-critic structure, where the hidden layers of Transformer-XL enable end-to-end implicit modeling of dynamic interactions between the robot and its environment. This approach facilitates robust decision-making by expanding the observation space through historical sequences and dynamic predictions.

The effectiveness of our method is demonstrated through experiments showing robust walking performance in diverse real-world environments, including zero-shot sim-to-real transfer. These results highlight our framework’s ability to model environmental dynamics and adapt to changing conditions. Future work will focus on extending this framework to achieve full-body coordination, including free arm movement, for more versatile and natural locomotion."

We hope that this revised version makes the explanation clearer and better reflects the contributions of our approach. We look forward to your further comments.

Reference

[1] Gu et al., Advancing Humanoid Locomotion: Mastering Challenging Terrains with Denoising World Model Learning, RSS 2024

Thank you for your review and comments. To respond/comment on your listed weaknesses:

Weaknesses:

W1: While hardware support was indispensable for implementation, the primary focus of this paper remains on algorithmic contributions.Our algorithmic framework is designed to be flexible and extensible. For example, it can integrate multimodal inputs, like vision, tactile, and auditory data. The Transformer's ability to process diverse inputs offers a promising path to enhance robotic systems.
W2: The explanations for some of the questions are as follows:
$\bullet$ Regarding the ‘understanding of the interaction process with the environment,’ we have provided a detailed explanation in $\color{red}{Section\ 3.1.2\ Dynamical\ Environment\ Understanding}$. In this section, we define the concept of the interaction process between the robot and the environment, highlighting its highly dynamic nature and strong temporal correlations. As for ‘comprehending interaction information during dynamic prediction,’ we thoroughly discuss this in $\color{red}{Section\ 3.1.3 Dynamic\ Prediction}$, where we emphasize that dynamic prediction is an integral part of understanding the interaction process. During dynamic prediction, the robot leverages interaction information to ‘imagine’ the complete states of the physical world and itself resulting from each possible action. This capability allows the robot to proactively anticipate future dynamic changes and evaluate the potential value of each action, thereby enhancing its adaptability to diverse scenarios, especially in complex and unpredictable environments. Furthermore, this process strengthens the generalization of the robot's walking capabilities. We have revised the content in $\color{red}{Section\ 3.1.3\ Dynamic\ Prediction}$ to make this clearer to understand.
$\bullet$ Partial observations can only capture local information and fail to comprehensively represent the full complexity of the environment, making them insufficient to support the robot's decision-making requirements in complex environments. Due to these limitations, state estimators have been adopted[1][2]. Additionally, [3] such as teacher-student distillation are used to incorporate privileged information, addressing the constraints of partial observations.
$\bullet$ We assume that the critic has access to the full observation of the environment and accumulates historical observation information through hidden variables. In the actor network, we use Transformer to encode partial observations into a hidden variable sequence ${z}_{t}$. Since the critic’s hidden variables contain richer interaction information, we allow the critic’s hidden variables to supervise the learning of the actor’s hidden variables, thereby expanding the actor’s observation space. This mechanism enhances the actor’s performance in partially observable environments, enabling it to better capture environmental interaction features. For further details, please refer to $\color{red}{Section\ 3.2.3.}$
W3: We have carefully reviewed the manuscript and addressed all the mentioned typos and formatting issues, including missing spaces before citations, missing words, and broken references.

Questions:

Q1: The order of updates in our approach is primarily determined by practical considerations rather than a strict theoretical rationale. In practice, we found that optimizing the policy first and then the regression module of the hidden variables yielded comparable results to other update orders. However, there are some potential benefits to this specific order:
Policy Adaptation First: By optimizing the policy first, we ensure that it is well-aligned with the current dynamics of the environment. This alignment provides a solid basis for the subsequent optimization of the regression module.
Stable Latent Representations: Delaying the regression module's optimization slightly until after the policy update reduces the risk of destabilizing the latent space early in training.
We will explore this aspect in future work to better understand the implications of update order on performance and stability.
Q2: As discussed in the manuscript $\color{red}{Fig. 11\ Appendix\ A.3}$, we conducted experiments to analyze the impact of varying the history lengthon the performance of our method. The results show that time window length determines the model's context range in sequential tasks. Longer windows capture more dependencies but increase costs. In our experiments, lengths of 16 and 8 perform similarly and significantly better than others.

Reference

[1] I Made Aswin Nahrendra et al., DreamWaQ: Learning Robust Quadrupedal Locomotion With Implicit Terrain Imagination via Deep Reinforcement Learning. ICRA 2023.
[2] Cui et al., Adapting Humanoid Locomotion over Challenging Terrain via Two-Phase Training. CORL 2024.
[3] Kumar et al., RMA: Rapid Motor Adaptation for Legged Robots. RSS 2021.

Dear Reviewer,

Please provide feedback to the authors before the end of the discussion period, and in case of additional concerns, give them a chance to respond.

Timeline: As a reminder, the review timeline is as follows:

November 26: Last day for reviewers to ask questions to authors.

November 27: Last day for authors to respond to reviewers.