Curriculum Reinforcement Learning via Morphology-Environment Co-Evolution

Thank you for your comments and suggestions! We address your main concerns as below:

The significance of the paper's contributions is a bit unclear.

• To the best of our knowledge, MECE is the first paper to discuss the co-evoltuion between the agent's morphologies and training environments and use the co-evolution to create an efficient curriculum for training embodied agent for better generalization. Would you mind kindly providing papers that have previously examined this problem?
• Our baseline methods include the SOTA algorithms of training agent to adpat to diverse environments (POET), the SOTA morphology optimization method utilizing RL to evolve the agent's morphology in a fixed environment (Transform2Act), and a widely-studied method that models an agent's morphology as GNN (NGE).

How general is the proposed approach, beyond the tasks and environments considered in the experiments?

• In MECE, the simulation is Mujoco, which is a widlely used simulator in RL.
• We did not design special constraints for environments because the learned $\pi_E$ automatically avoids generating unlearnable environments. Furthermore, tasks are not accompanied by any form of extrinsic reward signal. Hence, MECE has the potential to be applied to a wide range of tasks and circumstances, extending beyond the confines of experimental settings.

Is the proposed MECE method computationally efficient?

• MECE training is more computationally efficient than all the compared baselines. For all the test environments, MECE took around 44 hours to train on a standard server with 8 CPU cores and an NVIDIA RTX 3090 Ti GPU, while Transform2Act requires around 63 hours on the same server.

Have you encountered any scalability issues when applying MECE to more complex tasks or environments?

• The 3d-locomotion environment is a complex environment for RL navigation tasks. If we do not set any contraints on $\pi_e$, in earlier stages, the unmatural environment policy may change the environment at a large scale, which can make the training environments overly hard/simple. Hence, we limit the scale of changes by $\pi_e$ to environment in each step.

It is not clear to me how environments are produced and how the agents perform in your environment (Figure 4), do you have a video?

• We introdce how $\pi_E$ generates and controls the environments in Appendix A.
• We will release the source code with videos.

It seems that MECE's performance is not much better than Transform2Act, can you provide more results on different tasks?

• We respectfully disagree with the assertion that this is "not much superior to Transform2Act." In Figure 2, MECE outperforms the modified Transform2Act (trained on diverse environments) by around 20% in terms of rewards and outperforms Transform2Act-Original by approximately 60%.

Dear reviewer,

Would you mind validating whether our rebuttal addresses your concerns:

• We explained the generalization and contribution of MECE.
• We explained that MECE is not computationally expensive.

Please feel free to ask any additional queries after reading our rebuttal. During this author-reviewer discussion period, we hope to catch and answer any questions.

Thank you,

The authors

Thank you for your comments and suggestions! We will correct the typos following your suggestions. We address your main concerns as below:

The setting of experiments is different from the motivation.

• Our evaluation is consistent with the motivation and the 2nd question at the beginning of the experiments: "Does our method create agents more adaptive and generalizable to diverse new environments?". As stated in the 2nd line of Section 5.2, we evaluated MECE and its learned morphology in 12 diverse and unseen environments (random variants of 2d-locomotion, 3d-locomoation, or Gap-crosser) under 6 random seeds with the morphology fixed. The experimental results show that the morhology evolved by MECE can generalize better to the tasks in diverse unseen environments.
• MECE does not require finetuning for the unseen environments in the test, unlike the Transform2Act and NGE methods, which still require this additional step. Furthermore, we present the performance of baselines without finetunes in the table below.

How do $r_m$ and $r_E$ contribute to our statement on "a good morphology" and "a good environment"? + Why we need morphology policy $\pi_m$ and environment policy $\pi_e$?

• We explain the motivation of designing $r_m$ and $r_e$ in the 4th and 5th paragraphs of Section 3, in particular, the text above Eq. (1) and Eq. (2). They are in line with the statement of "a good morphology should make the agent learn faster and make more progress in different environments." (i.e., greater progress of control policy $\pi$ in Eq. (1)) and "a good environment should accelerate the evolution of the agent and help it find better morphology sooner" (i.e., greater progress of morphology policy $\pi_m$ in Eq. (2)).

• The training and co-evolution of $\pi_e$ and $\pi_m$ is the key for the morphology optimization and gaining generaliation to diverse unseen environments. They jointly create a special adaptive curriculum for the training phase for the agent to evolve its morphology and generalization capability. In particular, $\pi_e$ creates a curriculum of environments for optimizing the morphology and control policy $\pi$, while $\pi_m$ creates a curriculum for the environment changing and control policy learning. They improve the efficency of exploration through (1) interactions with MDP using different morphologies and environments; (2) structural constraint and correlation captured by the GNN; (3) more directed evolution than random mutation.

• In our thorough ablation study-Ⅰ,Ⅱ,Ⅳ, we have extensively evaluated the importance and impacts of $r_E$ and $r_m$, $\pi_e$ and $\pi_m$ in morphology optimization and generalization to new environments.

The clarity of the explanations could be improved.

• Thank you for this suggestion. We have updated all unclear notations and expressions following your suggestions.

Dear reviewer,

Would you mind validating whether our rebuttal addresses your concerns:

• The consistency of our experiments and motivation, and we provide additional results.
• The design of the reward function for $\pi_e$ and $\pi_m$.

Please feel free to ask any additional queries after reading our rebuttal. During this author-reviewer discussion period, we hope to catch and answer any questions.

Thank you,

The authors

Thank you for your response. We would want to further address your concern as follows:

• The evaluation tasks presented in the paper are characterized by a higher level of difficulty and generality compared to either morphology optimization alone or domain-specific generalization. Specifically, our objective is to acquire both a general morphology and general policy that can be directly applied to unseen environments and tasks without the need for further fine-tuning. To the best of our knowledge, the majority of existing work, including the baselines we have evaluated, does NOT directly tackle this problem and was NOT specifically developed to address this particular issue. To provide equitable comparisons, it is necessary to make some modifications to some baselines in experimental comparisons. The comparisons have already incorporated enhanced POET, as indicated in Figure 2 and the table provided in our rebuttal.
• Our current discussion is about "what is a good morphology or a policy for the test/inference phase". In our introduction section (and many places), we are discussing the good morphologies that exhibit higher learning efficiency during the training phase, rather than the test phase (since the adaption_cost =0 for the morphology evolved by MECE in the test). The MECE framework constructs a mechanism that encourages $\pi_m$ to discover the morphology that can achieve higher learning progress in diverse environments, utilizing the reward design $r_m$.

Please let us know if we misunderstood your comment regarding our response to your first question.

Thank you for the suggestions! we have revised the draft by following your comments. We address your questions as below:

The control complexity compared to Cheney et al..One could say the soft robots in Cheney et al. (2013) are more complex than the robots co-evolved in this paper.

• Whether a soft-bodied robot is more complex than a rigid-bodied robot is still an open problem with different opinions from different studies. In this paper, we only focus on the optimization in the space of rigid-bodied robots. We will make it clear in the new version.

How expensive is MECE co-evolving the three different policies?

• Training MECE is not costly compared to baselines. For example, MECE took around 44 hours to train on a standard server with 8 CPU cores and an NVIDIA RTX 3090 Ti GPU, while Transform2Act took around 63 hours on the same server.

Comparison with POET.

• Yes, we agree that POET is not designed for changing morphologies. But this baseline was required by a previous reviewer. For ensure a fair comparison, we have made several modifications (detailed in point (3) of the ''Baseline'' paragraph in Section 5.1).

"I would suggest a slightly more advanced baselines that samples environments of increasing complexity"

• Thank you for the suggestion! We already included such a baseline in our comparison (e.g., in Figure 2): Enhanced POET starts with a simple environment and then gradually creates and adds new environments with increasing diversity and complexity.
• As mentioned in the paper, MECE stands as the pioneering approach in discussing co-evolution between the agent's morphology and the training environment. Finding baselines that align properly with this settings might be challenging.

What happened if starting with the same initial robot of 2d-locomotion in other environments?

• The initial agent in gap-crosser is the same as that in 2d-locomotion. We tried to initialize the same agent in 3d-locomotion but it didn't work. This is because 3d-locomotion has one more dimension (XYZ-plane) and it is hard to ensure the final performance of evolving morphologies without initial limitations.

"It would be good to see some pictures of the evolved environments."

• We will release the code with videos.

Dear reviewer,

Would you mind validating whether our rebuttal addresses your concerns:

• We have explained the issues of the experiments in MECE.
• The computation of MECE is not cost.
• We will release the source code with videos.

Please feel free to ask any additional queries after reading our rebuttal. During this author-reviewer discussion period, we hope to catch and answer any questions.

Thank you,

The authors