Meta-Evolve: Continuous Robot Evolution for One-to-many Policy Transfer

We thank Reviewer NXFw for their insightful comments and the time they spent on reviewing our paper. Regarding the questions and comments by Reviewer NXFw:

1. How common is the problem addressed by this paper, of transferring an effective policy from one robot morphology to multiple others at once?

The problem is common and will become even more popular in the future.

One example: recent advances in the robot learning community started to research on learning across multiple different robot embodiments, such as the recently released Open X-Embodiment benchmark [A] (https://robotics-transformer-x.github.io/) which includes 22 different robots.

[A] Padalkar, Abhishek, et al. "Open x-embodiment: Robotic learning datasets and rt-x models.". arXiv preprint arXiv:2310.08864 (2023). 2023 Oct 13.

2. Why was only the 1-Steiner tree attempted in the door task?

This is because after ablation studies on Hammer and Relocate tasks, it already shows that 1-Steiner tree is the best design choice of our Meta-Evolve.

3. Why were so few seeds used for the results?; The number of training runs in the experiments was quite small, with only 5 random seeds.

We conducted statistical analysis and found that with 5 seeds, our method Meta-Evolve is better than the baseline HERD with more than 99.99% confidence, on all tasks.

This means 5 random seeds are enough for our experiments.

4. Why was the value of 80% selected?; it would be interesting to see if a change from 80% to 70% or 90% would change the order of the methods

We explained this in the 3rd paragraph of Section E of the appendix:

“We followed HERD (Liu et al., 2022a) and adopted the training overhead needed to reach 80% success rate as our evaluation metric. The core idea of using this evaluation metric is to compare the efficiency of the policy transfer when the difficulty of transferring each policy is unknown beforehand. Using an alternative success rate higher than 80% as the evaluation metric could also be feasible. However, since the success rate threshold for moving on to the next intermediate robot is 66.7%, the success rate increase from 80% to a higher one can only happen after the policy transfer is completed. Therefore, the remaining part of training for reaching a higher success rate is simply vanilla reinforcement learning on the target robots and is irrelevant to our problem of inter-robot policy transfer, so should not be included in the evaluation.”

5. Did the authors consider the relationship between the reduction in the total length of the paths in the graph (as compared to independent paths) and the reduction in the training time? Were these correlated?

Yes, the two reductions are positively correlated.

6. Why wasn’t DAPG run to completion? How effectively was it able to perform the task, given that it did not complete in the time provided?

During exploration on new robots, DAPG has trouble finding successful trajectories with non-zero reward due to the huge robot difference. This means extremely low sample efficiency under sparse task-completion reward setting, and means that DAPG is not effective in directly transferring the expert policies to new robots.

7. Do the authors have any comments on why (e.g. in the Hammer task, on path 8-10-11) their outperformed HERD on one-to-one transfer?

Sorry we don’t have a good explanation for that. We leave the detailed study of this phenomenon as a future work.

It is not obvious that the work has much to do with evolution, in either the biological or computational senses; It is also not obvious why the word “meta” was chosen, especially given that it has other connotations in Reinforcement Learning.

The word "evolution" has more than just biological or computational meaning. According to Webster's Dictionary (the most authoritative dictionary for modern English), another meaning of "evolution" is "a process of change in a certain direction" (see the second listed meaning in this link: https://www.merriam-webster.com/dictionary/evolution).

The current name for our method “Meta-Evolve” is a combination of “meta” (inspired by the idea of meta learning where the learning objective is across multiple tasks) and “evolve” (to show the connection to robot evolution).

We are open to changing the name to a better one if Reviewer NXFw could suggest a better name for our method.

We thank Reviewer Escv for their insightful comments and the time they spent on reviewing our paper. Regarding the questions and comments by Reviewer Escv:

However, I am concerned that they are claiming that just because two robots share a similar kinematic tree, that they also share a similar expert policy.

We apologize for the confusion. To clarify, we didn’t claim the relationship between similar kinematic trees and similar expert policies.

Our assumption is that, if two robots are similar in their morphology and dynamics (specified by their kinematic structures, physical parameters etc.), they may also be similar in their optimal policy and have small cost of policy transfer between them. It’s not just about the kinematic trees. We will include additional sentences in the next version of the paper to make it more clear.

In my understanding, the kinematic tree matching would relate a pair of robots with 2 fingers and 3 fingers more closely than a pair robot with 2 fingers and 4 fingers. However, it very well may be the case that a robot with 2 fingers will act more similarly to a robot with 4 fingers than one with 3 fingers.; Further, the convex hull of robot hardwares might be irrelevant if the robots close in hardware-space are far in policy space.

We agree that it is possible that in some cases, transferring policy to robots closer in hardware space can actually be more expensive.

Our assumption is that the policy transfer cost is locally proportional to robot hardware space distance when the hardware difference is small. Globally, this assumption may not be true.

Based on this assumption, we propose a heuristics to use physical parameter distance to estimate the cost of local policy transfer. The heuristics are used to construct evolution trees and are shown to be effective in our extensive experiments. As mentioned in the 2nd paragraph of Section 3.5, we leave the problem of finding the optimal way of estimating policy transfer cost and constructing evolution trees as future work:

“Policy transfer through robot evolution relies on local optimization of the robot evolution. On the other hand, optimizing the evolution tree requires optimizing the robot evolution paths globally and needs an accurate “guess” of the future cost of policy transfer. In fact, our proposed heuristics can be viewed as using $L^p$ distance of evolution parameters to roughly guess the future policy transfer cost for constructing evolution tree. We leave the problem of finding the optimal evolution tree and meta robots as future work.”

I am curious about the method the authors use to actually train the policies on the new robots. The description of generating new robots is clear but the method for actually transferring the policies is not.; How are the authors training the new policies on the new robots?

Sorry for the confusion. As mentioned in the 1st paragraph of Section 3.3 and line 4 of Algorithm 1, the policy is transferred by fine-tuning on the next intermediate robots, i.e. training the policy on the next intermediate robots using an RL algorithm. In our experiments, we use NPG algorithm as our RL algorithm.

I am concerned about the notion of neural plasticity in this problem. Simply fine-tuning the policy more times might lead to the degradation in performance that we see in the HERD and REvolveR methods.; Does the simple fact that the authors are fine-tuning the same policy more times on the baselines lead to poor performance?

We did a sufficient amount of experiments on REvolveR, HERD and our Meta-Evolve, and we did not see performance degradation problems.

The authors do not demonstrate the quality of the new policies. I know they are training to achieve some specified success rate but I think more experimentation on what the transferred policies are is important to discuss in this paper; Can you please describe quantitatively/qualitatively the quality of the learned target policies between the different methods?

Our method aims at improving the overall policy transfer efficiency. After policy transfer is completed, the performance/quality of transferred policies can be further improved by continuing to train on the target robots with RL algorithms.

Given sparse task-completion reward, success rate is the best way to evaluate the policy quality. Quantitatively, transferred target policies reach the same success rate at the end of policy transfer, though it takes much smaller overall training/simulation cost for our Meta-Evolve to do so than HERD.

Qualitatively, as suggested, we visualize the transferred policies on Hammer and Relocate tasks using different methods in this video: https://drive.google.com/file/d/1Ln8oLZhU3g1hw_lZhqAu-CC0rW2WvSW_/view?usp=sharing (select 1080p for best quality). As shown in the video, policies transferred using different methods show slightly different behaviors, though they are all able to complete the tasks.

We thank Reviewer e334 for their insightful comments and the time they spent on reviewing our paper. Regarding the questions and comments by Reviewer e334:

The experiments are mainly limited to hand manipulation tasks, which can be easily represented by tree structures. Does your main idea still works on modular robots (like 3d voxel-based robot)?

The main assumption of our method is that the robot can be defined by several continuous parameters so that the robots can be continuously interpolated by varying the parameters and the robot evolution tree can be constructed in high-dimensional space $[0,1]^D$.

For modular and soft robots, as long as the robot can be defined by multiple continuous parameters, our main idea should still be able to work.

Can and how your method transfer to the real world robotic problems? Like manipulation tasks.

Yes, it can.

We showed that our method can be applied to real-world robots. In Section A of the appendix (page 13), we included experiments where we transfer a manipulation policy from a five-finger dexterous robot hand to three real commercial robots: Jaco, Kinova3 and IIWA. Please refer to Section A for more details.

The videos of the experiments in Section A and the real-robot demo can be found on our anonymous project website: https://sites.google.com/view/meta-evolve.

We thank Reviewer 6cdy for their comments and the time they spent on reviewing our paper. Regarding the questions and comments by Reviewer 6cdy:

as the name suggests the method explores in the kinematic space of the robots. What happens if the dynamics differ drastically? An example would be, having a robot that has the same kinematic structure but twice the masses. Can the method handle this? It seems that no. Isn't changing the dynamics but keeping the kinematics the same a different robot? How can the proposed method handle significant dynamics differences between the source and target robots? Can it even do that?

Sorry for the confusion. To clarify, our method also explores different robot dynamics because it also explores different robot physical parameters such as mass and inertia, as described in Section 3.2 and Equation (3).

Our method can handle the case where the source and target robot dynamics differ, but of course the difference cannot be unreasonably large. For example, we cannot expect a manipulation policy on a multi-fingered robot gripper to be transferred to a multi-legged locomotion robot dog.

As long as the dynamics difference is reasonable, our method is able to handle that. This is because our method interpolates physical parameters after kinematic tree matching, which is essentially interpolating robot dynamics.

As mentioned by Reviewer 6cdy, we conduct an additional experiment where we double the mass of the agile locomotion robots. Both HERD and our method can still successfully transfer the policy to the new target robots with double masses.

I am not sure how we should interpret the additional experiments on commercial robots. I cannot see the added value of those experiments and how they contribute towards convincing us that the method is applicable to real-world situations/robots.

The goal of our method is to transfer an expert policy from one robot to multiple other predefined robots.

In the additional experiments in Section A of the appendix, we verified that our method can transfer a manipulation policy from a five-finger dexterous robot hand to three different existing commercial robots (Jaco, Kinova3, IIWA) in simulation while outperforming baseline HERD by roughly 2$\times$ in terms of overall efficiency. In addition, we verified that the transferred policy on Kinova3 in simulation can be deployed on the real Kinova3 machine in a real-world situation.

Therefore, the additional experiments support our claim and further show that our method can be applied to real-world situations/robots.

the comments on Sim2Real are weak imho. I do not see why the paper is not relevant to Sim2Real methods. How does the proposed method fit inside the Sim2Real literature?

Conceptually, sim-to-real transfer is not within the scope of our paper:

• Our Meta-Evolve deals with transferring policy from the source robot to target robots, in simulation.
• Sim-to-real transfer deals with transferring the policies on target robots from simulation to real machine.

Therefore, it is actually OK to not even discuss sim-to-real transfer in our paper.

We still discussed it anyway in Section B (and also conducted additional real-robot experiments in Section A as experimental evidence), simply to address people’s (potential) concerns over deploying the transferred policies to real machines.

The authors have chosen to consider a model-free approach to RL/transfer learning, while obviously the models are known (at least the kinematics part). This choice should be better motivated and highlighted in the text. How can the proposed method integrate model-based RL/learning? Why did the authors not experiment with this?;

In general, models are not easily known in robotic tasks even if the robot kinematics and dynamics are known. This is because there are unknown robot-object interactions that affect both the robot states and the object states. Robot-object interactions are not only determined by the robots but also depend on the objects in the environment.

It’s easy to apply model-based RL algorithms in our method though. We can just instantiate the policy optimization part of line 4 of Algorithm 1 with a model-based RL algorithm.

However, we believe our existing experiments and ablation studies are already sufficient to support the claims made in our paper.

One of the videos is not properly anonymized; we can clearly see the face of someone performing the experiments. Is this part of the video part of the DexYCB dataset? If this is the case, it should have been highlighted.

This part of the video is from public dataset DexYCB (other example videos of the dataset can be found in https://dex-ycb.github.io/).

Also, this information has already been highlighted in the supplementary video. At the start of the video (0:00:00-0:00:03), we have already highlighted with text “DexYCB Dataset Human Demonstration”.

Thanks for the additional discussions. Regarding Reviewer 6cdy’s concerns over comparison to generic Sim2Real methods/methodology:

Sim2Real transfer can be viewed as a special case of domain transfer. It specifically deals with the problem of transferring knowledge/skills from simulation to its counterpart real-world systems. Some common assumptions regarding the domain difference in Sim2Real:

1. Unknown Domain Difference: One assumption is that the precise nature and extent of the domain difference are not explicitly known. This implies that there are discrepancies between the simulation and the real world, but the exact details of these differences are not identified or quantified.
2. Small Range of Domain Difference: Another assumption is that the domain difference is within a relatively small range. This assumption suggests that, despite the differences, the simulation is considered precise enough for the purposes of training, and the variations between the simulated and real-world environments are manageable.
3. Parameter Errors as the Cause: Sim2Real often assumes that the primary cause of domain difference lies in errors in the model parameters or limitations in the simulation's fidelity. In other words, the assumption is that if the simulator were perfect and the parameters were precisely tuned, the domain difference would be minimal.

These assumptions justifies the most popular Sim2Real method domain randomization, which randomizes the domain parameters by adding a small random perturbation noise in order to train a domain-robust policy that can generalize to a range of domains.

However, in our problem, while the 3rd assumption of Sim2Real still holds true, the 1st and 2nd assumption does not. In our problem:

1. Known Domain Difference: The domain differences between the source and target robots are assumed to be known due to the source and target robots being pre-defined.
2. Huge Domain Difference: The domain differences between the source and target robots can be extremely huge due to huge differences in morphology.

Our approach is specifically designed for the assumption of our problem:

1. We construct robot evolution trees based on known robot physical parameters (known domains). This allows policy transfer to be along paths with known domain directions which is very likely to be better than completely random perturbations of domains.
2. We decompose the task of transferring the policy to an extremely different domain into multiple sub-tasks by introducing multiple intermediate robots (intermediate domains). This is especially beneficial for policy transfer in challenging sparse task-completion reward settings where maintaining sufficient performance is crucial for sample efficiency and the overall policy transfer efficiency.

Therefore, we believe that, in our problem, even if a Sim2Real method can work, it is very likely to show much worse overall performance than our method.

We hope the above discussions are sufficient to address your concerns on the comparison of Sim2Real and our method.

We thank Reviewer 6cdy for the additional comments. Regarding the additional comments by Reviewer 6cdy:

My comment actually says "The method explores the kinematic parameters and not the dynamic parameters". In other words, there is no systematic way that the difference in dynamics is explored.

We kindly point out that the definition of “kinematic parameters” and “dynamic parameters” are: “The Denavit-Hartenberg parameters constitute the kinematic parameters, and the link masses, link inertias, and center-of-mass vectors are the dynamic parameters.” (1st paragraph of Section 2 of [A])

Our method explores robot physical parameters including mass and inertia, as described in Section 3.2 and Equation (3). Therefore our method also explores different robot dynamics.

[A] Pradeep Khosla and Takeo Kanade. "Parameter Identification of Robot Dynamics." CDC 1985.

Having more experiments is always "helpful" and nice, but I still do not really see the contribution here.; I already expressed my concerns regarding the contribution of these experiments. … this looks like position control, which has very close to zero reality gap.

The focus of our paper is to transfer policies between robots with different morphologies, which we found in our experiments are already non-trivial even with zero sim-to-real gap. Sim-to-real is also an important and actively-studied field but not our focus nor contribution of this paper. The goal of the real-robot demo includes:

• To verify our assumption that our method tolerates a relatively small level of uncertainties (e.g. model errors, delay and noise from controller) required in real world deployment.
• For the demonstration purpose for the broader robotic community.

We also thank Reviewer 6cdy for acknowledging that the robot and the controller “has very close to zero reality gap”. This can help justify the value of our additional experiments: to verify that the transferred policies in simulation can be deployed on the real machines.

Too many important details are missing. Is this position control? Torque control? Velocity? What's the exact state? Overall, too many details are missing to evaluate the experiments.; There are very little details. And from the video, this looks like position control...

• Control: we use the Operational Space Controller (OSC) [B] that moves the end-effector to desired 6D pose with PD control schema. It is also detailed in the robotsuite package: https://robosuite.ai/docs/modules/controllers.html#Operational-Space-Control---Pose-with-Fixed-Impedance.
• Robot state: states of the ADROIT hand + 6D pose of the robot end-effector

We follow the default robosuite setting for the robot arm, therefore, we did not specify the details but refer to the readers to the documentations of robosuite. We have updated our paper to include the above details in the 3rd paragraph of Section A. We are happy to provide more details and will release source code to ensure the work is reproducible.

[B] O. Khatib. "A unified approach for motion and force control of robot manipulators: The operational space formulation." IEEE Journal on Robotics and Automation, 1987.

Sim2Real is an "umbrella" term covering the transferring of policies from a source environment to a target environment. The target environment is usually the real world but this is not necessary. Your Meta-Evolve deals with transferring the policy from a source robot to a target robot. These read and look like very similar.

We follow the definition of “Sim2Real” in literature such as [C,D,E] where the term “Sim2Real” specifically refers to transferring from simulation to its counterpart real systems. While in our setting, the source and target robots are different even with zero reality gap which some people refer to as “domain transfer”. We agree from a very high level, they are all transferring between different distributions of uncertainties.

In the context of this paper specifically, we believe there are two steps of transfer learning: (1) from one robot A to another robot B; (2) from simulation of B to physical deployment of B. These two steps are different as the uncertainties have different sources. In (1), the uncertainties come from the different morphologies of robots, while in (2) the uncertainties come from some uncontrollable factors that are hard to precisely incorporate in the simulation e.g. the delay of motors. We would be happy to review the sim-to-real domain and clarify the difference from and similarities with our approach.

[C] J. Truong et al. "Rethinking sim2real: Lower fidelity simulation leads to higher sim2real transfer in navigation." CoRL 2023.

[D] M. Kaspar et al.. "Sim2real transfer for reinforcement learning without dynamics randomization." IROS 2020.

[E] P. M. Scheikl et al. "Sim-to-real transfer for visual reinforcement learning of deformable object manipulation for robot-assisted surgery." RA-L 2022.