Response to Reviewer yoqm

Q1. How sensitive is EASI to the choice of hyperparameters for both the GAN and the ES components?

A1. We conducted a hyperparameter sensitivity analysis for EASI, as detailed in common response A5, and provided recommendations for parameter settings. EASI has low sensitivity to hyperparameters, and we rarely changed EASI's hyperparameters across all experiments in this paper. We believe that EASI can achieve plug-and-play effectiveness in other tasks.

Q2. How does EASI perform across more complex real-world scenarios?

A2. We tested EASI's sim-to-real performance in the real world using the Unitree Go2 robot in common response A2. The Go2 task is complex and challenging, yet EASI is able to adjust the simulation parameters using approximately 40 seconds of real demonstration data, making the simulation more realistic.

Q3. What are the computational requirements for training GANs and performing evolutionary searches in EASI?

A3. EASI's computation consists of three main parts: sim trajectory collection, discriminator training, and parameter evolution.

1. Sim Trajectory Collection: This is the most computational-consuming step. Assuming we have $\lambda$ offspring in the evolutionary strategy, we need to use same policy to collect $\lambda$ trajectories from environments with $\lambda$ different parameters. Due to the high parallelism of the Isaacgym simulator, we can generate $\lambda$ environments with different parameters in Isaacgym and sample $\lambda$ trajectories concurrently.
2. Discriminator Training: This involves sampling from both demonstration and environment trajectories to train the discriminator.
3. Parameter Evolution: In this part, we use the discriminator to estimate the similarity between the simulation trajectory and the demonstration. Then, using ES to generate the distribution of the next generation of simulation parameters. Based on this new parameter distribution, we sample and obtain the physical parameters of N simulated environments and set the parameters for each environment.

In our experiments with the most complex environment, Go2, setting $\lambda$=300 and trajectory_length=500，we test the time consuming on a PC equipped with Intel i5-13600KF and RTX4060 Ti. For one generation of evolution, the time consuming is as follows:

Typically, 30 to 50 generations are sufficient for parameter convergence.

Q4. Discuss potential optimizations to make the method more accessible for practical deployment.

A4. Currently, EASI has demonstrated strong performance for real-world deployment. However, we believe there is still significant potential for improvement. Real-world data collection often lacks the ideal conditions found in simulations, and the rough policy trained with UDR may not perform optimally in practical applications. As a result, the demonstration data we gather in real-world settings may be imperfect and unbalanced. As noted in the limitations, there are existing imitation learning methods designed to handle unbalanced and imperfect demonstrations. In the future, we aim to implement similar approaches to train a more effective discriminator based on these imperfect, unbalanced datasets, thereby reducing the deployment costs associated with EASI.

Q5. How scalable is EASI to high-dimensional state and action spaces?

A5. In the Go2 sim-to-real experiment detailed in common response A2, the task featured high-dimensional state and action spaces. Despite these complexities, EASI demonstrated excellent performance in this sim-to-real task.

Response to Reviewer ZLtE

Q1. The architecture of the used GAN and how are the tuple s, a, s' processed in the training. :

A1. The input to the discriminator consists of a state transition $(s, a, s')$. This $(s, a, s')$ data is combined into a tensor with a shape of $(2 \times \text{dim(state)} + \text{dim(action)})$, which is then fed into a multilayer perceptron (MLP) discriminator. The discriminator is trained using a WGAN-based approach to determine whether the $(s, a, s')$ data originates from the demonstration. Subsequently, we employ Evolution Strategies (ES) to select parameters that make the simulation closely resemble the real environment and to 'generate' the next generation of environment parameters. Unlike conventional GANs, EASI does not utilize a neural network generator; instead, it relies on ES to choose and generate the next set of parameters based on the discriminator's output. Thanks to the stability of ES, EASI achieves faster and more stable training compared to traditional GANs.

In our experiments, an MLP discriminator is sufficient to achieve good results. We also considered employing networks such as LSTM or TCN for processing $(s, a, s')$ data. However, since the MLP has already demonstrated exceptional performance in the current tasks, we plan to explore these alternative network architectures in future work.

Q2. In Figure 3, why the EASI can outperform the oracle that indicates the upperbound of the performance?

A2. BallBalance is a unique environment where the ball can only be controlled indirectly by a movable table and cannot be directly influenced by actions. In this task, if the ball falls off the table, the episode ends and a penalty is incurred. We hypothesize that during the early stages of ORACLE training, the policy may prioritize maximizing rewards by keeping the ball at the target position, which could inadvertently lead to the ball falling. In contrast, EASI narrows the parameter range but still allows for variations across different environments. As a result, the policy may adopt a more cautious approach in the early training stages, focusing on preventing the ball from falling off and thus avoiding penalties. As training progresses, both ORACLE and EASI can effectively keep the ball from falling while maintaining its position at the target, leading to similar performance in the later stages of training.

On the other hand, UDR, with its overly wide parameter range, leads to an overly conservative policy. This policy only achieves the basic reward of preventing the ball from falling off but is hard to maintain the ball at the target position.

Q3. For sim-to-transfer case, the oracle may not represent the upperbound, in which case some advanced compared algorithms need to be compared in the experiments.

A3.In the common response A4, we introduced additional comparative experiments, including the FineTune and GARAT (a GAT branch method). In these experiments, EASI consistently demonstrated the best performance in transferring simulation trained policies to the target environment.

Q4. It is interesting to see EASI can adjust the parameters out of the initial range. What makes it capable of doing so?

A4.In EASI, we utilize Evolution Strategies (ES) to adjust the distribution of parameters. During the evolution process, ES incorporates mechanisms such as "recombination" and "mutation."

For recombination: Parameters that contribute to a more realistic environment have a higher probability of producing offspring. As a result, parameters within the initial range that are closer to the target value exert greater influence on the next generation's parameter range, helping to refine the distribution towards the target. For mutation: Each time new offspring are generated, some undergo mutations that deviate from the original distribution. If these mutated individuals achieve higher fitness values than those within the original distribution, the population will gradually evolve in the direction of the mutated individuals. This means that even if the initial parameter range does not include the global optimum, EASI can potentially discover new parameters close to the optimum through multiple iterations.

After refining the simulator's parameter distribution with EASI, we can train or fine-tune the policy in this enhanced simulator, thereby improving sim-to-real performance.

Q5. How is the scalability of the EASI, I mean if the number of parameters increases, what will happen to EASI?

A5. In common response A3, we introduced additional simulation parameters, searching for a total of 25. EASI was still able to identify the appropriate target parameter range.

In our real-world Go2 sim-to-real experiment discussed in common response A2, we only searched across 7 parameters. We set the motors on the four legs to the same positions, resulting in identical parameters for the hip, thigh, and calf motors, each with two PD parameters. Despite this limited search, we achieved excellent sim-to-real performance. By focusing on a few key parameters with EASI, we were able to significantly minimize the gap between the simulation and the real environment.

Response to Reviewer zVKJ

Q1. How come in ball balance the oracle method performs worse than EASI.

A1. BallBalance is a unique environment where the ball can only be controlled indirectly by a movable table and cannot be directly influenced by actions. In this task, if the ball falls off the table, the episode ends and a penalty is incurred. We hypothesize that during the early stages of ORACLE training, the policy may prioritize maximizing rewards by keeping the ball at the target position, which could inadvertently lead to the ball falling. In contrast, EASI narrows the parameter range but still allows for variations across different environments. As a result, the policy may adopt a more cautious approach in the early training stages, focusing on preventing the ball from falling off and thus avoiding penalties. As training progresses, both ORACLE and EASI can effectively keep the ball from falling while maintaining its position at the target, leading to similar performance in the later stages of training.

Q2. Are dense rewards used for all tasks?

A2. Yes, all experiments were conducted using dense rewards during training. EASI mainly focuses on the state transitions of these tasks, and the training method of the task itself is not the primary focus of EASI.

Q3. What if UDR does not get any successful demonstrations?

A3. For EASI, our requirement for realworld demonstration is that the $(s,a,s')$ tuples contain sufficient information, meaning that the state transition distribution in the demonstration needs to partly overlap with that in the simulation. If the rough UDR policy can run in the real environment, even if the performance is not ideal, it can still provide valuable state transitions for EASI training.

If UDR completely fails to accomplish the task, but some meaningful $(s,a,s')$ tuples are still recorded during action execution. In this situation, the collected demonstration data may be imperfect and unbalanced, with only a small portion of state transition overlap with that in the simulation. In such cases, EASI may fail due to insufficient state transition information. We mentioned this limitation in the paper, and notice that current imitation learning methods address imperfect and unbalanced demonstrations. In the future, we will aim to use imperfect and unbalanced imitation learning methods to extract useful state transition information from failed real-world demonstrations and then use EASI to find suitable parameters. We believe that in future work, EASI could have lower requirements for real demonstrations, allowing it to identify optimal parameters even when the policy complete very poorly in the reality.

Q4. What does "domain priority" in section mean?

A4. What we meant to express is that Domain Randomization (DR) requires prior knowledge of the specific domain, including detailed parameters about real-world deployment. Thank you for pointing out the issues with the wording in the paper, and we will revise the wording to ensure clearer understanding.

Q5. Only control tasks are tested which generally are easier to solve with just domain randomization.

A5. The gap between simulation and real-world environments has long been a significant challenge for deploying RL-based robotic controllers in practical applications. Through our method EASI, we aim to advance the deployment of such controllers, focusing specifically on control tasks. And additionally, we tested EASI in a more complex sim-to-real tasks using the Unitree Go2 robot in common response A2.

Q6. it's not clear how much simulated data is needed.

A6. During EASI's process, $\lambda$ environments with different parameters are created in the Isaac Gym simulator (in the experiment, we set $\lambda=300$), and use a same policy to collect 1 trajectory for each sim environment, the max_trajectory length is setted in different tasks. Thanks to Isaac Gym's high degree of parallelism, we can parallelly run hundreds or even thousands of environments with different physical parameters. After each evolutionary iteration, all environments are reconfigured with new physical parameters based on the evolutionary strategy, initiating the next evolution cycle. The process of generating sim data is highly convenient, fast, and inexpensive, so the amount of simulation data used is not a major concern.

Q7. It might be fairer to run the UDR baseline for as many online steps EASI uses for training after EASI and before with the rough UDR.

A7. To ensure fairness, we used the same real demonstration data as EASI to fine-tune the UDR algorithm. The specific experimental details are described in common response A4. In more complex environments like Ant, FineTune's performance improvement is limited when the amount of real demonstration data is insufficient. EASI, on the other hand, consistently achieves significant improvements in policy transfer to target environment.

Q8. For more complex tasks involving manipulation of other objects, this approach will probably not work well.

A8. For complex tasks involving manipulated objects, if the parameters of the object are known, our ball balance experiment can be seen as such a task, where a 'manipulated ball' is placed on the table. If the parameters of the manipulated object are uncertain, we can still utilize EASI to adjust the manipulator's parameters, allowing the simulation to more closely resemble the target environment while accommodating a wider range of parameters for the manipulated object.

Response to Reviewer 2SzS

Q1. The study lacks some stronger comparison.

A1. In the common response A4, we introduce additional comparative experiments, including the FineTune and GARAT. In these experiments, EASI consistently demonstrates the best performance in transferring simulation trained policies to the target environment.

Q2. The comparative use of compute.

A2. On our platform, we use EASI for one of the most complex tasks, Go2. EASI completed the parameter search in less than 10 minutes and is able to adjust the simulation parameter distribution to an appropriate range. After determining the simulation parameter distribution, UDR is used for policy training or fine-tuning.

While using UDR directly can save the time spent on parameter search, as mentioned in the paper, we often lack the knowledge to set a reasonable parameter range for UDR. Without specialized knowledge to adjust the UDR parameter range, the trained policy might not transfer well to the real environment. Thus, directly using UDR may require extensive expertise and trial and error, which can be time-consuming compared to using EASI. By using EASI, we can leverage a small amount of real-world demonstration to quickly determine the UDR simulation parameter range, avoiding trial and error and improving the performance of the sim-to-real policy.

Q3. Some connections within the paper are very confusing.

A3. We carefully reviewed the issues you've raised, and we consider your feedback both valuable and important. We will promptly adjust the layout of the images in the paper and provide additional explanations for Fig. 1, offering a more detailed introduction to its contents. For Fig. 2, we will modify the layout of the paper to better link the legend to the corresponding content. Regarding the unclear phrasing, we will also make corrections in the next version, ensuring that readers can more easily understand our intended message. Thank you for your insightful feedback!

Q4. Eq. 4 is unclear to me. Does it contain a typo?

A4. The objective of Eq. 4 is to find a discriminator D that can better distinguish between simulated data and real data. We have revised Eq. 4 to make it more rigorous:

$\mathop{\max}\limits_{D} [E_{d^{\mathcal{M}}(\mathbf{s},\mathbf{a},\mathbf{s}')}[D(\mathbf{s},\mathbf{a},\mathbf{s}')]-E_{d^{\mathcal{B}}{(\mathbf{s},\mathbf{a},\mathbf{s}')}}[D(\mathbf{s},\mathbf{a},\mathbf{s}')]].$

Thank you for raising these issues. We will promptly revise the paper and add additional explanations to make the formula expressions clearer and easier to understand.

Response to Reviewer rASM

Q1. How do you capture/model the state transition

A1. In this paper, we draw on the training paradigm of GANs to estimate the distance between the simulated and real state transition distributions. According to GAN theory [1], Discriminator D and Generator G play the following two-player minimax game with value function $V(G,D)$:

$\mathop{min}\limits_{G} \mathop{max}\limits_{D}V(G,D)=E_{x\sim p_{data}(x)}[logD(x)]+E_{z\sim p_{z}(z)}[log(1-D(G(z)))].$

For any given fixed G, training D to maximize the quantity $V(G,D)$, we can proof that [1]:

$C(G)=\mathop{max}\limits_{D}V(G,D) =-log(4)+KL(p_{data}||\frac{p_{data}+p_{g}}{2})+KL(p_{g}||\frac{p_{data}+p_{g}}{2})$ $C(G)=-log(4)+2\cdot JSD(p_{data}||p_{g}).$

$C(G)$ could view as the estimication of the Jensen–Shannon divergence between two distributions.

We could leverage $C(G)$ to estimate the distance between state transition distributions. In each generation of evolution, we only need to randomly sample $(s,a,s')$ tuples from real state transition and simulated state transition to train the discriminator. Once the discriminator reaches its optimal, it can be used to estimate the distance between real and simulated state transition distributions.

In EASI, we use a WGAN-style discriminator

$C(G) = \mathop{\max}\limits_{D} E_{d^{\mathcal{M}}(\mathbf{s},\mathbf{a},\mathbf{s}')}[D(\mathbf{s},\mathbf{a},\mathbf{s}')]-E_{d^{\mathcal{B}}{(\mathbf{s},\mathbf{a},\mathbf{s}')}}[D(\mathbf{s},\mathbf{a},\mathbf{s}')],$

to tackle the discriminator vanishing gradients problem, and for this instance, $C(G)$ approximate the Earth-Mover distance between two distributions [2].

[1] Goodfellow, Ian, et al. "Generative adversarial nets." Advances in neural information processing systems 27 (2014). [2] Arjovsky, Martin, Soumith Chintala, and Léon Bottou. "Wasserstein generative adversarial networks." International conference on machine learning. PMLR, 2017.

Q2. Do they require the same initial state s_k?

A2. EASI does not have specific requirements for the initial agent state $s_k$, nor does it require the alignment of trajectories and state transitions. In fact, this is a key advantage of EASI. EASI uses discriminator to estimate the distance between the state transition distributions of simulation and demonstration. Unlike previous work, which required aligning sim and real trajectories and then comparing trajectory errors, EASI uses $(s,a,s')$ tuples to train a discriminator and estimate the distance of state transition distribution by the discriminator.

Q3. Can authors incorporate the GAT branch methods, and Domain Adaptation methods as well?

A3. In the common response A4, we introduced additional comparative experiments, including the FineTune and GARAT (a GAT branch method). In these experiments, EASI consistently demonstrated the best performance in transferring simulation trained policies to the target environment.

Q4. The generalizability of the proposed method

A4. Since EASI uses Evolutionary Strategies (ES) as the simulation parameter generator, it is much more stable in the training process compared to the original GAN algorithm and less sensitive to hyperparameters. In common response A5, we conducted a hyperparameter sensitivity analysis for EASI. The results showed that EASI has low sensitivity to hyperparameters, further demonstrating its stability.

Additionally, We tested EASI's sim-to-real performance in the real world using the Unitree Go2 robot in common response A2. The task in this experiment is complex and challenging, and EASI is still able to exhibit the expected performance.

Q5. Can authors explain Fig.3

A5. The training was conducted entirely in the original sim environment, and we saved models from various stages of the training process. These saved models were then tested in the target environment (since it's sim-to-sim, model testing was very convenient). The x-axis represents the number of policy updates in the original sim environment during RL training, while the y-axis shows the performance of the saved model tested in the target environment.

Through this figure, we want to show that EASI not only facilitates better transfer of policies trained in the original environment to the target environment, but also accelerating the training process because of the narrower Domain Randomization (DR) parameters range. We apologize for the confusion caused by Fig. 3. In future versions, we will include additional explanations to ensure Fig. 3 is better understood.

Q6. Code is not available anywhere in the paper.

A6. We have uploaded all the codes and supplementary materials.