A Trust Region Approach for Few-Shot Sim-to-Real Reinforcement Learning

Dear reviewer, thank you for your detailed report and your interest in our work. We took your suggestions into account and updated the paper. Here are the answers to your questions.

General paper update: We have improved the FOOD agent by considering a regularization between state-action-state visitation distributions. Not only FOOD is now directly aligned with our Proposition, but it also provides better results with lower variance in most environments. Sections 4 and 5 have been updated.

Robotics application: We chose to use the term "Sim-to-Real" as we believe "Off-Dynamics RL" is the first step for tackling the real Sim-to-Real problem. We understand that it can be seen as an oversail, but it is quite commonly accepted in the literature [1, 2, 3]. The application of our agent to a real system such as a robot would add an interesting dimension to this work. However, such experiments would be another contribution of its own due to the associated difficulties and costs. We also would like to stress that our experiments are more extensive than the previous works [1, 2, 3], where for instance DARC mainly considers broken environments. For those reasons, we believe the extensiveness of our experiments matches what has been proposed in the Sim-to-Real community.

Proposition 4.1: Although simple, to the best of our knowledge, Proposition 4.1. has never been proposed in the literature in the Sim-to-Real setting with a dynamics mismatch, nor in any RL-related literature. We would be happy to reconsider the novelty of our paper if the reviewer can point us to a work that proposes this result.

Imitation Learning (IL) answers: IL is known to work with few data samples of the target policy even in high dimensional environments (e.g. Humanoid with $4000$ transitions). Also, our method is agnostic to the IL technique used and will benefit from further advances in IL.

Experiments:

• Seeds: Since the experiments are long, having $4$ or $5$ seeds is the norm for RL papers.
• Statistical significance: On the contrary, our algorithm FOOD is the only statistically efficient method compared to the baseline as shown by Table 1 where the standard deviation was not divided by $2$. We performed an unpaired t-test with an asymptotic significance of 0.1 w.r.t. the best performer for each experiment and included it in the rebuttal. We divided the variance by $2$ in Fig. 1 because of the variance of $RL_{\text{sim}}$ which made the figure more difficult to read.
• "High value induce the agent to replicate real trajectories": There may be a misunderstanding here. The high value refers to our hyper-parameter $\alpha$, which controls the amplitude of the regularization. Since PPO's advantages are normalized, setting $\alpha$ to $5$ forces the agent to focus solely on imitating the transitions from the real environment. It may not always be possible, hence the potential drop in performance. Unlike other baselines (Cf. Appendix C.5 and C.6), setting $\alpha$ to a value close to $1$ provides a stable algorithm with the strongest results.

Suboptimal real trajectories: Due to the particular structure of our method, the quality of the collected trajectories is indeed an important matter. For example, in Broken Joint Cheetah, when collecting $5$ trajectories, the Cheetah falls directly to the ground in one of them. In such cases, practitioners should remove this trajectory as it leads the agent to the wrong parts of the simulator. Nevertheless, in our experiments, we have found that considering medium-quality trajectories for the regularization is sufficient for FOOD to improve over the baselines as it also maximizes the rewards around these trajectories.

Trust Region term: We used this term to have a clear continuity with existing algorithms. We understand that it was ill-applied and intend to change it to the more general term "Conservative".

Proposed paper: We thank the reviewer for providing us with the related paper "Virtual vs. Real". It combines data coming from two different sources, e.g. a simulator and a real environment, and unlike our baselines and our agent, uses Bayesian Optimization to compute the objective function. We will add this work to the related work section.

Clarity: We would like to thank the reviewer again for pointing out the parts of the paper that were unclear. We have taken your suggestions into account and updated the paper.

[1] Desai, S., Durugkar, I., Karnan, H., Warnell, G., Hanna, J., Stone, P. (2020) - An imitation from observation approach to transfer learning with dynamics mismatch - NeurIPS.

[2] Eysenbach, B., Asawa, S., Chaudhari, S., Levine, S., Salakhutdinov, R. (2021) - Off-dynamics reinforcement learning: Training for transfer with domain classifiers - ICLR.

[3] Yu, W., Liu, C. K., Turk, G. (2018) - Policy transfer with strategy optimization - ICLR.

[4] Ho, J., Ermon, S. (2016) - Generative adversarial imitation learning - NeurIPS.

Dear reviewer, thank you for your detailed report and your interest in our work. Here are the answers to your questions.

General paper update: We have improved the FOOD agent by considering a regularization between state-action-state visitation distributions. Not only FOOD is now directly aligned with our Proposition, but it also provides better results with lower variance in most environments. Sections 4 and 5 have been updated.

Modification DARC Baseline: Similar to DARC, our work focuses on investigating the different objectives that lead to a good transfer between a source and target environment, assuming no access except for a few trajectories in the target environment. Both of our works are agnostic to the underlying RL agent used. In their paper, they chose to optimize it with SAC while we did it with PPO. Their choice was arbitrary as explained in their Section 5. We chose to re-implement DARC in PPO to have a fair comparison between the different objectives. Our FOOD approach could very well have been applied with SAC.

Domain Randomization Baseline: Domain randomization is indeed important in the Sim-to-Real setting. To the best of our knowledge, such algorithms are usually difficult to apply when the simulator is a black box. We did not assume to have access to the parameters of the simulator to be as general as possible. In this setting, the baseline RARL [1] is known to drastically fail. ANE, which can be considered a Domain Randomization technique since the inclusion of noise in the action perturbs the transition probabilities, performs well. As explained in the Experiments section, both statements are taken from [2].

Robotics application: We chose to use the term "Sim-to-Real" because we believe that "Off-Dynamics RL" is the first step in tackling the real Sim-to-Real problem. It can be seen as an oversailing, but it is widely accepted in the literature [2, 3, 4]. We understand that applying our agent to a real system such as a robot would add an interesting dimension to this work. However, such experiments would be a contribution of its own due to the difficulties and costs involved. We would also like to emphasize that our experiments are more extensive than the previous works [2, 3, 4], where for instance DARC mainly considers broken environments. For example, the Low Fidelity Minitaur benchmark is highlighted as it illustrates a typical situation: the simulator is a simplification of the dynamics of the real world. For those reasons, we believe that the comprehensiveness of our experiments is consistent with what has been proposed in the Sim-to-Real community.

[1] Pinto, L., Davidson, J., Sukthankar, R., Gupta, A. (2017, July) - Robust adversarial reinforcement learning - ICML.

[2] Desai, S., Durugkar, I., Karnan, H., Warnell, G., Hanna, J., Stone, P. (2020) - An imitation from observation approach to transfer learning with dynamics mismatch - NeurIPS.

[3] Eysenbach, B., Asawa, S., Chaudhari, S., Levine, S., Salakhutdinov, R. (2021) - Off-dynamics reinforcement learning: Training for transfer with domain classifiers - ICLR.

[4] Yu, W., Liu, C. K., Turk, G. (2018) - Policy transfer with strategy optimization - ICLR.

Thank authors for the response. As for Q1, it will be much better if the proposed method can be combined with SAC, and it outperforms the original DARC. Also, sim2real experiments on real robot will be much more impressive to support your claim. Since my concern is not fully resolved, I'd like to keep my score.

Dear reviewer, thank you for your detailed report and your interest in our work. Here are the answers to your questions.

General paper update: We have improved the FOOD agent by considering a regularization between state-action-state visitation distributions. Not only FOOD is now directly aligned with our Proposition, but it also provides better results with lower variance in most environments. Sections 4 and 5 have been updated.

Simplicity and novelty: We do not agree that the complexity of a method is representative of its contribution to the field. On the contrary: if a simple but novel algorithm works in practice then it should be preferred to other more complicated ones. Furthermore, our contribution is not strictly related to our method, but also to the study of a relevant objective to optimize in the Off-Dynamics setting. We have empirically shown that current methods are not general enough for this setting and evaluated the relevance of the baselines and our agent in many scenarios.

Writing: Since the proposition is now directly aligned with our algorithm, we have merged Sections 4.1 and 4.2. We have also corrected the mentioned parts that were unclear:

• Select best trajectories is now Remove trajectories resulting in poor behavior. Due to the particular structure of our method, the quality of the collected trajectories is indeed an important matter. For example, in Broken Joint Cheetah, when collecting $5$ trajectories, the Cheetah falls directly to the ground in one of them. In such cases, practitioners should remove this trajectory as it leads the agent to the wrong parts of the simulator (hence the original sentence "select best trajectories" in Algorithm 1). Nevertheless, in our experiments, we have found that considering medium-quality trajectories for the regularization is sufficient for FOOD to improve over the baselines as it also maximizes the rewards around these trajectories.

• To clarify Algorithm 1: $\mathcal{D}_t$ is used in the Imitation Learning algorithm to build the imitation reward function, and $\mathcal{M}_s$ is the simulator: the agent only samples in the simulator.

• We are currently justifying the definition of $V^{\pi}_{\text{imit}}$ with proof to justify why it minimizes the JS-divergence between state-action-state visitation distributions when using a discriminator. It will appear in the Appendix in the following days.

• We have not mentioned the equation for updating the value function weights, as it depends on the RL algorithm $\mathcal{O}$ considered. Our agent FOOD is quite general: it focuses on constraining the overall objective to allow a better transfer. This objective can be optimized by any online RL agent that builds a Value function into its learning scheme.

CQL Baseline: To properly assess the achievable improvement gained using a simulator in addition to the real data, we consider that Offline Reinforcement Learning (ORL) agents are an important baseline for Few-Shot Sim-to-Real. It is seen as a lower bound to beat. Indeed, the agent has access to some real trajectories. If ORL agents were able to leverage this data to learn a near-optimal policy, then were would be no need for new Sim-to-Real agents. We arbitrarily chose CQL as the ORL agent as it is known to perform well.

Dear reviewer, thank you for your detailed report and your interest in our work. Here are the answers to your questions.

General paper update: We have improved the FOOD agent by considering a regularization between state-action-state visitation distributions. Not only FOOD is now directly aligned with our Proposition, but it also provides better results with lower variance in most environments. Sections 4 and 5 have been updated.

Suboptimal real trajectories: Due to the particular structure of our method, the quality of the collected trajectories is indeed an important matter. For example, in Broken Joint Cheetah, when collecting $5$ trajectories, the Cheetah falls directly to the ground in one of them. In such cases, practitioners should remove this trajectory as it leads the agent to the wrong parts of the simulator (hence the original sentence "select best trajectories" in Algorithm 1). Nevertheless, in our experiments, we have found that considering medium-quality trajectories for the regularization is sufficient for FOOD to improve over the baselines as it also maximizes the rewards around these trajectories.

Low data regime: According to the original GAIL paper [1], this technique is efficient in a low data regime in many high-dimensional environments (sometimes using only hundreds of transitions). In addition, our algorithm is not necessarily limited to GAIL to compute the regularization. It could benefit from future advances in the field that would require even less data than GAIL to be efficient.

Baselines:

• We have considered most of the current methods that fit our setting. DARC is the most prominent approach for tackling the Sim-to-Real gap with few data. ANE is an important baseline for domain randomization when the simulator is a black box [2].
• To properly assess the achievable improvement gained using a simulator in addition to the real data, we consider that Offline Reinforcement Learning (ORL) agents are an important baseline for Few-Shot Sim-to-Real. It is seen as a lower bound to beat. Indeed, the agent has access to some real trajectories. If ORL agents were able to leverage this data to learn a near-optimal policy, then were would be no need for new Sim-to-Real agents. We arbitrarily chose CQL as the ORL agent as it is known to perform well.
• A new algorithm named H2O [3] attempts to combine data from the real environment and the simulator in a CQL fashion, as you suggest. However, it is quite clear from their experiments that it requires a lot of real data to be efficient (Cf. Figure 6 of Appendix C.1 of their paper where it requires at least $50$ trajectories to be efficient on HalfCheetah.). We have mentioned this in the related work section, although we are currently making it clearer in a new version of the paper. We can also include this baseline if you think it is still relevant.
• We argue that TD3 + BC would not be a straightforward baseline to mix the data sampled from both the simulated and real data sets. One could imagine adding the following BC reward to each RL agent:

where $(s_c, a_c, s_c')$ is chosen as the closest transition of $\mathcal{D}_{\textrm{t}}$ to $(s, a, s')$ with respect to a metric $M$. First, it would require an additional computationally expensive step to select this transition. Second, this bonus would be subject to compounding errors: it only considers one-step transitions, not the whole trajectory induced by the real environment. Our agent FOOD, on the other hand, encompasses the whole real trajectories induced by the real dynamics in its regularization and is easy to implement.

[1] Ho, J., Ermon, S. (2016) - Generative adversarial imitation learning - NeurIPS.

[2] Hanna, J. P., Desai, S., Karnan, H., Warnell, G., Stone, P. (2021) - Grounded action transformation for sim-to-real reinforcement learning - Machine Learning.

[3] Niu, H., Qiu, Y., Li, M., Zhou, G., HU, J., Zhan, X. (2022) - When to trust your simulator: Dynamics-aware hybrid offline-and-online reinforcement learning - NeurIPS.

Dear reviewer agZp,

We fully take your remarks into account to enhance the quality of our paper in a future version. Specifically, we plan to:

• Add the additional baselines H2O, a modification of TD3+BC for our setting, and try SAC-based DARC on our environments.
• Add additional theory from Trust Region methods to back up the relevance of our algorithm and potentially improve it.
• Potentially add a robot experiment if it is possible.

Thank you for your insights and constructive feedback.