Robust Reinforcement Learning with Structured Adversarial Ensemble

Dear Reviewer APuc,

We really appreciate your detailed review. Please see below our responses to your concerns.

Related Works.

Thank you so much for referring us to these additional references. Following your advice, we have updated the manuscript to include a more thorough discussion of the related works. For your convenience, we replicate the updated Related Work section (only the paragraphs about robust RL) below:

One research topic closely related to our method is domain randomization. Domain randomization is a technique to increase the generalization capability over a set of pre-defined environments. The set of environments are parameterized (e.g., friction and mass coefficient) to allow the agent to encode the knowledge about the deviations between training and testing scenarios. The environment parameters are commonly uniformly sampled during training [19, 21, 22, 23]. Even though ADR [20] is proposed to learn a parameter sampling strategy on the top of domain randomization, all of the aforementioned methods are not learned over the worst case scenarios. Moreover, in real life application, if not chosen carefully, the environment set can also lead to over-pessimism with a larger range while selecting a smaller range of the set will be over-optimistic. Hence, our proposed method can be readily extended into domain randomization by considering the environments as fixed adversaries.

Robustness to transition models has been widely investigated. It was initially studied by robust MDPs [2,3,4] through a model-based manner by assuming the uncertainty set of environmental transitions is known, which can be solved by dynamic programming. In this approach, a base dynamic model is assumed and the uncertainty set is crafted as a ball centered around the base model with a predetermined statistical distance or divergence, e.g., KL-divergence or Wasserstein distance. Following works address scenarios where the base model is unknown but samples from the base model are available. For example, [6,10] propose model-based algorithms that first estimates the base model and then solve the robust MDP; [5,7] propose online model-free policy evaluation and policy iteration algorithms for robust RL with convergence guarantees; [1] proposes algorithms with polynomial guarantees for tabular cases where both the number of states and actions are finite.; [8,9] further extends the study of robust RL with only offline data. In contrast to these works, we follow the approach of RARL which does not explicitly specify the set of environments but learns a robust policy by competing with an adversary.

Subsequent works generalize the objective to unknown uncertainty sets, and formulate the uncertainty as perturbations/disturbance introduced into the environments [10,11, 12, 13,14]. Notably, RARL [14] introduces an adversary with the objective to affect the environment to minimize the agent’s rewards. Notably, while in this work we focus on robustness to the transition model, there are two other types of robustness: robustness to the disturbance of actions [15,16] and robustness to state/observation [17,18].

The current framework considers robustness against adversarial actions. Tessler et al., (2019) and thereafter are the closest to current work's setting. Of course then their algorithms need to be benchmarked against.

Thank you for this suggestion. It seems our work considers a rather different setup than Tessler et al. (2019), and hope our response here could help clarify the potential misunderstanding here. Specifically, our work pursues the goal similar to the RARL work, i.e., toward the robustness against perturbed environment by adversarial training (as part of our methodology), which would be rather different than solving the problem of robustness against adversarial actions provided by the other party (what Tessler et al. (2019) considered). In the framework of RARL, the adversary conducts actions to change the environment to minimize the agent’s rewards. The action space of the adversarial is often different to that of the agent while Tessler et al., (2019) assumes that the protagonist and the adversary share the same action space, which is distinct to the experimental setup in RARL and our work.

Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning, pp. 6215–6224. PMLR, 2019.

[1] Zaiyan Xu, Kishan Panaganti, and Dileep Kalathil. Improved Sample Complexity Bounds for Distributionally Robust Reinforcement Learning. International Conference on Artificial Intelligence and Statistics, 2023

[2] Iyengar, G. N. (2005). Robust dynamic programming. Mathematics of Operations Research, 30(2):257–280.

[3] J Andrew Bagnell, Andrew Y Ng, and Jeff G Schneider. Solving uncertain markov decision processes. Citeseer, 2001.

[4] Arnab Nilim and Laurent Ghaoui. Robustness in markov decision problems with uncertain transition matrices. Advances in neural information processing systems, 16, 2003.

[5] Panaganti, K. and Kalathil, D. (2021). Robust reinforcement learning using least squares policy iteration with provable performance guarantees. In Proceedings of the 38th International Conference on Machine Learning, pages 511–520.

[6] Panaganti, K. and Kalathil, D. (2022). Sample complexity of robust reinforcement learning with a generative model. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, pages 9582–9602.

[7] Roy, A., Xu, H., and Pokutta, S. (2017). Reinforcement learning under model mismatch. In Advances in Neural Information Processing Systems, pages 3043–3052.

[8] Panaganti, K., Xu, Z., Kalathil, D., and Ghavamzadeh, M. (2022). Robust reinforcement learning using offline data. Advances in Neural Information Processing Systems (NeurIPS).

[9] Shi, L. and Chi, Y. (2022). Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity. arXiv preprint arXiv:2208.05767

[10] ​​L Shi, G Li, Y Wei, Y Chen, M Geist, Y Chi (2023) The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Model, NeurIPS 2023

[11] Ian Abraham, Ankur Handa, Nathan Ratliff, Kendall Lowrey, Todd D Murphey, and Dieter Fox. Model-based generalization under parameter uncertainty using path integral control. IEEE Robotics and Automation Letters, 5(2):2864–2871, 2020.

[12]Daniel J Mankowitz, Nir Levine, Rae Jeong, Yuanyuan Shi, Jackie Kay, Abbas Abdolmaleki, Jost Tobias Springenberg, Timothy Mann, Todd Hester, and Martin Riedmiller. Robust reinforcement learning for continuous control with model misspecification. arXiv preprint arXiv:1906.07516, 2019.

[13]Eugene Vinitsky, Yuqing Du, Kanaad Parvate, Kathy Jang, Pieter Abbeel, and Alexandre Bayen. Robust reinforcement learning using adversarial populations, 2020a.

[14]Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In International Conference on Machine Learning, 2017.

[15] Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning, pp. 6215–6224. PMLR, 2019.

[16] Yutong Li, Nan Li, H Eric Tseng, Anouck Girard, Dimitar Filev, and Ilya Kolmanovsky. Safe reinforcement learning using robust action governor. In Learning for Dynamics and Control, pp. 1093–1104. PMLR, 2021.

[17] Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems, 33:21024–21037, 2020.

[18] Robust Multi-Agent Reinforcement Learning with State Uncertainty Sihong He, Songyang Han, Sanbao Su, Shuo Han, Shaofeng Zou, and Fei Miao. Transactions on Machine Learning Research, June 2023.

[19] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. Intelligent Robots and Systems (IROS), 2017 IEEE/RSJ International Conference on. IEEE, 2017.

[20] Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J Pal, and Liam Paull. Active domain randomization. Conference on Robot Learning, pp. 1162–1176. PMLR, 2020.

[21] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Sim-to-Real Transfer of Robotic Control with Dynamics Randomization,” in 2018 IEEE international conference on robotics and automation (ICRA), pp. 3803–3810, IEEE, 2018.

[22] Z. Li, X. Cheng, X. B. Peng, P. Abbeel, S. Levine, G. Berseth, and K. Sreenath, “Reinforcement learning for robust parameterized locomotion control of bipedal robots,” in 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 2811–2817, IEEE, 2021.

[23] J. Siekmann, Y. Godse, A. Fern, and J. Hurst, “Sim-to-real learning of all common bipedal gaits via periodic reward composition,” in 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 7309–7315, IEEE, 2021.

Thank you for the detailed response. I think the authors have improved the manuscript compared to the pre-rebuttal stage but there are concerns from other reviewers that need further iterations. I will update my score after the author-reviewer discussion period.

Dear Reviewer APuc,

As the discussion period is closing to an end, we would like to know if there are any other concerns that we can help resolve. The authors are more than happy and fully committed to providing further clarification and addressing any remaining questions you may have. If our responses have effectively addressed all of your primary concerns, we sincerely ask the reviewer to kindly consider an upward adjustment of the score to reflect the improvements made.

Thank you once again for the precious time and effort that you have invested in reviewing this work.

Best wishes,

Authors

Dear Reviewer c2QC,

We greatly appreciate your comments. We have found quite a few discrepancies between the comments from the reviewer about specific parts of our methodology, and what had been introduced in the paper -- we have enriched them with more details below, and hopefully they could help clarify the misunderstandings/confusions.

Limit technical novelty. RARL with an adversarial population is not a novel idea.

Compared to other works, our work has the following contributions:

1. We identified and addressed two major bottlenecks at once toward robust RL through adversarial training, the potential over-optimism and over-pessimism issues. As a result, our method, as demonstrated by our extensive empirical study (e.g., see Figure 2 and Table 1), significantly improves RARL and RAP which fail to address these problems, demonstrating the cruciality of the identified problems.
2. Our work, to the best of our knowledge, is the first that provides thorough theoretical understanding to illustrate the benefit of introducing ensembles to robust RL, which laid out the theoretical ground that future works can follow. Moreover, our method differs from the classic average/vote-based ensemble methods, as it considers only a (self-adapted) moving subset of the ensemble that are most helpful for combating potential over-optimism/pessimism.
3. We conducted extensive experiments and ablation studies to verify the efficacy of our method. The results have shown that our method consistently dominates baselines across different environments, with various types of disturbance applied, over a variety (i.e., on-policy and off-policy) of backbone RL algorithms.

While the amount of difference between algorithms is a challenging question to settle, the difference of our proposed method to other ensemble method-based algorithms has proven to lead to significant performance improvement. This exactly underscores the technical importance of our proposed method. As the other two reviewers are satisfied with the novelty of our method, we would greatly appreciate it if the reviewer could further pinpoint from which perspectives we could further improve the novelty of our work, in order to lead to your satisfaction.

Connection between theorem and the algorithm

We have detailed the insights brought by the theoretical results in the paragraph “Insights from the Theoretical Results” in Section 3.1. To reiterate, our theoretical results have shown that the number of the adversaries required is of a moderate polynomial order. There are two parameters to choose: the total number of adversaries and the k-value. With the analysis corroborating that the total number of adversaries is a relatively small number and the fact that k-value is upper bounded by the total number of adversaries, the search space is significantly reduced by optimizing over the worst-k adversaries (which does not necessarily compensate for optimality either).

Why to optimize the performance over the worst-k adversarial, how to choose the k-value, etc.

As presented in Section 3.2, we propose to optimize over the worst-k adversaries to address the potential over-pessimism caused by the misspecification of the adversary set – among other things, this is a novelty of our work that enables our method to significantly outperform existing (state-of-the-art) methods. In most practical real-world scenarios, it is often challenging to have a precise characterization of the adversary set, thus leaving the robust RL algorithms with an adversary set with agents mostly leading to irrelevant and infeasible outcomes, e.g., an adversary that applies extreme disturbance that is unfeasible to occur in realistic scenarios. The effectiveness of our methodology is concretely justified by extensive experiments in Section 4, because the objective of the protagonist can be significantly diverted if it optimizes against solely the worst adversary who may apply unrealistic disturbances. Hence, the choice of k-value depends on to what extent one would want to minimize the effects brought in by such unrealistic adversaries. In principle, larger $k$-value should be used for environments with straightforward dynamics, where the adversaries are more likely to discover extreme strategies easily.

However, there is no component in the algorithm that aims to improve diversity explicitly

Thank you for this question. By Occam’s Razor, we believe it is beneficial to retain only necessary components of an algorithm. As illustrated by empirical studies ( Section 6 Ablation A4), we have confirmed that as long as the adversaries are initialized differently, they will remain distinct. And the consistently improved performance has demonstrated that the distinction from random initialization is sufficient. Hence, we choose to not induce extra components that can potentially cause stability issues and consume more computation complexity.

Dear Reviewer c2QC,

As the discussion period is closing to an end, we would like to know if there are any other concerns that we can help resolve. The authors are more than happy and fully committed to providing further clarification and addressing any remaining questions you may have. If our responses have effectively addressed all of your primary concerns, we sincerely ask the reviewer to kindly consider an upward adjustment of the score to reflect the improvements made.

Thank you once again for the precious time and effort that you have invested in reviewing this work.

Best wishes,

Authors

Dear Reviewer 1Lc9,

Thank you so much for your dedicated effort in helping improve our work! Please see below our detailed responses.

Figure 1. A more straightforward illustration with an improved caption is needed.

We appreciate this advice and we have updated the manuscript to include a new Figure 1 with improved caption.

Can the authors elaborate on how the average performance optimization directly counteracts over-pessimism?

Thank you for this insightful question. In most practical real-world scenarios, it is often challenging to have a precise characterization of the adversary set, thus leaving the robust RL algorithms with an adversary set with agents mostly leading to irrelevant and infeasible outcomes, e.g., an adversary that applies extreme disturbance that is unfeasible to occur in realistic scenarios. Hence, we propose to optimize over worst-k adversaries so that robust RL can prevent the over-pessimism without precise knowledge of the adversary set. This approach is highly effective, as demonstrated by the compelling results from extensive experiments, because the objective of the protagonist can be significantly diverted if it optimizes against solely the worst adversary who may unfortunately be one of the erroneously chosen adversaries in the adversary set.

Moreover, as the max-min problems in robust RL are normally solved by iterative updates of the protagonist and the adversaries, where in each iteration we have an adversary $\phi$ against whom we will optimize the protagonist. However, if the adversary set is not precise, $\phi$ may be a misspecified scenario. If the rest $k-1$ adversaries (or the majority of the worst-$k$ adversaries) are indeed in the true interested scenarios, optimizing the average over the worst-k adversaries distracts the attention of the protagonist from the single uninterested worst case to the cases of interest. However, we cannot directly optimize the average performance of all the adversaries because this can lead to too little consideration of the worst-case performance, as demonstrated by our empirical study in Figure 2.

We find this comment very helpful for improving the clarity on the motivations of this work, and we have already updated our manuscript to include this detailed explanation (in red text).

Dear Reviewer 1Lc9,

As the discussion period is closing to an end, we would like to know if there are any other concerns that we can help resolve. The authors are more than happy and fully committed to providing further clarification and addressing any remaining questions you may have. If our responses have effectively addressed all of your primary concerns, we sincerely ask the reviewer to kindly consider an upward adjustment of the score to reflect the improvements made.

Thank you once again for the precious time and effort that you have invested in reviewing this work.

Best wishes,

Authors